AI and Citation Metrics: Exposing Bias in Science Funding

This article explores how science-of-science researchers combine citation-network disruption indices and grant-portfolio artificial intelligence (AI) to investigate systemic bias in academic peer review. By analyzing massive datasets of funded and unfunded proposals alongside publication outcomes, researchers can mathematically determine whether traditional funding committees penalize high-risk, interdisciplinary ideas in favor of safe, incremental research.

Understanding Citation-Network Disruption Indices

To measure whether research is truly groundbreaking, scientists use the disruption index (often called the CD index). This metric evaluates how a published paper alters the citation network around it.

• Consolidating Research: If future studies cite both a specific paper and the older references that the paper relied upon, the research is considered consolidating. It builds incrementally on existing knowledge.
• Disruptive Research: If future studies cite a specific paper but ignore its predecessors, the paper has successfully eclipsed prior work. It has disrupted the field and established a new paradigm.

By applying this index to millions of publications, researchers can objectively score the disruptiveness of scientific outputs and trace those papers back to the grants that funded them.

Analyzing Proposals with Grant-Portfolio AI

Before a paper is ever written, the idea exists as a grant proposal. Science-of-science researchers use Natural Language Processing (NLP) and machine learning to analyze the text of thousands of successful and rejected grant applications.

Grant-portfolio AI evaluates these documents to quantify two main factors:

1. Interdisciplinarity: The AI maps the semantic distance between concepts in a proposal. If a proposal combines terminology from distant fields—such as quantum physics and cellular biology—it scores high on interdisciplinarity.
2. Risk and Novelty: Machine learning models compare the proposal's text against existing literature. Proposals introducing entirely new combinations of methodologies or unverified hypotheses are flagged as high-risk.

Connecting the Data to Expose Peer Review Bias

To prove whether peer review systematically underfunds breakthroughs, researchers merge the AI proposal data with the citation-network disruption data. They track the lifecycle of ideas from the initial grant application to the eventual published results.

First, researchers use AI to identify a cohort of high-risk, interdisciplinary proposals submitted to funding agencies. Next, they look at the peer review outcomes for these proposals. Consistently, the data shows that review panels give lower scores to proposals with high semantic distance. Evaluators often struggle to assess fields outside their immediate expertise, leading them to view interdisciplinary work as flawed or overly speculative.

Tracking the "Near-Miss" Proposals

The most definitive proof comes from studying "near-miss" applicants—researchers whose high-risk proposals fell just below the funding threshold, compared to those who scored just above it.

Researchers use AI to track the scientists whose risky proposals were rejected, identifying cases where the scientists managed to complete the work anyway using alternative, smaller, or private funding sources. When the resulting papers from these rejected proposals are evaluated using the citation-network disruption index, they frequently score much higher in disruption than the safe, consolidating proposals that the peer review panel actually chose to fund.

Through this combination of AI text analysis and citation mapping, science-of-science researchers demonstrate that consensus-driven peer review inherently favors safe bets, systematically starving high-risk, high-reward interdisciplinary science of crucial early-stage funding.