Scaling Science: Using Data to Find “Hidden Gem” Research

Frequently Asked Questions

1. What is the goal of this research?

   The current system of scientific recognition and funding has significant bias towards work published in the top few percent of journals and by a relatively small number of research groups. In other words, a large amount of high-quality scientific research goes unnoticed due to current promotion and funding criteria. In this work, we explored whether we could use data science to find “hidden gem” research that would go on to become highly “impactful” but do not necessarily have the high out-of-the-box citation counts that are typical of well-known and highly-established research groups.

   We found that there are indeed a significant number of such “hidden gem” papers, which are missed by conventional citation metrics. As an example, there are a number of journals that would be considered mid-range on the basis of Impact Factor that have higher average predicted impact than some top-ranked journals. Furthermore, even the bottom-ranked journals (again based on Impact Factor) from our biotechnology-focused dataset each have a number of papers of predicted medium-to-high future impact.
   

   We think this is an important finding; it suggests that there exists untapped research and investigators that, despite creating high-quality work, may not be receiving the resources, recognition, and support they deserve. It also suggests that, if we can design new ways to identify these “hidden gems” projects, we can potentially start breaking out of a journal- and citation-drive status quo. Of course, our platform is simply an algorithmic way to find new research and investigators; its biases must be continually monitored by area experts and it must be carefully integrated into a larger, human-driven toolkit to ensure systemic biases are decreased and that innovation and creativity are incentivized.
   

   However, there is also correlation between research groups—some groups may be large, and work on consistently cited research, while others may be smaller, less connected, or less recognized via traditional metrics. Not only is focusing attention and funding on larger “clusters'' undesirable from a social equality perspective, but it also could lead to reduced risk-taking and thus lower overall scientific output. As such, we are interested in integrating our data-driven measures, along with constant human feedback and expertise, into funding strategies that reward, rather than penalize, research in new areas or with novel collaborators. 
   

   Overall, our goal is to develop tools that help us, collectively, discover interesting, exciting, impactful research—especially work that might be overlooked with current publication metrics, which are known to be biased. By combining a wider variety of metrics into a machine learning pipeline, we hope to start to move beyond reliance on any particular metric (e.g. citations, h-index, etc). It’s critical to note that we view DELPHI as an additional tool to be integrated into the researcher’s toolkit, and never as a replacement for human-level expertise and intuition. In other words, our framework can be understood as a way to search through the academic literature to help find important research and researchers that the current system may have missed. 

2. Your models use many features that we know to be biased; aren’t you going to augment, rather than combat, systemic biases in the academic network?

   This is an important question. Our hypothesis was that, by considering a broader array of features, we could find important research that would otherwise be missed by current bias-prone metrics, which we have demonstrated by DELPHI’s ability to highlight “impactful” work that citations (and other metrics) would not have otherwise uncovered in the same time period. The features used in our system have been demonstrated as containing signal by previous research for predictive tasks, and while we had to start somewhere, we expect future, improved versions of our model to be developed that incorporate larger and improved sets of relevant metrics, as well as new objective functions—which would ideally be community-curated to ensure relevance and lack of bias. That said, the avalanche of academic research makes the usage of search algorithms and metrics inevitable, and thus we believe that the development of continually-improved ways of exploring the academic landscape (and to highlight high-quality researchers that currently have less visibility) are important.

3. How exactly is your algorithm capable of knowing the future based on a paper?

    We are not predicting the future based on the contents of a paper—our approach is recognizing patterns in how the scientific community itself (composed of humans) react (by building on) a newly-released piece of research. By measuring the interaction of experts with new work, we hope to highlight valuable work earlier (regardless of where it was published or how many citations it has). 

4. How can any machine learning model replace human intuition?

   Our goal is certainly not to decide what work gets done, but instead to highlight important work—and most importantly “hidden gem” research that are treated unfairly by the current status quo. Our approach uses time-series changes in network-level features to quantify the interaction of the scientific community with new papers, and compresses this high-dimensional information down to a single "early-warning score" that can be used by researchers. Interestingly, it turns out that this approach can find important work earlier than citations or other existing metrics—simply by measuring the response of the academic community in a more comprehensive manner. In no way is our algorithm a replacement for the brain of a researcher (or funder); it is an additional data input that they can use to broaden their “search space” and potentially find research they may otherwise not be exposed to. Our hope is that such tools can be used in combination with existing systems to broaden the scope of researchers that are considered important and thus deserving of attention and funding. 

5. How are you quantifying scientific impact?

   There have been a plethora of attempts to quantify scientific impact, ranging from the more common citation count, h-index and journal impact factor to time- and field-normalized measures. However, as has been discussed extensively in previous literature, these metrics are not only suboptimal measures of quality but are also lagging indicators of impact—as such, their use can lead to suboptimal decisions in academic hiring, promotion and funding. As such, we have designed our framework to accommodate arbitrary, user-defined measures of “impact.” Further, we have demonstrated strong performance and compelling results using a time-rescaled measure of node centrality, which both removes age bias, allowing meaningful comparisons across years, and provides state-of-the-art performance in ranking milestone technologies. 

6. Do you think it may be possible for malicious actors to "game" this system?

   As with all machine learning-based systems, care must be taken to ensure that these methods do not provide opportunities for malicious actors to manipulate the system for their own gain. By considering a broad range of features and using only those that hold real signal about future impact, we think that DELPHI holds the potential to reduce the possibility for manipulation by obviating reliance on simpler (and often reputation-related) metrics. It is possible, therefore, that DELPHI scores will be more difficult for authors or journals to manipulate than, for example, simple citation counts (upon which h-index and journal impact factor are based). However, additional studies, as well as careful human examination of calculations, are critical to more fully understand these possibilities.

7. Won’t using machine learning in this way simply aggravate existing systemic problems in the academic and funding communities?

   We are acutely cognizant of the potential for our system to aggravate existing systemic biases, and want to strongly emphasize that any utilization of our platform should be conducted alongside area experts. Just as we would not blindly accept that the top cited articles are the most important, we should not blindly accept the top-scoring articles of our system to be the most valuable. Instead, the DELPHI score should be viewed as an alternative perspective into the academic system that can highlight work for consideration—even if it has a low citation count, or comes from an unexpected place.

8. Why does the heatmap of CRISPR researchers show many male investigators?

   The heatmap of CRISPR researchers, which is a figure from the discussion section of our recent manuscript, does not show any data from our machine learning pipeline. Instead, this is simply showing historical correlation between researchers in our biotechnology-focused dataset that have published in the CRISPR space. For readability (given the large number of researchers in the space), the graphing function sampled the names at regular intervals (they were not selected by us). The purpose of this figure is to highlight the existing correlation between certain parts of the academic network—the identification of which is the first step towards developing new funding models that drive resources towards the “uncorrelated” and less-connected researchers. While this was part of our discussion, and not a core result of our paper, we think the integration of data-driven approaches to identifying under-funded researchers could help construct a funding landscape where funding is allocated more broadly, and where risk-taking is appreciated on a fundamental level.

9. What would be the process for expanding this system into other scientific fields, and what might be the limitations of this?

   Our graph-based approach makes the expansion of our system into additional scientific fields possible—initially beginning with the inclusion of additional disciplines and academic journals, and subsequently potentially including other sources of high-quality research like arXiv. However, such expansion must be done carefully, as there are significant differences between the structure and dynamics among different disciplines. 

10. Who could use this platform, and how could it potentially help improve the academic system?

    The metrics widely used by the academic and funding ecosystem are known to be biased and suboptimal—our goal is to create a tool that suggests additional research that is worth of review—even if it does not have an enormous number of citations, or is not published in a high Impact Factor journal. Given the large amounts of funding that are allocated annually by government funding agencies (NSF, NIH, NASA, etc), broadening the scope of what is considered valuable research could potentially help move towards a more impactful and equitable place—especially if it helps us find undervalued but high-value research or investigators which the current system overlooks.

11. I have a serious scientific comment, methodology inquiry, or would like to implement your system myself. What should I do?

    Please contact us! We would be very happy to collaborate with those interested in designing new ways to make the academic system more accessible, equitable, and productive, and we welcome all serious feedback.