How are research outputs changing in the age of AI?
AI is accelerating research productivity, particularly in data-rich analysis and automated software generation, potentially shifting incentives away from creating new research resources. Yet there is currently no systematic way to measure how research communities produce, use, or extend essential artefacts such as data and software.
Paying It Forward builds a metric, or set of criteria, that describes whether a piece of research generates new research artefacts, and what kind of artefacts it produces.
We limit the scope to software and data. Arguably, there are many other types of research artefact, such as a new theoretical methodology, hardware or protocols.
Open each panel for the detail. Every paper gets one rating for software (S1–S3) and one for data (D1–D3).
| Definition | Typical evidence in a paper | |
|---|---|---|
| Research Data |
“Research data can be defined as ‘recorded factual material commonly retained by and accepted in the scientific community as necessary to validate research findings’. Although the majority of such data is created in digital format, all research data is included in the definition, irrespective of the format in which it is created.” – UKRI, 2026 (source)A collection of observations or information produced, collected, processed or released by the research. |
Dataset, corpus, database, annotations, benchmark data, supplementary data, model weights, lab books |
| Research Software |
“Research Software includes source code files, algorithms, scripts, computational workflows and executables that were created during the research process or for a research purpose. Software Components (e.g. operating systems, libraries, dependencies, packages, scripts, etc.) that are used for research but were not created during or with a clear research intent should be considered software in research and not Research Software.” (10.5281/zenodo.5504015)An executable computational artefact produced or used as part of the research. |
Source code, package, scripts, application, model implementation |
| Other | Research artefacts that are neither data nor software. | Hardware, physical materials, protocols, instruments, figures, documentation, experimental designs, or simply no research artefacts produced outside of the paper, such as a literature review |
New or substantially extended software that plays a substantial role in achieving the research goals, or is itself a research goal. It should not be a utility but rather represent substantial research effort. The paper should describe how the research created or significantly extended research software. Crucially, the software created should generate new knowledge.
Existing software that plays a substantial role in achieving the research goals, but is not substantially generated or extended by the research team. It should not be a utility but rather represent substantial research effort. The paper should describe how the research uses existing research software, and the software used should generate new knowledge.
Either software is not used, or the software used does not create new knowledge but instead acts as a utility, or does not play a substantial role in achieving the research goals.
New or substantially extended data that plays a substantial role in achieving the research goals, or is itself a research goal. The paper should describe how the research created or significantly extended research data. Crucially, the data created should generate new knowledge.
Existing data that plays a substantial role in achieving the research goals, but is not substantially generated or extended by the research team. The paper should describe how the research used existing research data, and the data used should generate new knowledge.
Either research data is not used, or the data used does not create new knowledge. Data does not play a substantial role in achieving the research goals.
If a paper both uses and generates software or data, it belongs in the generated category. We assume that anything generated is also used.
If the right category is unclear, err towards the higher number. Generated (1) needs clear evidence that the research produced it; otherwise classify it as used (2).
If a paper says it generated software or data but gives no way to access it, we still count it as S1 or D1. Future work packages will assess data stewardship and FAIRness.
Software runs down the side, data across the top. Pick a cell to jump to examples of that category.
Researchers develop and release a Python software package for training and evaluating pollution-prediction models. They also collect measurements from 50 monitoring stations and create and release a new dataset of air-quality observations.
Researchers develop and release a new image-analysis software package. They evaluate it using an existing publicly available microscopy dataset. The dataset is important to the research, but the researchers do not substantially create or extend it.
Often called a tool paper.
Researchers develop and release a software application that allows researchers to reconstruct and compare ancient musical scales from specified theoretical parameters. The software itself is the principal research output, and the study does not generate or analyse a substantive research dataset.
Researchers collect audio recordings from forests and create a new annotated dataset of bird calls and species observations. They use an existing acoustic-analysis software package to process the recordings, but do not substantially modify or extend the software or publish it as a research output.
Researchers conduct a controlled greenhouse experiment measuring plant growth under different soil conditions. Measurements from the experiment form a new dataset that is released with the paper. Data collection and analysis rely on existing laboratory and statistical software, which the researchers do not substantially modify or release.
Researchers use data-analysis software to analyse an existing agricultural dataset. Both the software and dataset make substantial contributions to the research, but neither is substantially developed or generated by the researchers in this paper.
Researchers analyse genetic, demographic, and health information from participants in an existing biobank. They use established bioinformatics and statistical software to identify associations between genetic variants and cardiovascular outcomes. The biobank data and software are both substantial to the research, but the researchers do not generate a new dataset or develop the software as part of the study.
This combination is very rare.
Researchers use an established theorem-proving system to formally verify a mathematical result. The proof assistant is central to the research but is not substantially developed or extended by the researchers.
Real-world examples: Archive of Formal Proofs · Example preprint
Researchers conduct a survey of 5,000 participants and create and release an anonymised dataset of survey responses. The dataset is a substantial research output. Although general use software like word processors is used, the research does not involve software playing a substantial role in achieving its goals.
Researchers collect physical specimens from several coastal plant populations and record measurements of leaf size, shape, and structure. These observations form a new dataset that is reported and made available as a research output. Although they record their data in a spreadsheet, software does not play a substantial role in achieving the research goals.
Researchers analyse an existing demographic dataset to investigate long-term trends in life expectancy. The dataset is central to the research, but no new or substantially extended dataset is produced. Software does not play a substantial role in achieving the research goals. Although data is stored and plotted in a spreadsheet, this is not deemed as software playing a substantial role.
Researchers examine an existing collection of digitised poems to investigate changing representations of nature. The corpus is an important source of evidence, but no new dataset is produced and software does not play a substantial role in the research.
Researchers develop a theoretical framework explaining how collaboration influences scientific discovery, drawing on existing literature and conceptual reasoning. Neither software nor data plays a substantial role in achieving the research goals.
A review of existing work that produces no research artefacts outside of the paper itself.
Argument or theory is the contribution; no software or data plays a substantial role.
Details of an experimental procedure published before any investigation is carried out. The protocol sets out what data will be collected and what software will be written to analyse the data, but neither are contributed in this paper.
This metric can be evaluated by taking a research paper and using the criteria to give it a classification (e.g. S2D3). For scalability, we are testing whether large language models, both local and frontier models, can classify papers into these categories reliably, and how their judgements compare with human classifications. Once we have established a reliable method, we can apply it to a larger dataset to gain insights into the current state of research output.
