AIGenerated
EN · IT · ES

AI scans 4,000 medical studies — most don't share useful code

Researchers used AI to find out whether doctors and scientists can actually rebuild results from published studies, and the answer was mostly no.

Two gaps remain in medical research code: sharing is uncommon, and public code often lacks the documentation needed for reuse.
Picture: news.mit.edu
How it works
  1. AI scans thousands of medical papers.
  2. AI extracts code‑sharing statements and links.
  3. Software fetches linked code repositories.
  4. Humans check a sample to verify findings.

What you could do with it

  • Researchers could use large language models to screen nearly 4,000 medical papers and identify statements about shared code, then follow links to assess the associated repositories 1.
  • Journals could use the findings to inform TRIPOD-Code, a reporting guideline designed to clarify exactly what researchers should share so their work is reproducible 1.
  • Human reviewers could independently check a sample of papers and code repositories to verify what AI tools have found, catching errors and disagreements before finalizing results 1.

From the announcement: two men at a vintage desktop computer with a graph on the screen

Picture: news.mit.edu

What the team did

Clinical prediction studies build computer models that estimate patient outcomes, like the risk of developing a disease. Since these models can affect actual patient care choices, it is important that other scientists are able to examine closely how they were developed and evaluated 1.

To see how often that actually happens, an international team led by researchers at MIT and Dana-Farber used large language models — AI systems trained on huge amounts of text that can understand and process unstructured information — to screen nearly 4,000 papers available through PubMed Central's open-access collection 1.

The LLMs identified statements about shared code, extracted links, and then software retrieved the linked code repositories for further assessment 1. Human reviewers independently checked a sample of 500 papers and repositories to verify the AI's work, resolving disagreements before comparing results .

Tom Pollard, who is the senior author of the study 1 and a research scientist at MIT's Laboratory for Computational Physiology as well as the Computer Science and Artificial Intelligence Laboratory, said that the biggest strength of LLMs is how they handle unstructured data such as text.

What they found

The review revealed two major gaps 1:

Code sharing remains uncommon. Only about one in eight studies reported sharing code, and where code was shared, the quality varied wildly 1. Some code repositories were highly structured and well documented, whereas others provided very little direction whatsoever .

Public code does not equal reusable code. Even when researchers made their code available online, many failed to include the documentation, software requirements, and supporting information needed for someone else to actually run it and reproduce the results 1.

Practices also varied substantially across journals and countries 1. Across different journals, the share of studies that made their code available varied from none to over 70 percent . Certain journals that had low code-sharing rates still maintained official policies requiring it, indicating that having rules on paper may not alone be sufficient to encourage researchers to share their work .

Thomas Sounack, the study's first author, explained that good code sharing means "giving enough information to understand what the code does and be able to run it to reproduce the results reported in the paper" 1.

Setting a new standard

The findings, recently published in Nature Medicine, are now informing the development of TRIPOD-Code, a reporting guideline designed to clarify what researchers should share alongside their code 1. TRIPOD-Code builds on existing checklists called TRIPOD and TRIPOD+AI, which help researchers report how clinical prediction models were developed and evaluated .

Pollard said the goal is practical: "We want to give researchers practical guidance on how to share code, and give journals clearer expectations to communicate to authors" 1.

This story reports on a trend for information only. It is not advice or encouragement to do any of it.

How good is it

Only about one in eight studies reported sharing code, where code was actually shared 1. When researchers did make code public online, many failed to include the documentation and software requirements needed for someone else to run it and reproduce results . The proportion of studies sharing code varied from zero to over 70 percent depending on the journal, suggesting inconsistent practices rather than uniform quality . Having written code-sharing policies appears insufficient on its own, since some journals with low sharing rates still maintained official policies requiring it .

Corrections and updates

  • Corrected on 20 November 2026. An earlier version called TRIPOD-Code 'a new reporting guideline'; the source describes it specifically as 'a reporting guideline being developed,' reflecting its current developmental stage rather than simply labeling it new.

Key facts

  • Nearly 4,000 medical studies and hundreds of code repositories were examined using AI
  • Only about one in eight studies reported sharing code
  • Where code was shared, quality varied enormously
  • - Findings are shaping TRIPOD-Code, a reporting guideline being developed
  • Study published in Nature Medicine by MIT and Dana-Farber researchers

Sources

  1. AI helps researchers investigate how thousands of medical studies share code

If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work

Keep reading