Evaluating AI performance on end-to-end data science projects.

Authors

Evelyn Hughes, Rohan Alexander

Published in

PloS one. Volume 21. Issue 11. Pages e0361887. Epub Nov 11, 2026.

Abstract

Generative AI is used for knowledge work but many current AI evaluations focus on isolated tasks and constituent parts such as writing code, rather than end-to-end workflows. We test whether AI models can generate end-to-end data science projects. This requires AI models to select a dataset, develop a research question, write code to download, clean, and test data, create figures, and write a reproducible paper. To do this we collect 40 real end-to-end data science projects, then grade them according to a rubric. We use these to build an automated grading pipeline that evaluates data science projects and aligns with the grades that human graders give. For each of nine AI models we then generate five projects and use the automated grader to evaluate them. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment such as explaining measurement, writing abstracts, and citing sources. Anthropic's Claude Opus 4.8 averaged 87 per cent and OpenAI's GPT-5.4 averaged 85 per cent, both around the level of an A for an undergraduate. No model was consistently the best across all aspects of the rubric. These findings suggest that while AI models could soon approximate entry-level data scientists on routine tasks, they require consistent verification and evaluation.

PMID:
43013607
Bibliographic data and abstract were imported from PubMed on 12 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.