An End-User Audit of Reproducibility, Data Leakage, and Overfitting of the Top-Ranked ADMET Prediction Models in TDC Leaderboards.

Authors

Ihor Koleiev, Roman Stratiichuk, Nazar Shevchuk, Mykola Melnychenko, Alex Nyporko, Daniil Todoryshyn, Vladyslav Husak, Sergii Starosyla, Semen Yesylevskyy, Alan Nafiiev

Published in

Journal of chemical information and modeling. Jul 02, 2026. Epub Jul 02, 2026.

Abstract

Public leaderboards such as the Therapeutics Data Commons (TDC) ADMET benchmark are widely treated as a ranking of state-of-the-art models. However, a high leaderboard position is only meaningful if the corresponding model can actually be reproduced and deployed by an independent researcher. In this work, we audit whether the top-ranked TDC ADMET models meet that bar. We assessed the top-ranked models of all 22 TDC ADMET leaderboards from the perspective of an end user with access only to the publicly released artifacts of each model─its publication, code repository, and installation instructions. For every end point, the top three models were screened with a unified protocol including an execution environment reproducibility check, a data-leakage assessment, verification of the hyperparameter-optimization procedure, and a reevaluation against the current leaderboard. Only three models (CaliciBoost, MapLight, and MapLight + GNN) passed all stages and reproduced their reported performance. The remaining models failed because of unavailable code, nonreproducible environments, runtime incompatibilities, or methodological flaws. We traced direct or indirect data leakage in the MiniMol, GradientBoost, and XGBoost models, and used deliberately overfitted variants of our own Mol2Vec-based models to show that tuning on the public test set─whether accidental or intentional─can substantially inflate both metrics and leaderboard rank. These results indicate that current TDC leaderboard positions cannot be read as a direct measure of model quality and practical applicability and emphasize the urgent need for better public ADMET benchmarks based on the hidden test sets, strict data set versioning and model submission with standardized inference environments.

PMID:
42392971
Bibliographic data and abstract were imported from PubMed on 03 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.