Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan.

Authors

Eiko Takaoka, S Ruban, B Mohammed Siraj, Varenya Vinay, Hilda Shanthini, Akshay Dias

Published in

Frontiers in digital health. Volume 8. Pages 1791066. Epub Oct 16, 2026.

Abstract

Machine-learning (ML) studies increasingly compare feature-importance rankings between small, unevenly sized cohorts and interpret the differences as evidence of culturally specific "risk architectures." We show that such comparisons can be highly misleading without leakage-free evaluation, using a worked example from our own cross-sectional lifestyle and health data collected during June-August 2024 from two universities in India and Japan. Three problems were identified in our preliminary analysis: (1) an outcome (obesity, defined as BMI ≥ 25) that was tautologically predicted by including BMI itself as a model input; (2) model evaluation without a held-out test set; and (3) feature-importance comparisons that ignored sampling variability from an eightfold difference in cohort size.
We addressed each problem using penalized logistic regression, Random Forest, and gradient-boosted trees with proper train/test splitting, held-out permutation importance, and bootstrap confidence intervals. We also ran a sensitivity analysis that repeatedly subsampled the Indian cohort down to the size of the Japanese cohort.
Removing body mass index (BMI) as a predictor collapsed obesity-model discrimination from perfect (AUC = 1.00) to chance level (AUC = 0.50-0.57), confirming that the BMI-obesity association was definitional rather than empirical. Held-out predictive performance for the remaining viable outcome, elevated self-rated stress, was modest at best (AUC 0.49-0.71). Critically, in the subsampling analysis, every feature-importance value observed in the actual Japanese data fell within the range produced by sample-size noise alone.
We present this analysis as a transparent, reproducible case study of specific, checkable pitfalls: label leakage, missing train/test evaluation, unadjusted small-sample comparison, and mislabeled psychometric instruments. These should be routinely screened for before ML feature importance is interpreted as evidence of population-level or cultural differences in digital health research.

PMID:
42964282
Bibliographic data and abstract were imported from PubMed on 31 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.