Effects of segmentation granularity on nutrient prediction from real-world food images.

Authors

Liam Hand, Guido Camps, Ricardo da Silva Torres, Albert Ali Salah

Published in

Frontiers in nutrition. Volume 13. Pages 1927530. Epub Oct 07, 2026.

Abstract

Diet quality is central to preventing non-communicable diseases, yet traditional dietary assessment methods are burdensome and error-prone. Estimating nutrients directly from food images offers a scalable alternative, but most methods treat a meal as a single visual entity and estimate calories and mass more accurately than individual macronutrients. It has not yet been systematically evaluated whether decomposing a meal into its ingredients - predicting nutrients per ingredient and aggregating to the dish level - improves estimation. We compared three input representations for nutrient regression from overhead food images: direct regression from the original image, regression from binary food-region masks, and regression from fine-grained ingredient-level masks. To enable this, we annotated a 1,056-image subset of the Nutrition5K dataset with 2,922 ingredient-level segmentation masks across 40 categories, and used an identical regression model for all conditions. We also evaluated a realistic setting with automatically generated masks. Contrary to our hypothesis, finer segmentation did not improve overall performance: the unsegmented and binary food-region segmentation conditions performed comparably, while ingredient-level segmentation resulted in the highest error. Ingredient-level predictions preserved the relative balance of macronutrients well per ingredient, but errors in each ingredient's estimated nutrient quantities accumulated once aggregated to the dish level, leading to a general, though not strict, tendency toward worse performance as dish complexity increased. Ingredient-level regression stayed competitive for simpler dishes with visually separable ingredients, but struggled when ingredients were highly mixed. These findings indicate that the principal limitation of ingredient-level nutrient estimation is accurate quantity estimation rather than modeling of nutrient composition. Future work should prioritize explicit quantity cues, such as depth or volume, to complement ingredient-level decomposition. We release our annotations and code to support reproducible research.

PMID:
42908751
Bibliographic data and abstract were imported from PubMed on 23 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.