Evaluating Visual Decision Support: How Does Preference Elicitation Shape Metric Sensitivity?

Authors

Lena Cibulski, Tobias Mertz, Evanthia Dimara, Stefan Bruckner

Published in

IEEE transactions on visualization and computer graphics. Volume PP. Jun 19, 2026. Epub Jun 19, 2026.

Abstract

Supporting decision-making is a central goal in visualization research, yet evaluating how effectively visualizations aid decision tasks remains difficult. Objective metrics for decision quality as a baseline for comparative evaluations are rare, with previous work proposing the consistency between a decision made and self-reported subjective preferences. However, the impact of preference elicitation design on such metrics is not well understood. Our focus is therefore on understanding how elicitation methods shape the sensitivity of decision quality metrics, not on comparing visualization techniques themselves. We report a preregistered study with 548 participants examining how varying the expressiveness of preference elicitation affects the sensitivity of choice consistency metrics when comparing parallel coordinates and tabular visualizations. Our baseline condition replicates a previous study using simple preference elicitation with significantly more participants and confirms its inconclusive findings that suggest a comparable performance of parallel coordinates and tabular visualization for the tested decision task. We further test two additional conditions to investigate if more expressive preference elicitation increases the metric's sensitivity. Our results suggest that more expressive elicitation may improve sensitivity in detecting performance differences, but further increases offer no clear additional benefit in our study. We observed these sensitivity differences even though the experimental setup, with its comparable baseline performances, was not optimized for detecting visualization performance differences. Our results provide initial evidence that elicitation design can affect the informational value of decisioncentric visualization studies, informing both the development of metrics and the interpretation of comparative studies. A preprint of this paper, along with the pre-registration, data analysis scripts, and all supplemental materials, is available at https://osf.io/u63zw/?view\_only=b8b04a7e505c45c89ae957674248c494.

PMID:
42319863
Bibliographic data and abstract were imported from PubMed on 20 Jun 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.