Influence of Evaluator Expertise on Inter-Rater Reliability in Crown Preparation Assessment Without Formal Calibration.

Authors

Eanas I Jellil, Mustafa N Abdulghani, Sundus H Naser, Amer A Mohammed, Farah Nabeel AlKhayyat

Published in

European journal of dental education : official journal of the Association for Dental Education in Europe. Sep 29, 2026. Epub Sep 29, 2026.

Abstract

To investigate the relationship between evaluator expertise and inter-rater reliability during crown preparation assessment, as well as whether students' academic level affects performance outcomes.
This research used a two-phase design. Initially, it enrolled 92 third-year and 90 fourth-year students; after subjecting the cohorts to predetermined inclusion criteria, 85 students of each cohort were retained to be included in the study. In Phase 1, the third-year students who prepared a metal ceramic crown on a typodont tooth were evaluated by two senior and two junior assessors using an analytic rubric that was validated on a 10-point scale. In Phase 2, the fourth-year students did the same task and it was only assessed by senior evaluators. The intraclass correlation coefficients (ICC) were used to analyse the inter-rater reliability, and t-tests were used to analyse differences in groups.
In Phase 1 the reliability of all the evaluators was low (ICC = 0.035). Within this phase senior evaluators showed moderate reliability in the absence of formal calibration (average-measure ICC = 0.634), and junior evaluators demonstrated systematic disagreement (ICC = -0.074). In Phase 2, senior evaluators demonstrated good-to-excellent agreement. Third- and fourth-year performance did not differ significantly (mean difference = 0.28; p = 0.255; d = 0.18).
Within Phase 1, inter-rater reliability differed descriptively between evaluator groups, with senior evaluators demonstrating more consistent scoring patterns than junior evaluators in the absence of formal calibration. There was no statistically significant difference observed in academic levels in this cohort.

PMID:
42806874
Bibliographic data and abstract were imported from PubMed on 29 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.