Evaluating Clinical Implementation of Risk Prediction Based Interventions Using Difference-In-Differences.

Authors

Maricela Cruz, Susan M Shortreed, Gregory E Simon, Yates Coley

Published in

Health services research. Pages e70015. Jul 21, 2025. Epub Jul 21, 2025.

Abstract

To compare alternative Difference-in-Differences (DID) methods for evaluating the effect of risk-stratified interventions, or interventions targeting at-risk groups, on binary outcomes.
In simulations, we compared operating characteristics of recycled prediction estimators for common average treatment effect on the treated (ATT) estimands across three DID models: the traditional two groups and two periods model, a risk score adjusted model, and a model adjusting for risk score and its interactions with risk group and period. We compared DID ATT estimates to randomized evaluation estimates of a risk-stratified intervention implemented at Kaiser Permanente Washington (KPWA), delivering additional text-message reminders to reduce missed clinic visits.
Our study included 588,503 KPWA visits, with 285,814 (49%) visits pre-evaluation (05/01/2018-10/30/2018) and 302,689 (51%) visits during the evaluation (02/01/2019-09/30/2019). Pre-evaluation, 120,350 visits were classified as high-risk. During the evaluation, 125,076 visits were labeled as high-risk, with 62,557 (50%) randomized to the intervention. We generated data in simulations based on this setting.
In simulations, the traditional DID and risk score adjusted models had smaller bias and standard errors, and better coverage probabilities. DID estimates closest to randomized evaluation estimates (-0.007, 95% CI [-0.010, -0.004]) were from the traditional DID model assuming the identity link (-0.008, 95% CI [-0.011, -0.005]) or the risk adjusted model with any link (-0.006, 95% CI [-0.008, -0.003] identity; -0.007, 95% CI [-0.011, -0.003] logit; -0.007, 95% CI [-0.012, -0.003] log) for the ATT on the absolute difference scale (usual DID ATT estimand), and the risk score adjusted model with log or logit links for all other estimands.
Compared with randomized evaluation results, the traditional DID model is appropriate for the ATT on the absolute difference scale, while the risk score adjusted model with log or logit links is appropriate for all ATT estimands considered.

PMID:
40685898
Bibliographic data and abstract were imported from PubMed on 21 Jul 2025.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.