Digital Endpoint Evidence Generation Toolkit

Overview

Digital health technologies (DHTs) such as wearables and sensors generate rich, continuous data—but translating that data into regulatory-ready endpoints requires rigorous validation evidence. This project addressed the challenge of standardizing how that evidence is generated across multiple studies and therapeutic areas.

The toolkit concept provides a modular, reproducible approach to generating the analyses required to demonstrate an endpoint's measurement properties: whether it measures what it claims to measure, whether it can detect clinically meaningful change, and whether it behaves consistently across populations.

The Problem

In pharma clinical development, digital endpoint validation evidence is essential for regulatory acceptance. The typical analyses include:

• Known-groups validity: Does the endpoint differentiate between groups expected to differ?
• Convergent/divergent validity: Does it correlate appropriately with related and unrelated measures?
• Anchor-based responsiveness: Does it detect change when patients report improvement or decline?
• Sensitivity-to-change: Can it detect treatment effects with adequate precision?

Without standardization, each study team reinvents these analyses from scratch. This leads to:

• Inconsistent methodological choices across programs
• Duplicated effort and longer timelines
• Difficulty comparing results across studies
• Knowledge loss when team members transition

Approach

1. Modular Function Library

Designed a set of R functions that encapsulate standard validation analyses. Each function follows consistent input/output patterns and includes built-in documentation. The modular design allows analyses to be combined flexibly depending on study needs.

2. Valid Dataset Specification

Defined clear rules for what constitutes a "valid" observation for analysis purposes. This includes minimum wear time requirements, data completeness thresholds, and handling of edge cases—documented upstream so that analysis results are traceable to their data foundations.

3. Reproducible Report Templates

Created parameterized RMarkdown templates that generate complete validation evidence reports. Users specify study-specific parameters (variable names, thresholds, population definitions), and the template produces publication-ready outputs with tables, figures, and interpretation guidance.

4. Shiny Interface Concept

Designed (but did not fully implement) a Shiny application interface that would allow non-programmer users to configure analyses, upload data, and generate reports through a web interface. This concept demonstrated how the toolkit could scale to broader organizational use.

Outcomes

• Reduced setup time: Analysts could begin validation analyses faster by starting from standardized templates rather than building from scratch
• Improved consistency: Standardized methods ensured comparable approaches across studies, supporting meta-analyses and cross-program insights
• Knowledge transfer: New team members could onboard faster by learning a documented system rather than deciphering ad-hoc scripts
• Regulatory readiness: The structured approach aligned with expectations from regulatory guidance on digital endpoint validation

Key Learnings

• Documentation is code: Well-documented functions with clear examples are essential for adoption. Code that others can't understand won't be reused.
• Start with the end in mind: Designing for regulatory submission requirements from the outset ensures that outputs meet the bar when it matters.
• Iterate with users: Early feedback from biostatisticians and clinical scientists improved both the technical design and the usability of outputs.
• Balance flexibility and standardization: The toolkit needed to accommodate study-specific variations while maintaining consistent core methods.

Tools & Methods