Open Analytics from inside any study to see how the study is performing while it’s still running — compliance, response times, drop-off, and per-participant engagement, all on one screen. The panels are designed around documented threats to ESM data quality rather than around generic product-analytics templates: each chart corresponds to a construct that the methodological literature flags as something you should be watching, not exporting a CSV to compute after the fact.
Why monitor at all?
Experience-sampling and ecological momentary assessment depend on repeated, in-situ self-reports collected over many days. Four findings from the literature make real-time monitoring more than a convenience:
- Compliance is high on average — and wildly variable.
- Across published ESM studies, compliance averages roughly 70–80%, but individual studies range from below 50% to above 95%. Wen et al., 2017; Rintala, Wampers, Myin-Germeys & Viechtbauer, 2020. The mean is comforting, the spread is not — you cannot assume a new study will sit near the average.
- Attrition is front-loaded.
- Drop-off is steepest in the first one to three days of the protocol. Eisele, Vachon, Lafit et al., 2022. Anything you do to retain participants — reminders, contact, schedule adjustment — has to happen inside that window.
- Missed beeps are not random.
- Symptom severity, time of day, and social context all bias which prompts get answered. Vachon et al., 2019; Sokolovsky et al., 2014. Missing-not-at-random is the default case in ESM, not the edge case.
- Late responses corrupt “momentary”.
- A reply that arrives forty minutes after the prompt is a different measurement than one that arrives within two. Stone et al., 2003. Without surfacing the latency distribution you can’t set, or defend, a threshold.
Panel 1 — Overview
Four headline numbers at the top of the page: Total sent in the selected window, response rate (responded ÷ sent), mean latency from prompt to first tap, and active today (participants who responded at least once). Below them, a daily-response-rate time series with a 7-day rolling mean. Use the day-range selector (1 / 7 / 14 / 30 / 90) in the page header to widen or narrow the window — every panel below respects that choice.
Treat this as the dashboard for the dashboard. If the headline rate is healthy and the curve is flat, you probably don’t need to drill further. If the curve is sloping down, the panels that follow will tell you where the loss is coming from.
Panel 2 — Retention curve
A line chart by relativestudy day, not calendar day: each participant counts from their own day 0. Two series:
- Eligible — participants who have been enrolled long enough to reach that relative day. This is the denominator.
- Active — participants who actually responded on or before that relative day. This is the numerator.
Relative-day alignment is standard in survival analysis but rare in ESM tooling. Eisele et al., 2022; Vachon et al., 2019. It separates the question “is my protocol losing people over time?” from the question “has my cohort had time to drop off yet?” — questions a calendar-day plot conflates and can therefore answer wrong.
Read the gap between the two lines as your dropout rate at that relative day. A widening gap in days 1–3 is the front-loaded attrition pattern from the literature.
Panel 3 — Delivery funnel
Three stacked stages over the selected window: sent → opened (tapped) → completed. The drop between adjacent stages is flagged in coral.
The point of the funnel is to decompose “non-response” into two remediations that are often confused:
- Engagement loss · sent → opened.
- The push went out but the participant didn’t tap. Two very different things hide inside this drop. Technical: permissions revoked, app uninstalled, battery optimisation, stale push token — the notification never surfaced on the device. Behavioural: the notification did surface, but prompt fatigue, inopportune timing, or irrelevant content meant the participant ignored it. The dashboard cannot separate these from server-side data alone — see the note below — so when this stage is leaking, work both angles: re-onboard one or two flagged participants from Panel 7 to rule out the technical case, then look at schedule timing and content.
- Abandonment · opened → completed.
- The participant tapped the notification but did not finish the survey. Causes are usually the survey itself: too long, too intrusive, broken on mobile. Action: shorten or restructure the instrument.
Panel 4 — Response-time distribution
A histogram of latencies from prompt to first tap, bucketed < 5m, 5–15m, 15–30m, 30–60m, 1–2h, 2h+. The panel header shows median and the share of responses under 15 minutes — a common operational threshold.
Stone and colleagues argued that an ESM reply only carries the “momentary” in EMA if it arrives close enough to the prompt for the participant’s state not to have meaningfully shifted. Stone et al., 2003; Shiffman, Stone & Hufford, 2008. Different protocols draw the line differently — 5 minutes for affective work, 30 minutes for behaviour, sometimes an hour for diary designs — so Samply surfaces the distribution rather than imposing a single threshold. Use it to set, and to defend, the cut-off you apply when you analyse.
Panel 5 — Hourly response pattern
Bar chart of response rate by hour-of-day, computed in each participant’s local timezone, then aggregated. Bars are coloured when the rate is above the daily average and grey when below.
Time-of-day compliance is a documented source of diurnal selection bias: if response rate sags in the evening, your evening sample is biased toward whatever predicts being available in the evening. Sokolovsky et al., 2014; Smyth et al., 1998. Use this panel to spot hours where the sampling protocol effectively breaks down, and either adjust the schedule window or weight the analysis accordingly.
Panel 6 — Schedule performance
One row per notification schedule defined in the study, showing sent, opened, and compliance. A row tagged (untracked schedule) aggregates results that were sent before Samply began attributing each Result document to its originating schedule — those rows have no config link and are bucketed together.
Comparing fixed vs. random vs. event-contingent schedules within a single study is one of the few places where the literature is unambiguous about what to look at. van Berkel, Ferreira & Kostakos, 2017. Use this panel to identify schedules that under-perform the others — a debriefing schedule at the end of the study, for example, will often sit far below your daily ones — and to decide whether the difference reflects participant burden, content, or timing.
Panel 7 — Participant compliance
A sortable, filterable table of every active participant, with sent, opened, compliance %, and last active. Rows below the configurable threshold (default 60%) are flagged in coral.
Group means hide the participants who actually need intervention. A study averaging 73% can perfectly well contain participants at 20% whose data is unreliable, and participants at 95% who are carrying the average — collapsing them produces a single number that is unactionable. Jones et al., 2019. Use this table to send targeted reminders, decide which participants to drop from the analytic sample, or simply to know who to contact before the dropout becomes permanent.
What this dashboard does not do
Four honest limitations, listed up front because they shape how you should read every panel above:
- Response time, yes. Response content validity, no.
- Careless responding, straight-lining, and inattentive answers are not detected. Those checks remain on the analyst. Meade & Craig, 2012.
- Beeps are weighted equally.
- A one-minute morning ping and a ten-minute end-of-day diary count the same in the compliance numerator. Whether to weight by participant burden, survey length, or design intent remains an open methodological question. Eisele et al., 2022.
- No inferential MNAR test.
- The dashboard surfaces what is missing and when it went missing, but missing-not-at-random must still be handled analytically. The field lacks a consensus method. Schafer & Graham, 2002; Silvia et al., 2014.
- Threshold defaults encode a position.
- The 60% low-compliance threshold is a methodological choice, not a natural constant. It’s configurable per study, but the default still shapes what you notice — read it as a starting point, not a rule.
Drill-down & export
Click any participant row in Panel 7 to jump to that participant’s result history. For analyses that need the raw timestamps and event log, use the Export CSV action from the History tab — every aggregation in Analytics is reproducible from that dump in your own statistical environment.
References cited above
- Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disease, 175(9), 526–536.
- Eisele, G., Vachon, H., Lafit, G., et al. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151.
- Jones, A., Remmerswaal, D., Verveer, I., et al. (2019). Compliance with ecological momentary assessment protocols in substance users: A meta-analysis. Addiction, 114(4), 609–619.
- Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.
- Myin-Germeys, I., Kasanova, Z., Vaessen, T., et al. (2018). Experience sampling methodology in mental health research. World Psychiatry, 17(2), 123–132.
- Rintala, A., Wampers, M., Myin-Germeys, I., & Viechtbauer, W. (2020). Momentary measurement of social-network compliance in experience sampling. Psychological Assessment, 32(1), 81–90.
- Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177.
- Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32.
- Silvia, P. J., Kwapil, T. R., Walsh, M. A., & Myin-Germeys, I. (2014). Planned missing-data designs in experience-sampling research. Behavior Research Methods, 46(1), 41–54.
- Smyth, J. M., Ottenbreit-Leftwich, A., et al. (1998). Stressors and mood measured on a momentary basis are associated with salivary cortisol secretion. Psychoneuroendocrinology, 23(4), 353–370.
- Sokolovsky, A. W., Mermelstein, R. J., & Hedeker, D. (2014). Factors predicting compliance to ecological momentary assessment among adolescent smokers. Nicotine & Tobacco Research, 16(3), 351–358.
- Stone, A. A., Shiffman, S., Schwartz, J. E., et al. (2003). Patient compliance with paper and electronic diaries. Controlled Clinical Trials, 24(2), 182–199.
- Trull, T. J., & Ebner-Priemer, U. (2014). The role of ambulatory assessment in psychological science. Current Directions in Psychological Science, 23(6), 466–470.
- Vachon, H., Viechtbauer, W., Rintala, A., & Myin-Germeys, I. (2019). Compliance and retention with the experience sampling method over the continuum of severe mental disorders. Journal of Medical Internet Research, 21(12), e14475.
- van Berkel, N., Ferreira, D., & Kostakos, V. (2017). The experience sampling method on mobile devices. ACM Computing Surveys, 50(6), 1–40.
- Wen, C. K. F., Schneider, S., Stone, A. A., & Spruijt-Metz, D. (2017). Compliance with mobile ecological momentary assessment protocols in children and adolescents. Journal of Medical Internet Research, 19(4), e132.