Best practices

Guidance for designing custom checks that surface real regressions without noisy false alarms. For an overview of how custom checks work, see Custom checks. For a worked example, see Writing a custom check.

Start with built-in snapshot types such as network-requests, js-bundle-sizes, and react-renders — they capture concrete, reproducible signals (request counts, URLs, bundle byte sizes, React commit counts) that usually only change when your application behaviour or assets change.

Metrics that fluctuate between replays even when nothing meaningful changed — CPU usage, wall-clock timing — are difficult to threshold reliably and tend to produce false alarms. Treat them with scepticism: at most use them as exploratory, non-blocking warnings, never as warnings that require a reviewer's acknowledgement.

Meticulous visual tests compare the head commit against its base and fail when they detect a visual difference. Custom checks should follow the same diff semantics: compare each metric on head against the same session's baseline on base, and only surface a warning or require acknowledgement when there is a meaningful difference relative to base.

Avoid absolute thresholds that alarm whenever a metric crosses a fixed number (for example, "require acknowledgement if network requests exceed 50"). That pattern ignores whether the change is a regression — a session that always made 60 requests would fail every run even when your commit did not touch networking code. Instead, compare the delta between base and head (for example, "require acknowledgement if head makes more than 20% more requests than base for the same session").

Many Meticulous sessions are short — a quick page view or a single interaction may produce only a handful of snapshots or requests. At that scale, a +1 delta or a large percentage swing is usually noise rather than a real regression (for example, 1 → 2 requests is +100%).

Exclude session pairs below a minimum data floor before they can warn. Writing a custom check uses MIN_REQUESTS_FOR_ALARM for this: neither base nor head must reach at least 3 meaningful requests before the check can alarm.

If your check compares base vs head at the session level rather than aggregating across the whole test run, include only sessions that ran on both base and head. Sessions that appear only on head — for example because a new flow was added to the test pool to exercise a feature under development — have no baseline to regress against. Omit them from the verdict rather than treating new head traffic as a failure.

Not every captured snapshot reflects the behaviour you care about. Filter out irrelevant data before comparing base and head so the check tracks real product changes, not background traffic.

In a network request capacity check, count only meaningful requests — for example, calls to your own backend (same-origin /api/* routes) — and exclude third-party endpoints such as analytics, error tracking, font CDNs, and the Meticulous recorder. See Writing a custom check for an example filter.

Reserve the acknowledgement-required verdict (warn-and-require-user-ack) for clear, actionable regressions — for example, when head exceeds base capacity by 20% or more. Requiring acknowledgement on smaller increases creates noisy PR friction and trains reviewers to ignore the check.

When you do track a metric that fluctuates naturally between runs, prefer a non-blocking warning (warn-without-requiring-user-ack) over one that requires acknowledgement, so the Checks tab surfaces a signal without gating merges.

A verdict on its own rarely tells a reviewer why a check passed or failed. Include enough context in the markdown report for someone to understand and act on the result without re-running anything: which sessions triggered the verdict, the base and head values being compared, the delta and the threshold it crossed, and a breakdown of the specific endpoints, bundles, or metrics responsible.

Where it helps, link directly to the relevant sessions in the Meticulous UI so a reviewer can jump straight to the replay. A clear, self-explanatory report is what lets reviewers confidently accept or address a check that requires acknowledgement.

All custom check results for a test run must be submitted together in a single reportCustomCheckResults call. Do not split checks across separate CI jobs that POST independently as each finishes.

Before posting results to Meticulous, run your reporter against a real completed test run and inspect the output. Plan for a dry-run mode (for example a --dryRun flag) that executes your check logic and prints verdicts and markdown reports without calling the reporting API. Confirm the output matches your expectations — filtering, thresholds, session links, and breakdowns — before reporting for real. Once results are posted, reviewers see them in the Checks tab.