Writing a custom check

This guide walks through building a network request capacity check from scratch: a custom check that makes sure sessions on the head test run do not issue many more network requests than they did on the base.

It is organised in four sections, mirroring what every custom check does:

1. Read the snapshot data captured during replay.
2. Compute the result of the check by comparing base and head.
3. Report the result back to Meticulous.
4. Set up CI so the check runs on every PR.

Everything lives in a single report.ts file that we build up as we go.

A complete, runnable version of this check lives in the custom-checks-examples repository.

Each recorded session has a baseline on the base test run — the number of meaningful HTTP requests that session normally issues. That baseline is the session's capacity. On every PR we compare head against base per session, and if head exceeds capacity by too much the check surfaces a warning or failure before the change merges.

Before following this guide, make sure you have:

• A Meticulous project with CI tests set up so every PR produces head and base test runs.
• Custom checks enabled for your project. Contact the Meticulous team to enable the custom checks.
• Your Meticulous API token.

Before writing any code, choose a completed test run to use as a running example while you build the reporter. You will point the script at it repeatedly to check that your filtering, thresholds, and report read the way you expect. Grab its id from the test run's URL in the Meticulous app — https://app.meticulous.ai/projects/<org>/<project>/test-runs/<testRunId>.

It helps to pick a test run that actually exhibits the regression you are checking for, so you can confirm the check fires, not just that it runs. For a network request capacity check, a simple way to produce one is to open a PR that deliberately increases network traffic — for example, adding a short polling interval that re-fetches an endpoint — and use the test run associated with that PR as your example. Once the check is working, you can drop the PR.

A custom check is just a small Node script — the reporter — that you run after a Meticulous test run finishes. It talks to Meticulous through the published @alwaysmeticulous/custom-checks SDK, which handles authentication, resolving test runs, downloading data, and posting results.

Create a dedicated directory (for example custom-checks/ at the repo root) with its own package.json and a single report.ts we will build up as we go:

{
  "name": "my-custom-checks",
  "private": true,
  "scripts": {
    "report": "ts-node report.ts"
  },
  "dependencies": {
    "@alwaysmeticulous/custom-checks": "^2.296.0"
  },
  "devDependencies": {
    "ts-node": "^10.8.1",
    "typescript": "^5.9.3"
  }
}

Use the latest @alwaysmeticulous/custom-checks release from npm — the version above is current at the time of writing. Install dependencies with your package manager (npm install, pnpm install, etc.).

The custom-checks-examples repo has this project ready to run if you'd rather skip the boilerplate.

Before computing anything, get a minimal report.ts running that connects to Meticulous and resolves your example test run. This confirms your token and SDK setup work before you add any check logic.

Two SDK functions get us started:

• createClient(...) opens an authenticated connection to Meticulous from your API token. Every other SDK call takes the client it returns.
• findTestRunForCustomChecks(...) takes a test run id and returns the resolved run together with its base run (waiting for the run to finish if it is still in progress). That head/base pair is what a check compares.

Start with all the imports we will need across the script:

import {
  createClient,
  findTestRunForCustomChecks,
  findTestRunByCommitForCustomChecks,
  getSnapshotsFromTestRun,
  reportCustomCheckResults,
  type MeticulousClient,
  type Snapshot,
  type CustomCheckVerdict,
  type ReportedCustomCheckResult,
} from "@alwaysmeticulous/custom-checks";

const testRunId = process.argv[2];

const client = createClient({
  apiToken: process.env.METICULOUS_API_TOKEN,
  appInfo: "my-app/custom-checks",
});

const { testRun } = await findTestRunForCustomChecks({
  client,
  testRunId,
  // We are only exploring here, not reporting — don't register this run as
  // expecting custom checks (explained under "Report the result").
  skipRegisteringExpectedCustomChecks: true,
});

console.log(`Resolved test run ${testRun.id} (${testRun.status}): ${testRun.url}`);

Run it against the example test run you picked earlier:

METICULOUS_API_TOKEN=<token> pnpm run report -- <testRunId>

We pass skipRegisteringExpectedCustomChecks: true here because we are only exploring and will not report results — the Report the result section explains this flag. Everything from here on is added to this same report.ts file.

The data a custom check compares comes from snapshots. While Meticulous replays each session — on both the head and base deployments — it records structured data points called snapshots and stores them alongside the replay. Each snapshot is tagged with:

• sessionId — which session it was captured in, so you can align the same session on base and head. A session is essentially a recorded user flow that Meticulous replays.
• sessionDescription — a short, human-readable summary of what the user was doing in that session (for example Added an item to the cart), useful for labelling sessions in your report. It is null when the session has no description, so treat it as optional.
• type — the kind of snapshot (for example network-requests).
• stageDuringSession — which screenshot in the session timeline the data belongs to.
• data — the payload, whose shape depends on the snapshot type.

Some snapshot types are built in and need no application code — Meticulous captures them automatically. The built-in types today are network-requests (every fetch / XHR a session made), js-bundle-sizes (the JavaScript bundles it loaded), react-renders (the cumulative React commit count), and react-component-renders (a per-component breakdown of which components re-rendered). You can also record your own from browser code; see Built-in snapshot types and recording custom snapshots.

This check uses network-requests. Each such snapshot's data describes a single request:

interface NetworkRequestSnapshotData {
  url: string;
  method: string;
  requestBody?: string;
  status: number | null;
  responseBody?: string;
  // ...plus request/response headers and other metadata
}

So one session that made 12 requests produces 12 network-requests snapshots, all sharing that session's sessionId.

The SDK function for reading snapshot data is getSnapshotsFromTestRun(...). You pass it the client, a test run id, and the snapshotTypes you care about; it downloads every matching snapshot for both the head run and its base and returns them as two arrays, baseSnapshots and headSnapshots. This is the entry point for any check, whatever snapshot type it reads.

Add the snapshot type constant and a computeNetworkRequestsCheck function near the top of report.ts, above the resolving code you added in Resolve a test run. For now it just fetches and logs how much data we got:

const NETWORK_REQUESTS_SNAPSHOT_TYPE = "network-requests";

const computeNetworkRequestsCheck = async (
  client: MeticulousClient,
  testRunId: string,
): Promise<void> => {
  const { baseSnapshots, headSnapshots } = await getSnapshotsFromTestRun({
    client,
    testRunId,
    snapshotTypes: [NETWORK_REQUESTS_SNAPSHOT_TYPE],
  });

  console.log(
    `Fetched ${baseSnapshots.length} base and ${headSnapshots.length} head snapshots.`,
  );
};

Then call it after resolving the run:

await computeNetworkRequestsCheck(client, testRun.id);

Re-run the script — you should see non-zero counts. For large test runs this download can take some time, since one snapshot is written per captured request across every session on both base and head. We will change the return type to a check result once we have something to report.

We now know the raw shape of the data. Next, capture the rules of the check as types and constants so the comparison logic stays readable. Add these near the top of report.ts:

/** Stable id shown in the Meticulous UI. */
const CHECK_ID = "network-requests";

/** Surface a non-blocking warning once a session exceeds base capacity by this much. */
const WARN_PERCENT_INCREASE_THRESHOLD = 10;
/** Require reviewer acknowledgement once a session exceeds base capacity by this much. */
const REQUIRE_ACK_PERCENT_INCREASE_THRESHOLD = 20;

/** Ignore low-traffic sessions where +1 request is noise. */
const MIN_REQUESTS_FOR_ALARM = 3;

interface EndpointComparison {
  label: string;
  baseCount: number;
  headCount: number;
  delta: number;
}

interface SessionComparison {
  sessionId: string;
  // Short description of what the user did in the session (e.g. "Added an item
  // to the cart"), used to label the session in the report. `null` when the
  // session has no description.
  sessionDescription: string | null;
  baseCount: number;
  headCount: number;
  delta: number;
  percentIncrease: number;
  endpoints: EndpointComparison[];
}

A custom check reports one of three verdicts, typed by the SDK as CustomCheckVerdict:

• pass — no regression; the check is green and no report is surfaced.
• warn-without-requiring-user-ack — surfaces a report in the Meticulous UI as a signal for reviewers to glance at, but does not block the pull request or require anyone to act on it.
• warn-and-require-user-ack — surfaces a report that a reviewer must explicitly acknowledge (review) in the Meticulous UI before the run is considered actioned, the same way an unreviewed visual diff blocks until someone accepts or ignores it.

Note there is no fail verdict: instead of failing outright, a check escalates by requiring acknowledgement. (A check that errors while running is a separate, run-level concern, not a verdict.)

That distinction drives the two thresholds here. We surface a non-blocking warning once a session exceeds its base capacity by 10%, and require acknowledgement at 20% — reserving the acknowledgement-required verdict for clear, actionable regressions so a single retried API call does not gate a PR. SessionComparison and EndpointComparison are the per-session and per-endpoint breakdown we will produce next.

Not every HTTP request reflects your application's behaviour. Analytics beacons, error trackers, font CDNs, and the Meticulous recorder itself all issue requests during replay. Counting them would let a session "regress" on telemetry noise rather than on product traffic.

Each downloaded snapshot is a Snapshot — the SDK type for the data points from Understand snapshots, carrying sessionId, sessionDescription, type, stageDuringSession, and a data field typed as unknown (the SDK does not know the shape of every snapshot type). So declare the minimal shape you need and decide which requests are meaningful:

interface NetworkRequestData {
  url?: string;
  method?: string;
  requestBody?: string;
}

const networkRequestData = (snapshot: Snapshot): NetworkRequestData =>
  (snapshot.data ?? {}) as NetworkRequestData;

/** Hostname substrings to exclude from the comparison. */
const IGNORED_REQUEST_HOST_SUBSTRINGS = [
  "sentry.io",
  "segment.io",
  "google-analytics.com",
  "googletagmanager.com",
  // Add the third-party hosts your app loads during replay.
];

const isMeaningfulRequest = (snapshot: Snapshot): boolean => {
  const { url } = networkRequestData(snapshot);
  if (!url) {
    return true;
  }
  try {
    const hostname = new URL(url).hostname.toLowerCase();
    return !IGNORED_REQUEST_HOST_SUBSTRINGS.some((needle) =>
      hostname.includes(needle),
    );
  } catch {
    // Relative URLs (e.g. "/api/graphql") are same-origin app traffic.
    return true;
  }
};

Tailor IGNORED_REQUEST_HOST_SUBSTRINGS to your stack. Same-origin and relative URLs should always count as meaningful.

To explain which endpoints regressed, group meaningful requests by a human-readable label, then bucket them into per-session, per-endpoint counts:

/** GraphQL calls by operation name; everything else as `METHOD /path`. */
const describeRequest = (snapshot: Snapshot): string => {
  const { url, method } = networkRequestData(snapshot);
  if (!url) {
    return "(unknown request)";
  }
  const verb = (method ?? "GET").toUpperCase();
  try {
    const { pathname } = new URL(url);
    return `${verb} ${pathname}`;
  } catch {
    return `${verb} ${url.split("?")[0]}`;
  }
};

type RequestCountsByEndpoint = Map<string, number>;

const countRequestsBySession = (
  snapshots: Snapshot[],
): Map<string, RequestCountsByEndpoint> => {
  const countsBySession = new Map<string, RequestCountsByEndpoint>();
  for (const snapshot of snapshots) {
    if (snapshot.type !== NETWORK_REQUESTS_SNAPSHOT_TYPE) continue;
    if (!isMeaningfulRequest(snapshot)) continue;

    let byEndpoint = countsBySession.get(snapshot.sessionId);
    if (!byEndpoint) {
      byEndpoint = new Map();
      countsBySession.set(snapshot.sessionId, byEndpoint);
    }
    const label = describeRequest(snapshot);
    byEndpoint.set(label, (byEndpoint.get(label) ?? 0) + 1);
  }
  return countsBySession;
};

Align sessions by sessionId and compare each session's head request count against its base capacity. Skip sessions that only ran on head — they have no baseline capacity to compare against:

const sumCounts = (byEndpoint: RequestCountsByEndpoint): number =>
  [...byEndpoint.values()].reduce((total, count) => total + count, 0);

const compareEndpoints = (
  base: RequestCountsByEndpoint,
  head: RequestCountsByEndpoint,
): EndpointComparison[] => {
  const labels = new Set([...base.keys(), ...head.keys()]);
  return [...labels].map((label) => {
    const baseCount = base.get(label) ?? 0;
    const headCount = head.get(label) ?? 0;
    return { label, baseCount, headCount, delta: headCount - baseCount };
  });
};

// First non-null `sessionDescription` seen per `sessionId`, so each session can
// be labelled by what the user did rather than by its opaque id.
const collectSessionDescriptions = (
  ...snapshotLists: Snapshot[][]
): Map<string, string | null> => {
  const descriptions = new Map<string, string | null>();
  for (const snapshots of snapshotLists) {
    for (const snapshot of snapshots) {
      const existing = descriptions.get(snapshot.sessionId);
      if (existing == null) {
        descriptions.set(snapshot.sessionId, snapshot.sessionDescription ?? null);
      }
    }
  }
  return descriptions;
};

const compareSessions = (
  baseSnapshots: Snapshot[],
  headSnapshots: Snapshot[],
): SessionComparison[] => {
  const baseCounts = countRequestsBySession(baseSnapshots);
  const headCounts = countRequestsBySession(headSnapshots);
  const descriptions = collectSessionDescriptions(baseSnapshots, headSnapshots);
  const comparisons: SessionComparison[] = [];

  for (const sessionId of headCounts.keys()) {
    // Only compare sessions that also ran on base.
    if (!baseCounts.has(sessionId)) continue;

    const baseByEndpoint = baseCounts.get(sessionId) ?? new Map();
    const headByEndpoint = headCounts.get(sessionId) ?? new Map();
    const baseCount = sumCounts(baseByEndpoint);
    const headCount = sumCounts(headByEndpoint);

    comparisons.push({
      sessionId,
      sessionDescription: descriptions.get(sessionId) ?? null,
      baseCount,
      headCount,
      delta: headCount - baseCount,
      percentIncrease:
        baseCount === 0
          ? headCount > 0
            ? Infinity
            : 0
          : ((headCount - baseCount) / baseCount) * 100,
      endpoints: compareEndpoints(baseByEndpoint, headByEndpoint),
    });
  }

  return comparisons.sort((a, b) => b.delta - a.delta);
};

Every check must end on a single verdict, typed by the SDK as CustomCheckVerdict — the union "pass" | "warn-without-requiring-user-ack" | "warn-and-require-user-ack". Turn the per-session comparisons into one of those values: did any session exceed its base capacity beyond the warn or acknowledgement thresholds? Use integer arithmetic on counts so boundary conditions (exactly +10% or +20%) are stable:

const hasEnoughTraffic = (c: SessionComparison) =>
  Math.max(c.baseCount, c.headCount) >= MIN_REQUESTS_FOR_ALARM;

const requiresAck = (c: SessionComparison) =>
  hasEnoughTraffic(c) &&
  c.baseCount > 0 &&
  c.headCount * 100 >=
    c.baseCount * (100 + REQUIRE_ACK_PERCENT_INCREASE_THRESHOLD);

const isWarning = (c: SessionComparison) => {
  if (!hasEnoughTraffic(c) || c.delta <= 0 || requiresAck(c)) return false;
  if (c.baseCount === 0) return true; // new meaningful traffic on head
  return (
    c.headCount * 100 >=
    c.baseCount * (100 + WARN_PERCENT_INCREASE_THRESHOLD)
  );
};

const computeVerdict = (
  comparisons: SessionComparison[],
): CustomCheckVerdict => {
  if (comparisons.some(requiresAck)) return "warn-and-require-user-ack";
  if (comparisons.some(isWarning)) return "warn-without-requiring-user-ack";
  return "pass";
};

A verdict on its own does not tell a reviewer why the check failed. Meticulous renders a markdown report in the Checks tab, so build one that lists the sessions that exceeded capacity, with a per-endpoint table showing where the extra requests came from.

Because this check compares per session, link each session in the report to its view in the Meticulous app. That page shows the base vs head comparison for the session — screenshots, timeline, and simulation details — so reviewers can see what changed without hunting for the session id. Label the link with the session's sessionDescription (e.g. "Added an item to the cart") so reviewers recognise the flow at a glance, falling back to session1, session2, … by position when a session has no description. The URL pattern is:

https://app.meticulous.ai/projects/<organization>/<project>/test-runs/<testRunId>/sessions/<sessionId>

<organization> and <project> are the URL slugs from your project's path, <testRunId> is the head test run you are reporting against, and <sessionId> is the session id from the snapshot data.

const PROJECT_PATH =
  "https://app.meticulous.ai/projects/<organization>/<project>";

const sessionUrl = (testRunId: string, sessionId: string): string =>
  `${PROJECT_PATH}/test-runs/${testRunId}/sessions/${sessionId}`;

const buildReport = (
  verdict: CustomCheckVerdict,
  comparisons: SessionComparison[],
  testRunId: string,
): string => {
  const alarming = comparisons.filter((c) => requiresAck(c) || isWarning(c));
  const lines = [
    "# Network request capacity",
    "",
    `**Verdict:** ${verdict}`,
    "",
  ];

  alarming.forEach((comparison, index) => {
    // Prefer the session's recorded description; fall back to its position when
    // the session has none.
    const label = comparison.sessionDescription ?? `session${index + 1}`;
    lines.push(
      `## [${label}](${sessionUrl(testRunId, comparison.sessionId)}) — ${comparison.baseCount} → ${comparison.headCount} requests`,
      "",
      "| Endpoint | Base | Head | Δ |",
      "| --- | ---: | ---: | ---: |",
    );
    for (const endpoint of comparison.endpoints.filter((e) => e.delta > 0)) {
      lines.push(
        `| ${endpoint.label} | ${endpoint.baseCount} | ${endpoint.headCount} | +${endpoint.delta} |`,
      );
    }
    lines.push("");
  });

  return lines.join("\n");
};

A check hands its outcome back as a ReportedCustomCheckResult — the SDK shape that pairs your checkId with the verdict, a one-line summary shown inline in the UI, and a report (here { type: "markdown", markdown }). Every check, whatever it measures, returns this same shape.

Now finish the computeNetworkRequestsCheck function we started in Fetch the snapshots. Change its return type to Promise<ReportedCustomCheckResult> and, after fetching the snapshots, compute the comparisons, verdict, and report:

const computeNetworkRequestsCheck = async (
  client: MeticulousClient,
  testRunId: string,
): Promise<ReportedCustomCheckResult> => {
  const { baseSnapshots, headSnapshots } = await getSnapshotsFromTestRun({
    client,
    testRunId,
    snapshotTypes: [NETWORK_REQUESTS_SNAPSHOT_TYPE],
  });

  const comparisons = compareSessions(baseSnapshots, headSnapshots);
  const verdict = computeVerdict(comparisons);

  return {
    checkId: CHECK_ID,
    verdict,
    summary:
      verdict === "pass"
        ? "No sessions exceeded their network request capacity"
        : `${comparisons.filter(requiresAck).length || comparisons.filter(isWarning).length} session(s) exceeded network request capacity`,
    report: {
      type: "markdown",
      markdown: buildReport(verdict, comparisons, testRunId),
    },
  };
};

Two more SDK functions complete the round trip:

• findTestRunByCommitForCustomChecks(...) resolves a run from a commit SHA (waiting for it to finish). It is the CI-friendly counterpart to findTestRunForCustomChecks, which takes a run id — CI usually knows the commit, not the run id.
• reportCustomCheckResults(...) posts your computed results back to Meticulous in a single call (more on the "single call" rule below).

Wire the bottom of report.ts to resolve the test run, compute the check, and either print the result (while iterating) or post it back to Meticulous. Accept either a positional test run id (handy while developing against your example run) or --commitSha (what CI uses). Replace the resolving block from Resolve a test run with:

const dryRun = process.argv.includes("--dryRun");

const resolveTestRun = async () => {
  const commitShaFlagIndex = process.argv.indexOf("--commitSha");
  const commitSha =
    commitShaFlagIndex !== -1
      ? process.argv[commitShaFlagIndex + 1]
      : undefined;

  if (commitSha) {
    return findTestRunByCommitForCustomChecks({
      client,
      commitSha,
      // On a dry run we are only experimenting, so don't tell the backend that
      // results are coming for this run (see below).
      skipRegisteringExpectedCustomChecks: dryRun,
    });
  }

  // First positional argument after `pnpm run report --`.
  const testRunId = process.argv
    .slice(2)
    .find((arg) => arg !== "--" && !arg.startsWith("-"));
  if (testRunId) {
    return findTestRunForCustomChecks({
      client,
      testRunId,
      skipRegisteringExpectedCustomChecks: dryRun,
    });
  }

  throw new Error(
    "Pass a testRunId argument or --commitSha to identify the test run.",
  );
};

const { testRun } = await resolveTestRun();

const checks = [await computeNetworkRequestsCheck(client, testRun.id)];

if (dryRun) {
  for (const check of checks) {
    console.log(`${check.checkId}: ${check.verdict}\n${check.report.markdown}`);
  }
} else {
  await reportCustomCheckResults({
    client,
    testRunId: testRun.id,
    results: { status: "complete", checks },
  });
}

Both findTestRunForCustomChecks and findTestRunByCommitForCustomChecks register the run as expecting custom check results by default — that is what makes the Checks tab appear in the Meticulous UI and show a "waiting for checks" state until you report. On a real run that is exactly what you want. But while iterating locally with --dryRun you are not going to report anything, so registering would leave that run stuck showing a "waiting for checks" tab that never resolves.

Pass skipRegisteringExpectedCustomChecks: true to suppress that signal during dry runs. Tying it to the dryRun flag (as above) means real runs still register and report normally, while local experiments stay invisible. Note that calling reportCustomCheckResults always marks the run regardless, so this only matters on the dry-run path that never reports.

While building the check, keep using your example test run id:

METICULOUS_API_TOKEN=<token> pnpm run report -- <testRunId> --dryRun

In CI you pass the commit SHA instead (see Set up CI below). Read the printed verdict and markdown carefully before reporting for real — confirm that the sessions linked, thresholds, and endpoint breakdowns all make sense for the change under review. When you are satisfied, drop --dryRun to report results to Meticulous.

If anything misbehaves, compare against the runnable version in the custom-checks-examples repo.

A test run accepts custom check results exactly once. Every check you run must be computed and submitted together in one reportCustomCheckResults call, with all of them in the checks: [...] array.

You cannot report checks separately — for example, posting the network requests result from one CI job and the bundle-size result from another. A test run only accepts one set of results, so a second report is rejected.

So when you add more checks later, compute them all in report.ts and send the complete set in a single API call — one script (or one CI step) that computes every check, then includes every result in the single checks array passed to reportCustomCheckResults.

Open the test run in the Meticulous app and select the Checks tab. Each reported check shows its verdict, one-line summary, and rendered markdown report. For checks that require acknowledgement, reviewers can accept or ignore them from the UI, similar to reviewing visual diffs.

Add the reporter as a CI step after the step that kicks off your Meticulous test run. When resolved by commit SHA, the reporter waits for the test run to complete before computing and reporting results, so it can run right alongside the rest of your pipeline. A typical GitHub Actions step:

- name: Report custom checks
  if: always()
  working-directory: custom-checks
  env:
    METICULOUS_API_TOKEN: ${{ secrets.METICULOUS_API_TOKEN }}
  run: pnpm run report -- --commitSha ${{ github.sha }}

Place this in the same workflow that triggers Meticulous tests (see GitHub Actions setup). The job needs the same API token you use for CI uploads.

This check compares per session: it aligns each session's snapshots on base and head by sessionId and judges every session independently. That suits network request capacity, where each user flow has its own expected amount of traffic.

Other checks may want a different granularity:

• Across the whole test run. Aggregate the data over every replay without distinguishing sessions — for example, the total JavaScript bundle size shipped, or the set of unique endpoints called anywhere in the run — and compare the base aggregate against the head aggregate.
• Per phase. Use each snapshot's stageDuringSession to compare only the events that fired before a particular screenshot — for example, the requests made during page load versus after a specific interaction. This catches regressions localised to one part of a flow that a whole-session total would average out.

Pick whichever granularity makes the regression you care about easiest to detect and explain.