Testing Feature Flags with Meticulous

Note: To learn how to register feature flags in your sessions, see Recording the context of a user session. This page explains how Meticulous tests feature flags.

When a session is recorded, Meticulous snapshots its network responses, local storage, cookies and session storage, and stubs out and replays those values when it later replays the session. So each replay keeps the flag values its session was recorded with.

That already tests your flags in more than one state. If two sessions executed different code because a flag was on in one and off in the other, Meticulous's coverage-based test suite will often keep both. For example, if user 1 records session A with my_new_feature off and user 2 records session B with it on, session A replays with the feature disabled and session B replays with it enabled.

Two gaps remain, and the rest of this page covers them:

• A flag that was off in every recorded session never gets tested on. Letting Meticulous override named flags closes this gap for the flags a pull request touches.
• Sessions recorded before a flag existed have no value for it. Defaulting new flags to on closes this gap.

Meticulous can force a named feature flag to a specific value during a replay, so a replay can exercise code behind a flag that was off — or did not exist — when the session was recorded.

How it works:

1. Recording doesn't change. Real users get your normal flag values, and your app reports them with recordFeatureFlag. These reports are how Meticulous learns which flags exist and which values they take.
2. Meticulous picks flags to force. On a pull request, it looks at which flags the change touches and adds replays with those flags set to the values the change is about, alongside your normal replays.
3. Your app applies the forced value. During those replays your app uses the forced value in place of its normal evaluation, and keeps reporting the value it actually used with recordFeatureFlag. Real users being recorded never receive a forced value.

Your app receives forced values through:

• The JavaScript API, for flags your browser code resolves. Meticulous starts forcing flags for your project once it sees your app call getFlagOverride during a replay.
• A cookie your server reads, for flags your server resolves, for example while rendering a page. Meticulous starts forcing flags once you turn on cookie delivery for your project.

Wrap the helper your app uses to read a flag. Resolve the value once: prefer getFlagOverride, otherwise use your own evaluation, then record that same value with recordFeatureFlag before returning it. getFlagOverride asks Meticulous which value to use, and recordFeatureFlag tells Meticulous which value your app actually used. Recording a snapshot from getAllFlags() (or similar) will miss a forced value, because the SDK never saw the override.

const resolveFlag = (flagKey: string) => {
  const override = window.Meticulous?.context?.getFlagOverride?.(flagKey);
  const value = override?.overridden
    ? Boolean(override.value)
    : flagsFromYourApp[flagKey] || false;
  window.Meticulous?.context?.recordFeatureFlag?.(flagKey, value);
  return value;
}

Hook the helper your application actually calls. getFlagOverride returns { overridden: false } whenever Meticulous has no override for that flag, and always does so for real users being recorded, so it is safe to leave in production code. Use optional chaining as shown so your application also works when the Meticulous snippet isn't loaded. For complete wrappers for specific SDKs, see Feature flag providers.

How you consume override.value depends on the helper you wrap — Boolean() is not always right. Always record the flag's resolved value (the thing that describes the cohort), not a comparison boolean:

• On/off gate (checkGate(name) returns a boolean): Boolean(override.value), as above. Record that boolean.
• Value-read (getExperimentValue(name) / variation(name) returns the cohort): return and record override.value as-is. Boolean('control') is true, which would turn every variant on.
• Equality-check (isTreatment(name, expected) / editorExperiment(name, expected) returns a boolean meaning "is this flag set to expected"): compare, don't coerce. Record override.value, not the === result. Wrapping with Boolean(override.value) makes every expected match; returning the string is equally wrong because callers expect a boolean:

const isTreatment = (name: string, expected: string | boolean) => {
  const override = window.Meticulous?.context?.getFlagOverride?.(name);
  if (override?.overridden) {
    window.Meticulous?.context?.recordFeatureFlag?.(name, override.value);
    return override.value === expected;
  }
  return originalIsTreatment(name, expected);
}

If you wrap the value-read underneath an equality-check (getExperimentValue under editorExperiment), use the value-read rule and record there — the === already happens above you.

If your app resolves all of its flags in one place — a flag map built at startup, or a client initialised with a full set of values (such as LaunchDarkly's bootstrap option) — use getAllFlagOverrides instead of asking about each flag in turn. It returns every flag the current replay forces, keyed by flag name, so you can layer the overrides on top of your own values:

const flags = {
  ...flagsFromYourBackend,
  ...window.Meticulous?.context?.getAllFlagOverrides?.(),
};
for (const [name, value] of Object.entries(flags)) {
  window.Meticulous?.context?.recordFeatureFlag?.(name, value);
}

The same consumption rules apply: an override for an on/off gate may be a string, so coerce it where your app expects a boolean. Flags missing from the returned object are not overridden. It returns {} for real users being recorded, so it is safe to leave in production code.

If your server resolves flags while rendering a page, it can read forced values from the meticulous_flag_overrides cookie instead of your app calling getFlagOverride.

Turning it on. Turn on Send forced flags in a cookie on Settings → Feature flags. Meticulous can see your app calling getFlagOverride, but it can't see your server reading a cookie, so it only forces flags for an app that relies on the cookie once cookie delivery is on. If your app already reads overrides from a cookie of its own in the format below, set that name as the Cookie name there instead, and you won't need to change any code.

Once it's on, every replay that forces at least one flag sets the cookie before the first page load, so it is sent on the very first document request and on every request after that. Its value is URL-encoded JSON in the same shape as getAllFlagOverrides, for example {"new-checkout":true,"pricing-test":"treatment-b"}:

const overridesCookie = request.cookies["meticulous_flag_overrides"];
const overrides = overridesCookie
  ? JSON.parse(decodeURIComponent(overridesCookie))
  : {};
const flags = { ...resolveFlagsForUser(user), ...overrides };

The cookie is not httpOnly, so your browser code can read it too. The same consumption rules apply as for the JavaScript API. Your app must still record the values it actually used with recordFeatureFlag, for example by recording the flag values your server sends to the browser. Requests your frontend makes with fetch or XMLHttpRequest are normally answered from the recording during a replay, so the cookie only changes what your server sends for requests that actually reach it, such as page loads. With backend testing, your app's own API requests reach your server too, so it applies the cookie there as well.

Sessions recorded before a flag was introduced replay old network responses and local storage values that have no entry for it, so they usually won't test the new feature until new sessions are recorded with the flag on.

You can close that gap without waiting: when your app is running as part of a Meticulous test (checked with window.Meticulous?.isRunningAsTest) and a flag has no configuration at all, default it to on. Check for a named override first, then apply this default, then record the value you return. We recommend this if you often develop new features behind feature flags. The provider sections below show how.

Each wrapper below combines a named override, the default-on fallback for unrecognised flags, and recording. Wrap your client once and use the wrapper everywhere, rather than updating every call site.

Wrap checkGate. Statsig reports an Unrecognized reason for a gate it has no configuration for, which is the signal to default it to on during a Meticulous test. The same approach works for Statsig's React integration.

import { StatsigClient } from '@statsig/js-client';

const myStatsigClient = new StatsigClient(
  YOUR_CLIENT_KEY,
  { userID: 'a-user' },
  ...
);
await myStatsigClient.initializeAsync();

// Use this wrapper in place of myStatsigClient.checkGate
const checkGate = (gateName: string) => {
  const override = window.Meticulous?.context?.getFlagOverride?.(gateName);
  if (override?.overridden) {
    const value = Boolean(override.value);
    window.Meticulous?.context?.recordFeatureFlag?.(gateName, value);
    return value;
  }

  // During a Meticulous test, default gates with no configuration to on, so that old sessions
  // can test new features. See https://docs.statsig.com/sdk/debugging
  const value =
    window.Meticulous?.isRunningAsTest
      && myStatsigClient.getFeatureGate(gateName).details.reason.endsWith("Unrecognized")
      ? true
      : myStatsigClient.checkGate(gateName);
  window.Meticulous?.context?.recordFeatureFlag?.(gateName, value);
  return value;
}

Wrap variation. Check for a named override first, and during a Meticulous test pass true as the default so a flag with no configuration resolves to on:

const variation = (flagKey: string, defaultValue: boolean) => {
  const override = window.Meticulous?.context?.getFlagOverride?.(flagKey);
  const value = override?.overridden
    ? override.value
    : client.variation(
        flagKey,
        window.Meticulous?.isRunningAsTest ? true : defaultValue,
      );
  window.Meticulous?.context?.recordFeatureFlag?.(flagKey, value);
  return value;
}

If you initialise LaunchDarkly with a full set of values through its bootstrap option, use getAllFlagOverrides to layer the overrides on top of them instead.

The same three steps work for any flag system, including one you've written yourself: check getFlagOverride, fall back to your own evaluation (defaulting unconfigured flags to on during a Meticulous test if you want), and record the value you return. See Using the JavaScript API.

No forced replays appear on pull requests that change a flag. Meticulous only forces flags once it has seen your app call getFlagOverride during a replay, or once cookie delivery is on for your project, and only for flags your app reports with recordFeatureFlag. Check that the helper your code uses does both.

A forced flag didn't change what the replay rendered.

• The flag is read through a path that never asks for an override. Make sure every path your components read flags through either checks getFlagOverride or was built with getAllFlagOverrides. A flag map built from your provider alone (such as getAllFlags()), or a value restored from local storage, won't include the override.
• The value doesn't match your helper's shape. A boolean override on a helper that returns a variant name, or a Boolean() wrapped around a variant, quietly picks the wrong branch. See Matching the override to your helper.
• The flag is evaluated only on your server. The JavaScript API runs in the browser, so a flag your server resolves while rendering a page doesn't see it. Read the cookie on your server instead.
• The code behind the flag has no visible effect in that session. A forced flag can only change screenshots on pages the recorded session actually visits.

A forced flag shows a question mark. Meticulous delivered the forced value but your app didn't report a value for that flag with recordFeatureFlag, so it can't confirm your app used it. Record the value your helper returns, including when it came from an override.

A forced flag shows a warning. Your app reported a different value from the one Meticulous forced, so the override didn't reach the code that reported it. Check the cases above.

A forced flag shows a dash. Your app never asked getFlagOverride for that flag, so the JavaScript API couldn't deliver it. Check that your flag helper asks for each flag by the same name your app records.

• Recording the context of a user session - Learn how to record feature flags and other session context
• TypeScript Types for window.Meticulous - Get type definitions for the window.Meticulous object