Set up Agent review

Agent review uses a Meticulous-hosted agent to explore and test your application for a pull request. Meticulous uploads the build from your CI job and opens it in a recorder-instrumented browser, where the agent follows your instructions to exercise happy paths, edge cases, and potential regressions. The Agent review page reports each test case's result with step-by-step screenshots, and the discovered flows are saved as sessions that extend your replay-test coverage.

Use this guide to add Agent review to your CI workflow. It complements ordinary Meticulous replay tests; it does not replace the sessions you record or the normal PR-test workflow.

Agent review is currently in beta and is not enabled until we turn it on for your project. Contact us before merging the workflow and wait for confirmation that your project is opted in. If you add the workflow first, launches will fail until we enable it.

When you reach out, include your Meticulous org/project, whether you plan to use local mocks or a staging backend, and — if you need staging login — the details in Login setup.

You need:

• A Meticulous project linked to the repository and opted in to the beta (see above).
• A project-scoped API token stored in your CI provider as METICULOUS_API_TOKEN.
• A build command that produces a directory of static frontend assets.
• A GitHub Actions workflow (or equivalent CI job) that runs when a pull request is opened, plus a way to re-run manually.

Choose one environment for the agent:

The rest of this guide walks through the first two options.

Create .github/agent-review/instructions.md. Describe the app in terms the agent can act on: routes, important controls, test accounts, and the expected flows. Keep secrets out of this file.

For example:

# Storefront

Start at `/`. Browse the catalogue, add an item to the cart, and complete the checkout form. Use `test@example.com` and the test account details supplied by the environment. Also visit `/orders` and check empty and populated states.

Good instructions name the goal and the UI affordances that lead to it. They should also call out variants worth exercising, such as validation errors, filters, feature flags, or a dark-mode toggle.

If a first-time tour, tip, or consent banner can cover the application, name it and its visible dismiss control in the instructions. Agent review automatically closes common consent banners and high-confidence onboarding tips before starting a flow, but app-specific guidance is useful when the control has an unusual label. Do not tell the agent to close a dialog that is part of the behavior you want tested.

For applications with configured staging login, the most deterministic option is to complete the onboarding state as part of that project login setup. Meticulous captures the resulting cookies and local storage and seeds each case browser with them, so the popup never appears or adds a dismissal action to the recorded session. Tell us which tour-completion state or action is required when you send the login setup details.

Add a separate pull-request workflow, or a job after your existing build. Agent review runs are relatively expensive, so prefer not running on every push. Trigger on PR opened / reopened, and use workflow_dispatch when you want a fresh run after later commits.

The following GitHub Actions job builds a static site in build/ and launches Agent review for the PR's head commit:

name: Agent review

on:
  pull_request:
    types: [opened, reopened]
  workflow_dispatch:

jobs:
  generate-sessions:
    runs-on: ubuntu-latest
    env:
      METICULOUS_API_TOKEN: ${{ secrets.METICULOUS_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm build
      - name: Launch Agent review
        run: |
          npx -y @alwaysmeticulous/cli@latest ci agent-test \
            --assetsDir build \
            --commitSha "${{ github.event.pull_request.head.sha || github.sha }}" \
            --instructionsFile .github/agent-review/instructions.md

Replace build with your build output directory. The command runs the agent in Meticulous; your CI runner does not need Docker or LLM credentials.

Do not use bare on: pull_request: (that fires on every push to the PR). To re-run after later commits, use Actions → Agent review → Run workflow.

On a pull_request event, github.sha can be GitHub's temporary merge commit. Pass github.event.pull_request.head.sha || github.sha so the generated sessions and their result are associated with the commit that Meticulous tests and displays for the PR.

To prove the command and paths are valid without launching an agent, append --dryRun. You can also run the same command locally after building, with METICULOUS_API_TOKEN exported.

If the uploaded frontend needs a live backend, serve a disposable staging environment over public HTTPS and proxy the frontend's relative API paths to it:

      - name: Launch Agent review with staging API
        env:
          METICULOUS_STAGING_USERNAME: ${{ secrets.STAGING_AGENT_USERNAME }}
          METICULOUS_STAGING_PASSWORD: ${{ secrets.STAGING_AGENT_PASSWORD }}
        run: |
          npx -y @alwaysmeticulous/cli@latest ci agent-test \
            --assetsDir frontend/dist \
            --backendUrl "https://staging.example.com" \
            --backendProxyPaths /api \
            --commitSha "${{ github.event.pull_request.head.sha || github.sha }}" \
            --instructionsFile .github/agent-review/instructions.md

Your frontend must make same-origin requests under the configured prefixes (for example, fetch('/api/tasks')). --backendProxyPaths defaults to /api, so you can omit it when that is the only prefix you need. Absolute API URLs bypass this proxy: pass those hosts as --trustedOrigins https://auth.example.com --trustedOrigins https://api.example.com. Each value must be an HTTPS origin (no path or credentials). Those hosts must allow CORS from http://localhost:8000 (the default assets origin; override with --appPort), and cookie sessions on a trusted origin need SameSite=None; Secure. The backend must be publicly reachable on HTTPS, accept the proxied (or cross-origin) requests, and use a disposable account because the agent can perform writes.

If the agent needs to sign in against your staging backend, we configure the login flow for your project — it is not something you set in CI. Send us the following before your first credentialed run (ideally when you request beta access):

You can paste this into Slack or email:

Project: <org/project>
Staging backend URL: https://…
Login type: username/password on app | SSO/IdP | other (describe)
Login URL/path: …
Username field: (default ok / selector: …)
Password field: (default ok / selector: …)
Submit: (default ok / selector or button text: …)
After login: lands on … / session cookie …
Post-login tour/tip: none | completion action/state: …
Test account: (username/password go in CI secrets only)
MFA/captcha on staging for this account: yes/no

Once we have configured login, provide credentials and any other login options only through METICULOUS_STAGING_* environment variables: METICULOUS_STAGING_USERNAME and METICULOUS_STAGING_PASSWORD, plus any flow-specific values we agree on — for example METICULOUS_STAGING_TOTP_SECRET (base32 TOTP seed) for an MFA flow, or METICULOUS_STAGING_SKIP_EMAIL_CLIENT_ID (trusted-automation client id) when the login flow must bypass an email verification challenge. Every environment variable with the METICULOUS_STAGING_ prefix is forwarded to the login flow, so adding a new option never requires a CLI upgrade. Do not put these values in agent instructions or commit them to the repository.

1. Open a pull request with a small, visible change (or manually re-run the workflow).
2. Confirm the Agent review job uploads the build and prints a workflow run identifier.
3. Open the Meticulous test run for the PR commit, then open Agent review to inspect the generated test cases, results, and screenshots.
4. Review the recorded flows and adjust instructions.md if important routes or states were missed.

If the agent cannot reach an API request, first check that the frontend uses a relative URL under --backendProxyPaths, or that absolute cross-origin hosts are listed in --trustedOrigins; then check that the staging deployment is healthy and the test credentials can sign in. For command syntax and the container option, see CLI commands.