Setting up Meticulous tests to run in your CI provider

1. Install the Meticulous GitHub App

Visit https://github.com/apps/alwaysmeticulous/installations/new to install our GitHub App.

2. Add your Meticulous API token as a secret to your GitHub repo

Select the project below that contains the sessions you wish to simulate, copy and paste the API token, and add it to your Github repo as a secret named METICULOUS_API_TOKEN:

METICULOUS_API_TOKEN:

<API_TOKEN>

Be very careful with this API token, since it allows the holder access to your recorded sessions.

3. Add a github actions workflow to run your tests

To run Meticulous on CI add a new .github/workflows/meticulous.yaml file, or, if you already use Github actions, you can add it as a job to a an existing workflow. The workflow needs to run on both pushes to your main branch and on pull requests.

This workflow file should use the alwaysmeticulous/report-diffs-action to run your tests. This can be either against a local build of your app or against the latest preview URL for the commit (if you use a service such as Vercel or Netlify that generates preview URLs).

If you hit any issues then book a call with us and we'll help you get set up.

Please select the appropriate option:

See below for an example workflow file, which you can add to your repo as '.github/workflows/meticulous.yaml'. Note that you'll need to update it to build and serve your frontend app.

File name: .github/workflows/meticulous.yaml.

File contents:

# Workflow for serving app locally & running Meticulous tests against it

name: Meticulous

# Important: The workflow needs to run both on pushes to your main branch and on
# pull requests. It needs to run on your main branch because it'll use the results
# from the base commit of the PR on the main branch to compare against.
on:
  push:
    branches:
      - main
  pull_request: {}
  # Important: We need the workflow to be triggered on workflow_dispatch events,
  # so that Meticulous can run the workflow on the base commit to compare
  # against if an existing workflow hasn't run
  workflow_dispatch: {}

# Important: The workflow needs all the permissions below.
# These permissions are mainly need to post and update the status check and
# feedback comment on your PR. Meticulous won’t work without them.
permissions:
  actions: write
  checks: write
  contents: read
  discussions: write
  pull-requests: write
  statuses: write

jobs:
  test:
    name: Report diffs
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Use Node.js LTS
        uses: actions/setup-node@v3
        with:
          node-version: "16"
          cache: yarn

      - name: Cache node_modules
        uses: actions/cache@v3
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            node-modules-${{ runner.os }}

      - name: Install dependencies
        run: |
          yarn install --frozen-lockfile --non-interactive

      - name: Build project
        run: |
          yarn build

      - name: Serve project
        # TODO: Update these commands to serve your app's frontend locally, and
        # then update the app-url to match.
        #
        # Notes:
        #
        # 1. Please make sure your serve command listens 0.0.0.0 rather than
        #    localhost or 127.0.0.1 (i.e. listens to all requests via all network
        #    interfaces rather than just the loopback interface - see
        #    https://stackoverflow.com/a/20778887 for more details)
        # 2. The sleep is often required to ensure your app is readily being served
        #    by the time the Meticulous tests start
        run: |
          yarn serve &
          sleep 5

      - name: Run Meticulous tests
        uses: alwaysmeticulous/report-diffs-action@v1
        with:
          api-token: ${{ secrets.METICULOUS_API_TOKEN }}
          # TODO: Update the port and protocol below to match your app's frontend
          app-url: "http://localhost:3000/"

4. Merge the PR to add your new GitHub workflow, and open a new PR to test Meticulous

Merge the PR to add the above workflow. You won't see any results on the PR that adds the workflow because you need to wait for the workflow to run on your main branch for it to detect any diffs.

Once the PR has merged and Meticulous has run on your base branch you can open a new PR to test Meticulous. It should post a comment if your new PR changed any of the screens or logic for the workflows you've recorded sessions for:

Meticulous comment

You can click the link in the comment to review the diffs before merging your PR.

5. (Optional) Require approving diffs before merging a PR

If you've installed the Meticulous GitHub App Meticulous will add a check on your PR that is red if there are diffs that haven't been approved yet and becomes green once you click the green 'Approve all Visual Differences' button. This button can be found by clicking on the link in the Meticulous comment.

If you wish, you can make this check blocking by following the instructions here. Doing so will prevent developers from merging a PR which has visual differences until they have clicked the button to acknowledge the differences.

6. (Recommended) Move to a larger GitHub runner

Meticulous test speeds scale roughly linearly with the number of CPUs. You can speed up your tests by switching to a larger GitHub Actions runner. A 2x larger runner will cost twice as much per minute but the tests complete twice as fast. So, assuming most of the job time comes from running the Meticulous tests, rather than building your app, then you'll pay the same amount but your tests will complete twice as fast. We recommend going for a runner with at least 16 cores, preferably 32.