Troubleshoot Comparisons Against an Older Commit
I see "Compared against an older commit" on my PR test run
Meticulous compares your pull request against a test run on its base commit. If that commit has no test run when your pull request's tests are ready, Meticulous compares against the nearest earlier commit that has one, rather than waiting.
d2a8, which is still building, and the run on c4e9 was cancelled. Meticulous compares your pull request against b7c2, the nearest earlier commit with a test run, so changes made in c4e9 and d2a8 can show up as differences.Before approving a difference, check that your pull request could have caused it.
Stacked pull requests can show "Compared against a merge parent" instead: the base is a temporary merge commit GitHub created, so Meticulous used the tip of the branch the pull request is stacked on.
Seeing this now and then is expected, for example when your pull request branched from a commit whose main-branch build was still running. If you see it on most pull requests, check the following.
1. Meticulous runs on every main-branch commit that can change your app
Meticulous needs to run on your main branch as well as on pull requests. If you test preview deployments, every main-branch commit needs a deployment. Skipping commits that only touch files outside your app is fine. To skip a commit that touches your app's files but can't change it, label it with ci label-commit --labels not-relevant.
2. Main-branch runs finish
A failed or cancelled main-branch run leaves that commit without a test run. On GitHub Actions, a concurrency group shared by main-branch runs cancels queued runs even without cancel-in-progress, so give each non-pull-request run its own group:
concurrency:
group: meticulous-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: true
On GitLab, don't mark the main-branch job interruptible, so newer pipelines don't cancel it.
3. Your workflow matches the latest example
Compare your workflow with the example workflow: update the Meticulous actions to the latest release, and include the ensure-base step, which builds a missing base commit alongside your pull request. With other CI providers, use ci prepare.
Once the base commit has a test run, push a new commit or re-run your tests to compare against it.
Issues / questions?
If you still see this on most pull requests, or would rather your pull requests wait for the base commit's tests, email support@meticulous.ai or book a call with us, and we'll look at your project's setup.