Troubleshoot Simulation Accuracy

I see inaccurate simulation differences on my PR test run

If you see inaccurate simulation differences on your PR test run, it means that the simulation against the new app version could not reproduce critical user events and page navigations that occured during the simulation against the old app version. Inaccurate simulations can be caused by:

  1. A change in your application since the session was recorded. Changes such as moving a navigational button or modifying the network API can cause the session to replay incorrectly against new versions of your app. If your pull request made this type of change, then this is likely the cause, and you can safely ignore the inaccurate simulation diffs. If you want to check directly whether the inaccuracies are due to a change, you can look for any network requests or “click” user events that were successful in the replay timeline before your change and unsuccessful in the replay timeline after your change.
  2. A difference between the environment the session was recorded on and the environment it is being replayed in. You can test if this is the case by replaying the session against the environment it was originally recorded on and the environment it is being replayed in - see Troubleshooting failed simulations for more detailed instructions and for advice on how to fix this.
  3. Something else You can debug what may be causing it by replaying the session locally, or looking at the replay timeline. In this case reach out to Meticulous support by emailing eng@meticulous.ai, and we'll help to get Meticulous working for your application.

I see a warning about low simulation accuracy on my project dashboard

If you see a warning about low simulation accuracy, it means that less than 40% of recent simulations can reproduce critical user events and navigate to all pages within the first 5 minutes of a recorded user session. The primary impact of this issue is reduced test coverage for your app until the simulation accuracy improves. See Troubleshooting failed simulations for more detailed instructions and for advice on how to fix this.

Issues / questions?

We're always happy to help you with any issues you encounter while setting up or anything you might be unsure about.

Get in touch by emailing eng@meticulous.ai, or book a call with us, and we'll help you get setup.