Incremental Asset Upload

Most apps should use the standard upload-assets workflow, which uploads your whole build on every commit. Incremental asset upload is an optimisation for extremely large builds where only a small fraction of files change between commits. It lets your CI pipeline upload only the chunks that changed, while still running tests against a complete build.

With incremental asset upload, you split your build into named chunks and upload each chunk to Meticulous independently. A chunk that hasn't changed since a previous build is already stored under its version id, so re-uploading it is a no-op. Once every chunk that was modified in the build has been uploaded, you trigger a test run by pointing Meticulous at a manifest that specifies the version to use for each chunk, so that Meticulous can re-assemble a full build.

This splits the work into two commands:

1. ci upload-asset-chunk — upload a single chunk (skipped instantly if already uploaded).
2. ci run-with-uploaded-asset-chunks — assemble the referenced chunks and trigger a test run.

Meticulous downloads the referenced chunks into each test run, assembles them — using each chunk's directory prefix — into a single directory, serves that directory from an asset server, and runs your tests against that localhost URL.

Each chunk has:

• chunkName — a logical name for the chunk (e.g. app, vendor, home-page-app). Chunks are deduped by the (chunkName, chunkVersionId) pair.
• chunkVersionId — a version identifier for the chunk's contents. See Choosing a version id.
• chunkAssetsDirectory — the local directory whose contents are packaged into the chunk.
• chunkAssetsDirectoryPrefix — an optional path prefix prepended to every file in the chunk. Files in chunkAssetsDirectory are served under this prefix at replay time. Leave it empty to serve the files at the root.

When Meticulous assembles a build, it lays each chunk's files down under its prefix. For example, given:

• chunk1 — prefix "", contains file1.js and sub-folder/file2.js
• chunk2 — prefix "", contains file2.js
• chunk3 — prefix "sub-folder2", contains file3.js

the assembled directory is:

file1.js
file2.js
sub-folder/file2.js
sub-folder2/file3.js

If two chunks with the same prefix contain a file at the same path, Meticulous cannot decide which one to serve and the test run fails with an error. Make sure your chunking scheme partitions files so that no two chunks produce the same final path.

It is a hard requirement that two chunks with different bytes never share the same version id — otherwise Meticulous may serve stale files and your test results will be incorrect.

It is a soft requirement that two chunks with identical bytes share the same version id — if this is violated too often you lose the deduplication benefit (the chunk is re-uploaded unnecessarily), but correctness is unaffected.

A version id can be:

• A hash of the chunk's contents (e.g. the directory's content hash), or
• A hash of the inputs to the build task that produced the chunk (e.g. the build cache key).

You'll need to provide Meticulous with a manfiest of the versions of all chunks to be used, not just the versions of the chunks modified in the build. For this reason using the hash of the inputs to the build task that produced the chunk (e.g. the build cache key) can work well, since for build tasks that have been skipped it's easier to compute the hash of the inputs than it is to compute the hash of the contents of the chunk (hash of the output).

For every chunk that makes up your build, call ci upload-asset-chunk:

npx @alwaysmeticulous/cli ci upload-asset-chunk \
  --apiToken="$METICULOUS_API_TOKEN" \
  --chunkName="home-page-app" \
  --chunkVersionId="ad8a8da9aaaweaad9" \
  --chunkAssetsDirectory="dist/home-page-app" \
  --chunkAssetsDirectoryPrefix="" \
  --commitSha="$CI_COMMIT_SHA"

If a chunk with the same chunkName and chunkVersionId is already uploaded, the command exits 0 after compressing but before uploading.

For the first build on your main branch you run you'll need to upload every chunk in your application. If you consistently run builds for every commit in the chain from thereon then you'll only need to upload the chunks that changed in that commit -- assuming that every ancestor commit successfully uploaded all of its changed chunks, all the way back to a commit where you uploaded every chunk. We therefore recommend either calling upload-asset-chunk for every commit, or using your build cache to skip chunk versions which you know for certain have already been uploaded, since there is already a cache entry from that build task.

For more information on the command run npx @alwaysmeticulous/cli ci upload-asset-chunk, or view the source code.

Once every chunk has been uploaded, write a manifest that references every chunk required to assemble the full build — not just the ones that changed in this commit — and pass it to ci run-with-uploaded-asset-chunks.

The manifest is a JSON array of { name, versionId } references:

cat > assets-manifest.json <<'EOF'
[
  { "name": "home-page-app", "versionId": "ad8a8da9aaaweaad9" },
  { "name": "settings-page-app",   "versionId": "dd8ffdaa9dfedebb3" }
]
EOF

npx @alwaysmeticulous/cli ci run-with-uploaded-asset-chunks \
  --apiToken="$METICULOUS_API_TOKEN" \
  --assetReferencesManifest="./assets-manifest.json" \
  --commitSha="$CI_COMMIT_SHA" \
  --waitForBase

Because the version ids are required, your build needs to know — or be able to regenerate — the version id of every chunk that forms the build, including the unchanged ones. Using a content hash or build cache key as the version id (see Choosing a version id) makes this deterministic.

This command fails if any referenced chunk has not been fully uploaded, so make sure every chunk in the manifest has been uploaded (step 1) before triggering the run, and that your pipeline handles failed uploads.

To restrict the run to sessions starting on specific routes, pass --sessionFilter — see Filter Sessions by Start URL.

For more information on the command run npx @alwaysmeticulous/cli ci run-with-uploaded-asset-chunks, or view the source code.

The manifest is a non-empty JSON array. Each entry must be an object with non-empty name and versionId string properties:

[
  { "name": "charts-app", "versionId": "ad8a8da9aaaweaad9" },
  { "name": "plugin-1",   "versionId": "dd8ffdaa9dfedebb3" }
]

Names should be as stable as possible across builds.

run-with-uploaded-asset-chunks will print out a URL at which you can download the assembled build. Download it and check that the build has been re-assembled correctly, with the correct versions of the correct assets mounted in the correct folders. If you have a backend running at the URL hardcoded into the assets then you can also spin up a local server to serve the directory and check your app loads and runs correctly.

An example for a SvelteKit app;

#!/usr/bin/env bash
set -euo pipefail

# Build your app into per-chunk directories (e.g. dist/charts-app, dist/plugin-1).
#
# Note: some build outputs can vary slightly between builds even if the input
# source files are identical.
#
# For example some builds may embed the commit SHA in one of the built assets,
# or some builds may not be be fully deterministic. In the case of builds that
# embed information like a commit SHA you may want to make sure this is split out
# into a seperate chunk, rather than injected into every chunk. In the case
# of builds that are not fully deterministic (bytes can change slightly on a rebuild)
# you may want to use build caching on a stable cache key to ensure chunk version
# ids stay stable when the outputs are functionally identical.
yarn build

# Split the SvelteKit static build into two chunks along a natural boundary:
#   - "html": root-level pages + static assets, served at /
#   - "app":  the hashed _app/ bundle, served under the _app prefix

rm -rf chunks
mkdir -p chunks/html

# Root-served files (everything except the _app bundle).
find build -maxdepth 1 -mindepth 1 ! -name _app \
  -exec cp -R {} chunks/html/ \;
# The _app bundle is uploaded as its own chunk under the _app prefix.

# Version each chunk by a content checksum (SHA1 over relative path + bytes)
# so chunks dedupe across commits when their contents are unchanged.
hash_dir() {
  ( cd "$1" && find . -type f -print0 | sort -z | xargs -0 sha1sum ) | sha1sum | cut -d' ' -f1
}
HTML_VERSION=$(hash_dir chunks/html)
APP_VERSION=$(hash_dir build/_app)

# Upload the html chunk (short circuits if already uploaded)
npx -y @alwaysmeticulous/cli@latest ci upload-asset-chunk \
  --chunkName=html \
  --chunkVersionId="$HTML_VERSION" \
  --chunkAssetsDirectory=chunks/html \
  --commitSha="${{ github.sha }}"

# Upload the app chunk (short circuits if already uploaded)
npx -y @alwaysmeticulous/cli@latest ci upload-asset-chunk \
  --chunkName=app \
  --chunkVersionId="$APP_VERSION" \
  --chunkAssetsDirectory=build/_app \
  --chunkAssetsDirectoryPrefix=_app \
  --commitSha="${{ github.sha }}"

# Trigger the run
cat > manifest.json <<JSON
[
  { "name": "html", "versionId": "$HTML_VERSION" },
  { "name": "app",  "versionId": "$APP_VERSION" }
]
JSON
npx -y @alwaysmeticulous/cli@latest ci run-with-uploaded-asset-chunks \
  --commitSha="${{ github.sha }}" \
  --assetReferencesManifest=manifest.json \
  --waitForBase=true

Meticulous persists uploaded chunks so they can be reused across builds. A chunk is retained as long as it has been used by a test run in the last N days, even if it was uploaded more than N days ago. N is set to match the data retention period you have configured for your project (default ~90 days).

• Split your build into named chunks and give each a version id derived from its contents or build inputs.
• Upload at least the changed chunk on each commit with ci upload-asset-chunk.
• Reference every chunk required for the full build in the manifest, then trigger the run with ci run-with-uploaded-asset-chunks.