Testing and Scheduling

Once separates which tests should run from where and when they run. That separation keeps affected-test selection conservative, makes exact requests auditable, and lets scheduling evolve without changing test identity.

Start here whether you are declaring a test target for the first time or planning work for an existing target. Full ecosystem guides cover build and test targets together. Test-runner reference pages provide focused, copyable declarations for runner-only integrations.

Start With Your Runner#

Keep the native runner as the source of truth for test collection and execution. Choose the matching target kind, make its prerequisite available, and let Once record stable test identities around it:

Runner	Target declaration	Prerequisite	Default automatic batch
pytest	`pytest_test`	The selected Python interpreter can import pytest	One test file
Ruby Specification	`rspec_test`	The selected Ruby interpreter can load the runner	One specification file
Minitest	`minitest_test`	The selected Ruby interpreter can load Minitest	One test file
Vitest	`vitest_test`	The package-local Vitest entry point is installed	One test file
Jest	`jest_test`	The package-local Jest entry point is installed	One test file

Each target kind ships a runnable starter. Retrieve the one that matches your runner when you want a complete manifest and source layout:

bash

# Python with pytest
once query example pytest_test pytest-test-minimal --format json

# Ruby Specification
once query example rspec_test rspec-test-minimal --format json

# Ruby with Minitest
once query example minitest_test minitest-test-minimal --format json

# JavaScript or TypeScript with Vitest
once query example vitest_test vitest-test-minimal --format json

# JavaScript or TypeScript with Jest
once query example jest_test jest-test-minimal --format json

After copying or adapting the declaration, use the same first-run workflow for every runner. The bundled starters name their target tests:

bash

once query schema pytest_test
once test tests --format json
once query test-manifest tests --format json
once test tests --jobs 4 --format json

Replace pytest_test with the chosen kind when querying its schema. The first complete run discovers stable units and writes the manifest. Its structured response keeps that executed work in plan and shows the batches available to the next run in next_plan. The next run can schedule those units as independent file batches across the requested workers. Set batching = "case" in a pytest, Ruby Specification, Vitest, or Jest target only when its fixtures and startup costs make case-level batches worthwhile. Python, Ruby, and Node.js executable attributes accept names on the executable search path, absolute paths, and workspace-relative paths. A pytest target with no python attribute automatically tries .venv/bin/python before the host executable search path. Set a portable project path explicitly when the repository uses a different runtime layout.

Start With the Safety Model#

Layer	Decides	Stable output
Selection	Which test targets or explicit units belong in the run	Selection reasons and safety level
Plan	How selected work is divided into semantic batches	Plan and batch identifiers
Schedule	Which worker takes each batch and attempt	Placement, timing, and status
Results	What the runner observed	Normalized suites, cases, attempts, and artifacts

Affected selection is conservative. Once starts from changed graph inputs, walks reverse dependencies, and selects complete test targets. Missing ownership also selects tests instead of silently skipping them.

Exact unit selection is different. It is allowed only when the target kind declares a lossless translation from a stable Once unit identifier to native runner arguments. Unsupported or stale unit requests fail during planning.

The boundary is deliberately ecosystem-neutral. Rust validates generic test information and result records, builds graph-based plans, and schedules opaque batches. A Starlark target kind owns runner discovery, command arguments, filter translation, result normalization, and platform requirements. Adding a test ecosystem should therefore require a target kind, not a branch in the planner or executor.

Discover the Test Surface#

List test targets before choosing one:

bash

once query tests --format json

Run a complete target once to establish current discovery data:

bash

once test tests/unit --format json
once query test-manifest tests/unit --format json

The manifest reports:

source, which distinguishes normalized discovery from a whole-target fallback;
listing_supported, which states whether the target kind can expose stable units;
case_filtering, which is runner_args only when exact execution is safe;
sharding, which states whether Once may create automatic batches and whether their granularity is file, case, or target;
discovery_fingerprint, which ties the unit list to the declared inputs that can change discovery; and
units, the stable identifiers observed in the latest complete run.

A filtered run does not replace the complete manifest. Agents can run one case for quick feedback without forgetting the other cases discovered by the last whole-target run.

Plan Affected Tests Before Running#

Every plan query exports the same replayable document, including discovery inventories and input bindings. Inspect one for changed workspace paths:

bash

once query test-plan \
  --changed-path src/library.rs \
  --format json

Each selected test includes a reason. The current affected policy emits one complete test scope per selected target. When that target has a current manifest and supports exact filtering, the plan divides the complete scope into stable file or case batches. This changes scheduling granularity without using test impact as permission to omit a discovered case.

The first run, a changed test input, or an incomplete manifest produces one whole-target batch. That run refreshes discovery. Later plans may then split the same target automatically. A stale manifest never filters against an old unit list.

Run the same selection through the dynamic scheduler:

bash

once test \
  --changed-path src/library.rs \
  --jobs 4 \
  --format json

--jobs caps concurrent local workers. It does not change selection, plan identity, or batch identity.

Inspect recent attempts and measured durations afterward:

bash

once query test-attempts --limit 20 --format json

Run One Exact Unit#

Choose an identifier returned by the current complete manifest:

bash

once query test-manifest tests/unit --format json

once test tests/unit \
  --test-unit 'tests/unit::returns_greeting' \
  --format json

Before execution, Once verifies both conditions:

the target declares case_filtering = "runner_args"; and
the requested identifier is present in the persisted complete manifest.

Run the whole target again when discovery is missing or stale. Do not construct runner-specific filters from names that Once did not return.

Rerun One Batch#

A batch is a stable group of test units, not a worker number. Inspect the current target plan and copy a batch's id:

bash

once query test-plan --target tests/unit --format json
once test tests/unit --test-batch '<batch-id>' --format json

The batch request requires one explicit target. It cannot be combined with --test-unit, --all, or changed-path selection. Once resolves it against the current plan before starting work. A missing or obsolete ID fails with current batch suggestions and the command to inspect them. Changed discovery inputs can make an old exact batch unavailable until a whole-target run refreshes the manifest.

Replaying a file batch preserves every case in that batch, including its fixture boundary. It does not replace the complete discovery manifest with the filtered result. Worker count does not change the batch ID. This reproduces the scope of the work, not necessarily its outcome after code or environment changes.

The agent tool once_query_test_plan exports the complete replayable plan. once_run_tests accepts test_batch for local replay with the same rules. The returned execution scope and schedule identify the exact work and attempt.

Replay Saved Plans on CI Workers#

Export the plan after a complete discovery run:

bash

once test tests/unit --format json
once query test-plan --target tests/unit --format json > plan.json

Plan queries always cover complete selected targets. Choose opaque batch IDs from plan.batches at execution, not during export. An external scheduler distributes the document unchanged, and each fresh worker runs its assignments:

bash

once test --plan plan.json \
  --batch '<first-batch-id>' --batch '<second-batch-id>' \
  --jobs 4 --format json

Omit --batch to execute the complete exported plan. A repeated or unknown ID fails before execution. Do not reconstruct native runner filters, split a file batch, or transfer private discovery caches. --jobs, --memory-limit, --cpu-limit, and --sandbox retain their ordinary scheduling and resource limits. Portable replay cannot be combined with target selectors or --config; the worker must use the workspace build configuration bound by the export.

The document has schema once.portable_test_plan.v1, a content-derived id, the native plan, and opaque bindings for its targets. Bindings tie the complete discovery inventory to declared input contents, resolved relevant configuration, dependency inputs, and loaded target-kind implementations. Missing or stale discovery exports a whole-target fallback, never an old exact inventory. Execution rejects stale bindings, malformed identities, overlapping scopes, incomplete inventory coverage, and split file fixtures before scheduling. The content digest detects changes, not authorship: retrieve plans through a trusted, authenticated coordinator and treat them like repository configuration.

Built-in dynamic-language runners separate tool versions from installation paths, allowing matching checkouts and tool versions at different locations. Explicit environment values remain relevant, including inherited variables declared by the target. Changing them, runner versions, test arguments, sources, or configuration requires a new plan. Other target kinds remain conservative: without an explicit portable configuration contract, their complete metadata is bound and a host-dependent change may require re-exporting the plan.

Replay returns once.portable_test_execution.v1:

json

{
  "schema": "once.portable_test_execution.v1",
  "plan_id": "<portable-plan-id>",
  "selected_batch_ids": ["<batch-id>"],
  "complete": true,
  "success": false,
  "execution": {"plan": {}, "next_plan": {}, "schedule": {}, "runs": []}
}

execution uses the ordinary scheduler report. Each runs entry includes its batch_id, target, exit_code, success, validated normalized results or null, and any results_error or process error. Case IDs, suites, ordered attempts, optional attempt durations in milliseconds, and failures stay native normalized results, not batch assignment identities. All assigned batches are attempted even when tests fail. Ordinary test failures return the complete JSON and a nonzero exit. complete means every selected batch has terminal evidence; it does not mean tests passed. Whole-target runners may still omit normalized results when their completed process verdict is authoritative.

Require the expected plan ID, exactly the assigned batch IDs, complete: true, success: true, and a successful process exit before passing a CI partition. Validation errors, interruptions, or infrastructure errors can produce an error envelope or no final document. They must never pass a partition. Cancellation forwards the signal to running batch processes and preserves native signal exit behavior. Exact batches retain isolated outputs and exact-set result validation.

For hosted attribution, add --external-group <UUID> --partition-index <N>. The index is a nonnegative 32-bit integer. Authenticated native batch runs carry these fields, the portable plan ID, and their native batch ID in RunStarted.test_context, independently of normalized command arguments. An external group indicates that its coordinator owns the single consolidated test report per partition. Native batch events remain available for diagnostics and must not independently replace or duplicate the partition's shared test report. This context does not assert partition completion.

Distribute Tests Through Tuist#

Hosted sharding requires a Once release with --shards and --shard-plan. Once 0.71.0 supports portable plans only.

Configure the existing native Tuist provider in the root once.toml:

toml

[infrastructure.cache]
provider = "tuist"

[infrastructures.tuist]
kind = "tuist"
account = "acme"
project = "app"

The default server is https://tuist.dev; set url on the provider for another server. Once reuses its native OAuth credentials, TUIST_TOKEN, or supported CI OIDC authentication. No Tuist executable is needed.

After complete discovery, the planning job creates hosted assignments from opaque portable batch IDs and uploads the portable bytes bound to the hosted UUID:

bash

once query test-plan --shards 4 --reference "$CI_EXECUTION_ID" --format json

The output is {"shard_plan_id":"<UUID>","shards":[0,1,2,3]}. Read the actual returned indices to construct your CI matrix. An empty plan returns an empty UUID and array. --target and repeated --changed-path retain their usual selection behavior. The reference defaults to TUIST_SHARD_REFERENCE or the detected CI run; pass it explicitly outside supported CI environments.

Each worker uses the exact hosted UUID, not a reference that another planning job can replace:

bash

once test --shard-plan "$SHARD_PLAN_ID" --shard-index "$SHARD_INDEX" --jobs 4

TUIST_SHARD_PLAN_ID and TUIST_SHARD_INDEX can supply the worker selection. Once downloads and validates the UUID-bound artifact, executes every assigned batch through its resource-budgeted scheduler, and submits one immutable REST report. An empty assignment runs nothing. Missing results, failed processes, and interrupted work cannot pass the partition. Cancellation attempts a bounded failed report upload, then exits by the original signal. Native events remain diagnostic and do not assert partition completion.

Case retries retain their ordered attempts. Shared Tests identity remains the native target, suite, and case name; distinct native case IDs with the same target/suite/name share that existing identity. Batch IDs are never case names.

For a rerun, increase --shard-attempt. It defaults to TUIST_SHARD_ATTEMPT, then the GitHub run attempt, then 1. --shard-report-revision defaults to 0 and can increase for a corrected report within the same attempt. Each invocation submits one report with one UUID and the supplied attempt and revision. These counters are separate from retries within a test case.

Understand Current Ecosystem Coverage#

Target kind	Stable unit discovery	Exact unit execution	Automatic granularity
`pytest_test`	Yes	Yes	File by default, optional case
`vitest_test`	Yes	Yes	File by default, optional case
`jest_test`	Yes	Yes	File by default, optional case
`rspec_test` for Ruby Specification	Yes	Yes	File by default, optional case
`minitest_test`	One unit per file	Yes, by file	File
`rust_test`	Yes	Yes	Target
`android_local_test`	Yes	Yes	Target
`android_instrumentation_test`	Yes, from a completed device run	Yes, through the instrumentation `class` argument	Target
`apple_test_bundle` with Swift Testing	Yes	Not yet	Target
`shellspec_test`	Yes	Not yet	Target
`zig_test`	Suite-level fallback	Not yet	Target
`elixir_test`	Summary counts only	Not yet	Target

Unsupported ecosystems still participate in conservative affected selection and target-level scheduling. Once reports the limitation instead of guessing a native filter.

Treat Sharding As Scheduling#

The scheduler creates stable batches, orders them using recent uncached durations, and lets idle workers pull the next batch. This avoids the duration skew of a fixed partition while keeping the plan independent from the requested worker count.

Dynamic-language runners default to file batches because files commonly share imports, fixtures, database setup, and interpreter startup. Every discovered case from one file stays in the same batch. Historical batch durations let Once balance those stable files without coupling the plan to four, eight, or any other fixed number of workers. A target can opt into case granularity when its runner and setup make smaller batches worthwhile. Use batching = "target" for the dynamic-language runners when repeated startup or shared fixtures cost more than parallel execution saves. More workers do not force a finer batch size, and more batches are not a guarantee of a faster run.

Concurrent batches write results, logs, and native output below isolated batch directories. After all batches finish, Once validates and merges them into one canonical target result. The plan contains no provider-specific or worker-specific assignment, so local or future remote placement does not redefine test identity. Each worker runs a local schedule; an external CI coordinator can assign the exported portable batches across workers.

Before scheduling a complete target scope, Once verifies that exact batches are disjoint and cover its current discovered units. Exact batch results must report every requested unit exactly once and no unrequested units. A runner that ignores a filter, reports duplicate IDs, or exits successfully without valid passing evidence fails the batch. A startup failure retains its exit status and diagnostic output instead of being replaced by a capability acknowledgement error. Results must remain in the batch's isolated output directory.

Discovery freshness depends on declared collection inputs. A runner must report a complete whole-target inventory and declare inputs that change dynamic test registration. A fingerprint cannot prove that an adapter reported every real case. Periodic whole-target runs help detect inventory drift.

Add Selection to a Project Module#

Project-local Starlark test kinds use the same discovery and planning surfaces as built-in kinds. Query the live contract instead of copying a provider shape:

bash

once query module-contract --format json

The returned test_starter includes the complete once_test_info provider, normalized result paths, action declaration, and filter plumbing. test_target_starter shows the matching manifest table, and normalized_test_result_example is the machine-readable output contract. Its adapter receives:

--once-results <path>;
--once-log <path>;
--once-target <target>; and
one repeated --once-test-unit <identifier> for each exact filter.

The adapter writes once.test_results.v1, uses stable target-qualified case identifiers, honors every requested filter, and exits unsuccessfully for a failed or unknown selection. Once rejects incomplete normalized results before using them for discovery or accepting a scheduled batch. Validate the module and workspace before the first run:

bash

once query validate-module modules/testing.star --format json
once query validate-workspace --format json

After a complete run, the normal manifest, exact-plan, affected-plan, and scheduler commands work without a parallel project-specific interface. See the modules reference for the full authoring contract.

Agent Checklist#

For a coding agent, the safe loop is short:

Query test targets and capabilities.
Inspect an affected plan before running change-scoped tests.
Run a complete target before relying on its manifest.
Request an exact unit only when the manifest declares runner argument filtering.
Treat prioritization and worker count as ordering, never as permission to remove tests.
Read normalized results and schedule attempts instead of scraping native runner output.

With --format json, command failures return an once.error.v1 record on standard output and keep diagnostic logs on standard error. Agents can use the process status for control flow and the structured error.message field for repairs.

The same operations are available through the Model Context Protocol tools. The tool reference maps each tool to its matching terminal command.