Live Run Event Protocol

The Once client streams typed events over gRPC to a compatible ingest server as they happen during a run: run and target lifecycle, per-target phases, cache decisions, and (framework-dependent) individual test cases. The protocol is versioned as once.events.v1.

This page documents the shape of the wire, the event catalog, opt-in configuration, the delivery guarantees the client honours, and the current scope of what a Once run emits today.

What the client sends#

Every run publishes a stream of events to a single ingest service. The service defines four methods and one bidirectional streaming call:

  • GetServerCapabilities returns supported protocol versions, batch and event size caps, the finalization grace window, deduplication retention, and the safe-literal allowlist version.
  • GetArgvHashKey returns the project-scoped keyed-hash key the client uses to normalize argument values without leaking their content. The key rotates on a server policy; retired keys remain valid on ingest for a documented grace window.
  • PublishRunEvents is the bidirectional streaming call. The client streams batches; the server streams one BatchAck per batch.
  • GetRunAck is a unary read-only probe used on reconnect to discover the server's durable state before the client resumes.

Every event carries a monotonic sequence number, a wall-clock stamp for display, and a monotonic offset from run.started that is authoritative for durations. A batch carries a client-generated id, an optional list of loss-interval control records, and a contiguous range of events. The server processes any loss controls first, then validates the batch's sequence range against the current expected next sequence, then durably projects the events under a single transaction that also advances the acknowledged sequence.

Delivery guarantees#

The delivery model is at-least-once with server-side deduplication by (run_id, seq). Duplicates with identical payload are silently accepted; duplicates with divergent payload are logged and dropped. The client keeps an in-memory ring of unacknowledged events with a reserved slot for the run's terminal event, so overflow of ordinary events can never suppress the run's terminal state.

Ordinary overflow drops the oldest unacknowledged events and records the dropped sequence range as a loss interval. On the next batch send, the loss interval is drained into the batch's gap_advances list as a canonical sorted non-overlapping set. Loss is never carried as an event that consumes a sequence number, so a dropped range cannot collide with a retained event's sequence.

Terminal delivery uses an explicit intent event. When the run is ending, the client emits a finalization intent event and enters a bounded local drain. The server starts its own grace window on receipt and transitions the run to a distinct finalization state: finalized on receipt of the terminal event, finalization pending if the grace elapses first, or lost if no intent and no heartbeats arrive past the liveness timeout. A late terminal event within the deduplication retention window transitions a pending or lost run to finalized while preserving the fact that the run was previously marked pending or lost.

Reconnect uses GetRunAck to reconcile the client's mirror of the server's durable state before resuming. Ring buffer and loss intervals survive across reconnects because they live in the caller-owned session state; producers continue publishing through the outage without observing it.

Event catalog#

Every event carries scope information sufficient for the projector to resolve per-target and per-case correlations without ambiguity.

Run lifecycle#

  • run.started records Once version, protocol version, host class, git revision, a normalized argument vector, working directory relative to the workspace, an environment fingerprint, a graph digest, the project identifier (validated against the bearer token), and the effective limits negotiated with the server.
  • run.finalizing announces the client's intent to close the run and starts the server's grace window.
  • run.completed records the terminal result (succeeded, failed, cancelled, timed out, or infrastructure error), an optional cancellation reason, the wall duration, and aggregate totals.
  • run.heartbeat fires every five seconds while the run is active so the server can distinguish a network partition from a clean exit.

Graph#

  • graph.compiled records the graph digest, target count, target kind histogram, and the requested roots.
  • target.instance records a target instance identifier for the logical occurrence of a target within a run.

Target execution#

  • target.queued and target.started bracket a target attempt.
  • target.phase records transitions along the six exclusive ordered phases: queued, cache checking, preparing, executing, capturing, and publishing.
  • target.wait announces what a queued target is waiting on (dependency wait, resource wait, worker wait, throttled, or infrastructure).
  • target.completed records terminal result, whether a cache restore covered the outcome, an optional exit code, an evidence reference, and duration.
  • target.cancelled records a cancellation reason.
  • target.retried links a previous attempt identifier to the new attempt identifier a following target.queued will mint.

Tests#

  • test_suite.started and test_suite.completed bracket the tests produced by one test target.
  • test_case.started and test_case.completed describe every individual case with a stable case identifier, its display name, and per-case duration and terminal result.
  • test_case.retried links previous and new attempts across a case retry.

Logs#

  • log.chunk streams a slice of standard output or standard error scoped to the run, a target execution, or a test case execution. Chunks in a scope never overlap.
  • log.truncated records dropped bytes when a stream exceeds its size threshold.

Cache#

Cache is a first-class concern in this protocol; every event in one lookup is grouped by a single cache decision identifier so the projector renders one cache decision row per target execution.

  • cache.probe records a single tier lookup with its ordinal and outcome (hit, miss, error, or bypassed).
  • cache.miss_reason records why an action digest did not match a baseline. A required resolution field states whether a comparison baseline is present (with a reference to a previous run and digest), none is available, none was attempted, or the lookup failed. A primary reason plus the full set of detected reasons, differing input paths (with a truncation indicator), and an analysis status accompany the resolution.
  • cache.upload, cache.download, and cache.store_reused each record a single content transfer or a skipped-upload signal. cache.store_reused carries the bytes an upload would have transferred had the digest not already been present.

Artifacts and diagnostics#

  • artifact.published announces a declared output with its kind, a content reference, and a workspace-relative path.
  • diagnostic.emitted carries a normalized lint or type-check finding with a stable fingerprint for cross-run deduplication.

Identity and redaction#

Tenancy is derived from the bearer token; the wire never carries a tenant field. The project identifier in run.started is validated against the token's authorized set.

Raw absolute paths, raw environment values, raw hostnames, and unredacted argument vectors are not part of the protocol; the schema has no fields for them. The client sends only the sanitized shapes:

  • A normalized argument vector as an ordered list of safe literals (from an explicitly enumerated allowlist), flag keys, combined --key=value records with an opaque value hash, and generic opaque value hashes for every other positional. The keyed hash uses a project-scoped key retrieved via GetArgvHashKey.
  • A repository-relative working directory validated for traversal at ingest. Leading /, .., or absolute-path shapes are rejected.
  • Coarse host and worker class tokens rather than resolvable hostnames.
  • A content-addressed environment fingerprint over the actionable subset of the environment.

Log ingestion is off by default and requires explicit per-project opt-in on the server. Diagnostic messages, test names, parameters, and artifact paths remain author-controlled but are validated at ingest; suspicious content (absolute paths, secrets-shape tokens) is quarantined and surfaced as a data-quality warning on the projected state.

Every event field carrying a source path (test case files, diagnostic locations, artifact paths, differing inputs) must be workspace-relative and is validated for traversal at ingest. Violations clear the offending field and record a data-quality warning; the enclosing event is still projected.

Enabling the client#

Live ingest is a provider capability. Configure a provider by its origin in once.toml and authenticate against it; the client then asks the provider what it supports and uses the events endpoint the provider advertises.

The workflow is:

  1. The workspace names a provider (for example a provider bound to https://tuist.dev) in its infrastructure configuration, and the user authenticates against it.
  2. On run start the client asks the provider for its capabilities. The provider advertises which of cache, remote_execution, and events it supports, along with the endpoint (and any auth metadata) for each supported surface.
  3. If events is advertised, the client dials the endpoint the provider named, using the same bearer credentials the provider already established. No separate configuration is needed.
  4. If events is not advertised, the client simply does not send any events. The run behaves exactly as it did before ingest was available.

This makes ingest a property of the provider rather than a per-project deployment concern: turning it on for a workspace is a matter of switching to a provider that advertises it.

The client subscribes to the run's local event bus, streams events through a bidirectional call while the run executes, and drains for up to three seconds before the CLI exits. If the endpoint is unreachable or the connection is lost, the client backs off, calls GetRunAck to reconcile, and resumes. Ingest failures never abort the run itself; the CLI logs a warning and continues.

Local override#

For development and manual testing against a server that is not behind a configured provider, an environment variable can point the client at any endpoint directly:

bash
export ONCE_EVENTS_ENDPOINT=https://events.example.com
once build //some:target --ui

The environment variable, when set, takes precedence over the provider's advertised endpoint for that run only.

What Once emits today#

Not every event in the catalog above is fired yet; the wire vocabulary is complete but the client fire-points multiply over time. As of the current release:

  • Run and target lifecycle: run started, finalizing, completed; target queued, started, completed. Fired from the local UI publisher.
  • Target phases: all six phases (queued, cache checking, preparing, executing, capturing, publishing) are fired from the main graph command around the natural boundaries in the run loop.
  • Log chunks: every subprocess chunk is forwarded target-scoped from the local output observer.
  • Test suites and test cases: the wire schema is in place, and a parser for cargo test --format=json output is available in the CLI. Wiring the parser into the test command emits per-case events end to end; other test frameworks follow the same shape.

The following event kinds are defined in the wire vocabulary but not fired yet in Once because they need executor internals to grow richer signals:

  • Cache probes and cache miss reasons.
  • Artifact publications with the full content reference shape.
  • Diagnostic emissions aligned with the lint findings work.
  • Target wait reasons for the critical-path view.

Adding a fire-point never requires changing the wire; the transport already carries every payload the vocabulary defines.

Compatibility#

Version is expressed in the proto package: once.events.v1. A future breaking change lands as once.events.v2 served alongside v1 for a deprecation window. Non-breaking additions land as new payload variants and are silently tolerated by older clients through the proto #[non_exhaustive] shape.

A server may serve this protocol alongside other gRPC services on the same endpoint, so there is no requirement for a dedicated hostname, TLS certificate, or ingress rule.