Producer

Producer ingests data by opening stream (implicitly) on feed, appending batches of records, and sealing stream into commit (optional). What belongs in one stream and when to seal are producer decisions. See Architecture.

Lifecycle

Step	API	What Happens
Open	open(feed)	Allocates new stream, acquires writer lease, returns AppendSession
Append	append / appendAsync	Submits batch, assigns sequence numbers, persists to WAL, returns after durability
Seal	seal(message)	Closes stream, finalizes artifacts, publishes commit, trims WAL

Feed holds many sealed streams over time. Typically one open stream per feed at time, enforced by writer lease.

WAL (Durability)

Batches are not written to bucket one at a time. Server buffers them into WAL segments and flushes when either threshold is reached:

Trigger	Default
Time	~250 ms
Buffered size	~8 MiB

Each segment is one atomic object PUT. When PUT succeeds, every batch in that segment is acknowledged. When PUT fails, those batches fail.

After ack, data survives process crash. May not yet appear in pack or manifest form.

Publish / Append

append and appendAsync complete when batch is durable in WAL segment. Response is AppendResult with assigned sequence range (first, last, recordCount).

Unsealed records are readable on feed before seal. Seal publishes them into commit history with full storage artifacts.

API	Behavior
append	Blocks until WAL ack
appendAsync	Returns future that completes on WAL ack

On server, sequence assignment and fence or match_seq checks run at submit time. On client, appendAsync respects maxUnackedBytes (default 16 MiB) and blocks when unacked payload would exceed cap. seal drains all in-flight async appends before calling server.

Continuous Flush

After WAL ack, records are handed to background flusher that incrementally builds pack, manifest, and key index state while stream stays open. Append throughput is not blocked on artifact upload.

Seal waits for this lane to drain, then publishes commit. Seal latency can exceed last append ack on large streams.

Seal Stream

On seal, server:

Operation	What Happens
Seal	Marks stream sealed and rejects further appends
Flush	Flushes remaining WAL data
Finalize	Drains continuous flusher and finalizes artifacts
Publish	Publishes commit record
Trim	Trims WAL segments for that stream

Batches

AppendBatch carries one or more records plus optional coordination fields:

Field	Purpose
Records	Payload bytes, optional key, version, tombstone marker, optional fingerprint
fencingToken	Must match stream writer epoch or append is rejected
matchSequence	Optimistic tail check: must equal current last sequence

Fingerprint is an opaque 32-byte identity. Omit it and the server defaults to SHA-256 of the payload. If supplied, the server stores it as-is.

CLI binds writer fencing token automatically. Java client does same when batch omits one.

Keyed and keyless records can mix in one stream when feed allows keys. Key index artifacts produced only for keyed feeds.

Leases & Fencing

Each feed has one active writer lease at time. Opening or resuming stream acquires lease and bumps writer epoch. Superseded writer receives WriterFencedException on later appends or seal.

In multi-node deployment, write-path requests that hit non-owner may proxy to lease holder. See Installation.

Recovery

Operation	Purpose
resume(feed)	Reattach to latest unsealed stream after restart
admin resume	Server-side equivalent for abandoned open stream
admin discard	Drop unsealed streams without publishing commit
admin repair	Finish seal that failed mid-publish

Resume discards staged flush state for that stream and continues from WAL.

Examples

CLI one-shot append and seal:

./streamstack append orders \
  --record k1=hello --record k2=world --message 'hourly poll'

Java SDK with explicit session control:

AppendSession session = client.open("orders");
session.append(new AppendBatch(List.of(
    AppendRecord.of("hello".getBytes()),
    AppendRecord.keyed("k2".getBytes(), 0L, "world".getBytes())
), Optional.empty(), Optional.empty()));
Commit commit = session.seal("hourly poll");

Pipelined ingest with byte window before seal:

AppendSession session = client.open("orders").maxUnackedBytes(4 * 1024 * 1024);
for (AppendBatch batch : batches) {
    session.appendAsync(batch);
}
Commit commit = session.seal("bulk load");