StreamStack

StreamStack is a durable stream store for unlimited data feeds.
Feeds hold ordered commit history. Producers append and seal. Consumers read forward from a cursor.

StreamStack is a commit-oriented stream store. Producers append records to open streams, where StreamStack assigns sequence numbers and durably persists batches in a write-ahead log. Sealing a stream publishes a commit that references immutable storage artifacts and advances the feed's ordered history. Consumers can read at feed scope through commit cursors (RAW, deduplicated, or latest-per-key) or at stream scope by reading sealed and unsealed streams directly, maintain named subscription positions, and manage history through compaction, retention, and export.

Why StreamStack?

The natural model of a stream: records append over time and readers resume from a position, that is the base case to build on. In practice, sources and consumers rarely stay in the same shape end to end.

A source may arrive through polls or batches, which gives each ingest run a boundary that may need to be recorded. A destination may impose a different boundary entirely: the downstream store may not accept a continuous stream, or may need to know what changed within a specific batch. Ingestion may also be batched on purpose, even when the source could stream continuously, such as CDC from a busy application where changes are grouped into time-bounded batches so one endless update stream does not overwhelm the pipeline.

StreamStack carries both models (Stream and Batch) on one feed. The feed is a continuous history. A stream is a logical boundary for one batch within it, sealed as a commit when that batch is complete. Readers that want continuous consumption follow the feed with cursors. Readers that want batch-shaped delivery can target a single commit or read across a range, including deduplicated or latest-per-key views when the same record shows up in more than one batch.

Polling Example

An ingest job polls an external source every hour, opens a stream on the feed, and appends the results as pages arrive the same way records are appended to any other stream. When the batch is complete the stream is sealed, and that seal publishes a commit with open time, seal time, and the stream that produced it. The next hour opens the next stream on the same feed and seals again.

So, a full day of hourly polls leaves 24 streams on one feed, each sealed into its own commit. Consumers are not forced to the same cadence. The feed can be followed continuously with cursors or subscriptions, read one sealed stream for a single hour, or read a wider window later in RAW, deduplicated, or latest-per-key mode.