Overview

S3Stream is a core stream storage component within AutoMQ that adheres to the concept of storage-compute separation. It offloads Apache Kafka®'s native ISR-based log storage layer to cloud storage EBS and object storage.

S3Stream is a stream storage library rather than a distributed storage service. AutoMQ innovatively implements a set of core stream storage APIs on top of object storage, including position management, Append, Fetch, and Trim data. The following code snippet shows several core interfaces of these APIs.

public interface Stream {
    /**
     * Get stream id
     */
    long streamId();

    /**
     * Get stream start offset.
     */
    long startOffset();

    /**
     * Get stream next append record offset.
     */
    long nextOffset();

    /**
     * Append RecordBatch to stream.
     */
    CompletableFuture<AppendResult> append(RecordBatch recordBatch);

    /**
     * Fetch RecordBatch list from a stream. 
     */
    CompletableFuture<FetchResult> fetch(long startOffset, long endOffset, int maxBytesHint);

    /**
     * Trim stream.
     */
    CompletableFuture<Void> trim(long newStartOffset);
}

Core Characteristics of Stream Storage

All data on the internet is generated in a streaming manner, then stored and computed in streams to extract the business value of real-time data. This also implies that stream data imposes at least the following requirements on storage:

• Low Latency: The greatest value of streaming data lies in its freshness. For example, businesses related to ad recommendations have very high real-time requirements. The faster data is stored and computed, the more value it can provide.

• High Throughput: Since all data is generated in a streaming manner, stream storage must support extremely high throughput. Many businesses require at least GiB/s bandwidth.

• Low Cost: Massive amounts of streaming data mean high storage costs. Additionally, many businesses need data playback and re-computation capabilities, making daily storage of stream data a business norm.

With the rapid development of big data, the demands on stream storage in terms of cost, latency, and throughput have correspondingly increased. However, no cloud storage service currently available can simultaneously meet all of these requirements.

• Block storage has low latency but high costs.

• Object storage is cost-effective, but each API call incurs a latency of around 100ms.

• File storage charges based on bandwidth, making it unsuitable for high-throughput stream storage scenarios.

AutoMQ innovatively combines block storage EBS and object storage, leveraging their strengths to provide low-latency, high-throughput, cost-effective, and nearly "infinite" capacity stream storage capabilities.

S3Stream Architecture

In the core architecture of S3Stream, data is first persistently written to the WAL and then uploaded to S3 storage in near real-time. Additionally, to efficiently support both Tailing Read and Catch-up Read models, S3Stream has a built-in Message Cache component to accelerate reads.

• WAL Storage: Low-latency storage media is selected, with each WAL disk requiring only a few GiB of space, typically using cloud storage EBS.

• S3 Storage: The largest object storage service offered by cloud providers is selected to provide high-throughput and cost-effective primary data storage services.

• Data Caching: Both hot data and prefetched cold data are stored in the cache to accelerate reads. At the same time, an efficient eviction mechanism based on consumer focus ensures optimal memory utilization.