WAL Storage

In the stream storage repository S3Stream, WAL is one of the core components. WAL storage primarily addresses issues such as high latency when using object storage as the main data store and low efficiency in writing to multiple partitions. It has two primary responsibilities:

Providing low-latency, high-performance data persistence writes. Once the data is successfully written to WAL, confirmation is returned to the client.
During a Broker node failure that requires failover, it recovers data from WAL that has not been timely uploaded to S3.

WAL Storage Implementation

WAL is implemented as a fixed-size, cyclically written storage space. Various storage media can be selected, but the implementation mainly considers the following points:

Centralized writes, unlike Apache Kafka®, AutoMQ does not need to write separate log files for each partition. By mixing the data from all partitions into the WAL, it supports efficient writing in scenarios with a large number of partitions.
Sequential writes and group commits, data is written sequentially into the WAL, combined with the group commit mechanism, requiring only a small amount of IOPS to achieve high-throughput writes.
Direct IO Writing: Data is directly written through to the storage medium to fully utilize the durability of cloud storage. Write operations are considered successful only upon confirmation, unaffected by the operating system's Page Cache dirty page collection.
Support for Raw Device Writing[1]: AutoMQ needs to write only one file. If EBS is used as the storage medium, it can be used directly as a raw device for writing, without the need to mount a file system, thereby avoiding additional overhead brought by the file system. This ensures optimal performance and latency.

Through the aforementioned technology, WAL storage significantly reduces the dependency of stream storage systems on IOPS, addressing the issues of low write efficiency and high API call costs associated with object storage, thus making S3Stream widely applicable. AutoMQ supports both latency-sensitive and high-throughput systems.

WAL Storage Medium Selection

Public Cloud providers generally offer three types of storage services, namely:

- Block storage, such as AWS EBS, Azure Zone-redundant Disk, GCP Regional Persistent Disk, and Alibaba Cloud Regional ESSD. Among these services, EBS adopts a single AZ multi-copy architecture, while the other block storages use a multi-AZ multi-copy architecture, collectively referred to as Regional EBS.
- Object storage is the most standard storage service provided by cloud providers, with virtually all cloud providers supporting the standard S3 protocol.
File storage, primarily NFS protocol-based file storage services, is also widely used in big data and other business scenarios, such as AWS EFS and AWS FSx series.

The aforementioned storage mediums can be used as WAL storage mediums, with specific differences as shown in the table below.

-	EBS & Regional EBS WAL	S3 WAL	NFS WAL
Multi AZ	EBS has a single AZ multi-replica architecture Regional EBS has a multi AZ multi-replica architecture	S3 offers both single AZ and multi AZ product options	NFS offers both single AZ and multi AZ product options
Durability	Between 5 nines and 9 nines	Around 11 nines	Around 11 nines
Latency	Sub-millisecond	Hundred-millisecond	Millisecond-level
Cost	Low	Low	Moderate
Applicable Scenarios	Regional EBS is the best option, suitable for all Kafka use cases	Suitable for most latency-insensitive scenarios, such as logging and monitoring	Low-latency solutions on AWS, such as core transaction matching scenarios

When AutoMQ selects S3 WAL, S3 provides both WAL storage and primary storage services. WAL storage is optimized for writes and will consume fewer S3 API calls, especially in scenarios with a large number of partitions. S3 WAL has efficient data ingestion capabilities, while the primary storage part provides efficient data consumption through various compaction mechanisms.