- Providing low-latency, high-performance data persistence writes. Once the data is successfully written to WAL, confirmation is returned to the client.
- During a Broker node failure that requires failover, it recovers data from WAL that has not been timely uploaded to S3.
WAL Storage Implementation
WAL is implemented as a fixed-size, cyclically written storage space. Various storage media can be selected, but the implementation mainly considers the following points:- Centralized writes, unlike Apache Kafka®, AutoMQ does not need to write separate log files for each partition. By mixing the data from all partitions into the WAL, it supports efficient writing in scenarios with a large number of partitions.
- Sequential writes and group commits, data is written sequentially into the WAL, combined with the group commit mechanism, requiring only a small amount of IOPS to achieve high-throughput writes.
- Direct IO Writing: Data is directly written through to the storage medium to fully utilize the durability of cloud storage. Write operations are considered successful only upon confirmation, unaffected by the operating system’s Page Cache dirty page collection.
- Support for Raw Device Writing[1]: AutoMQ needs to write only one file. If EBS is used as the storage medium, it can be used directly as a raw device for writing, without the need to mount a file system, thereby avoiding additional overhead brought by the file system. This ensures optimal performance and latency.
WAL Storage Medium Selection
Public Cloud providers generally offer three types of storage services, namely:- Block storage, such as AWS EBS, Azure Zone-redundant Disk, GCP Regional Persistent Disk, and Alibaba Cloud Regional ESSD. Among these services, EBS adopts a single AZ multi-copy architecture, while the other block storages use a multi-AZ multi-copy architecture, collectively referred to as Regional EBS.
- Object storage is the most standard storage service provided by cloud providers, with virtually all cloud providers supporting the standard S3 protocol.
- File storage, primarily NFS protocol-based file storage services, is also widely used in big data and other business scenarios, such as AWS EFS and AWS FSx series.
In scenarios where AutoMQ chooses S3 WAL, S3 simultaneously offers both WAL storage and primary storage services. The WAL storage is optimized for writes, requiring fewer S3 API calls. Especially in environments with numerous partitions, S3 WAL provides efficient data ingestion capabilities, while the primary storage component supports efficient data consumption through various compaction mechanisms.
Notably, for AutoMQ Open Source, only S3-compatible storage services are supported as WAL storage options. This indicates that AutoMQ Open Source is a cloud-native Kafka implementation that relies solely on object storage, with a minimal deployment architecture. For AutoMQ commercial editions, different WAL options are available on various cloud providers to support more workloads.