Object Storage PUT/GET Lifecycle Visualization

Mental Model

An S3-compatible Object Storage system must store petabytes of data reliably on cheap commodity hardware. Unlike traditional filesystems, object storage decouples file metadata from the raw byte chunks, utilizing erasure coding and active self-healing to maintain high durability.

For interactive exploration of these behaviors, you can run the live sandbox: Object Storage System Simulator.

Components and Why They Exist

Client

Simulates active client applications putting (writing) or getting (reading) large files. Issues HTTP REST commands (PUT/GET/DELETE) and downloads/uploads object streams. In the demo, watch requests flow from the client to the API Gateway.

API Gateway

Serves as the public-facing HTTP interface that validates incoming requests and handles access control/throttling. Verifies authentication, checks bucket policies, and manages request routing. In the demo, watch requests validate here before proceeding.

Multipart Coordinator

Manages upload lifecycle for large files by splitting them into smaller, parallelizable chunks. Receives object chunks, coordinates upload state, and reconstructs files once all parts arrive. In the demo, watch it split large object uploads into chunks.

Erasure Coder

Optimizes storage efficiency while maintaining high durability by creating parity data. Splits data payloads into $N$ data chunks and generates $M$ parity chunks (e.g. 4+2 EC scheme). In the demo, watch it compute data and parity chunks.

Placement Engine

Selects storage paths that maximize fault tolerance by separating chunk locations across different racks/regions. Maps chunks to specific disks/nodes in accordance with anti-affinity rules. In the demo, observe chunks scattering across different Storage Nodes.

Metadata Service

Acts as the catalog that maps keys to logical files and physical storage disk locations. Stores manifests containing checksums, version IDs, chunk locations, and metadata. In the demo, watch it commit the manifest last, completing the PUT lifecycle.

Storage Nodes (A, B, C)

The physical disks and servers storing actual raw byte chunks. Exposes block/disk level API to read and write individual chunk files. In the demo, watch chunks land on nodes. Red highlights indicate offline storage nodes.

Rebuild Monitor

Reconstructs lost chunks automatically to maintain system durability in the background. Detects missing chunks from disk failures, reads surviving chunks, recalculates missing data, and writes to new disks. In the demo, watch it pull surviving chunks and repair the failed storage node chunks.

Disk Sweeper

Performs garbage collection to reclaim storage space after object deletion. Deletes physical chunks in the background after the object metadata has been marked with a Delete Marker. In the demo, observe chunks getting garbage collected and wiped from Storage Nodes.

Edge Cases and Misconceptions

Metadata Availability vs. Data Plane Health

A common misconception is that if storage nodes are healthy, data is accessible. In object storage, the Metadata Service is the critical brain. If the metadata database suffers an outage, the system cannot resolve object keys to physical chunk paths, causing all GET/PUT requests to fail, even if storage disks are fully operational.

Replication vs. Erasure Coding Tradeoffs

Replication (e.g. 3-way mirroring) is simple and has low CPU overhead, but requires a $200\%$ storage overhead. Erasure Coding (e.g. 4+2) only requires a $50\%$ storage overhead for similar durability but consumes significant CPU resources for calculating parity and reconstructing files.

Delete Markers vs. Physical Deletion

When an object is deleted, it is not immediately wiped from disk. Instead, the Metadata Service commits a “Delete Marker” version. This makes the object appear deleted to users (GET returns 404). The physical data is only deleted later when background Disk Sweeper jobs run asynchronously.\n