Cloud Storage System (Google Drive/Dropbox)
Understand how cloud storage systems like Google Drive and Dropbox sync files. Explore block-level chunking, content-based deduplication, delta sync, push notifications
Continue with the guided workspace
Use the paired workspace to connect this guide's mental model with live component state, visible signals, and the outcome each step or scenario is meant to prove.
Mental Model
A cloud storage system allows users to store, sync, and share files across multiple devices. Rather than transferring entire files (which wastes bandwidth and fails frequently on large payloads), the system breaks files into smaller, immutable blocks or chunks. It keeps track of file state via a lightweight Metadata Database and uses real-time WebSockets to notify other devices of changes.
Components and Why They Exist
Client (Mobile & Desktop)
This component exists to run the local application, monitor the file system for edits, and orchestrate the synchronization flow on the user’s device. It performs block-level chunking, calculates SHA-256 hashes of chunks, checks with the server for deduplication, and handles data transfers. In the demo, watch how the client splits files into small grid-based blocks and coordinates local cache updates.
Block Server & Object Storage (S3)
This component exists to save file contents efficiently without storing duplicate chunks. The Block Server matches chunk hashes to filter out duplicates, while Object Storage preserves the unique, immutable data payloads. In the demo, watch blocks being uploaded to S3 and note how duplicate chunks are skipped.
Sync Service & Metadata DB
This component exists to coordinate transfers and act as the single source of truth for file versions and block mappings. It indexes file paths, versions, and lists of chunk hashes, ensuring ACID metadata updates are committed correctly. In the demo, observe the updates to the Files Index and Block Catalog tables inside the Metadata DB node.
Notification Service (WebSockets)
This component exists to inform peer clients of updates instantly without having them constantly poll the server. It maintains persistent connection channels and broadcasts metadata change events to online devices. In the demo, watch animated notification packets travel from the Notification Service to the peer client.
Edge Cases and Misconceptions
Global Deduplication vs. User Privacy
Deduplication is global across the system, meaning if another user has already uploaded the exact same file chunk (e.g., a common OS installer or software library), your client skips uploading it. This saves massive storage but requires the system to protect block storage keys (SHA-256 hashes) from leakage, so users cannot guess other users’ files based solely on their hashes.
Metadata vs. Block Immutability
Actual data blocks in S3 are immutable and never updated in place. When a file is modified, new blocks are uploaded for the modified parts, and the metadata points the file’s latest version to the new list of block hashes. Stale blocks are cleaned up later via garbage collection if no files reference them.
Common Questions
Why split files into block-level chunks?
Breaking files down (typically into 4MB chunks) avoids uploading massive files at once, allowing the system to resume interrupted uploads and execute delta syncs.
What is Content-Based Deduplication?
It is the technique of hashing blocks (e.g., using SHA-256) and comparing them to existing blocks in S3. If a block already exists, the upload is skipped, saving network bandwidth.
What is a delta upload?
A delta upload sends only the specific blocks that were modified when a file was edited, rather than re-uploading the entire file.
How are notifications sent to other clients?
The Sync Service updates the Metadata DB and triggers the Notification Service, which uses active WebSocket connections to notify other clients to sync.
How does the system resolve concurrent edit conflicts?
If Client B tries to save a version based on a stale metadata version, the request is rejected with a conflict. Client B renames its local file to a conflicted copy, uploading it separately.
How does the client know which chunks to download during a delta sync?
The client fetches the latest metadata representation of the file (the chunk map) from the Metadata DB. It compares the list of chunk hashes in the remote version to its local cache, downloading only the chunks whose hashes are missing locally.