Object storage (S3) for files and media
Object storage keeps files ('objects') such as images, videos and backups as a key pointing to bytes plus metadata, in a cheap, very durable, almost unlimited store like Amazon S3.
Why it matters
Photos, videos, backups and documents are big and numerous. Storing them in a database makes it huge, slows backups and costs a lot; storing them on one server's disk breaks scaling.
The idea
Databases are bad at holding big files. Instead, store the file in object storage and keep only its key or URL in the database. Clients can upload straight to storage with a pre-signed URL, so big files never pass through your servers. Put a CDN in front for fast downloads.
A cloakroom: you hand in your coat (the file) and get a ticket (the key). The ticket goes in your pocket (the database).
In a cloakroom you could swap a coat's buttons in place. Objects are usually replaced whole, not edited in the middle, so object storage is for files, not for records that change often.
How it works
- The client asks your API: 'I want to upload a photo'.
- The API checks the user, then asks the storage service for a pre-signed URL: a link that allows one upload, to one key, for a few minutes.
- The client uploads the file straight to that URL. Your servers never touch the bytes.
- The API saves a row: photo ID, owner, object key, size and created time.
- To show the photo, the app uses a CDN URL for that key, or a short-lived signed download link for private files.
Example
Profile photo upload: the app asks the API for a pre-signed URL, uploads straight to S3, then the API saves 'users/42/avatar.jpg' in the users table.
Key points
- Big binary data goes in object storage; the metadata and key go in the database.
- Pre-signed URLs let clients upload or download directly and securely, for a limited time.
- Very durable (S3 is designed for 99.999999999%, 'eleven nines') and cheap per GB.
- Storage classes: hot (frequent access) and cold or archive (cheap, slow to get back).
- Large uploads use multipart upload: in chunks, retryable and resumable.
Common mistakes
- Putting files in database BLOB columns.
- Making a storage bucket public by accident.
- Sending big uploads through your own app servers.
Not true: Videos and other large files: stored in the database, next to their metadata.
In fact: Big files make the database huge and slow its backups. Put the file in object storage, and keep only its key and metadata in the database.
Not true: Big uploads: sent through your own app servers on the way to storage.
In fact: That ties up app servers with slow transfers. Give the client a pre-signed URL, and it uploads straight to object storage.
In the interview
Files go in object storage like S3, never in the database. The database keeps the object key and metadata. Clients upload directly using pre-signed URLs, and downloads go through a CDN.
Part of Unit 3: Data: databases, storage and scaling data in the free System Design course.