Object storage (S3) for files and media

Object storage keeps files ('objects') such as images, videos and backups as a key pointing to bytes plus metadata, in a cheap, very durable, almost unlimited store like Amazon S3.

Why it matters

Photos, videos, backups and documents are big and numerous. Storing them in a database makes it huge, slows backups and costs a lot; storing them on one server's disk breaks scaling.

The idea

Databases are bad at holding big files. Instead, store the file in object storage and keep only its key or URL in the database. Clients can upload straight to storage with a pre-signed URL, so big files never pass through your servers. Put a CDN in front for fast downloads.

A cloakroom: you hand in your coat (the file) and get a ticket (the key). The ticket goes in your pocket (the database).

In a cloakroom you could swap a coat's buttons in place. Objects are usually replaced whole, not edited in the middle, so object storage is for files, not for records that change often.

How it works

  1. The client asks your API: 'I want to upload a photo'.
  2. The API checks the user, then asks the storage service for a pre-signed URL: a link that allows one upload, to one key, for a few minutes.
  3. The client uploads the file straight to that URL. Your servers never touch the bytes.
  4. The API saves a row: photo ID, owner, object key, size and created time.
  5. To show the photo, the app uses a CDN URL for that key, or a short-lived signed download link for private files.

Example

Profile photo upload: the app asks the API for a pre-signed URL, uploads straight to S3, then the API saves 'users/42/avatar.jpg' in the users table.

Key points

Common mistakes

Not true: Videos and other large files: stored in the database, next to their metadata.
In fact: Big files make the database huge and slow its backups. Put the file in object storage, and keep only its key and metadata in the database.

Not true: Big uploads: sent through your own app servers on the way to storage.
In fact: That ties up app servers with slow transfers. Give the client a pre-signed URL, and it uploads straight to object storage.

In the interview

Files go in object storage like S3, never in the database. The database keeps the object key and metadata. Clients upload directly using pre-signed URLs, and downloads go through a CDN.

Part of Unit 3: Data: databases, storage and scaling data in the free System Design course.