System design: file uploads with presigned URLs, multipart uploads and processing

Key points

• Send the bytes from the client straight to object storage with a presigned URL; your API only authorizes, picks the key and records the file.
• A presigned PUT can pin the exact size and content type it was signed for; a presigned POST policy can enforce a size range.
• Use multipart uploads or a resumable protocol such as tus for large files, and add a lifecycle rule that aborts abandoned multipart uploads.
• Never trust the filename or the Content-Type header: generate the key yourself and check the stored bytes against an allowlist.
• Uploads land in a quarantine bucket, are scanned and processed by idempotent queued jobs, and only then move to the bucket you serve from.
• Serve private files through a CDN with short-lived signed URLs, and delete originals, derivatives and cached copies together.

Requirements: what an upload system has to handle

File upload looks like one form field and turns into a small distributed system: a browser on a weak connection, an object store, a scanner, a queue of processing jobs and a CDN. Before choosing an approach, write down what the feature has to handle.

Keep the bytes in object storage, not in your database or on an app server’s disk. Object storage is built for large, immutable blobs addressed by a key, grows without capacity planning and has lifecycle rules built in; see what object storage is. Your database keeps one row per file: the key, the owner, the declared and detected type, the size, a checksum, a status and timestamps.

Through your server or straight to object storage

There are two places the bytes can go first. In a proxy upload the browser posts the file to your API, which streams it on to the bucket. It is simple and lets you inspect every byte on the way, but each upload holds an app server connection for its whole duration, crosses your network twice, and runs into body-size limits and timeouts on load balancers and serverless platforms.

In a direct upload your API only authorizes. It checks the user, the declared size and type, chooses the object key and returns a presigned URL: a URL carrying a signature made with your cloud credentials, valid for one operation on one key for a short time. The browser sends the bytes to the storage service itself. The AWS documentation describes this as letting someone upload an object without having AWS credentials or permissions of their own; Google Cloud Storage offers the same idea as signed URLs, which it describes as giving limited permission and time to make a request.

Proxying still has a place for small files that must be transformed before they are stored, or where clients cannot reach the storage endpoint. For everything else, direct upload is the default, and the rest of this post assumes it.

Presigned URLs: what to sign and how to limit them

1. The client asks your API for an upload slot, sending the file’s size, its declared type and its name, which is kept for display only.
2. The API checks that the user may upload, their quota and rate limit, and that the size and type are within policy. It creates a file row with status pending and a key it generated: a prefix, the user id and a random UUID.
3. The API returns a presigned URL that expires in minutes and is signed for that key, that size and that content type.
4. The client uploads straight to storage, then tells the API it has finished. A storage event notification can confirm the upload independently, so a client that never calls back is still processed.
5. The API checks the object exists with the expected size, marks the row uploaded and queues the scan.

import uuid
import boto3
from botocore.config import Config

MAX_BYTES = 25 * 1024 * 1024
ALLOWED = {"image/jpeg", "image/png", "application/pdf"}
s3 = boto3.client("s3", region_name="us-east-1", config=Config(signature_version="s3v4"))

def presign_put(user_id: str, size: int, content_type: str) -> str:
    """Called by your API after it has checked the user may upload."""
    if not 0 < size <= MAX_BYTES or content_type not in ALLOWED:
        raise ValueError("file too large or type not allowed")
    key = f"incoming/{user_id}/{uuid.uuid4()}"  # never the user's filename
    return s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": "uploads-quarantine", "Key": key,
                "ContentType": content_type, "ContentLength": size},
        ExpiresIn=300,  # five minutes
    )

url = presign_put("42", 1_048_576, "image/jpeg")
print(url.split("?")[0].rsplit("/", 1)[0])
print([p for p in url.split("&") if "SignedHeaders" in p])
# https://uploads-quarantine.s3.amazonaws.com/incoming/42
# ['X-Amz-SignedHeaders=content-length%3Bcontent-type%3Bhost']

The code above was run offline with placeholder keys; signing needs no network call. With Signature Version 4, the URL lists the headers it was signed with in X-Amz-SignedHeaders. Passing ContentType and ContentLength puts content-type and content-length on that list, so an upload with a different size or type fails the signature check. The AWS guide to presigned uploads makes the same point for content type: the request must send the type the URL was signed with.

A presigned PUT pins one exact size. When you want a range instead, for example because the client cannot report the size up front, use a presigned POST: a signed policy whose conditions S3 checks against a form upload, including content-length-range. boto3 exposes it as generate_presigned_post, whose documentation lists content-length-range and Content-Type among the allowed conditions.

import boto3
from botocore.config import Config

s3 = boto3.client("s3", region_name="us-east-1", config=Config(signature_version="s3v4"))
post = s3.generate_presigned_post(
    Bucket="uploads-quarantine",
    Key="incoming/42/7f3c2a",
    Fields={"Content-Type": "image/jpeg"},
    Conditions=[
        {"Content-Type": "image/jpeg"},
        ["content-length-range", 1, 25 * 1024 * 1024],  # bytes
    ],
    ExpiresIn=300,
)
print(post["url"])
print(sorted(post["fields"]))
# https://uploads-quarantine.s3.amazonaws.com/
# ['Content-Type', 'key', 'policy', 'x-amz-algorithm', 'x-amz-credential', 'x-amz-date', 'x-amz-signature']

• Keep the expiry short: minutes, not the 7-day maximum the AWS SDKs allow. AWS calls presigned URLs bearer tokens: anyone who holds one can use it, more than once, until it expires.
• A presigned URL cannot outlive the credentials that signed it. AWS notes that a URL signed with temporary role credentials stops working when the role session ends, whatever expiry was requested.
• Generate the key on the server and never reuse the user’s filename. OWASP recommends a random name such as a UUID. It also stops one upload replacing another, because S3 overwrites an existing object that has the same key.
• Sign uploads into a quarantine bucket or prefix that nothing serves from.
• Rate-limit the signing endpoint per user, with the token bucket from designing a rate limiter. It is the gate in front of your storage.

Multipart and resumable uploads for large files

A single PUT of a 4 GB video that fails near the end starts again from zero. Multipart upload splits one object into parts that upload independently, in parallel and in any order, and a failed part is re-sent on its own. In S3 it takes three kinds of call: CreateMultipartUpload returns an upload id, each UploadPart sends one numbered part and returns an ETag, and CompleteMultipartUpload sends the list of part numbers and ETags so S3 assembles the object. S3 documents parts of 5 MiB to 5 GiB (the last part may be smaller), up to 10,000 parts, and recommends multipart for objects of 100 MB or more.

1. The client asks your API to start an upload. The API calls CreateMultipartUpload with a key it chose and stores the upload id on the file row.
2. The client splits the file and asks for presigned URLs for a batch of part numbers. Each part is a plain PUT to its own URL, so several can run at once.
3. The client records each part’s ETag. After a crash or a page reload it asks the API which parts S3 already holds, through ListParts, and sends only the rest.
4. The client sends its list of part numbers and ETags to your API, which calls CompleteMultipartUpload. AWS advises keeping your own record of parts for this call rather than building it from a ListParts listing.

Abandoned uploads are the trap. AWS says a multipart upload has no expiry once started: its parts stay stored, and billed, until you complete or abort it. Add a lifecycle rule with the AbortIncompleteMultipartUpload action, as AWS recommends, so uploads nobody finished are cleaned up after a few days.

tus is an open protocol for resumable uploads over plain HTTP, independent of the storage behind it. A client creates an upload with a POST carrying an Upload-Length header and gets a Location back. It sends bytes with PATCH requests that carry an Upload-Offset, and after a drop it sends HEAD to learn how many bytes the server holds, then continues from there. A request whose offset does not match gets 409 Conflict. A tus server, your own or a hosted one, writes the result to object storage. Google Cloud Storage has its own resumable uploads: the first request opens a session, later requests go to the session URI, and its signed-URL documentation notes a signed URL is generally only needed for that first request.

Pick multipart when you upload to an S3-compatible store and control the client code. Pick tus when you want off-the-shelf clients, resumption across browser restarts and a storage backend you can change later.

Validating type and size: never trust the client

Everything the client says about a file is a claim: the name, the extension and the Content-Type header. The OWASP File Upload Cheat Sheet says not to trust the Content-Type header because it can be spoofed, and to allow only the extensions the feature needs rather than block a list of dangerous ones.

The bytes are better evidence. Most binary formats begin with a fixed signature, often called a magic number: JPEG starts with FF D8 FF, PNG with an eight-byte header beginning 89 50 4E 47, PDF with %PDF-. Read the first few bytes of the stored object with a ranged GET, not the whole file, and compare them with what the client declared.

SIGNATURES = {  # leading bytes -> the type they prove
    b"\xff\xd8\xff": "image/jpeg",
    b"\x89PNG\r\n\x1a\n": "image/png",
    b"%PDF-": "application/pdf",
    b"GIF87a": "image/gif",
    b"GIF89a": "image/gif",
}

def sniff(head: bytes) -> str:
    for magic, mime in SIGNATURES.items():
        if head.startswith(magic):
            return mime
    if head[:4] == b"RIFF" and head[8:12] == b"WEBP":
        return "image/webp"
    return "unknown"

def check(head: bytes, declared: str) -> str:
    actual = sniff(head)
    return "ok" if actual == declared else f"reject: declared {declared}, bytes say {actual}"

print(check(b"\xff\xd8\xff\xe0\x00\x10JFIF", "image/jpeg"))      # ok
print(check(b"<html><script>alert(1)</script>", "image/jpeg"))  # reject: declared image/jpeg, bytes say unknown
print(check(b"%PDF-1.7\n%\xe2\xe3", "image/png"))                # reject: declared image/png, bytes say application/pdf
print(check(b"RIFF\x24\x00\x00\x00WEBPVP8 ", "image/webp"))      # ok

• A matching magic number is one check, not proof. OWASP warns that signature validation should not be used on its own because bypassing it is common and easy: a file can open with a valid JPEG header and carry a script further in. Use it as a fast filter before scanning and re-encoding.
• Check size twice: the declared size before signing, the stored object’s real size after the upload. For archives, cap the decompressed size as well, as OWASP advises, and the number of entries, since a small zip can expand enormously.
• Allow types per feature: an avatar accepts JPEG, PNG and WebP; a document upload accepts PDF.
• Re-encode images you will display. Decoding and re-encoding into an allowed format strips metadata such as GPS coordinates and a good deal of hidden content, though OWASP notes that rewriting does not remove everything.
• For PDFs and Office documents, OWASP lists content disarm and reconstruct (CDR), which rebuilds the file without active content such as macros.
• Store each object with the type you detected, not the declared one, and serve user files from a separate domain, with Content-Disposition: attachment for files that should download rather than render.

Quarantine, malware scanning and the processing queue

Use two buckets, or two prefixes with different permissions: a quarantine bucket that receives uploads and that no user can read, and a clean bucket you serve from. Nothing moves to clean until it passes. OWASP recommends running uploads through an antivirus or a sandbox where one is available.

1. The upload completes in quarantine. A storage event, or the client’s completion call, enqueues a scan job with the file id.
2. A worker reads the object, checks its size and magic number, and runs it through a scanner: an antivirus engine on your own workers or a vendor’s scanning API.
3. If it is clean, the worker copies it to the clean bucket under the same key, deletes the quarantine copy, marks the row clean and enqueues processing jobs. If it is infected or invalid, the worker marks it rejected, leaves it in quarantine for a short investigation window and tells the user.
4. Processing jobs make the derivatives: image sizes, a PDF preview, video renditions. Each writes to a key derived from the source key and the job type, then records that it is done.

Every step runs on a queue with retries, so every step will sometimes run twice. Make each one idempotent: derive output keys deterministically, so a rerun overwrites the same object, and move the row through guarded status changes (uploaded to clean, never back). Idempotency and job queues covers the patterns. Video transcodes can run far longer than image jobs, so give them their own queue and workers, so they cannot starve the quick jobs, and record progress the interface can show.

• Scanners have limits: archive depth, file size, password-protected files. Decide in advance what happens to a file the scanner cannot open; rejecting it is the cautious default.
• Keep the scan result, the engine and its signature version on the file row, so you can find and rescan files if a detection is added later.
• Emit an event when a file becomes usable, so the rest of the app reacts to clean rather than to the upload itself.

Serving files: CDN, signed URLs and storage classes

Public, unchanging files such as product images go through a CDN with long cache lifetimes. Put a content hash or a version in the key, so a changed file is a new URL and nothing has to be purged; what a CDN is explains the caching side.

Private files need two things. First, only the CDN can read the bucket: the CloudFront documentation recommends requiring users to come through CloudFront rather than straight to the origin, so they cannot bypass the restrictions in signed URLs or signed cookies. Second, your API hands out short-lived signed URLs or signed cookies, only after checking that the user may see that file. A signed URL suits a single download; a signed cookie suits a page that shows many private images.

• S3 lifecycle rules have two kinds of action, according to its documentation: transitions, which move objects to another storage class after a set time, and expirations, which delete them.
• Originals that are rarely read after their first month can move to an infrequent-access or archive class. Derivatives you can regenerate may not be worth keeping in a cold class at all.
• Colder classes have minimum storage durations, and archive classes have retrieval delays, so check both before writing a rule for short-lived files.
• Give temporary prefixes, such as quarantine, exports and incomplete multipart uploads, short expiration rules of their own.

Deletion, retention and privacy

User files are often personal data: photos, identity documents, medical records. Under the GDPR a person can ask for their data to be erased, and data should not be kept longer than its purpose needs; GDPR for apps covers the rules. This is not legal advice. The engineering requirement is the same in any jurisdiction: you must be able to find and delete every copy.

• Keep an index. The file row knows the original key and every derivative key, so deletion is a lookup, not a bucket scan.
• Delete in order: mark the row deleted so access checks fail at once, delete the original and its derivatives, then purge CDN caches for any public URL. Short signed-URL expiries take care of the private ones.
• Versioned buckets keep old versions after a delete. Add a lifecycle rule for noncurrent versions, or a deletion only hides the file.
• Backups and logs hold copies too. Document how long they keep them, and keep object keys free of personal details such as names or email addresses.
• Strip location metadata from images you republish, and encrypt sensitive buckets with keys whose use you can audit.
• Set retention per kind of file: rejected uploads for a short investigation window, user content until the user or their account is deleted, exports and temporary files on short expiries.

BYD Tech builds web apps with uploads, background jobs and the storage behind them: the code lives in the client’s GitHub from day one, and AI-assisted work is reviewed by an engineer.