Upload authorization need not transfer the file itself
Two kinds of load
The API checks the user, accepts metadata and authorizes uploads. File transfer is a large byte flow. At 100 files/s of 15 MB each, throughput must be 1500 MB/s. One API here transfers only 600 MB/s, although it can handle the small metadata workload.
Proxying or a direct path
In the proxied design, the client sends bytes through the API to object storage. This requires enough API throughput. In the direct design, the API issues a limited permission and the client transfers bytes straight to storage. Access is still checked; checking is simply separated from transfer. Signed URLs are not declared single-use.
Why split files into chunks
Small chunks transfer easily but add overhead. Large chunks need more buffer. Here, chunk size determines an estimated useful storage rate, with one chunk reserved per stream. This compares options; it does not measure actual browser, API or S3 memory.
Comparing two architectures
Try three APIs with proxying and storage, then one API with direct permission and the same storage. Parts of 8–10 MB balance speed and buffering. On the direct path, permission lifetime is limited to 300 seconds. Virus scanning, real accounts, upload resumption and cloud traffic cost are outside this model.
Task
100 files/s of 15 MB each. One API transfers 600 MB/s. Build a proxied or direct upload to object storage; buffer must not exceed 1200 MB and direct-upload authorization must last no longer than 300 s.
Files are all 15 MB. With direct authorization, the API only checks access/metadata and supports up to 200 files/s; bytes go directly to object storage. Proxying is limited to 600 MB/s per API. Storage throughput is modeled as 2000×chunk/(chunk+2) MB/s. One chunk is notionally reserved per parallel stream: this is a teaching buffer estimate, not measured RAM.
- Add object storage and choose a direct or proxied path.
- Choose API capacity, part size and, if needed, permission lifetime.
- Check completed files, buffering and cost; compare the other path.