S3 - Streaming Files

Streaming Streaming Moving data in small chunks as it arrives instead of loading the whole thing into memory first. Memory stays flat no matter how big the file is. moves a file in chunks instead of loading the whole thing into memory at once. The bytes flow straight from storage to the client as they arrive. It's drinking through a straw instead of swallowing the whole bottle. You take it a sip at a time, no matter how big the bottle is.

Streaming isn't an S3 feature. It's a general idea that shows up all over: HTTP request and response bodies, reading a file off disk, piping between processes, and network sockets. S3 just happens to hand you a stream.

Why stream

• A 2 GB video loaded into memory might crash your server; streamed, it barely registers
• Users see the first bytes of a video or PDF immediately, not after a full download
• Ten concurrent downloads cost the same memory as one

How to use it

Say you're serving a video from S3 over HTTP. Buffering Buffering Loading an entire file into memory before doing anything with it. Simple, but a big file can exhaust your server's memory. reads the whole object into memory before sending a byte. For a 2 GB file that's 2 GB of RAM held at once:

import { GetObjectCommand } from "@aws-sdk/client-s3";

app.get("/videos/:id", async (req, res) => {
  const object = await s3.send(new GetObjectCommand({
    Bucket: "my-app-uploads",
    Key: `videos/${req.params.id}.mp4`,
  }));

  // Collect every chunk first: the whole video sits in memory at once.
  const chunks = [];
  for await (const chunk of object.Body) {
    chunks.push(chunk);
  }
  res.setHeader("Content-Type", "video/mp4");
  res.end(Buffer.concat(chunks));
});

pipeline connects the readable stream to a writable one and handles backpressure Backpressure A stream's way of slowing the sender when the receiver can't keep up, so data doesn't pile up in memory. , errors, and cleanup for you. Here it pipes the object straight to the HTTP response, so memory stays flat no matter how big the file is:

import { GetObjectCommand } from "@aws-sdk/client-s3";
import { pipeline } from "node:stream/promises";

app.get("/videos/:id", async (req, res) => {
  const object = await s3.send(new GetObjectCommand({
    Bucket: "my-app-uploads",
    Key: `videos/${req.params.id}.mp4`,
  }));

  res.setHeader("Content-Type", "video/mp4");

  // Body is a readable stream; pipe it straight to the HTTP response.
  // Chunks flow through one at a time and are never all held in memory at once.
  await pipeline(object.Body, res);
});

When to stream (and when not to)

Streaming should be your default: stream unless you have a concrete reason not to. If you find yourself reading a whole file into a variable before sending it, that's the signal to stream instead. The exceptions are narrow:

• The file is tiny and you need it all in memory anyway (parsing a small JSON or CSV, transforming the whole thing).
• You need random access, jumping around the file rather than reading start to finish.
• A library or API you're handing the data to only accepts a full buffer.

Exercise

Here’s a download that works but buffers the whole object in memory before writing it to disk. Refactor it to stream the file straight through, so memory stays flat no matter how big the object is.