File Upload Pipeline: Direct-to-Cloud S3/GCS Presigned URLs in Node.js

Optimizing backend performance and eliminating server bottlenecks by routing large file uploads directly to cloud storage using presigned URLs in Node.js.

In traditional web application architectures, when users upload large media files—such as high-resolution images, profile avatars, video recordings, or data archives—the files are typically sent through the application server via multipart form-data middleware like Multer. The Node.js server then buffers the incoming stream in memory or temporary disk storage before streaming it onward to cloud storage providers like Amazon S3 or Google Cloud Storage (GCS).

This multi-hop upload pattern creates severe architectural bottlenecks:

• Server Bandwidth Exhaustion: The Node.js single-threaded event loop and network bandwidth are consumed entirely by passing large binary payloads.

• Memory Overload: Storing incoming multipart streams risks hitting container heap limits, causing out-of-memory (OOM) crashes under high concurrency.

• Cost Inefficiencies: You pay for double data transfer—ingress from the user to your backend server, and egress from your backend server to the cloud.

By implementing a **Direct-to-Cloud Presigned URL Pipeline**, your Node.js server acts purely as an authorization gateway. The server generates a cryptographically signed temporary URL with strict expiration windows and permissions, allowing the client browser to upload files directly to AWS S3 or Google Cloud Storage.

How Presigned URLs Work

The direct-to-cloud upload workflow consists of three distinct phases:

1. Request Authorization: The authenticated client requests an upload authorization token and a presigned URL from your Express backend, specifying the target file name, size, and MIME type.

2. Direct Cloud Upload: Your backend validates user permissions, generates an S3 or GCS presigned PUT URL using cloud SDK credentials, and returns it to the client. The client then performs a direct `PUT` request with the binary file data straight to cloud storage.

3. Confirmation and Persistence: Upon successful upload completion, the client notifies your backend to record the file metadata (such as S3 object key or public URL) in your primary PostgreSQL or MongoDB database.

Generating Presigned PUT URLs with AWS SDK v3

To implement presigned uploads with Amazon S3 in Node.js, install the official AWS SDK v3 packages.

              
            
BASH
Installing AWS SDK v3 S3 client and presigner packages.
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

              
            
TypeScript
Express endpoint for generating AWS S3 presigned PUT URLs.
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { Request, Response } from 'express';

const s3Client = new S3Client({
  region: process.env.AWS_REGION || 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID || '',
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || '',
  },
});

export async function generatePresignedUrlHandler(req: Request, res: Response): Promise<void> {
  try {
    const { fileName, fileType } = req.body;

    if (!fileName || !fileType) {
      res.status(400).json({ success: false, message: 'fileName and fileType are required.' });
      return;
    }

    const uniqueKey = `uploads/${Date.now()}-${fileName}`;

    const command = new PutObjectCommand({
      Bucket: process.env.AWS_S3_BUCKET_NAME || 'my-app-bucket',
      Key: uniqueKey,
      ContentType: fileType,
    });

    // Generate presigned URL valid for 5 minutes (300 seconds)
    const presignedUrl = await getSignedUrl(s3Client, command, { expiresIn: 300 });

    res.status(200).json({
      success: true,
      uploadUrl: presignedUrl,
      fileKey: uniqueKey,
    });
  } catch (error: any) {
    res.status(500).json({ success: false, message: error.message });
  }
}

Generating Signed URLs with Google Cloud Storage

If your infrastructure runs on Google Cloud Platform, you can implement an equivalent direct-to-cloud upload pipeline using the `@google-cloud/storage` SDK.

              
            
BASH
Installing Google Cloud Storage SDK.
npm install @google-cloud/storage

              
            
TypeScript
Generating GCS signed PUT URLs in Node.js.
import { Storage } from '@google-cloud/storage';
import { Request, Response } from 'express';

const storage = new Storage({
  keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS,
});

const bucketName = process.env.GCS_BUCKET_NAME || 'my-gcs-bucket';

export async function generateGcsSignedUrlHandler(req: Request, res: Response): Promise<void> {
  try {
    const { fileName, fileType } = req.body;
    const uniqueFileName = `uploads/${Date.now()}-${fileName}`;
    const bucket = storage.bucket(bucketName);
    const file = bucket.file(uniqueFileName);

    const options = {
      version: 'v4' as const,
      action: 'write' as const,
      expires: Date.now() + 5 * 60 * 1000, // 5 minutes
      contentType: fileType,
    };

    const [url] = await file.getSignedUrl(options);

    res.status(200).json({
      success: true,
      uploadUrl: url,
      fileKey: uniqueFileName,
    });
  } catch (error: any) {
    res.status(500).json({ success: false, message: error.message });
  }
}

Uploading Files Directly from React to Cloud Storage

On the React frontend, obtaining a presigned URL and executing a direct `PUT` request bypasses your Node.js backend entirely, ensuring optimal network performance.

              
            
TSX
React component handling direct-to-cloud file uploads.
'use client';

import { useState } from 'react';

export default function FileUploadComponent() {
  const [uploading, setUploading] = useState(false);
  const [progress, setProgress] = useState(0);

  async function handleFileUpload(e: React.ChangeEvent<HTMLInputElement>) {
    const file = e.target.files?.[0];
    if (!file) return;

    setUploading(true);
    try {
      // 1. Request presigned URL from backend
      const res = await fetch('/api/upload/presign', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ fileName: file.name, fileType: file.type }),
      });

      const { uploadUrl, fileKey } = await res.json();

      // 2. Upload file directly to S3/GCS using presigned PUT URL
      await fetch(uploadUrl, {
        method: 'PUT',
        headers: { 'Content-Type': file.type },
        body: file,
      });

      alert(`File uploaded successfully! Key: ${fileKey}`);
    } catch (err) {
      console.error('Upload failed:', err);
    } finally {
      setUploading(false);
    }
  }

  return (
    <div className="p-6 max-w-md mx-auto bg-white rounded-xl shadow">
      <h2 className="text-xl font-bold mb-4">Direct Cloud File Upload</h2>
      <input type="file" onChange={handleFileUpload} disabled={uploading} className="mb-4" />
      {uploading && <p className="text-indigo-600 font-medium">Uploading directly to cloud...</p>}
    </div>
  );
}

Summary

Implementing direct-to-cloud file upload pipelines using presigned URLs in Node.js transforms your backend architecture by eliminating bandwidth bottlenecks, reducing server memory overhead, and cutting cloud infrastructure egress costs.

By offloading binary payloads straight to Amazon S3 or Google Cloud Storage while keeping your Node.js server strictly focused on authorization and metadata persistence, engineering teams can build highly scalable, resilient applications.