🚀 Handling Large CSV Downloads Gracefully with Queues & Background Jobs

We’ve all been there—hitting that “Download CSV” button and watching the browser hang for what feels like forever. If your app is generating large datasets on the fly over an HTTP request, you’re probably familiar with the frustration: timeouts, memory spikes, slow performance, and a horrible user experience.

At tejaya.tech, we believe there’s a better way. Let’s dive into how you can offload large CSV generation to a background job using queues, and notify users via email when it’s ready. Smooth, efficient, and user-friendly.

🔥 The Problem with Direct Downloads (Synchronous HTTP)

In a typical setup, when a user clicks to download a CSV:

1. The request hits your server.

2. You generate the CSV in-memory.

3. You stream the file back to the browser.

Sounds simple, right? But as data grows, this can get ugly.

❌ Drawbacks:

• Request timeouts – Most web servers and browsers can’t handle requests lasting more than 30–60 seconds.

• Memory bloat – Holding massive datasets in memory is a recipe for disaster, especially if multiple users do it simultaneously.

• Poor UX – Users are stuck staring at a spinner with no idea when it’ll finish—or if it’ll crash.

✅ The Better Way: Background Jobs + Queues

Instead of tying up server resources and the user's browser, offload heavy lifting to a background worker.

Here’s how the flow works:

1. User triggers the CSV download request.

2. Backend enqueues a background job to generate the file (using a queue like Redis + Sidekiq, BullMQ, Celery, etc.).

3. The worker generates the CSV in the background, saves it (e.g., in S3 or local storage).

4. Once done, the system emails the user with a secure link to download the file.

This pattern is used by major apps like Shopify, Stripe, and GitHub for heavy report exports.

🛠️ Implementation Overview

Let’s break it down.

1. Queue the Job

// Example in Node.js using BullMQ
import { Queue } from 'bullmq';

const csvQueue = new Queue('csv-generation');

app.post('/download-report', async (req, res) => {
  const job = await csvQueue.add('generateCSV', { userId: req.user.id });
  res.status(202).send('We’ll email you when it’s ready!');
});

2. Background Worker

import { Worker } from 'bullmq';

const worker = new Worker('csv-generation', async job => {
  const data = await fetchUserData(job.data.userId);
  const filePath = await generateCSV(data);
  await sendEmailWithDownloadLink(job.data.userId, filePath);
});

3. Email Notification

You can use SendGrid, Postmark, SES, etc. to send a secure link:

Subject: Your CSV report is ready!

Hey there,  
Your data export is ready.  
👉 [Download Now](https://yourdomain.com/downloads/file.csv)

This link will expire in 24 hours.

⚡ Benefits of the Background Job Pattern

🧠 Pro Tips

• Use UUIDs or signed URLs for secure download links.

• Expire download links after a certain period.

• Show job status in your UI (e.g., “Generating report…” with polling or websockets).

• Log and monitor queue failures (e.g., Sentry, Prometheus).

• Chunk large data while generating CSVs to avoid memory spikes.

📬 Final Thoughts

Handling large file downloads asynchronously isn’t just about performance—it’s about respecting your user’s time. By embracing queues and background workers, you can provide a smoother, faster, and more reliable experience.

Thinking of implementing this in your product? Let’s nerd out about architecture or share lessons learned—hit reply or drop a note on tejaya.tech!

Let us know if you want to Implement similiar workflow for your application, or tailor this for a specific audience like product teams or backend engineers.