Next.js — job queue

A background job queue with zero external infrastructure: POST enqueues work, the pool drains it on worker threads, and a progress endpoint reads counters straight out of shared memory — no Redis, no database polling, no postMessage per status update.

The shape

Three moving parts, all inside the Next.js server bundle: a shared counter contract, a pooled process task, and a route handler that writes queued before fire-and-forget dispatch. Because the queue state is shared memory, the GET handler is a pure read — it never touches a worker.

api/jobs/jobs.contract.tsbackend

// src/app/api/jobs/jobs.contract.ts — counters all routes share
import { z } from 'zod';
import { defineSharedMemory, field, type TaskContract } from '@atolljs/core';

export const jobsMemory = defineSharedMemory({
  queued: field.number(),
  completed: field.number(),
  failed: field.number(),
  lastMs: field.number(),
});

export const ProcessJob: TaskContract<[id: number, workMs?: number], JobResult> = {
  taskId: 'jobs.process',
  resultSchema: jobResult,
};

api/jobs/route.tsbackend

// src/app/api/jobs/route.ts — enqueue on the API thread, drain in workers
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';

let nextId = 0;

// POST /api/jobs {"count": N, "workMs"?: number} — returns immediately;
// the pool queues dispatches internally when saturated.
export async function POST(req: Request) {
  const { count = 1, workMs = 50 } = await req.json().catch(() => ({}));
  const { pool, memory } = getJobs();
  for (let i = 0; i < Math.min(1000, count); i++) {
    memory.queued.write(memory.queued.read() + 1);
    void pool.process(nextId++, workMs)
      .catch(() => memory.failed.write(memory.failed.read() + 1));
  }
  return NextResponse.json({ accepted: count, queued: memory.queued.read() });
}

// GET /api/jobs — progress read straight off the buffer. No dispatch.
export async function GET() {
  const { memory } = getJobs();
  const queued = memory.queued.read();
  const completed = memory.completed.read();
  return NextResponse.json({ queued, completed, failed: memory.failed.read(),
    inFlight: queued - completed - memory.failed.read(),
    lastMs: memory.lastMs.read() });
}

api/jobs/jobs.worker.tsbackend

// src/app/api/jobs/jobs.worker.ts — each completion is a memory write
import '@atolljs/node/shim';
import '@atolljs/core/worker/workerBootstrap';
import { TaskRegistry } from '@atolljs/core';
import { jobsMemory, ProcessJob } from './jobs.contract';

TaskRegistry.register(ProcessJob, (id, workMs = 50) => {
  const t0 = performance.now();
  let acc = 0;
  while (performance.now() - t0 < workMs) acc += Math.sqrt(acc);
  const completed = jobsMemory.completed.read() + 1;
  jobsMemory.completed.write(completed);
  jobsMemory.lastMs.write(performance.now() - t0);
  return { id, ms: performance.now() - t0, completed };
});

Why not just await

Awaiting a batch in POST couples response latency to total work time and invites platform timeouts. Writing the counter and releasing the dispatch means the handler answers in microseconds while workers drain at their own pace — the client polls GET /api/jobs (or binds the same fields with useSharedValue in a client component) for live progress.

Where it runs

Same contract as every server-side use case: export const runtime = 'nodejs', a globalThis-held pool (see Next.js server), and a long-lived process. One subtlety the example demonstrates: Next may evaluate a contract module in more than one bundle graph (instrumentation's import graph vs the route's), so handlers read memory through getJobs().memory — the instance the pool actually bound — instead of a second jobsMemory import that could be unbound on this thread. On serverless the pool — and the queue state — dies with the invocation, so this pattern targets self-hosted next start or standalone output, not per-request functions.

Runnable source: examples/nextjs/src/app/api/jobs/, tested by test/apiJobs.test.ts against a real worker_threads pool.