Child Processes and Worker Threads

Go beyond a single thread: run other programs with execFile and spawn, move CPU-heavy work to worker threads so the event loop stays responsive, share work across CPU cores with a pool or the cluster module, and pass messages safely.

Lesson 23 of 43Advanced17 min read

Node.js runs your JavaScript on a single thread, which is perfect for I/O such as files, databases and network calls: while one request waits, others are served. But some work isn't waiting, it's computing: resizing images, generating a large PDF, hashing thousands of passwords, parsing a huge file. While that runs, the event loop is blocked, and every other request on the server has to wait.

Node offers two ways to escape the single thread. Child processes run a separate program, which might be another Node script or any command-line tool such as ffmpeg, git or ImageMagick. Worker threads run JavaScript in parallel threads inside the same process, ideal for CPU-heavy calculations in your own code. For using every CPU core to serve web traffic, the cluster module, or simply running several instances behind a load balancer, does the job.

In this lesson you'll run commands safely and stream their output, move a slow calculation into a worker thread and see the main thread stay responsive, build a small worker pool, and learn which tool fits which problem.

Running other programs

JavaScript
import { execFile, spawn } from 'node:child_process';
import { promisify } from 'node:util';
const execFileAsync = promisify(execFile);

const { stdout } = await execFileAsync('git', ['log', '-1', '--format=%s']);

const child = spawn('ffmpeg', ['-i', 'in.mov', 'out.mp4']);
child.stderr.on('data', (chunk) => process.stdout.write(chunk));
child.on('close', (code) => console.log(`Exited with code ${code}`));
  • execFile runs a program and collects its output when it finishes; good for short commands with small output.
  • spawn streams output as it's produced; good for long-running or chatty programs.
  • Pass arguments as an array. Avoid exec with user input: it runs through a shell, where characters like ; and && can inject extra commands.
  • fork starts another Node script with a built-in message channel between parent and child.

Worker threads

JavaScript
// main.js
import { Worker } from 'node:worker_threads';

const worker = new Worker(new URL('./primes.js', import.meta.url), { workerData: { limit: 5_000_000 } });
worker.on('message', (count) => console.log(`Found ${count} primes`));
worker.on('error', (err) => console.error(err));
JavaScript
// primes.js
import { parentPort, workerData } from 'node:worker_threads';
parentPort.postMessage(countPrimes(workerData.limit));
  • Each worker has its own event loop and memory; data is copied between threads with postMessage.
  • Large binary data can be transferred or shared with SharedArrayBuffer to avoid copying.
  • Creating a worker takes time and memory, so long-running servers keep a pool of workers and reuse them. Libraries such as Piscina do this for you.

Cluster and scaling across cores

A single Node process uses one CPU core for JavaScript. To use all cores for web traffic, run several processes. The built-in cluster module forks workers that share one port; in production it's more common to let the platform handle it, such as several containers behind a load balancer, PM2's cluster mode, or a platform's instance count. Each process has its own memory, so shared state such as sessions and caches must live in Redis or the database.

Node.js Advanced Syntax

Processes and workers

import { execFile, spawn, fork } from 'node:child_process';
import { Worker, isMainThread, parentPort, workerData } from 'node:worker_threads';
const { stdout } = await promisify(execFile)('node', ['--version']);
const p = spawn('npm', ['run', 'build']);   p.on('close', (code) => {});
const w = new Worker(new URL('./job.js', import.meta.url), { workerData: input });
w.on('message', handle);   w.postMessage({ type: 'start' });
parentPort.postMessage(result);           // inside the worker
import cluster from 'node:cluster';   if (cluster.isPrimary) cluster.fork();

Node.js Advanced Summary

  • CPU-heavy JavaScript blocks the event loop and delays every other request.
  • execFile runs a command and collects output; spawn streams output; fork runs Node scripts.
  • Pass command arguments as arrays and avoid exec with user input.
  • Worker threads run JavaScript in parallel; communicate with workerData and postMessage.
  • Use cluster or multiple instances to serve traffic on every CPU core.

Node.js Advanced Examples

Each example shows the code, the output it produces and what each part does.

Example 1

Running a command with execFile and spawn

Node.js
import { execFile, spawn } from 'node:child_process';
import { promisify } from 'node:util';

const run = promisify(execFile);

const { stdout } = await run(process.execPath, ['-e', 'console.log(6 * 7)']);
console.log(`Child said: ${stdout.trim()}`);

try {
  await run(process.execPath, ['-e', 'process.exit(3)']);
} catch (err) {
  console.log(`Failed with exit code ${err.code}`);
}

const child = spawn(process.execPath, ['-e', "for (let i = 1; i <= 3; i++) console.log('step', i)"]);
let lines = 0;
child.stdout.on('data', (chunk) => { lines += chunk.toString().trim().split('\n').length; });
child.on('close', (code) => console.log(`spawn finished: code ${code}, ${lines} lines`));
Output
Child said: 42
Failed with exit code 3
spawn finished: code 0, 3 lines
Line-by-line explanation
  1. Line 6process.execPath is the path to Node itself; any program could be run the same way, with arguments passed as an array, never through a shell.
  2. Lines 9 to 12A non-zero exit code makes the promise reject, with the code on the error.
  3. Lines 15 to 17spawn streams output as it arrives and reports the exit code on close.
Example 2

Keeping the event loop responsive with a worker

Node.js
// count-primes.mjs
import { parentPort, workerData } from 'node:worker_threads';

function countPrimes(limit) {
  let count = 0;
  for (let n = 2; n <= limit; n++) {
    let prime = true;
    for (let d = 2; d * d <= n; d++) if (n % d === 0) { prime = false; break; }
    if (prime) count++;
  }
  return count;
}

parentPort.postMessage(countPrimes(workerData.limit));
Node.js
// main.mjs
import { Worker } from 'node:worker_threads';

let ticks = 0;
const timer = setInterval(() => ticks++, 10);

const worker = new Worker(new URL('./count-primes.mjs', import.meta.url), { workerData: { limit: 2_000_000 } });
worker.on('message', (count) => {
  clearInterval(timer);
  console.log(`Primes up to 2,000,000: ${count}`);
  console.log(`Main thread stayed responsive: ${ticks > 5}`);
});
worker.on('error', (err) => console.error(err));
Output
Primes up to 2,000,000: 148933
Main thread stayed responsive: true
Line-by-line explanation
  1. Line 2The worker reads its input from workerData.
  2. Lines 4 to 12A deliberately CPU-heavy calculation that would block a server for a noticeable time.
  3. Line 14postMessage sends the result back to the main thread. Because the work runs in a worker, the main thread's timer keeps ticking the whole time, just as a server would keep answering requests.
Example 3

A small worker pool

Node.js
// hash-worker.mjs
import { parentPort } from 'node:worker_threads';
import { scryptSync } from 'node:crypto';

parentPort.on('message', ({ id, password }) => {
  const hash = scryptSync(password, 'demo-salt', 32).toString('hex');
  parentPort.postMessage({ id, length: hash.length });
});
Node.js
// pool.mjs
import { Worker } from 'node:worker_threads';
import { availableParallelism } from 'node:os';

const size = Math.min(4, availableParallelism());
const workers = Array.from({ length: size }, () => new Worker(new URL('./hash-worker.mjs', import.meta.url)));
const pending = new Map();
workers.forEach((w) => w.on('message', ({ id, length }) => { pending.get(id)(length); pending.delete(id); }));

let next = 0;
function hash(password) {
  const id = next++;
  const worker = workers[id % workers.length];
  return new Promise((resolve) => {
    pending.set(id, resolve);
    worker.postMessage({ id, password });
  });
}

const passwords = Array.from({ length: 12 }, (_, i) => `password-${i}`);
const lengths = await Promise.all(passwords.map(hash));
console.log(`Hashed ${lengths.length} passwords, each ${lengths[0]} hex characters`);
console.log(`Pool size between 1 and 4: ${size >= 1 && size <= 4}`);
await Promise.all(workers.map((w) => w.terminate()));
Output
Hashed 12 passwords, each 64 hex characters
Pool size between 1 and 4: true
Line-by-line explanation
  1. Line 5Each worker keeps running and handles many messages, so the startup cost is paid once.
  2. Line 6scryptSync is slow on purpose; in a worker it doesn't block the main thread.
  3. Line 7The id lets the main thread match each answer to the right request.

Common Node.js Advanced Mistakes

Heavy computation on the main thread

Don't
app.get('/report', (req, res) => {
  res.json(buildHugeReport());      // every other request waits until this finishes
});
Do
app.get('/report', async (req, res) => {
  res.json(await runInWorker('build-report.js', req.query));
});

exec with user input

Don't
exec(`zip archive.zip ${req.query.file}`);         // "a.txt; rm -rf ~" runs a second command
Do
execFile('zip', ['archive.zip', safeName(req.query.file)]);

A new worker for every request

Don't
app.post('/hash', (req, res) => {
  const w = new Worker('./hash.js', { workerData: req.body });   // slow to start, memory grows under load
});
Do
const pool = new Piscina({ filename: new URL('./hash.js', import.meta.url).href });
app.post('/hash', async (req, res) => res.json(await pool.run(req.body)));

Node.js Advanced in Real Projects

Processes and workers in production

Build tools such as esbuild and Vite spread work across worker threads. Image services use sharp (which uses native threads) or worker pools for resizing, and video platforms spawn ffmpeg. CI tools and deployment scripts run git and docker through spawn. Web APIs scale across cores by running several processes in containers, and long jobs such as reports and exports usually move to a background queue like BullMQ processed by separate worker services.

Node.js Advanced FAQ

What is the difference between worker threads and child processes?

Worker threads run JavaScript inside the same Node process, start faster, and can pass messages cheaply or share memory with SharedArrayBuffer. A child process is a completely separate program, which can be any executable, such as git or ffmpeg, and is fully isolated, so a crash there doesn't take down your app. Use workers for CPU-heavy JavaScript and child processes for running other programs.

How many worker threads should I create?

For CPU-bound work, about one worker per available core is a sensible maximum; os.availableParallelism() tells you how many that is. Many apps leave one core for the main thread, which still handles requests. Creating more workers than cores doesn't speed anything up, because they just take turns on the same processors.

How do I stop a child process that runs too long?

Pass a timeout in milliseconds to execFile or exec: when it runs out, Node kills the child with SIGTERM and the call fails with err.killed set to true. You can also pass signal from an AbortController and call abort() whenever you like, or call child.kill() on a process started with spawn.

Will worker threads make my database queries or API calls faster?

No. Waiting on I/O doesn't use the JavaScript thread, so Node can already run thousands of queries and requests at the same time without workers. Workers help only when your own JavaScript does heavy computation, such as resizing images, parsing huge files or hashing large amounts of data.

Why does spawning npm or a .cmd file fail on Windows?

Commands like npm and npx are .cmd batch files on Windows, not executables, and recent Node versions refuse to run .bat or .cmd files through spawn or execFile unless you pass shell: true, as a protection against command injection. If you do use shell: true, never include user input in the arguments.

Node.js Advanced Exercises

Answer each one and check it instantly. If you get stuck, open the hint before the answer.

Fill in the blankseasy

Complete the line that sends a result from a worker back to the main thread.

parentPort.(result);
Multiple choiceeasy

What happens to other requests while a Node server runs a long CPU-heavy loop on the main thread?

Multiple choicemedium

Which function streams a child process's output as it's produced?

Multiple choicemedium

How does data get into a worker thread when it starts?

Multiple choicehard

Would worker threads speed up an API whose time is spent waiting on database queries?

Node.js Advanced Quiz

5 questions to check what you learned.

  1. Which module provides Worker and parentPort?

    • node:child_process
    • node:worker_threads
    • node:cluster
    • node:os

  2. Why pass command arguments as an array to execFile?

    • They aren't interpreted by a shell, which prevents command injection
    • It's faster
    • Arrays are required by Windows
    • To run in parallel

  3. What does the cluster module do?

    • Groups database rows
    • Creates worker threads
    • Compresses responses
    • Starts several Node processes that share a server port to use more CPU cores

  4. Why use a worker pool rather than a new worker per task?

    • Pools are required
    • Pools share variables
    • Starting workers is slow and uses memory; a pool reuses a fixed set
    • Workers can only run once

  5. Is data passed with postMessage shared or copied between threads by default?

    • Copied
    • Shared
    • Deleted
    • Encrypted

Related Node.js topics

08Async PatternsMaster asynchronous code in Node.js: error-first callbacks, promises, async and await, running tasks in parallel with Promise.all and allSettled, limiting concurrency, timeouts with AbortSignal, and converting callback APIs with promisify.24Debugging & PerformanceFind and fix problems in Node.js apps: debug with breakpoints using --inspect, log in a structured way, measure timings with performance hooks, spot event-loop blocking, detect memory leaks, profile CPU use, and speed up slow endpoints with caching.25DeploymentTake a Node.js app to production: prepare configuration and scripts, run with NODE_ENV=production, add health checks and graceful shutdown, package it with Docker, deploy to a hosting platform, automate releases with CI/CD, and monitor it once it's live.
Esc
↑↓ to navigateEnter to open