Bulk email for developers: batches, broadcasts, and queues
Reviewed
Bulk email is fan-out: one application job produces messages for many recipients. For developers, the important choice is who owns audience selection, scheduling, and recovery. Use a personalized batch when your application owns individual messages, a broadcast when you need a subscriber campaign, and a durable queue whenever work must survive a restart. This guide helps you choose that boundary before writing the send loop.
Separate purpose, orchestration, and transport
These are three independent decisions. Purpose describes why someone receives a message: a requested account notification, an opted-in newsletter, or outreach. Orchestration describes how work starts and stops: an event, a campaign, or a sequence. Transport is the API or SMTP service that submits mail. Calling a promotional message from code doesn’t make it transactional.
MailKite, which we build, supports personalized batches and broadcasts, but its terms prohibit unsolicited bulk or commercial mail, sending without consent, and sending after opt-out. A batch endpoint is not permission to use an imported prospect list. For outreach architecture and platform evaluation, read the cold email guide.
Choose the smallest system that owns the whole job
Work through these questions in order:
- Does each recipient need application-specific content? A usage report or invoice notice usually belongs in your application’s recipient ledger. Group submissions into batches, while preserving one record per message.
- Is this an edition sent to subscribers? Use a campaign system that owns subscriptions, drafts, unsubscribe handling, and progress. Start with programmatic newsletters.
- Does a later event change whether you should send? A completed setup or paid invoice should stop reminders. Use programmatic email orchestration rather than scheduling every follow-up in advance.
- Who resumes unfinished work? If the answer is “someone reruns the script,” add persistent jobs and checkpoints before increasing volume.
The table below is an architecture decision aid, not a provider feature matrix.
| Job | Practical starting point | State you must preserve |
|---|---|---|
| Personalized product notifications | Application queue and batch submission | Recipient, content version, attempt, provider ID |
| Monthly subscriber edition | Campaign manager and managed transport | Subscription evidence, audience, edition, opt-outs |
| Reminder that stops after completion | Event-driven sequence or durable workflow | Enrollment, business object, exit event |
| Long-running one-off import | Checkpointed worker with bounded concurrency | Import identity, cursor, recipient outcomes |
Batch size is not a sending rate
MailKite’s Send API accepts 1–50 recipients per personalized batch, returns results in request order, and can partially succeed. Each entry produces its own message to exactly one address. That limit reduces request overhead; it doesn’t promise 50 deliveries per second or tell you how many requests to run concurrently.
Size chunks to the endpoint limit. Set concurrency separately. Then apply the account’s actual volume and rate constraints. For example, SES quotas are regional and count recipients; sandbox accounts have a 200-recipient daily allowance and a one-recipient-per-second sending rate. A large legal request can still exceed the available sending budget.
For implementation, choose JavaScript, Python, or PHP. For transport selection, use the bulk email API comparison.
Build a recipient ledger before the worker
Give each intended message a stable application identity, such as job plus recipient plus edition. Store the normalized destination separately from its display name. Record eligibility, the content version, current state, attempt time, and any provider message or scheduled-send ID. Enforce uniqueness in the database so two workers cannot independently create the same intended delivery.
Use explicit states: pending, submitting, accepted, retryable failure, permanent failure, suppressed, and unknown. “Unknown” matters. A network timeout can happen after the provider accepted the message but before your worker received the response. Treating that timeout as definite rejection creates duplicates.
On MailKite, inspect every result rather than treating HTTP 200 as universal success. Persist successful results before releasing the job. Retry only eligible failed recipients. The sending reference permits custom MIME headers, but a header named like a deduplication key is not a documented server-side exactly-once guarantee.
The failures worth designing for
Consider a hypothetical three-recipient job: one accepted message, one temporary rejection, and one opted-out address. A whole-job retry resends the first message and ignores the third recipient’s eligibility. Recovery should select the temporary failure alone, check eligibility again, and leave an audit trail explaining why the other records were excluded.
Other common failures are less visible:
- Recipient leakage: multiple addresses on one visible To line expose the audience. Verify that your provider’s personalization mode creates separate envelopes.
- Checkpoint gaps: advancing an import cursor before storing outcomes loses work after a crash.
- Stale eligibility: an audience selected yesterday may include someone who unsubscribed today. Recheck before submission.
- Retry storms: workers waking together after throttling amplify the same failure. Use bounded retries, backoff, jitter, and shared rate control.
- Misleading progress: “accepted” and “delivered” are different stages. Neither proves inbox placement.
Close the feedback loop
Persist bounce and complaint events, suppress affected destinations according to the provider contract, and connect replies to the originating job. Authenticate webhook requests using the provider’s documented mechanism. Store events durably before acknowledging them, then process duplicate deliveries idempotently.
A useful operations view shows pending work, unknown outcomes, retryable failures, suppressions, and oldest queue age. Include cancellation as a first-class job state. Cancellation can prevent future submissions; it cannot recall messages already accepted by a transport.
Before a real campaign, exercise a local fixture containing duplicate rows, missing personalization, one opt-out, a partial failure, and an ambiguous timeout. Restart the worker during recovery and inspect its ledger. The result you want is explainable recipient state, not a console line saying “batch complete.” Once that boundary is dependable, choose your language guide and implement the smallest production job that uses it.