Notification — How it works

Overview

The notification engine manages reusable message templates and the events/deliveries that send them. An event is rendered from a template, dispatched through a pluggable provider, and every attempt is recorded append-only — so provider failures and retries never lose history. Recipient, body, and payload may be PHI and are never logged.

Data model & ownership

Table	Purpose
notification_template	Reusable template; relational channel (EMAIL/SMS/PUSH/IN_APP/WEBHOOK), code unique per tenant, {{variable}} subject/body + required_variables.
notification_subscription	Who has opted out of what (M10-009). Absence means opted in.
notification_event	An enqueued message; relational status (PENDING/SENDING/SENT/FAILED/UNDELIVERABLE/CANCELLED/SUPPRESSED), rendered subject/body, payload, plus claimed_at/next_attempt_at for the dispatcher.
notification_delivery	Append-only per-attempt outcome; relational status (SENT/FAILED), provider + non-sensitive response snapshot.
critical_result	Shared closed-loop acknowledgement row (M10B-009A). Source keys are generic (source_module + source_record_id); imaging and lab raise through Feign.

Key rules & invariants

• Template validation rejects an EMAIL template without a subject and any declared required variable not referenced in the subject/body; voided templates cannot be used for new deliveries.
• Enqueue renders the subject/body from the template ({{variable}} substitution) and rejects a missing required variable / invalid payload JSON.
• Send dispatches via the NotificationProvider SPI (a SimulatedNotificationProvider ships by default) and records a new notification_delivery per attempt; a provider failure is captured as a FAILED attempt (not thrown) and a retry appends another attempt — history is never lost.
• No PHI/secrets are logged — only ids and statuses. This matters most on the dispatcher's failure path: a password-reset body carries a live redeem token, so a dispatcher that logged event bodies on failure would undo SEC-010 by itself.
• Enqueued events are delivered automatically by the dispatcher, which runs only on a pod with the scheduler profile.

Content protection at rest (M33-010)

recipient, subject, body and payload_json hold rendered patient-facing content: a name, an appointment time, a clinic, a phone number or an email address. They are encrypted at rest with the platform PiiProtector (pii:v1: envelopes), and recipient additionally carries a blind index in recipient_hash.

Everything goes through NotificationEventProtector, which is the only place these fields are encrypted or decrypted.

• Encrypt at enqueue, after the opt-out check. The opt-out reads person_id and channel, not the address, so it is unaffected.
• Decrypt at the provider boundary, and nowhere earlier. Revealing at fetch would hold every patient's address and rendered body in memory for the whole dispatch batch. A retry re-reads the encrypted row rather than re-sending a cached plaintext.
• Decrypt onto a detached copy, never the managed entity. Decrypting an attached entity in place lets a dirty-checking flush write the plaintext back over the ciphertext.

⚠ recipient is both PHI and a lookup key, which is why it is encrypted and blind-indexed. AES-GCM uses a random IV, so two encryptions of the same address never compare equal; a bare encrypt would silently break M10-009's SUPPRESSED opt-out matching and delivery de-duplication, and an opted-out patient would simply start receiving messages again. The channel picks the normaliser, so +234 801 234 5678 and +2348012345678 hash alike.

⚠ channel, status and enqueued_at are never encrypted. idx_notification_event_status_channel drives the dispatcher's queue scan; encrypting any of them turns it into a sequential scan of every notification ever queued.

⚠⚠ Turning pii.encryption.enabled off does not decrypt existing rows. PiiProtector.reveal is a passthrough when the flag is off, so an already-encrypted event would otherwise be delivered with the literal text pii:v1:… as the patient's SMS or email. The protector refuses to dispatch such an event instead: the dispatcher catches per event, logs, and releases the claim, so the batch continues. Re-enable encryption with the key that wrote the row rather than sending ciphertext to a patient.

⚠⚠ Recipient search changes shape once encryption is on, and this is a real loss of behaviour. With plaintext it was a case-insensitive substring match, so an administrator could search doe@zhenus and find JohnDoe@zhenus.com. A blind index can only answer exact equality, so with encryption enabled the search matches the whole normalised address and nothing else. It is still the right trade: LIKE against a pii:v1: envelope would match the ciphertext and return nothing for every real address anyone typed, and a search box that silently answers "no results" about events that exist is worse than one that cannot do substrings. The term is normalised the same way the stored hash was, so +234 801 234 5678 and +2348012345678 still find each other.

Rows written before this ticket stay plaintext and pass through unharmed; NotificationProtectionBackfillRunner rewrites them in batches and skips anything already enveloped.

API

See the API Reference. Endpoint groups under /api/v1/notification: templates, and events (enqueue/get/search) + /{id}/send, /{id}/deliveries.

Delivery providers and routing (M10-013, NOTIF-002)

Five real adapters sit behind the NotificationProvider SPI, alongside the built-in simulator:

Provider name	Channel	Notes
aws-ses	EMAIL	Needs a region and a verified sender; SES refuses a send from an unverified address.
aws-sns	SMS	Publishes as Transactional, not Promotional — SNS deprioritises Promotional traffic and some carriers drop it, which for a result notification is the difference between delivered and lost.
africas-talking	SMS	Real coverage across much of sub-Saharan Africa, where SNS reach is patchy.
resend	EMAIL	Bearer-token JSON API. Needs a key and a sender address on a domain verified with Resend.
mailgun	EMAIL	Form-encoded API, HTTP Basic with the literal username api. ⚠ baseUrl selects the region that holds the message — api.mailgun.net (US) vs api.eu.mailgun.net (EU); a domain created in one does not resolve in the other, and the mismatch returns a 401 that reads like a bad key.
simulated	any	The fallback; accepts anything except a blank recipient or one starting with fail.

An adapter with no credentials reports itself unconfigured rather than accepting sends and failing them, so a deployment that has not set AWS up simply routes elsewhere.

⚠ A sender set to a bare domain counts as unconfigured, deliberately. resend and mailgun both require an @ in the sender address. A domain-only value (zhenus.com rather than noreply@zhenus.com) is rejected by the provider with a 422 on every send, so an adapter that accepted it would look configured and fail one message at a time. Reporting unconfigured lets the router resolve elsewhere instead. Found live on 2026-08-30, where SMTP_EMAIL held the verified domain rather than an address.

How a provider is chosen

NotificationProviderRouter resolves, each step falling through to the next:

1. the sending tenant's preference for the channel — a facility-scoped row beats the tenant-wide one;
2. the platform default for the channel (notification.providers.defaults.*);
3. any configured provider that supports the channel;
4. the simulator.

…with notification.providers.allowed applied to all four, see below.

Every step is a fall-through, never a failure. A preference naming an undeployed or unconfigured provider resolves to the next option rather than refusing the send — a typo in routing must not be why a clinical notification goes undelivered. The chosen provider is stamped on the delivery attempt, so a fall-through is visible afterwards rather than silent. Preferences are validated on write, so the administrator finds out about the typo while looking at the screen.

The residency allow-list (NOTIF-002)

notification.providers.allowed (NOTIFICATION_ALLOWED_PROVIDERS) is a comma-separated list of the provider names a deployment may use. Empty means no restriction, which is what every deployment before NOTIF-002 had.

It exists because the platform is deployed where health data may not leave the host country, and naming a platform default is not a residency guarantee. The fall-through above is deliberate, and it leaves two routes to a provider the deployment never intended, neither visible until after a message has been sent:

1. a tenant/facility preference row outranks the platform default outright (step 1 beats step 2), so a tenant administrator can route around the operator's configuration;
2. with no preference and no default matched, step 3 selects any adapter reporting itself configured — a stray AWS_REGION in a shared environment file is enough to send through SES.

So the allow-list is not a fifth step but a filter over all four, including the .../available?channel= picker, so an administrator is never offered a provider the deployment may not use.

⚠ The simulator is not exempt. It reports every send as a success, so leaving it reachable as an implicit last resort would mark clinical notifications SENT that were never sent — strictly worse than not sending. A deployment that wants it lists simulated explicitly. When the allow-list leaves nothing usable, resolve() returns empty and the send fails loudly, which is the correct outcome under a residency rule.

Routing is not a way to stop a send: an inactive preference falls through to the default. Whether a person is contacted at all is notification_subscription (M10-009), which is the recipient's decision.

The scope used to route comes from the event, not the current session — an event enqueued now is sent later, by a different caller than the one that enqueued it, and must follow the tenant it belongs to rather than whoever happens to trigger the send.

That rule is what lets the dispatcher below work at all: it runs on a timer thread where there is no session, so anything that reached for one would throw.

APIs: /api/v1/notification/notification-provider-preferences (CRUD, notification.provider.read / .write), plus .../available?channel=SMS listing the providers that can actually deliver here.

The dispatcher (NOTIF-001)

Enqueued events are delivered by an in-process scheduler owned by this engine — no external trigger, no separate job service. One pass claims a batch of due events and sends each through the same sendEvent path the admin endpoint uses.

:::warning Jobs only run on a pod with the scheduler Spring profile The same jar is deployed twice: the API Deployment runs without the profile and serves requests; a separate Deployment runs with it and runs jobs. Both the @EnableScheduling configuration and the job bean are @Profile("scheduler"), so an API pod has no timer at all — a job can never take threads or heap from work clinicians are waiting on. notification.dispatch.enabled=true on its own changes nothing on an API pod. If you enabled delivery and nothing happened, this is why. :::

The claim — why events are not sent twice

Every replica of the scheduler deployment runs the same timer, so N pods select the same PENDING rows within milliseconds. Two mechanisms combine, and neither is sufficient alone:

1. SELECT … FOR UPDATE SKIP LOCKED settles the race during the claim — concurrent claim transactions step over each other's rows rather than reading the same ones or blocking;
2. a PENDING → SENDING transition committed before any provider is called settles it afterwards, once the row locks are gone.

Sending happens outside the claim transaction on purpose. Holding row locks across a network call to SES would tie a database connection to a third party's latency, so one slow gateway would exhaust the pool and take the clinical API down with it.

The profile does not make the claim redundant. It decides where jobs run; the claim decides what happens when more than one runs anyway — true on every rolling restart, when outgoing and incoming pods overlap, and true the moment the scheduler deployment is scaled past one replica.

A pod that dies mid-dispatch leaves an event in SENDING. The claim re-takes such rows once claimed_at is older than notification.dispatch.claim-timeout, which makes delivery at-least-once: a stranded notification is unrecoverable without it, a duplicate one is merely unfortunate.

Retry, backoff, and the poison ceiling

A failed attempt sets next_attempt_at to backoff-base × 2^(attempt-1), capped at backoff-max, so a provider that is down is not retried once per poll by every replica. Each try still appends its own notification_delivery row — the history is append-only and is also where the attempt count comes from; there is deliberately no counter column on the event to drift out of step with it.

The attempt that reaches max-attempts leaves the event UNDELIVERABLE, which is terminal and which the claim never selects. Without that, one permanently-failing event (a malformed address, a blocked recipient) is re-selected by every poll forever and a batch full of them starves real notifications. An administrator can search for UNDELIVERABLE and read the whole attempt history behind it.

Adding another scheduled job

@EnableScheduling is context-wide and cannot be scoped, so a @Scheduled method written anywhere in core starts running on every scheduler pod the moment it exists — and a one-line annotation does not read like a deployment change. ScheduledJobConventionArchitectureTest therefore fails the build unless the declaring class is on its allow-list and carries @Profile("scheduler"). Before adding an entry, answer: does the job claim its work so two replicas cannot both do it, and does it bound how much it does per pass?

Configuration & feature flags

Domain-module Feign URL notification.service.url (NOTIFICATION_SERVICE_URL).

Provider configuration is env-driven under notification.providers.*, all empty by default so an unconfigured deployment keeps the simulator:

• notification.providers.defaults.EMAIL / .SMS — the platform default provider name per channel.
• notification.providers.aws.* — region, senderEmail, smsSenderId, and optionally accessKeyId/secretAccessKey. Leave the keys blank in production: the default AWS credential chain then uses the pod/instance role instead of long-lived keys in configuration.
• notification.providers.africas-talking.* — username, apiKey, senderId, baseUrl.
• notification.providers.resend.* — apiKey, senderEmail, baseUrl. RESEND_API_TOKEN is honoured as a fallback for RESEND_API_KEY. ⚠ senderEmail must be a full address, not the verified domain.
• notification.providers.mailgun.* — apiKey, domain, senderEmail, baseUrl. ⚠ baseUrl is the region setting, not a latency setting.
• notification.providers.allowed (NOTIFICATION_ALLOWED_PROVIDERS) — the residency allow-list. ⚠ Empty means no restriction, so a deployment under a data residency obligation must set it explicitly; see The residency allow-list.

Dispatch tuning lives under notification.dispatch.* (all env-driven, sensible defaults in code) and only takes effect on a pod running the scheduler profile:

Key	Default	What it does
enabled	true	Delivery on by default — a pipeline that has to be switched on is off in the deployment nobody checked.
poll-delay	PT30S	Fixed delay between passes, so a slow pass never overlaps itself.
initial-delay	PT60S	Lets the context finish booting before competing for connections.
batch-size	50	Events claimed per pass; bounds what one replica takes on.
max-attempts	5	Poison ceiling — the attempt that reaches it abandons the event as UNDELIVERABLE.
backoff-base / backoff-max	PT1M / PT1H	Exponential backoff and its cap.
claim-timeout	PT15M	How long a claim is honoured before another dispatcher may re-take it. Far longer than any provider call, on purpose.
pool-size	2	Bounded, named scheduler pool. Without an explicit one Spring uses a single thread shared by every scheduled method, where one blocked job silently stops the others.

Credentials are never logged, and only ids and outcomes appear in log lines — never the recipient, subject, or body.

Related features

• Billing (receipts), Concept/Demographic (recipients). Domain plug-ins call via exchange.client.notification.

Notification preferences (M10-009)

Preferences are opt-out. No subscription row means the person receives the notification; only a row with active = false suppresses it. Opt-in would mean a deployment that silently delivers nothing until configured — and the failure would be invisible, because there would be no error and no delivery record to look at.

A preference with no template covers the whole channel; one naming a template narrows to it, and the narrower preference wins in both directions, so muting appointment reminders does not mute results.

A suppressed notification is still recorded, as a SUPPRESSED event. That is deliberate: a notification that intentionally did not happen must remain answerable afterwards, and "we have no record" is not an answer to "why was I not told?". SUPPRESSED is distinct from CANCELLED, which is somebody calling a particular send off.

Critical-result acknowledgement loop (M10B-009A)

A critical finding is not closed by sending a message — it is closed by somebody acknowledging it. Imaging (M10B-009) and lab (M20-008) both raise through CriticalResultClient; neither module owns an escalation table or scheduler.

Step	What happens
Raise	POST /api/v1/notification/critical-results persists the row, places a QueueItem on the facility's CRITICAL_RESULTS queue (QueueType.CRITICAL_RESULT), and enqueues a notification with the opt-out carve-out (enqueueCriticalEvent) so an opted-out recipient still receives it.
List	GET …/critical-results?source_module=&include_closed= — the shared worklist (FE-371).
Acknowledge	POST …/{id}/acknowledge — attributable; required before close.
Close	POST …/{id}/close — stamps closed_at and completes the queue item. Unclosed stays visible forever.
Escalate	Property-driven ladder on the scheduler profile (notification.critical.escalation.*). Default rungs 15m / 30m / 60m up to max-level. Never ages a result out silently.

Permissions: notification.critical.read / .write (raise, acknowledge, and close). Nav route key notification.criticalResults.

Form-approval templates (M34-009)

Three Global-tenant IN_APP templates ship for routable form routing (notification/008):

Code	When used
form_approval_pending	A stage approval task is created — one event per resolved receiver
form_approval_rejected	An approver rejects — notifies the initiator with the mandatory comment
form_approval_completed	Routing reaches a terminal stage — notifies the initiator

Required variables: formName, submissionRef, stage, actorDisplay; comment on rejection. The workflow engine resolves context through ApprovalRoutingNotificationSupport (form engine implementation) and enqueues via NotificationTemplateService.resolveTemplate(tenantId, code) with Global-tenant fallback.

Per-facility CRUD for the ladder is deferred; the first ship matches the commitment-ageing pattern (property-driven). Clinical criteria (which findings/values are critical) stay in the calling modules.

In-app shell inbox (M34-009 / FE-410)

Method	Path	Purpose
GET	/api/v1/notification/in-app-notifications	Lists delivered IN_APP events for the signed-in person (safe title/body only)
POST	/api/v1/notification/in-app-notifications/{notificationId}/read	Marks one notification read (read_at)

Form-approval templates populate templateCode, routeKey (workforce.staff-portal), and an envelope deep link (/workforce/staff-portal/forms?envelope=INBOX|REJECTED|COMPLETED).