Form Builder & Submission — How it works

Overview

The form engine lets the platform define data-capture forms, publish immutable versions, target them to modules/facilities/clinical domains, and capture + process submissions. Form fields bind to concept ids (never raw tables), so captured data is coded and portable.

Data model & ownership

Area	Tables
Definition & versions	form_definition, form_version
Structure	form_definition.schema_jsonb / form_version.schema_jsonb (the whole form)
Targeting	module_form_mapping, facility_form_mapping, domain_form_target
Visibility scoping	form_visibility_role, form_visibility_org_unit
Submission	form_submission, form_submission_processing_log
Per-form storage	one table per form in domain_engine
Schema-change audit	form_schema_change
Lookup sources	form_lookup_source, form_lookup_allowed_host
Retired (FORM-001)	form_section, form_field, form_field_option, form_validation_rule, form_conditional_rule

One authoring family

Forms are authored with the drag-and-drop builder. The whole form is one schema_jsonb document; on publish the engine creates a per-form table domain_engine.<module>_<table> with a k_<fieldName> column per field. The owning module is declared by the author and denormalized onto both the submission (form_submission.module_key) and the dynamic row.

This is not a preference — it is modelling rule 3: patient-health data is collected by a drag-and-drop form, either a clinical observation form releasing into Observation or a form with a custom FormSubmissionHandler when it must also write module tables. There is no hand-written clinical CRUD screen and no second way to author a form.

The relational (M6) family is retired — FORM-001

The platform originally shipped a second, relational authoring family: structure in form_section / form_field / form_field_option, rules in form_validation_rule / form_conditional_rule, edited through /form-sections/*, /form-fields/*, /form-field-options/*, /form-versions/{id}/validation-rules, /form-versions/{id}/conditional-rules and /form-versions/{id}/field-events.

It is historical. Do not author against it, and do not plan work on it. Every one of those endpoints is marked deprecated in the OpenAPI document, and the Java types behind them (FormSection, FormField, FormFieldOption, their repositories, FormStructureService's section/field/option methods, FormRuleService, FormFieldEventService, DynamicFormTableWriter, and the FormSectionDto / FormFieldDto / FormFieldOptionDto / FormValidationRuleDto / FormConditionalRuleDto / FormFieldEventResultDto contracts) carry @Deprecated.

Nothing is removed and no table is dropped. Two reasons, and both are the point:

1. Existing submissions reference historical versions, and the raw payload must stay readable — the same rule that governs form submissions everywhere else. Deleting the structure would leave old submissions describing fields nothing can name.
2. Four shipped frontend routes still call the structure API (/forms/versions/[id], /forms/render/[id], /forms/preview/[id], and the section/field panels behind them). Removing the endpoints would 404 pages that are live today. The frontend retires those routes first; the backend removal is a separate, later change, and dropping the tables is later still.

What still touches the relational tables, as of FORM-001:

Reader	What it does	Family it serves
FormStructureServiceImpl	the deprecated authoring CRUD	relational only
FormRuleServiceImpl	validation/conditional rule CRUD, keyed by form_field_id	relational only
FormFieldEventServiceImpl	SET_VALUE / COMPUTE / CASCADE evaluation, keyed by field UUID	relational only
DynamicFormTableWriter	writes the per-submission row for a form with no author-chosen table	relational only
FormPublicationServiceImpl	collectRelationalProblems — publish gate for a version with no schemaJson	shared, branches on family
DynamicFormTableServiceImpl	relationalFieldColumns — column set for a version with no schemaJson	shared, branches on family
ObservationSubmissionHandlerImpl	loadFields — reads sections/fields first, falls through to the schema path only when they come back empty	shared, branches on family

The last three are why the tables cannot simply be dropped: they are the fallback branch taken when a version has no schemaJson, i.e. exactly for the historical forms.

⚠ Note the direction of that last one. ObservationSubmissionHandlerImpl queries the relational structure on every observation submission, drag-and-drop included, and only falls through to resolveSchemaObservations when the query returns nothing. The relational read is therefore on the live clinical path — two queries per submission that can only ever return empty for the family we actually use. Correct, but inverted: the drag-and-drop branch should be the one taken first. Not changed here, because reordering a clinical write path is a behaviour change and not a deprecation.

Note the seeded engine forms are safe — form/013-seed-engine-forms writes only form_definition and form_version rows, never form_section / form_field, so the seed does not depend on the retired family.

Two known gaps carried by the retirement, neither introduced by it:

• form_schema_change is written only by the relational FormStructureServiceImpl, so GET /form-versions/{id}/schema-changes returns an empty list for every drag-and-drop form. The drag-and-drop lock exists (FormSchemaLockService, M22-026) and refuses the unsafe edit; the history of accepted edits does not. The endpoint is therefore not deprecated — it is the drag-and-drop family's audit read, waiting to be populated.
• SubmissionProcessor calls DynamicFormTableWriter for every non-observation submission. For a drag-and-drop form it resolves the f_-plus-hex fallback table name, finds no such table and returns without writing — harmless, but it means the call is dead weight on the live path. Drag-and-drop rows are written by DynamicSchemaFormWriter from FormSubmissionServiceImpl.

Where a submission is stored

A drag-and-drop form picks one of two destinations, and they are mutually exclusive:

• Generic dynamic table — the form declares an owning module and a backing table name, and the engine writes one row per submission into domain_engine.<module>_<table>.
• Registered submission handler — the form declares submissionHandlerKey, and the engine dispatches the submission to the FormSubmissionHandler bean registered under that key. Use this when one form has to write across several real tables (patient registration writes person, patient, identifiers and addresses), which no single generic table can express.

GET /api/v1/form/submission-handlers lists the registered handlers (key, label, known field keys) so the builder can offer a picker. The key is validated against that registry, never against a URL — a form cannot name an arbitrary endpoint, so there is no request-forgery surface and nothing to allow-list. Both the destination choice and the module/table name are frozen once the form is published, because the physical storage exists by then.

Reference fields — file and staffPicker (M34-002, FORM-109)

Two input types whose value names another record rather than carrying data:

• file — the value is a document_id. The bytes go to the document engine at fill time (multipart POST /api/v1/document/documents); the submission carries only the reference. The server refuses a submission whose document does not exist in the session tenant, was not uploaded by the submitter, is scanned INFECTED, or whose stored content type is outside uhp.form.file-field.allowed-content-types (default application/pdf,image/png,image/jpeg — checked against what the document engine sniffed, never the client's claim). On acceptance the form engine stamps document.form_submission_id in the same transaction, so a rolled-back submission leaves no link.
• staffPicker — the value is the chosen staff member's person id (ID-001: one UUID names the account, the person and the staff profile). The server refuses a person that does not exist.

Both are exempt from the observation concept requirement — an attachment or a reliever is not an observation — and on an engine form they store as TEXT columns like any other reference. Format (UUID) is checked by the schema validator; existence and ownership by SubmissionReferenceValidator beside the rule engine, before anything is persisted.

Key rules & invariants

• Patient-record forms are filled from the patient dashboard. A form that collects a patient's record opens from the patient's dashboard (or their journey's workflow stage), which supplies the patient context the submission requires — there is no context-free entry to patient capture, and the backend refuses clinical submissions without a patient and an attributable provider. Context-free sidebar links are for admin/config forms only.

• Order widgets (M11-012). A clinical observation form may carry three system-block widgets — prescriptionWidget, labOrderWidget, imagingOrderWidget — that submit under fixed payload keys (prescriptionOrders, labOrders, imagingOrders). At submission, the observation pipeline persists them to the relational medication/order tables before the observations, through the same clinical services the screens use (concept validation, RxNorm resolution, CDS, attribution), inside the same processing pass. Safety rules: a payload key is acted on only when the schema declares the widget; an amendment never re-places (a PROCESSED original already placed, and correcting a typo must not duplicate care instructions); placed rows carry source_submission_id with a partial-unique retry floor; and a widget failure compensates — everything the submission placed is voided in the same transaction, so a failed form never leaves a half-placed prescription. Widgets are create-only: lifecycle (discontinue, dispense, results) stays on the screens. A facility extends what a form captures by adding ordinary fields next to the widget — the widget's own drug/dose/route contract maps to relational columns and is fixed per platform version.

• Programme enrolment widgets (M36-005). Two more system blocks — programEnrolmentWidget and programEnrolmentClosureWidget — submitting under programEnrolment and programEnrolmentClosure. Enrolment carries programme, start date and the register number (M36-004: pre-filled and read-only for an AUTO programme, required for MANUAL, absent for NONE); closure carries programme, stop date and the outcome concept.

  They are two widgets, not one with a mode. The fields differ, and the refusals are opposites: enrolment is refused when the patient already has an OPEN enrolment in the chosen programme (PO 2026-08-29 — enrolling somebody already enrolled is a mistake the clinician should see, not something to silently reconcile), and closure is refused when there is none (closing an enrolment that is not running writes an outcome onto nothing, and reports a completion that never happened). Both refusals name the programme, so "already enrolled" is distinguishable from "never enrolled" without opening another screen.

  ⚠ The refusal must not eat the retry, and this is the part that is easy to get wrong. "Already enrolled" is judged on the patient's open enrolment; a resubmission of the SAME formSubmissionId is one act arriving twice — a network retry — and returns the enrolment it already created. The floor is a unique index on patient_program.enrolment_submission_id (and closure_submission_id), because two concurrent retries both read "not yet enrolled" in Java and only the database can pick a winner.

  ⚠ Ordering: enrolment runs BEFORE the encounter is created, and its id is threaded into EncounterContextRequestDto. M36-002 attributes an encounter to whatever enrolment is open at creation time, so enrolling afterwards leaves the very encounter that enrolled the patient unattributed — and nothing reports it, because an unattributed encounter is the ordinary case. Closure runs with the other widgets, after the encounter: attribution matches on the encounter's DATE rather than on "open now", so the encounter that closed an enrolment still counts toward it.

• category is the FormCategory enum, not a string (TERM-004). This is load-bearing: the category is what decides whether a definition is treated as clinical. It used to be a free-text VARCHAR(80) re-parsed at a dozen call sites through FormCategory.fromString(), which returns empty on a miss — so a typo such as OBSERVATIONS skipped concept enforcement on every field, nulled the encounter type, skipped both publication gates, and stopped the observation handler firing at submission. The form published cleanly and recorded nothing clinically, with no error at any step. An unrecognised value is now a 400 at deserialization.

  The same applies to the category search filter, which is the one place a category still arrives as a string: an unrecognised value is rejected, not dropped. Dropping it would silently widen the search to every family — the read-side twin of the same defect.

• form_lookup_source.auth_type is the FormLookupAuthType enum (TERM-008A). The proxy used to switch on a free-text scheme and its default branch silently sent no credentials — so a typo such as BEARER_TOKEN registered cleanly, looked configured, and every fetch went out unauthenticated. An unrecognised value is now a 400 at deserialization; the database CHECK constraint rejects anything else at rest; the service switch is exhaustive over the enum (no silent default). Null on create still defaults to NONE.

• form_submission.encounter_type dropped (TERM-008G). The column duplicated form_definition.encounter_type_id: it was copied from FormClinicalContextDto.encounterType at submit time but never read — SubmissionProcessor resolves the encounter type from the definition and ObservationSubmissionHandlerImpl opens the encounter from that id. Clients must not send encounter type in clinical context; declare it on the observation form definition instead.

• clinicalContext.patientProgramId carries the clinician's programme choice (M36-006). Sent only where the form's encounter type is claimed by more than one programme the patient is enrolled in, which is the one case programme attribution refuses to guess at. It is persisted on the submission, not merely read from the request: the handler rebuilds its context from the row on every retry and reprocess, so a value held only on the request would survive the first attempt and vanish on the second. ⚠ Never trusted — revalidated against the patient's own open, mapped enrolments — and an enrolment made by the submission's own enrolment widget outranks it.

• Published versions are immutable; structure is edited only on a draft (published = false), enforced by an assertDraft guard.

• Observation fields require a concept binding; coded options validate against the concept engine.

• Drag-and-drop submissions validate before persisting (M22-022a) — this deliberately differs from the retired relational path, where a submission was persisted first (RECEIVED) and only then validated and processed by a strategy. SchemaSubmissionValidator reads the field rules out of schema_jsonb and enforces required, number min/max/integerOnly, date minDate/maxDate, and text minLength/maxLength/pattern (with an optional custom patternMessage). Violations raise InvalidRequestException → HTTP 400 listing every violation, and nothing is written to either form_submission or the per-form table. There is no REJECTED row to retry, by design: an invalid drag-and-drop submission leaves no trace.

• Making a field required is forward-only once the form has submissions (M22-027). Adding a new required field to a form whose backing table already holds records is refused with a 400 naming the field, because PostgreSQL cannot add a NOT NULL column to a populated table and the only way round it would be to invent an answer for records nobody asked. The two remedies are in the message: give the field a default value (which backfills the existing rows with something the author chose), or leave it optional. Flipping an existing field to required is always allowed and is deliberately a no-op at the database — that column stays nullable, so historical rows that never answered it survive and required-ness is enforced only on submissions from then on.

• Not yet enforced on the drag-and-drop path (M22-022b): cross-field rules, the declarative SET_VALUE / COMPUTE / CASCADE events, conditional visibility/required, and SSRF-safe endpoint-backed dropdown lookups. Do not assume a rule of those kinds is applied server-side.

Who sees which form — visibility scoping (M22-017)

GET /api/v1/form/form-definitions does not return every form in the tenant. It returns the forms this caller may fill, narrowed on four axes.

Axis	Declared on the form by	Matched against the caller's
Tenant	form_definition.tenant_id (Global-tenant forms are visible everywhere)	signed session tenant
Facility	form_definition.facility_id (null = every facility)	signed session facility
Role	form_visibility_role rows	user_role grants
Department / unit	form_visibility_org_unit rows	user_department_access grants, resolved by UserOrgAccessResolver

Three properties hold across all four, and each is a deliberate decision rather than a default:

• Declaring a scope is optional. A form with no role and no department/unit rows is visible to everyone in the tenant. Plenty of forms genuinely are facility-wide, and refusing to publish until a scope is declared would be friction on the common case. The consequence to accept is that a form that should be unit-scoped and was published without a scope is visible to everyone — reached by omission, which is why the builder's selectors and their labelling are doing real safety work.
• Multiple values are OR'd. A form scoped to Surgery + Theatre + Laboratory is three rows, and a caller in any one of them sees it. Same for roles.
• A unit restriction needs the unit. form_visibility_org_unit.unit_id null means the whole department; set means that unit only, and being in the parent department is not enough — otherwise unit scoping would collapse into department scoping.

The caller side

user_department_access answers which departments and units does this account work in. A row with unit_id null is department-wide; a row with it set is that unit. Grants carry a start_date / end_date day window that UserOrgAccessResolver honours, so a week-long cover grant genuinely lapses. Grants are administered at /api/v1/access-control/user-accounts/{userAccountId}/department-access, gated on accesscontrol.department-access.read / .write, and containment-checked: an administrator may only grant departments inside a facility their own scope already reaches.

The resolver is cached per account (userOrgAccess, 45s TTL) and evicted by every grant write path. The TTL is not decoration — it is what makes a grant that expires at midnight stop counting without a write, so the cache must stay registered in CacheNames.ALL.

Two axes that must not be conflated. The form list filters on the caller's department/unit; what a submission records is the patient's care location. A lab technician drawing a sample from a surgical inpatient fills a lab form, stored against the patient's surgical bed. Merging them would mean the technician either sees surgical forms or cannot record where the patient was.

Which caller is being filtered

The caller is resolved delegated id first, authenticated account second: a domain module calling core on a user's behalf sends X-Delegated-User, and that names the human the module is acting for; a direct request carries no such header, so the authenticated user_account is the human. Both arms of the filter fail closed — a caller that cannot be resolved at all sees only unscoped forms, because resolving an unknown caller to "everything" would make scoping trivially bypassable.

That combination is load-bearing. Reading the delegated id alone — which is how this shipped until M22-017 closed — means a clinician's own request resolves to no caller, and every role- and department-scoped form is invisible to everyone regardless of what is granted.

Where the decision is made

Core-side, from user_department_access — not from session-token claims. The caller's department/unit is authorization data, and the platform resolves authorization data server-side on every request (UserAuthorizationFactsResolver, AccessDecisionServiceImpl, UserAccessScopeService all work this way); only tenant and facility ride in the signed token, because they are a choice the user made when scoping their session. Putting department in the token would mean a revoked grant kept working until the token expired, and would make the roster swap below a token-format change.

Rostering stays in workforce. Which department an account may reach (access) and which one it is scheduled in (roster) are different facts that legitimately diverge — a clerk covering another department for a week has access with no roster entry. Core never imports workforce; the optional CurrentOrgAssignmentSource SPI in exchange lets workforce contribute live assignments, which the resolver unions with standing grants. When the bean is absent, standing grants alone drive visibility. Implementations must answer active now, not "dated today", or a shift crossing midnight would drop the night nurse's forms at 00:00.

What a clinician can fill right now — /form-definitions/available (FORM-119)

GET /api/v1/form/form-definitions/available?patientId=… answers a narrower question than the list above: of the forms this caller may fill, which ones apply to this patient at this moment. It is what the patient dashboard's forms drawer renders, and it exists because the answer depends on three things the client cannot see — the patient's open visit, the workflow stage that visit is standing in, and the stage's form bindings.

Three narrowings, applied in order:

1. Observation forms only. FormCategory.OBSERVATION, published, active. A DOMAIN form writes a module table and a DEMOGRAPHIC form edits a record; neither is patient-health data captured against a visit, and both have their own entry points.
2. Visibility scoping, exactly as above — the endpoint calls the same searchDefinitions path, so role and department/unit scoping cannot drift between the two lists.
3. The workflow stage, when there is one. If the patient's open visit is running a workflow instance, only the forms bound to its active stages are offered, each carrying required (the stage binding's minimum is unmet) and atMaximum (the maximum submissions are already in).

The response states which rule produced it, in mode:

mode	Means
WORKFLOW_STAGE	A workflow instance is running; the list is that stage's bound forms
ALL_ACCESSIBLE	An open visit with no workflow — every observation form the caller may fill
ALL_ACCESSIBLE_OVERRIDE	A workflow is running and the caller asked to see past it

⚠ The mode is not decoration. An empty list has three unrelated causes — no visit is open, the stage asks for no forms, and nothing observation-shaped is published for this caller — and a panel that renders all three as the same blank list is the M28-010 failure exactly: absence and success become indistinguishable. visitId null says the first; mode separates the other two.

⚠ includeAllAccessible=true is an escape hatch that must stay. A stage binding describes the expected pathway, not the limit of clinical judgement: a patient in a triage stage who needs an allergy recorded must not be blocked by configuration (M28-001). The widening is labelled in the UI and reported in mode, so the wider list is never mistaken for the pathway's own.

Never resolve-or-create here. The open visit is read through VisitService.searchVisits(…, ACTIVE), not EncounterContextService, whose resolve-or-create contract would have a read of the chart open a visit for a patient nobody has seen yet.

Stage forms come from activeStageIds, not currentStageId. On a parallel journey (bloods and X-ray at once) currentStageId is null, and reading it would offer no forms at precisely the moment two stages are both asking for them. Forms bound to several active stages are merged, keeping the strictest requirement.

A workflow lookup that fails degrades to ALL_ACCESSIBLE rather than propagating: the workflow engine being unavailable must not remove a clinician's ability to record observations.

The endpoint is gated on form.form.read and calls PatientAccessGuard.authorizeOrNotFound with AccessMode.READ — it names the patient, so it is a patient-scoped read like any other (SEC-006).

⚠ A denial answers 404, not 403 (SEC-011). The patient id arrives in the query string, so refusing out loud would confirm that the id names a real patient belonging to someone else. Unknown id and forbidden patient are deliberately indistinguishable.

API

See the API Reference. Live endpoint groups under /api/v1/form: form definitions (the schemaJson document is edited here), versions, module/facility/domain mappings, visibility roles and org units, form types, field-lookup sources, publication, the schema-change audit, and submissions (+ validation, retry, override, void, amend, edit).

Marked deprecated in the spec (FORM-001), kept routed for shipped screens only: /form-sections/*, /form-fields/*, /form-field-options/*, /form-versions/{id}/sections, /form-versions/{id}/validation-rules, /form-validation-rules/{id}, /form-versions/{id}/conditional-rules, /form-conditional-rules/{id}, and /form-versions/{id}/field-events. A new client must not call any of them.

Configuration & feature flags

FORM_OVERRIDE_SUBMISSION_ENABLED (default off) gates exactly one thing: the POST .../form-submissions/{id}/override endpoint, which reprocesses a FAILED and non-retryable submission — the escape hatch for a submission the normal retry guard refuses.

It does not gate which handler a form declares. Choosing a submissionHandlerKey is an authorization question, answered by the form.form.write permission and the closed handler registry, and CORE_PLAN.md is explicit that feature flags must not replace permissions. Putting handler selection behind this flag would also break a shipped capability: the flag defaults off, so every drag-and-drop form that routes through a handler would stop being editable.

Updating a definition

PUT /api/v1/form/form-definitions/{id} is a full replace, not a merge. A field omitted from the body is written as null — a partial body will blank schemaJson, formTypeId, backingTableName and submissionHandlerKey on a draft. (On a published definition the immutability guards reject the change instead.) Send the whole definition, which is what the builder's save path does.

M34-006 — routable forms (approval workflow binding)

A form may declare approvalWorkflowDefinitionId — a published workflow in the FORM_APPROVALS category with subject type FORM_SUBMISSION. Publication refuses a binding that is draft, voided, or wrong category/subject. After validation and any custom handler, an accepted submission auto-starts routing in the same transaction: workflow instance + routing_status = IN_PROGRESS + stage-1 approval task via ApprovalRoutingService.startRouting. Drafts never start routing.

POST /api/v1/form/form-submissions/{id}/resubmit (initiator-only, from REJECTED) voids the rejected row, recreates with replaces_submission_id, and resumes the same instance with a RESUBMITTED action. GET /api/v1/form/routing/consistency-gaps compares routing_status to workflow instance/task state (configuration-health style).

M34-008 — staff-portal routing envelopes

Envelopes are queries, not tables. Each staff member sees seven tabs — draft, inbox, in-progress, rejected, approved, completed, archived — backed by FormRoutingService:

Envelope	Query shape
DRAFT / IN_PROGRESS / REJECTED / ARCHIVED	initiator (created_by) + matching routing_status
INBOX	open APPROVAL task where caller is assignee or materialized candidate
APPROVED	caller recorded an APPROVED action on the instance
COMPLETED	routing_status = COMPLETED and caller participated (initiator, assignee, candidate, or action actor/on-behalf-of)

APIs (permission form.submission.read):

• GET /api/v1/form/routing/envelopes/{envelope} — paged FormEnvelopeItemDto
• GET /api/v1/form/routing/submissions/{id}/history — merged workflow_approval_action + workflow_event timeline

Workforce proxies the same shapes for the staff portal BFF (bearer propagates over loopback Feign):

• GET /api/v1/workforce/staff-portal/forms/envelopes/{envelope}
• GET /api/v1/workforce/staff-portal/forms/submissions/{id}/history

Form engine services call workflow through FormApprovalRoutingQueryService (not workflow repositories) to satisfy engine boundary rules.

M34-010 — pilot leave-request recipe (FORM-109)

Global leave-request form (form/036): DOMAIN workforce form with reliever (staffPicker) and file attachment fields, handler leave_request, and approval_workflow_definition_id bound to platform starter UHP_STARTER_LEAVE_APPROVAL. Gate proof: Milestone34FormApprovalsIntegrationTest.

Related features

• Concept (field bindings), Module / Facility (targeting), Clinical (captured observations). Domain plug-ins read via exchange.client.form.