Skip to content
Message lifecycle

Message lifecycle

Status: describes the system. This note is the design of how a message moves through MicroB2B: one routing decision made in the database, one record of progress per target, a lossy wake-up on the bus, claims with leases, a recovery sweep, and two operator actions. The operator’s view is in Routing and Troubleshooting; this note names the code.

The database is the one and only system of record for a message’s progress and the only place where routing is decided. A source plugin stores a message and dispatches it in the same transaction (or, for AS2, in a second transaction it controls); the dispatch matches the route table once, in Go, and inserts one queued delivery per target plugin instance. NATS carries nothing but a lossy wake-up per delivery on a subject that names only the target (delivery.<target>). A target claims a delivery with a compare-and-swap and a lease before it works on it, records the outcome on the delivery, and the message status users see is derived from its deliveries. A periodic sweep republishes wake-ups that were lost, a bounded number of times, after which a delivery is stalled and raises an alert.

Considered and rejected. Each target creating its delivery when it first handles the message, with routing staying the NATS subscription, was rejected because a message nobody handled would have no rows, so “nothing matched” could not be told from “lost”, which is the one distinction recovery needs; it also leaves the status of a message with several targets undecidable. JetStream was rejected because it would be a second place that holds message state and would still need the database claim, the sweep and idempotent handlers.


1. Vocabulary

1.1 Message stage

Stored on the message (messages.stage), one domain string persistence.MessageStage with ParseMessageStage, canonical slice and validating JSON.

StageMeaningSet by
acceptedThe row exists and its source plugin still owns it (AS2: being unpacked and checked); no deliveriesInsertMessage
dispatchedThe deliveries exist; at least one is not finalDispatchMessage / InsertAndDispatchMessage, and again by redispatch or redeliver
settledEvery delivery is delivered, failed or stalled, or the message has none (no route matched, or the source refused it)the persistence, in the transaction of the delivery transition that makes it true, or RefuseMessage

1.2 Delivery

One delivery per (message, target plugin instance), persistence.DeliveryState (domain string, canonical slice, ParseDeliveryState, validating JSON, Final(), Redeliverable(), Label(), Tone() for the badge):

StateMeaningFinal
queuedCreated or put back; nobody holds it. Woken by the dispatch and the sweepno
deliveringA target holds the claim (token and lease)no
retryingAn attempt failed in a way another may fix; next_dispatch_at is the next attemptno
awaiting_ackHanded off; the partner’s end-to-end receipt (AS2 asynchronous MDN) has not arrivedno
deliveredDone and, where a receipt exists, confirmedyes
failedEnded for good, with the reasonyes
stalledWoken the maximum number of times and never claimed (section 5). Final until the target claims it at its start or an operator actsyes (counts as settled)

1.3 The status a user sees

persistence.MessageStatus is the one value of the list, the filter, the badge and REST, with this canonical set (labels in brackets):

Received (“Received”), InProgress (“In progress”), Retrying (“Retrying”), AwaitingReceipt (“Awaiting receipt”), Stalled (“Stalled”), Delivered (“Delivered”), NotRouted (“Not routed”), Refused (“Refused”), Error (“Error”).

The core names no protocol: “MDN” is AS2’s word, and only the AS2 message panel says it.

The derivation rule is one pure function, persistence.DeriveStatus(stage MessageStage, refusal Refusal, deliveries []DeliveryState) MessageStatus; the first match wins:

ConditionStatus
stage acceptedReceived
no deliveries and refusal is RefusalPartnerRefused
no deliveries and refusal is RefusalInternalError
no deliveriesNotRouted
any failedError
any stalledStalled
any retryingRetrying
any queued or deliveringInProgress
any awaiting_ackAwaitingReceipt
otherwise (all delivered)Delivered

Refusal (persistence.Refusal, ParseRefusal, validating JSON) is the reason class RefuseMessage takes (section 3.4). Work still to be done on our side (retrying, queued, delivering) ranks above waiting for a partner, so a message with one delivery awaiting a receipt and one still queued shows “In progress”.

MessageStatus.Settled() is true for Delivered, NotRouted, Refused, Error and Stalled. In flight (InFlightMessageStatuses, the dashboard card) is Received, InProgress, Retrying and AwaitingReceipt. The actions have their own rules (section 6).

1.4 Stored, not computed per query

The status is derived and stored: messages.status is written by the persistence, and only by it, in the same transaction as every stage or delivery transition, from DeriveStatus over the message’s deliveries (SELECT state FROM deliveries WHERE message_id = ?, at most a handful of rows, on the unique index). Nothing else writes it and no plugin names a message status.

Why stored: the message list, its filter and count, the dashboard cards (polled by the live dashboard) and the live list would otherwise each need a GROUP BY over deliveries joined to messages for every page. On SQLite’s single connection that serializes behind every delivery write; stored, they stay the indexed status = ?. The cost is one small read in each transition transaction. TestStoredStatusIsTheDerivedStatus (persistence) drives random transitions and compares the column with DeriveStatus after each.

The status transition is what logStateChange records (status_changed) and what raises message_failed (for Error only), so alerting stays in one place.

2. Data model

All in the core: deliveries names targets by instance name only, so the core names no module. The models are in internal/persistence/bun/models.go and CoreSchema (dbinit.go). There are no migrations: a changed model means resetting the database.

2.1 deliveries (Bun model delivery)

ColumnTypeNotes
iduuid, pkthe wake-up payload, the REST id, delivery_id in logs
message_iduuid, not nullno foreign key (plain id columns)
targetstring, not nullthe route’s target_plugin
statepersistence.DeliveryState, not nullvalidated on write and read
attemptsint, not null, default 0incremented by each claim
dispatch_countint, not null, default 0wake-ups published since the last claim (section 5)
requeuesint, not null, default 0times the delivery was put back in the queue after it had started (redeliver, redispatch); a target that sees a count above zero knows an earlier pass may have left its work behind
next_dispatch_attimestamp, nullzerowhen the sweep looks at it next: queued the next wake-up, retrying the next attempt, delivering the end of the lease, awaiting_ack the receipt deadline; null for final states
claim_tokenuuid, nullzerothe current claim; every write by a claimant is conditional on it
last_errorstring, not null, default ''bounded like history text; never a URL, header or payload
created_at, updated_attimestamp
finished_attimestamp, nullzerowhen it became delivered or failed
tracetelemetry.TraceParent, not null, default ''the W3C traceparent of the dispatch (or redispatch) that created it; its wake-ups, claims and attempts continue that trace (DeliveryClaim.Trace, not in the REST resource)

Indexes (CoreSchema.Indexes):

IndexColumnsFor
unique deliveries_message_targetmessage_id, targetone delivery per target; the derive read; idempotent dispatch
deliveries_duestate, next_dispatch_atthe sweep (due queued/retrying, expired delivering, awaiting_ack deadlines)
deliveries_target_duetarget, state, next_dispatch_ata target’s catch-up at start
messages_status_createdstatus, created_at on messageslist filter, the in-flight count
messages_createdcreated_at on messagesthe dashboard’s and the statistics’ windows

persistenceBun.Index has no partial index; the composite indexes are enough at this scale.

2.2 messages

stage (persistence.MessageStage, not null); status, written only by the derivation; error, the source’s refusal reason or the last_error of the first failed delivery (written with the status). persistence.Message carries Stage and Deliveries []Delivery (filled by GetMessage, empty in lists). A message holds no retry fields: retries belong to the delivery.

2.3 message_history

Columns delivery_id (uuid, nullzero), target, old_state, new_state (both DeliveryState, empty for other kinds). Kinds:

KindWritten when
status_changedthe message status changes (old_status, new_status)
delivery_changedon every delivery transition; the text is the outcome text the target gave (a file written, an HTTP answer, a retry and why); the status change it causes repeats no text
redispatch_requested, redeliver_requestedan operator’s action; the text names the operator’s subject
mdn_received, mdn_refused, duplicate_receivedAS2 receipt and repeat events

2.4 Settings

persistence.Settings has Delivery DeliverySettings{MaxDispatches int} (JSON delivery.max_dispatches, default 5, 1 to 20), on the Settings page through internal/api/resource/settings.go and in /api/v1/settings. Like every setting it is read when a process starts. The back-off steps are constants (section 5).

2.5 AS2

as2_messages has delivery_id (outgoing rows only): Sender records it with the outgoing details, and MDNReceiver.locateSent reads it to settle the right delivery. The remote profile has receipt_timeout_hours (OverridableProfileParameters, default as2.DefaultReceiptTimeoutHours = 24, 1 to ReceiptTimeoutHoursCeiling = 720), the time the sender waits for an asynchronous receipt.

3. State machine

3.1 Message

FromEventToWhoCondition
(none)InsertMessageacceptedAS2 receiver
(none)InsertAndDispatchMessagedispatched, or settled with no deliveriesfile pollerone transaction
acceptedDispatchMessagedispatched, or settled with no deliveriesAS2 receiverafter unpack, claim and unpacked stored
acceptedRefuseMessage(refusal, reason)settled (status Refused or Error)AS2 receiver (unpack, claim, storage failure, repeat, replay); a source at start for abandoned rows (section 4.4)only from accepted
dispatchedlast delivery becomes finalsettledpersistencein the transition’s transaction
settledredispatch adds or wakes deliveries, redeliverdispatchedoperatorsection 6
settledstalled delivery claimed at target startdispatchedtargetsection 5

3.2 Delivery

FromEventToWhoCondition / effect
(none)dispatchqueuedpersistencedispatch_count = 1, next_dispatch_at = now + 10 s, wake-up after commit
queuedClaimDeliverydeliveringtarget workerattempts + 1, new claim_token, next_dispatch_at = lease end, dispatch_count = 0
retryingClaimDeliverydeliveringtarget workeronly when next_dispatch_at <= now, or when the sweep has woken it (dispatch_count > 0: the wake-up moved next_dispatch_at on to the next back-off step)
stalledClaimDeliverydeliveringtarget workeronly with fromStalled, from the target’s start-up catch-up
deliveringCompleteDelivery delivereddeliveredtarget workertoken matches
deliveringCompleteDelivery awaiting_ackawaiting_ackAS2 sender (async MDN requested)token matches; next_dispatch_at = the receipt deadline; a token that was cleared (the MDN arrived first) is ErrClaimLost, logged at INFO
deliveringCompleteDelivery retryretryingtarget workertoken matches; next_dispatch_at = At, dispatch_count = 0
deliveringCompleteDelivery failedfailedtarget workertoken matches
deliveringReleaseDelivery (stop)queued (attempt not counted) or retrying due now (the attempt had started its I/O)target workertoken matches; written with a context that is not cancelled
deliveringlease expiredretrying, next_dispatch_at = nowsweephistory “attempt interrupted: lease expired”
queued, retryingdue, dispatch_count < MaxDispatchessame statesweepdispatch_count + 1, next_dispatch_at = now + DispatchBackoff(count), wake-up
queued, retryingdue, dispatch_count = MaxDispatchesstalledsweepraises delivery_stalled
awaiting_ackdeadline passedfailedsweep“no receipt within ”
delivering, awaiting_ack, retryingSettleDelivery delivereddeliveredAS2 MDN receiverclears the claim
delivering, awaiting_ack, retryingSettleDelivery failedfailedAS2 MDN receivernever downgrades delivered; the first failure’s reason stands
failed, stalled, retrying, deliveredredeliverqueuedoperatorattempts = 0, dispatch_count = 0, requeues + 1, claim and finished_at cleared, wake-up
queued, stalledredispatchqueuedoperatordispatch_count = 0, requeues + 1, wake-up

3.3 What each plugin returns

A target implements delivery.Deliverer (section 4.5) and returns an outcome; it never writes a state itself.

PluginSituationOutcome
as2-senderprofile missing, build (compress/sign/encrypt), file name, MDN settingsFailed
payload not loadable, TLS client, POST error or non-2xx, async MICs not recordedRetry{At: policy.delay(attempts)} while attempts < policy.maxRetries, else Failed("max retries exceeded: ...") (the resend policy of the partner profile)
success, no MDNDelivered
success, sync MDN confirms / does notDelivered / Failed("synchronous MDN: ...") (not retried)
success, async MDN requestedAwaitingAck with the deadline now + receipt_timeout_hours; MDNReceiver later SettleDelivery(as2_messages.delivery_id, ...); a forged or unattributable MDN leaves it awaiting_ack
file-writerwrittenDelivered (text FileWrittenText)
payload unavailable, I/O errorRetry on the writer’s fixed schedule (file.WriteRetryDelays = 30 s, 2 min, 10 min, then Failed)
stage not recorded, pattern error, name exists without OverwriteFailed
re-attempt after an interrupted attempt or a requeue and the target file exists with exactly the payload’s bytesDelivered (the write had completed)
http-poster2xxDelivered
retryable (httpout.Classify) and attempts < MaxAttemptsRetry{At: now + retryWait(attempts)} (the interval, doubled per attempt, up to an hour; Retry-After is not honored)
otherwiseFailed; the message’s message_failed alert reports it

3.4 AS2 inbound specifics

CaseWhat happens
Unknown parties, receipt URL not allowed, too largerefused before a row exists
Unpack fails (signature, decryption, unsigned where signing is required), repeat or replayRefuseMessage(RefusalPartner, ...): settled, status Refused, no deliveries, no alert; failed MDN. For a repeat the reason is “Repeat of message …” and the first message gets duplicate_received
AcceptedDispatchMessage commits, then CompleteInboundMessage, then the positive MDN (sync in the response, async queued). A message is accepted for the partner exactly when its dispatch has committed
DispatchMessage or storage failsRefuseMessage(RefusalInternal, ...): status Error, message_failed; failed MDN FailureReasonDatabase or its storage counterpart; the deferred ReleaseInboundMessage frees the key so the partner’s resend is accepted
Route matches nothingNotRouted and one message_not_routed alert; the partner got a positive MDN

4. Dispatch, routes and subjects

4.1 Dispatch API and transaction boundaries

Methods of persistence.Persistence:

// InsertMessage stores a message in stage accepted with its first payload records.
InsertMessage(ctx, msg *Message, payloads []MessagePayload) error
// InsertAndDispatchMessage stores a message and dispatches it in one transaction.
InsertAndDispatchMessage(ctx, msg *Message, payloads []MessagePayload) (dispatch *Dispatch, err error)
// DispatchMessage dispatches an accepted message; on a message that is not accepted it changes nothing and returns its deliveries.
DispatchMessage(ctx, id uuid.UUID) (dispatch *Dispatch, err error)
// RefuseMessage settles an accepted message with no deliveries; the Refusal picks Refused or Error.
RefuseMessage(ctx, id uuid.UUID, refusal Refusal, reason string) error

Dispatch{Stage, Status, Deliveries []Delivery}. The dispatch, in one transaction: read the source, from and to from the row (stage = accepted as the condition of the update), match routes (4.2), insert one queued delivery per distinct target, set stage and status, write the history lines (delivery_changed per delivery, or “No route matched (source, from, to)”). After commit the persistence publishes one wake-up per delivery through its delivery notifier (4.3) and, for a message nothing matched, raises message_not_routed. A route whose target is no usable instance name fails the dispatch with ErrInvalidTarget and leaves the message accepted.

  • File poller: InsertAndDispatchMessage: there is no moment when the row exists undispatched. A crash before commit leaves the file to be picked up again; a crash after commit and before the file is moved makes a second message at the next scan (risk R9).
  • AS2 receiver: the row must exist before unpacking (audit, refusal), so it is InsertMessage then DispatchMessage. A crash between the two cannot lose the message: the partner received no positive MDN and resends, and the claim’s lease (InboundClaimLease) lets the resend in. A crash after DispatchMessage and before CompleteInboundMessage would let a resend in after the lease; so ClaimInboundMessage reports a stale incomplete holder (Stale), the receiver asks GetMessage(holder).Stage, and when that message is past accepted the claim is completed and the resend is answered as a repeat; otherwise TakeOverInboundMessage takes the claim over.
  • Publish after commit, never inside the transaction: a wake-up must never name a row the claimant cannot see yet. The window between commit and publish is covered by the sweep (next_dispatch_at = now + 10 s).

4.2 Route matching and the route cache

  • persistence.MatchRoutes(routes []Route, source, from, to string) []string (route.go): the distinct targets of the routes with SourcePlugin == source and, in each position, patternCovers(pattern, name), the token rule Route.Covers uses. Deduplicated by target, so overlapping routes to one target make one delivery. This is the only routing decision in the system.
  • The cache lives in the core Bun persistence (bunPersistence.routes, routes_cache.go), because every dispatch and every route write runs there. It holds the whole route table (small), loaded lazily on the first dispatch (so fixtures and seeding written straight into the table at startup are seen).
  • Invalidation: CreateRoute, UpdateRoute, DeleteRoute drop the cache after commit, in the same persistence; they also publish a persistence.RouteChange on events.routes (WithRouteChanges(disp)), and every Bun persistence subscribed to it drops its cache on a change from another instance. A backstop TTL (RouteCacheTTL = 30 s) covers writes that bypass the persistence (the developer database console) and a lost event.
  • Staleness window: none for a route written through the persistence that dispatches; at most 30 s for a write that bypasses the persistence. A route takes effect at the next dispatch, with no restart; a message dispatched before keeps its deliveries (redispatch adds the missing ones).

4.3 Subjects

SubjectBuilt byCarries
delivery.<target instance>persistence.DeliverySubject(target) (string, error), which refuses a name that is not one token of lower case letters, digits, _ and - (at most MaxInstanceNameLength)persistence.Wakeup{DeliveryID, MessageID} (JSON), dispatcher.Event.MessageID set for traces and logs
events.routespersistence.RouteChangesSubjectRouteChange{Persistence instance id}
  • A fixed first token keeps the namespace apart from events.* and notification.test.*, so no instance or party name can collide. An instance name is unique across plugin types (decision 1, section 11), so the name alone identifies the target.
  • Each target subscribes to exactly one static subject, its own, at Start; nothing reads routes in a target. persistence.UnmarshalWakeup decodes the payload.
  • The persistence publishes wake-ups through WithDeliveryNotifier(persistence.PublishWakeups(disp)), the same pattern as WithMessageHistoryNotifier; the notifier never blocks (a failed publish is a DEBUG line and a metric; the sweep recovers it).
  • Sources publish nothing on NATS for messages.
Plugin typeSubscribesPublishes
file-poller, as2-receivernothing for messagesnothing (the persistence publishes the wake-ups)
as2-sender, file-writer, http-posterdelivery.<own name> through delivery.Workernothing
persistence (standalone or process)events.routesdelivery.<target>, events.routes, events.message_history

4.4 Abandoned accepted messages

The sweep never dispatches: only the source knows whether an accepted message is complete. A source plugin that leaves accepted rows behind settles its own at Start with RefuseMessage through ListAcceptedMessages(source, olderThan, limit). The AS2 receiver uses olderThan = now - InboundClaimLease (the partner’s resend is let in after the lease, so the old row is superseded), reason “interrupted before it was accepted; the partner resends it”. The poller never leaves one.

4.5 The delivery worker (internal/delivery, core)

One package, naming no module. The worker is generic over its jobs: a Store[T] keeps them and a Deliverer[T] attempts one. The deliveries of messages to target plugin instances are one kind of job; a module keeps jobs of its own in a store of its own (an AS2 receiver’s asynchronous MDNs, section 4.7; a notifier’s notifications, docs/notifications.md) and works them with the same worker.

type Deliverer[T any] interface {
    // Deliver makes one attempt. ctx ends at the attempt timeout or when the worker stops.
    Deliver(ctx context.Context, work T) Outcome
}
type Store[T any] interface {
    Claim(ctx, id, now, leaseUntil, fromStalled) (Claim[T], error) // compare-and-swap, counts the attempt
    Complete(ctx, id, token, outcome) error                         // conditional on the claim (ErrClaimLost)
    Release(ctx, id, token, started, reason) error                  // a stop gives the claim back
    ListDue(ctx, now, limit) ([]Due, error)
}
type Outcome // Delivered(text) | AwaitingAck(text, deadline) | Retry(at, reason) | Failed(reason)
func NewWorker[T](Config[T]{Name, Store, Deliverer, Wakeups, Annotate, Options}) *Worker[T]
// Options: MaxConcurrent, AttemptTimeout (the lease is 2*AttemptTimeout + 1 min), DrainGrace, PollInterval, Now
func (w *Worker[T]) Start(ctx context.Context) error   // subscribe (if Wakeups), then look at the store; poll if PollInterval
func (w *Worker[T]) Poke()                             // a polling worker looks now
func (w *Worker[T]) RunOnce(ctx context.Context) error // work off what is due, return when it is recorded
func (w *Worker[T]) Quiesce()                          // unsubscribe, stop polling, claim nothing more
func (w *Worker[T]) Stop(ctx context.Context) error    // drain, cancel, release (section 7)

// The worker of a target plugin instance: its deliveries, woken on delivery.<target>.
type Attempt struct { Delivery persistence.Delivery; Message persistence.Message }
func NewTargetWorker(target string, p persistence.Persistence, d dispatcher.Dispatcher, impl Deliverer[Attempt], opts Options) (*Worker[Attempt], error)

Per wake-up: the subscription callback decodes the wake-up, acquires a slot of the semaphore (MaxConcurrent; selects on the lifecycle context), then starts a goroutine (tracked by a sync.WaitGroup) that calls Store.Claim (ClaimDelivery for a target); not claimed: release the slot and count a claim conflict; claimed: run Deliver with the attempt context (section 7), record the outcome with Store.Complete on a context that is not cancelled and has its own 10 s deadline. Backpressure: while every slot is taken the callback waits on the semaphore, so wake-ups queue in the subscription’s pending buffer (tiny messages; an overflow drops wake-ups, which the sweep recovers).

The worker keeps no timer. A recorded Retry is a row with a due time; the sweep wakes it when it is due, so a retry starts up to SweepInterval after its time.

The claim is the unit of concurrency: max_concurrent_deliveries is real for the AS2 sender, the file writer (default 1) and the HTTP poster (default 5). Deliveries of one target may complete out of order; the docs say so.

Catch-up at Start: after subscribing, the worker lists its due jobs (Store.ListDue; for a target ListDueDeliveries(target, now, limit): queued, retrying due or woken, stalled) and handles them like wake-ups, so a target that was down picks up its backlog without waiting for the sweep, and its stalled deliveries recover when it comes back.

Polling: a worker without wake-ups sets PollInterval and looks at its store at that interval and whenever it is poked. Its store is its own sweep: ListDue includes the jobs whose claim’s lease ran out, and Claim takes them over.

4.7 Asynchronous MDNs (AS2)

An asynchronous MDN the receiver owes a partner is not a delivery of a routed message: its outcome does not change the message’s status, and redispatch does not apply. It is a job of the receiver instance’s own worker over as2.AsyncMDNStore (table as2_pending_async_mdns: Pending, Delivering with a claim token and the lease in next_attempt_at, Retrying, Delivered, Failed), with as2.AsyncMDNSender as the Deliverer. The worker polls every mdn_poll_interval and the receiver pokes it when it queues an MDN, so an MDN is attempted at once. CompletePendingAsyncMDN refuses AwaitingAck and raises async_mdn_failed after the commit of a failure. The receiver plugin quiesces and stops the worker like a target’s.

4.6 Claim API and errors

MethodDoes
ClaimDelivery(ctx, id uuid.UUID, target string, now, leaseUntil time.Time, fromStalled bool) (*DeliveryClaim, error)conditional UPDATE ... WHERE id = ? AND target = ? AND (state = 'queued' OR (state = 'retrying' AND (next_dispatch_at <= now OR dispatch_count > 0)) OR (fromStalled AND state = 'stalled')); returns {Claimed, Token, Delivery, Message} (Delivery and Message are nil when not claimed) so the target needs no second round trip
CompleteDelivery(ctx, id, token uuid.UUID, outcome DeliveryOutcome) errorconditional on state = 'delivering' AND claim_token = token; ErrClaimLost when not
ReleaseDelivery(ctx, id, token uuid.UUID, started bool, reason string) errorstop path (section 7)
SettleDelivery(ctx, id uuid.UUID, outcome DeliveryOutcome) (applied bool, err error)async MDN receipt
ListDueDeliveries(ctx, target string, now time.Time, limit int) ([]Delivery, error)catch-up
ListDeliveries(ctx, messageID uuid.UUID) ([]Delivery, error)message page and REST
ListAcceptedMessages(ctx, source string, olderThan time.Time, limit int) ([]Message, error)section 4.4
Redispatch(ctx, messageID uuid.UUID) (message *Message, changed bool, err error), Redeliver(ctx, deliveryID uuid.UUID) (*Message, error)section 6

DeliveryOutcome is a closed struct (Kind domain string delivered|awaiting_ack|retry|failed, At, Deadline, Text) with Check, validated like every domain type: an awaiting_ack has a Deadline, no other kind has. The sweep is not part of Persistence (it is not called over RPC): persistence.Sweeper (Sweep(ctx, now, SweepLimits) (SweepResult, error)) is implemented by the Bun persistence only (bun.NewSweeper).

Error vocabulary (persistence.ErrorVocabulary):

CodeEntry
claim_lostrpcerr.Sentinel(ErrClaimLost)
not_redispatchablerpcerr.Typed(&NotRedispatchableError{Stage})
not_redeliverablerpcerr.Typed(&NotRedeliverableError{State})
not_acceptedrpcerr.Sentinel(ErrNotAccepted) (RefuseMessage on a dispatched message)
invalid_targetrpcerr.Sentinel(ErrInvalidTarget) (a route row with an unusable name, surfaced by dispatch)

5. The sweep and bounded dispatch

Who and where: a loop (delivery.StartSweep, over persistence.Sweeper) in the process that owns the database. It starts after manager.StartAll, so the first pass finds every target subscribed. It is not a plugin and not on the Services page, like the core persistence. Interval SweepInterval = 5 s; its first pass runs inside StartSweep, before it returns. Every write is conditional, so two sweepers would still wake a delivery once per due time.

One pass, each step an indexed query plus one conditional update per row, in batches of SweepBatch = 500 with a short pause between batches:

  1. Expired leases: state = 'delivering' AND next_dispatch_at <= now -> retrying, next_dispatch_at = now, dispatch_count = 0, claim_token cleared, history “attempt interrupted: lease expired”.
  2. Due wake-ups: state IN ('queued','retrying') AND next_dispatch_at <= now ORDER BY next_dispatch_at: with dispatch_count < MaxDispatches: dispatch_count + 1, next_dispatch_at = now + DispatchBackoff(count), wake-up after commit; with dispatch_count = MaxDispatches: stalled, last_error = "not claimed by <target> after <n> wake-ups", status recomputed, alert.
  3. Receipt deadlines: state = 'awaiting_ack' AND next_dispatch_at <= now -> failed, last_error “no receipt within ” (measured from the hand-off to the deadline), history line, the message status derived as usual. The deadline is set by the target when it completes the attempt as AwaitingAck (DeliveryOutcome.Deadline), so the core knows no protocol: the AS2 sender takes it from the partner’s profile (receipt_timeout_hours). A receipt that arrives later is refused as unattributable (mdn_refused), as a receipt for any settled delivery is.
  4. Gauges: the number of deliveries per state (one GROUP BY state, every state has an entry) and the due count, for metrics.

Back-off: DispatchBackoff = 10 s, 30 s, 2 min, 10 min (constants; the last step repeats when MaxDispatches is raised). With the default of 5 a never-claimed delivery is woken at 0 s, 10 s, 40 s, 2 min 40 s and 12 min 40 s and stalls at 22 min 40 s.

Rules that make it safe:

  • Only unclaimed wake-ups count: a claim resets dispatch_count to 0, and a delivering delivery is never woken (its next_dispatch_at is the lease end).
  • A retrying delivery the sweep woke is claimable although its next_dispatch_at now holds the next back-off step (dispatch_count > 0); the catch-up list includes it.
  • A message queued behind a busy target is claimed when a slot frees and is not stalled by being seen often; a backlog larger than a target can claim in 22 minutes does stall, which is the alert an operator needs (and the target’s next start claims it again).
  • stalled differs from failed: nothing was tried. An operator’s stop on the Services page stalls the target’s deliveries after the wake-ups run out; its next start claims them (catch-up includes stalled), so a maintenance window needs no manual action.
  • “No route matched” is decided at dispatch and never swept.

Alert: alert.CodeDeliveryStalled = delivery_stalled, critical, in alert.CodeTable, raised by the sweep once per delivery that stalls: subject the message, details target, wake_ups, from, to, link /ui/messages/<id>, dedupe key delivery_stalled:<target> (so the notifier’s cooldown and grouping fold a stopped target’s backlog into one notification). A delivery that fails for good is not an alert of its own: the message reaches Error and message_failed is raised through the status.

6. Redispatch and redeliver

ActionOnAllowed whenDoes
Redispatcha messagestage is not accepted (persistence.CanRedispatch; its source owns an accepted message)matches the routes again, inserts the deliveries of targets the message has none for, sets queued and stalled ones to queued with dispatch_count = 0, stage dispatched, status recomputed, history redispatch_requested, wake-ups after commit. Delivered, failed, retrying, delivering and awaiting deliveries are left alone. A call that changes nothing (no new target, nothing queued or stalled) succeeds, writes no history line and reports changed = false
Redeliverone deliveryfailed, stalled, retrying or delivered (DeliveryState.Redeliverable)queued, attempts = 0, dispatch_count = 0, last_error cleared, claim and finished_at cleared, requeues + 1, history redeliver_requested, wake-up after commit; a delivered message is InProgress again until the delivery ends
  • One function each, the persistence methods Redispatch and Redeliver (one transaction), called by the UI and REST. There is no permission check (an authenticated user is an administrator); the operator’s subject comes from auth.IdentityFrom and is the history text and the user of the log line (operator action, action, message_id, delivery_id). They are not audit events: the audit log records configuration, and the message’s history names the operator beside what happened.
  • REST: POST /api/v1/messages/{id}/redispatch and POST /api/v1/messages/{id}/deliveries/{delivery_id}/redeliver (Content-Type: application/json, no body, under RequireContentType), 200 with the message resource; 404 NOT_FOUND; 409 NOT_REDISPATCHABLE / NOT_REDELIVERABLE with details.stage or details.state and details.allowed, mapped in rest.WriteFailure (the one mapping). persistence.FindDelivery checks that a delivery belongs to the message of the request.
  • UI: POST /ui/messages/{id}/redispatch (a plain form with the CSRF field, 303 back with webui.SetFlash), POST /ui/messages/{id}/deliveries/{delivery_id}/redeliver (a button with hx-post, hx-confirm naming the target, hx-swap="none", answered with a flash and webui.PageRefreshed, because it sends again). Both under CSRF; a refused redispatch is the page again with 409 and the reason as flash, a refused redeliver a flash with the reason and webui.PageRefreshed.
  • Send again is Redeliver of a delivered delivery, for every target type: an operator may have lost the file or the downstream copy. The page words the button “Send again to ” and its confirmation is the generic text (the target receives the payload a second time) unless the target’s module contributes its own (webui.Extension.SendAgainWarning; the core names no module). For an AS2 sender the message goes out with the same Message-ID (decision 2), so the partner sees a repeat: it may answer with the receipt it stored or with none, and without a receipt the delivery fails at the receipt deadline and the message ends in Error. The file writer treats a file with exactly the payload’s bytes as its write when the delivery has been requeued (Delivery.Requeues > 0), writes a missing file, and fails on a taken name with other content when Overwrite is off. awaiting_ack, queued and delivering are refused.

7. Graceful stop

  1. Dispatcher: nats.Dispatcher owns a base context, cancelled by Close (which process.Run calls last); the subscription handler’s context derives from it and keeps the extracted trace. It is a backstop only.
  2. Manager: StopAll first calls Quiesce() on every plugin that implements the optional plugin.Quiescer (unsubscribe, claim nothing more; non-blocking), then stops the plugins in reverse start order (PluginStopTimeout 10 s each inside process.ShutdownTimeout 30 s). No target claims new work while an earlier one drains.
  3. Worker Stop(ctx): wait for in-flight attempts (the WaitGroup) for DrainGrace = 5 s or until 2 s before ctx’s deadline, whichever is first; then cancel the lifecycle context so stuck I/O aborts; then wait for the attempts to return.
  4. Per attempt: the attempt context is context.WithoutCancel(handlerCtx) (keeps message id, delivery id, trace), then context.WithCancel plus context.AfterFunc(lifecycle, cancel), then context.WithTimeout(AttemptTimeout). as2.sendMessage and every other I/O of a target use this context.
  5. Cancelled attempts: the worker calls ReleaseDelivery with a context that is not cancelled and has its own 2 s deadline: queued when the attempt had not started its I/O, retrying with next_dispatch_at = now when it had (the partner may have received it; AS2’s duplicate detection answers a resend of the same Message-ID with the stored MDN). Nothing becomes failed because of a stop. Attempts that finish within the grace period complete normally.
  6. A process killed without a stop leaves delivering rows; the sweep turns them into retrying at lease end.

8. UI, REST, observability

8.1 UI (internal/api/ui)

  • status_badge: classes from MessageStatus.BadgeClass(); each colour pair is in style_test.go (pairs, AA in both themes, the dark blocks kept identical).
  • Messages list filter: rendered from MessageStatusOptions. Dashboard: the in-flight card’s footer is “Received, in progress, retrying or awaiting receipt”; the failed card counts Error and Stalled in 24 h (“Error or stalled”, MessageWindowCounts.Attention over AttentionMessageStatuses); a separate Refused card counts Refused (MessageWindowCounts.Refused). Every card’s figure, and /api/v1/stats, comes from Persistence.CountMessageStatistics: the in-flight count on messages_status_created, and one pass over the messages created since the earliest window (messages_created) with a conditional sum per figure. Nothing counts the whole table (the first card is the messages of the last 7 days).
  • Message page: a Deliveries table (target, state badge via DeliveryState.Tone, attempts, next attempt for a retrying delivery, last error, updated; nowrap/wrap-anywhere by role; no row links) with an Actions column holding one icon button per redeliverable delivery (“Redeliver to ”, “Send again to ” for a delivered one) and, above the table, the message-level Redispatch. A delivery_changed history line streamed on the page also carries the delivery’s table row out of band (hx-swap-oob on id="delivery-<id>"), so the table follows live. History rows show target: Old -> New: text for delivery_changed.
  • Every link and form target is served (TestEveryLinkInATemplateIsServed).

8.2 REST and OpenAPI (internal/api/handlers/messages.go, internal/api/openapi/openapi.yaml)

  • Message: status enum = the canonical set; stage; deliveries (array of Delivery: id, target, state, attempts, next_attempt_at, last_error, created_at, updated_at, finished_at) on GET /messages/{id}.
  • GET /messages?status= accepts the canonical set (INVALID_STATUS lists it from MessageStatusOptions).
  • GET /stats: in_flight_messages; failed_last_24h counts Error and Stalled.
  • History kinds enum (/messages/{id}/history, /events): section 2.3; delivery_changed lines carry delivery_id, target, old_state, new_state.
  • RouteRequest description: matching happens at dispatch; a route applies to messages dispatched after it is saved; overlapping routes to one target deliver once.
  • The paths of section 6. TestEveryRequestBodyIsDescribedWithTheTypeTheServerTakes and TestEveryStateChangeDescribesTheTypeItDeclares cover them.

8.3 Observability

WhatWhere
delivery_id log attributelogctx.WithDelivery(ctx, id), set by the worker and the actions; a forbidden hand-written key in .golangci.yml (sloglint) like message_id
microb2b_delivery_transitions_total{state}persistence, per transition
microb2b_delivery_wakeups_total{source=dispatch,sweep,action,catch_up} and microb2b_delivery_wakeups_failed_totalnotifier and sweep
microb2b_delivery_claim_conflicts_total{target}worker (double wake-ups; expected small)
microb2b_deliveries{state}, microb2b_deliveries_due (gauges)sweep
microb2b_deliveries_stalled_total{target}sweep
Spansdelivery.dispatch, delivery.claim, delivery.attempt, delivery.sweep
Log levelsan idle sweep or worker is silent at INFO; a stall, a lease expiry and an operator action are INFO/WARN; a wake-up publish is DEBUG

telemetry names no module: labels are instance names and states.

9. Tests and risks

9.1 Test strategy

LevelTests
PureDeriveStatus table (every row of 1.3, precedence pairs); MatchRoutes (Any in each position, overlapping routes to one target give one target, other source, no match) checked against Route.Covers on the same fixtures; DeliverySubject refuses ., *, >, space, empty, over-long names; domain types parse/JSON; back-off schedule
Persistence (SQLite in memory, and PostgreSQL where the suite runs it)dispatch writes rows, stage, status and history in one transaction; idempotent DispatchMessage; claim CAS (two claimants, one wins; retrying not due is refused; stalled only with fromStalled); CompleteDelivery with a stale token is ErrClaimLost; SettleDelivery never downgrades; TestStoredStatusIsTheDerivedStatus; sweep steps (lease expiry, back-off, stall at the limit, alert once, alerttest, receipt deadlines); route cache invalidated by own write, by events.routes, by TTL
WorkerN deliveries run concurrently up to MaxConcurrent and never more; catch-up at start; a duplicate wake-up does one attempt; stop drains a short attempt to delivered, releases a stuck one to retrying (not failed); no attempt after Quiesce
Moduleseach target’s outcome table of 3.3; AS2 async MDN before CompleteDelivery(AwaitingAck); MDN settles via as2_messages.delivery_id; inbound accepted only after dispatch; stale claim takeover of a dispatched message is a repeat; writer’s content-equal re-attempt
End to end, real in-process NATSpoller -> AS2 sender -> AS2 receiver -> writer reaches Delivered on both sides; a route saved while running applies to the next file without restart; no route -> NotRouted; target stopped -> stalled with an alert, start -> delivered; redispatch and redeliver through REST and form (parity: same outcome); a wake-up dropped on purpose (dispatcher wrapper) is recovered by the sweep
UITestEveryLinkInATemplateIsServed, TestHTMXRequestsDeclareWhereTheAnswerGoes, style pairs, TestEndToEnd_APageOfAMissingRecordIs404 for the delivery path
Self load test (just test-load)acceptance: SIGTERM to the sending process mid-run with a backlog (-sigterm-after N), restart: every staged file is one Delivered message on A and one Delivered (written) message on B, none in Received, InProgress, Retrying or Stalled, no AS2 Message-ID received twice by B’s writer, output equal to the input; throughput scales with max_concurrent_deliveries
Interop matricesthe validators wait for messages to leave Received, InProgress, Retrying and AwaitingReceipt; inbound success is Delivered (written)
Playwrighta message page shows its deliveries; Redeliver on a failed delivery asks for confirmation; Redispatch on a NotRouted message without a new target says nothing happened, and after adding a route routes it
just test-shutdowntest/graceful-shutdown.sh checks that no delivery is failed after a stop

9.2 Risks

RiskMitigation
R1 Route cache stalenessown writes drop it synchronously; 30 s TTL for bypassing writes; redispatch adds missing deliveries
R2 Sweep load on SQLite’s single connectiontwo indexed queries per 5 s when idle, batch updates of 500 with a pause between batches; the derive read is on the unique index
R3 Double wake-ups (dispatch and sweep, catch-up and sweep)the claim CAS; the loser does nothing and counts a conflict
R4 Subject token rulesDeliverySubject refuses anything that is not one identity token; instance names are already IdentityPattern; fixed delivery. prefix
R6 Dual write (commit, then publish)publish after commit only; the sweep wakes anything whose wake-up was lost within 10 s
R7 A stopped or disabled target stalls its deliveriesthat is the alert; its next start claims them; MaxDispatches is a setting
R8 Re-attempt after an interrupted attempt sends twiceAS2: same Message-ID, the partner’s duplicate detection; writer: content-equal check; poster: the endpoint must be idempotent (documented, as for any retry)
R9 Poller crash after commit, before the file movesthe file becomes a second message; listed in the poller’s docs
R10 Out-of-order completion with concurrencydocumented; nothing in MicroB2B depends on order
R11 Cross-module check in the inbound claim takeoverthrough the core Persistence interface (GetMessage), not SQL across module tables

10. Where it lives

ConcernCode
Vocabulary: stage, state, status, outcome, refusal, wake-up, derivation, matchinginternal/persistence/{delivery.go,message_stage.go,message_status.go,route.go,settings.go,actions.go,sweep.go}
Dispatch, claims, transitions, route cache, sweep, actionsinternal/persistence/bun/{deliveries.go,persistence.go,routes_cache.go,sweep.go,actions.go}
Worker and sweep loopinternal/delivery/{worker.go,sweep.go}
Sourcesinternal/protocol/file/poller.go, internal/protocol/as2/receiver.go
Targetsinternal/protocol/as2/sender.go (+ mdn_receiver.go), internal/protocol/file/writer.go, internal/protocol/httppost/poster.go
UI and RESTinternal/api/ui/{routes_messages.go,message_actions.go,templates/message_detail.html}, internal/api/handlers/messages.go, internal/api/openapi/openapi.yaml
Graceful stopinternal/dispatcher/nats/nats.go, internal/plugin/manager.go, internal/delivery/worker.go
Acceptancetest/selftest-load (-sigterm-after), test/graceful-shutdown.sh

11. Decisions of the maintainer

  1. An instance name is unique across plugin types. Routes, messages and deliveries name an instance by its name alone, and the delivery subject is delivery.<instance> (section 4.3), so the plugin type travels nowhere. The rule is enforced where a name is chosen: saving a plugin instance (form and REST, one rule set) is refused when another plugin type holds the name, with a field error that names that type and instance. Each module lists the instance names it holds, enabled or not, and plugin.InstanceNames asks them all, so the core names no module (the same shape as the endpoint vocabulary). Seeded and fixture names carry their type (as2_receiver_default, file_writer_default).
  2. The AS2 Message-ID is per message. A delivery is a local concern; the Message-ID is globally unique and shared with the partner, so it is the message id. as2_messages keeps its key and the MDN lookup its rule; as2_messages.delivery_id (section 2.5) links the AS2 record to the delivery that sent it.
  3. awaiting_ack times out, per partner, default 24 h. The AS2 remote profile has the receipt timeout (receipt_timeout_hours, 1 to 720, validated like every field of the profile form and REST, one rule set); the AS2 sender passes the deadline when it completes the attempt as AwaitingAck, and the sweep fails the delivery at the deadline (section 5, step 3) with a reason that names the duration.
  4. A message that matches no route raises an alert. The alert.CodeTable row message_not_routed (warning, subject kind message, link to the message) is raised once by the dispatch step when it settles the message as NotRouted; the content rules apply (ids, party and instance names; no payload).
  5. Repeats and refused inbound messages have their own status and raise no alert. RefuseMessage takes a Refusal: the partner’s message turned down on its merits (a repeat or replay, a failed verification, an unknown or unmatched party, over the size limit, no signature where one is required) settles the message as Refused; a failure on our side (storage, database) is Error and raises message_failed. Refused is a message status of its own (label “Refused”, its own badge and filter value; logStateChange raises no alert for it; the dashboard counts it apart from failures; telemetry.Status knows it). The history line carries the reason.