Dispute evidence

Status: proposal. Nothing in this note is built yet. It describes a feature that answers one question in a dispute with a trading partner: “this document: did it go through MicroB2B, when, between whom, and can we prove what the other side signed for?”

1. The problem

MicroB2B is not where business documents end up. An inbound invoice is handed to an ERP or accounting system by a file writer or HTTP poster, and an outbound one comes from such a system through a file poller. Those systems keep the document, sometimes re-saved or converted, and nothing else: no AS2 Message-ID, no signature, no MIC, no MDN.

When a partner disputes a document (“we never received that invoice”, “the amount we received was different”, “we never sent you that order”), the operator has the document from the back office and needs MicroB2B to:

1. find the transaction that carried it, starting from the document alone;
2. show the protocol evidence of that transaction: who signed what, which MIC the receipt confirmed, with which certificates, and when;
3. verify that evidence again, now, from what is stored;
4. hand over that evidence in a form a third party (the partner, an auditor, a court) can check without trusting MicroB2B.

Each protocol has its own non-repudiation mechanism (AS2: signed MDN with Received-Content-MIC; AS4: a receipt with NonRepudiationInformation, a digest per payload part; others will follow), so the feature has a protocol-neutral core and a verifier contributed by each module, as with message panels.

Evidence is not a second store. It is the payload stages a message already has, a few stages it does not have yet, a digest on each, and the certificates its records reference. Payload retention decides per stage what is kept, so “evidence” is simply the stages that are kept for the evidence period.

2. What is kept today, and what is missing

The consequences:

• No lookup by content. A document can only be found by message id, party and date.
• The MIC cannot be recomputed from the document. The AS2 MIC is computed over the signed MIME entity (its MIME headers plus the canonicalised body), not over the bare file. Without that entity the stored MIC can be shown but not checked against anything.
• Inbound re-verification depends on our decryption key, which is rotated and may be deleted.
• Outbound has nothing on the wire to show: the MDN confirms a MIC over bytes that are gone, so we cannot show that the MIC belongs to the document in hand.
• A deleted certificate makes a signature unverifiable, years before the document’s legal retention ends (7 years for invoices in the Netherlands).

3. Design

3.1 Two new payload stages

persistence.PayloadStage stays a closed, protocol-neutral set: a stage says what the bytes are, never which protocol produced them. Two stages join it:

Neither exists for an unsigned message or one without a receipt; the verifier reports what is missing (§3.5).

• signed is the same thing for every signed message whatever the order and number of its other layers (compression before or after signing, encryption or not), which is why it can be a stage where an arbitrary “intermediate layer” could not. AS4 has the same notion (the signed parts of the user message).
• receipt exists: the MDN is stored and recorded like every other stage (storage.Payloads.Store), offered by the message page’s stage selector and GET /api/v1/messages/{id}/payload. mdn_recorded, mdn_signed and mdn_opened stay on as2_messages; they describe the MDN, not its storage, and RecordMDN is the claim that makes the first MDN the one stored. No target can be configured to deliver it (PayloadStage.Deliverable).
• received and headers are also recorded by the sender, for the attempt the receipt settled, so both directions keep the wire form. Their documentation says “the request as it went over the wire” instead of “as it arrived”.
• The outbound sender keeps the attempt’s signed, received and headers in memory (or as blobs named per attempt) until the delivery is settled, and records only the settled attempt’s as stages, so a message with retries does not accumulate copies.

3.2 A digest on every payload record

message_payloads gains digest (hex SHA-256 of the blob), computed while the blob is written by Payloads.Insert and Payloads.Store and never recomputed. That serves three purposes:

1. Lookup by content is a query on the unpacked stage’s digest, indexed on (stage, digest).
2. Integrity of storage: the database vouches for the blob. A verification reads the blob, hashes it and compares, so a blob changed in the object store or on disk is detected.
3. The chain (§3.7) covers digests, not blobs.

Normalised digests for documents the back office re-saved go in one more table, message_digests (message_id, kind, digest), written with the unpacked stage:

DigestKind is a domain string with a canonical slice and ParseDigestKind. The exact digest is not a kind there: it is the stage’s own.

3.3 Lookup

A lookup computes the exact digest and every normalised kind the uploaded document qualifies for and returns the messages matching any of them, labelled with the strongest match:

• identical: the unpacked stage’s digest matched; the document is the payload byte for byte.
• equivalent: only a normalised kind matched; the content is the same after normalisation, and the report says which normalisation.

A lookup goes by content only. MicroB2B does not interpret payloads, so it extracts no business identifiers (invoice or order numbers) from them: a document the back office only re-keyed, such as a PDF rendering of an EDI invoice, matches no digest, and finding its transaction by party and date is the operator’s job on the message list.

UI: Messages > Find by document (/ui/evidence), a certificate-style upload form (multipart, X-Requested-With, its own size limit) answering with the matches, each linking to its message page. REST: POST /api/v1/evidence/lookup (multipart) and POST /api/v1/evidence/lookup-digest (JSON {kind, digest}, so a back office can look up by hash without uploading the document).

The uploaded document is never stored. A lookup is recorded in the audit log with the document’s digest, since a dispute investigation is something an auditor will want to see.

3.4 Certificates: references, not copies

A signature is verified against the certificate that signed it, years later. Instead of copying certificates into every message, the protocol record names the certificates the exchange used, and the existing reference mechanism keeps them:

• as2_messages gains the fingerprints of the certificates of the exchange: the message signer (signer_fingerprint) and the receipt signer (receipt_signer_fingerprint). The fingerprint, not the id, because it identifies the certificate itself.
• AS2’s plugin.ModulePersistence.Certificates reports a certificate as used by the messages that name it while those messages’ evidence stages are kept. DeleteCertificate refuses it (CertificateInUseError) and the certificate page’s Used by says “N messages (evidence until )”.
• A certificate replaced in a profile stays in certificates, referenced by the messages it signed, until their evidence expires; then it is deletable like any unused certificate. The certificate list gets a filter for certificates no profile uses (“retired”).

One copy per certificate, and the reason it cannot be deleted is shown where the operator looks.

3.5 Verification

The evidence of a message is a view, assembled when asked for: the message and its history, the protocol record, the stages it holds with their digests, and the referenced certificates. Nothing is written for it.

A module contributes an evidence.Verifier through module.Module (like webui.Extension.MessagePanel); the core assembles the view and calls the verifier of the message’s protocol:

type Verifier interface {
    Verify(ctx context.Context, evidence Evidence) (Report, error)
}

type Report struct {
    Checks []Check // ordered; each {Name, Outcome (passed|failed|not_applicable), Detail}
}

The core first checks every stage’s blob against its recorded digest. AS2’s checks, in order:

1. Payload in entity: the unpacked stage is the content of the signed stage (after its content-transfer-encoding and decompression).
2. Message signature: the signed stage verifies against the certificate signer_fingerprint names; the signing time and the certificate’s validity at that time.
3. MIC: the MIC recomputed over the signed stage with the stored algorithm equals the stored MIC.
4. Receipt signature: the receipt stage verifies against the certificate receipt_signer_fingerprint names.
5. Receipt content: the MDN’s Original-Message-ID names this message, its disposition is processed, and its Received-Content-MIC equals the MIC of step 3.
6. Chain: the message’s chain entry is intact (§3.7).

Verification reads only the message’s own records, stages and the certificates they name. It never reads the live profile and never needs our private keys.

The message page gets an Evidence section (the stages held, the report on demand) for every message, so the lookup is only one way to reach it.

3.6 Retention per stage

Retention is one mechanism for payloads, with a period per stage, not a separate store for evidence:

A prune deletes the blob and its message_payloads row together, writes a history line on the message (the stages removed), and never touches a message that is not settled. The message row, its history and its protocol record are kept with the longest-kept stage. The Settings page gets the two periods in a “Retention” section.

That is the answer to “what is the point of purging payload if we keep a copy”: there is no copy. Purging removes the stages that are not evidence and leaves the ones that are.

3.7 Integrity and the bundle

• Tamper evidence. When a message settles, a chain entry is written over its evidence: hash = HMAC-SHA256(key, prev_hash || Encode(message id, protocol, stage digests, certificate fingerprints, MICs)), in a table evidence_chain numbered like the audit log. Pruning the short-lived stages does not break the chain because they are not in it. Key: the same AUDIT_KEY or a separate EVIDENCE_KEY (open question).
• Trusted time (optional). A configured RFC 3161 timestamp authority stamps each chain entry’s hash (or one hash per batch, to bound cost). Without it, times are MicroB2B’s own clock, and the report says so.
• Bundle. The partner and an auditor will not trust our UI, so the evidence is exportable as a ZIP (GET /ui/messages/{id}/evidence.zip, GET /api/v1/messages/{id}/evidence), compiled from the view:

README.txt               what each file is and the openssl commands to check it
report.json              the verifier's report, the message's metadata and history
report.pdf               the same, readable
payload/<original name>  the unpacked stage
signed.eml               the signed stage
receipt.eml              the receipt stage
certificates/*.pem       the certificates the record names
chain.json               the chain entry, its predecessor's hash and the anchor
timestamp.tsr            the RFC 3161 token, when there is one

The README’s commands (openssl cms -verify, openssl dgst) let someone without MicroB2B check signatures and MIC themselves. Bundle generation is a pure function of the view, golden-tested per protocol.

3.8 AS4 and others

Nothing in the core is AS2-specific. An AS4 module records signed (the signed parts of the user message), receipt (its receipt) and the certificate fingerprints, and its verifier checks each part’s digest in the receipt against the payload and the receipt’s signature. Stages, digests, retention, the chain, the lookup and the bundle are shared.

4. Phases

1. Digests and lookup: message_payloads.digest, message_digests, the lookup page and REST. Immediately useful: it answers “did this pass through us, when, and what happened to it” with the message page as the result.
2. Evidence stages: signed (receipt exists, AS2 both directions), wire stages on send, certificate fingerprints on as2_messages and the reference rule that keeps those certificates.
3. Verification and bundle: evidence.Verifier, AS2’s checks, the Evidence section on the message page, the bundle export.
4. Retention and hardening: per-stage retention with the two periods, the evidence chain, optional RFC 3161.

Retention comes last on purpose: until evidence stages exist, a prune could only delete everything or nothing.

5. Open questions

• Is the evidence chain keyed by AUDIT_KEY or by its own key? One key is simpler to operate; two keep a leak of one from compromising both logs.
• Should the lookup be scoped (a user sees only some partners’ evidence)? Today every authenticated user is an administrator, so not until authorization exists.
• Is the evidence period global or per organization (different jurisdictions)?
• Bundle format: is a ZIP with a README enough, or do partners expect a specific format (e.g. an ASiC-E container with an XAdES signature by us over the bundle)?
• Is the settled attempt’s signed stage enough outbound? The other attempts matter only when a partner claims to have received an attempt we consider failed; keeping them would mean stages per attempt.
• Does unpacked need the evidence period when signed holds it (decompressed and stripped of its signature)? Keeping it makes the lookup and check 1 simple at the cost of one extra copy per message.