Monitoring

This page explains how to keep an eye on MicroB2B: its logs, the health check, alerts, and the export of traces, metrics and logs to a monitoring system.

Logs

MicroB2B writes its log to standard error, as key=value text. With systemd you find it in the journal (journalctl -u microb2b). In a container, use docker logs.

Lines about a message carry its message_id. Search for it to find everything that happened to one message.

Setting	What it does	Default
`LOG_LEVEL`	`debug`, `info`, `warn` or `error`. Anything else means `info`	`info`
`LOG_STREAM_LEVEL`	Lowest level shown on the Logs page. It can be higher than `LOG_LEVEL`, not lower	`info`

At info, MicroB2B logs things that happened: starts, stops, messages, deliveries. Use debug only to look into a problem. It logs much more.

The Logs page

The Logs page of the web UI follows the log live. You can filter by level, by text and by message id. It keeps the latest 2000 lines in memory, so it shows nothing from before the last start. For older lines, use the log on the server.

Everyone who can sign in can read the Logs page. MicroB2B never logs passwords, keys or message contents.

Health check

GET /health on the web UI port tells you whether MicroB2B is up and can reach its database. It needs no sign-in. Use it for a load balancer or a monitoring probe.

When all is well, it answers 200:

{"status":"healthy","timestamp":"...","database":"up"}

When the database does not answer, it answers 503 with "status":"unhealthy" and "database":"down".

An AS2 receiver has no health check of its own. A probe can check that its port accepts connections. A GET on the receiver’s path answers 405.

Alerts

When something needs your attention, MicroB2B raises an alert. Examples are a message that failed, a message that matched no route, a certificate that is about to expire, or a part of MicroB2B that could not start. A notifier sends alerts to Slack, Teams, Discord, Google Chat, ntfy or a system of your own. See notifications.

You do not need to search the logs for these events. Set up at least one notifier.

OpenTelemetry

MicroB2B can send traces, metrics and logs to a monitoring system over OpenTelemetry (OTLP over gRPC). That can be an OpenTelemetry collector in front of Jaeger and Prometheus, or a hosted service such as Uptrace. It is off by default.

Setting	What it does	Default
`OTEL_ENABLED`	`true` turns the export on	`false`
`OTEL_EXPORTER_OTLP_ENDPOINT`	Where to send it, as `host:port`. Uses TLS	`localhost:4317`
`OTEL_EXPORTER_OTLP_INSECURE`	`true` sends without TLS, to a collector without TLS	`false`
`OTEL_EXPORTER_OTLP_HEADERS`	Headers sent with every export, as `key=value,key2=value2`. A hosted service’s access key goes here	none
`OTEL_SERVICE_NAME`	The name MicroB2B reports itself as	`microb2b`
`OTEL_SERVICE_VERSION`	The version it reports	the version of the program

For example, for Uptrace:

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=<your Uptrace endpoint>
OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your DSN>

The repository has otel-collector-config.yaml, a collector setup that sends traces to Jaeger and offers metrics to Prometheus on port 8889.

Traces

A trace shows one message’s way through MicroB2B, step by step. It starts where the message came in: at the receiver or at the file poller. Every delivery attempt joins the same trace, even when it happens hours later.

Traces never hold message contents, passwords, headers or the full address of an outgoing request. An outgoing request shows only the host it went to.

Logs in your monitoring system

With OpenTelemetry on, MicroB2B also sends its log lines, at or above LOG_STREAM_LEVEL. Each line is linked to its trace. Your monitoring system can then show a message’s log lines beside its trace. MicroB2B still writes every line to standard error as well.

Metrics

MicroB2B sends metrics about the server as a whole. No metric names a partner. Use the traces to look at one partner’s messages.

Useful metrics to watch:

Metric	What it tells you
`microb2b_messages_total`	Messages handled, by protocol and by `success` or `error`
`microb2b_message_processing_duration_seconds`	How long handling a message takes
`microb2b_message_size_bytes`	Message sizes
`microb2b_deliveries`	Deliveries in each state
`microb2b_deliveries_due`	Deliveries waiting to be sent. Stays near zero while all is well
`microb2b_deliveries_stalled_total`	Deliveries that got stuck because nothing picked them up. Alert on this
`microb2b_certificates_expiry`	Certificates that are expired or about to expire
`microb2b_inbound_requests_in_flight`	Requests a receiver is handling at the moment
`microb2b_inbound_requests_refused_total`	Requests a receiver turned away because it was busy. See AS2 receiver

MicroB2B also sends standard metrics: HTTP request times, the Go runtime, and the database connections.

Histograms use the exponential format of OpenTelemetry. A system that only reads histograms with fixed buckets cannot read them.

Reverse proxy Backup and restore