Admin Alerts

UTM writes structured error rows to the audit log whenever a critical trading function fails. The audit log is the forensic record. Admin alerts are the page: when an error row of a known shape lands, UTM emails the admin recipient list and posts a webhook so an operator hears about the problem before the next daily audit.

This page describes what fires an alert, how to configure the recipients, and how the dedup window stops a burst of failures from paging repeatedly.

When an alert fires

There are eleven alert types. Each alert carries a type, severity (HIGH or CRITICAL), a one-line title, a longer message, and a details object with the entity ids relevant to the failure.

Type	Severity	Trigger
AUTO_CLOSE_FAILURE	CRITICAL	The auto-close worker could not place a close order for a tracked trade at market close, or the post-close discrepancy check found a position that did not flatten after the settle window.
AUTO_CLOSE_UNRESOLVED_CLOSE_ORDER	CRITICAL	Auto-close found a live close-side order on the trade that it cannot safely act on: reason: "placeholder_id" (the order's broker id, or its status, is still a UTM placeholder, not confirmed live at the broker) or reason: "unrecognised_status" (the raw status is still unrecognised after the read-back refreshed it). No second close order is placed for that trade.
AUTO_CLOSE_UNPROVEN_ORDER	CRITICAL	Auto-close could not read back one or more orders at the broker before closing (the entries it had just tried to cancel, or a live close-side order). The close went ahead, sized from UTM's own fills; a trade with no fills got this alert only. Check the named orders at the broker.
SYNC_FAILURE	HIGH	The periodic sync worker failed an entire user iteration (not a transient single-account flake).
ORDER_FAILURE	HIGH	A non-retryable broker rejection landed an order in FAILED. Normal CANCELLED and EXPIRED order lifecycle does not trigger this. The live trading guard also sends it, as CRITICAL, when it refuses a close on a live account (LIVE_TRADING off, or windowed with the window closed): the position may still be open. Every refused close also logs an error with event set to live_trading.refused_close.
QUEUED_ORDER_LOST	HIGH	A queued order is older than a minute with no order-queue job under its id, or a processing order's job was failed by bullmq's stall path. UTM only reports it: it does not re-create the job, write the order or call the broker. Sent once per order. Dedups on the order id.
CREDENTIAL_ERROR	HIGH	A broker account transitioned to credentialStatus=invalid after the circuit-breaker threshold (three consecutive 401s).
ORDER_OUTCOME_UNKNOWN	CRITICAL or HIGH	Placing an order gave no clear answer, so the order may be live at the broker. It was left pending and flagged, and the trade was not failed. Dedups on the order id. The order lookup also pages under this type: see Order outcome pages.
ORDER_OUTCOME_RESOLVED	HIGH or CRITICAL	An order flagged by ORDER_OUTCOME_UNKNOWN was confirmed, and the flag was cleared. details.resolvedBy says which path confirmed it (lookup, sync or placement), and details.status is the broker's reported status. Dedups on the order id, separately from ORDER_OUTCOME_UNKNOWN. See Order outcome pages.
ORDER_OUTCOME_CONFLICT	CRITICAL	The placement's own late answer was a refusal, but the order lookup or sync had already confirmed the order from the broker's report. UTM keeps the broker's report and does not fail the trade; a person checks the order at the broker. details.brokerOrderId and details.observedStatus are what the broker reported, and details.refusal is the placement's answer. Dedups on the order id. It does not fire when the broker reports the order rejected too: the two answers agree, and the placement records and returns the refusal.
SETTINGS_UNAVAILABLE	CRITICAL	The trading.halt setting could not be read when an order that adds risk was placed, so UTM refused the order. Closes are still placed. Sends only when the alert settings can be read. Every refusal also logs an error with event set to settings.unavailable.

The details object is always populated with the relevant ids (tradeId, strategyId, accountId, brokerId, userId, orderId) so a webhook recipient can deep-link straight to the affected entity. AUTO_CLOSE_UNRESOLVED_CLOSE_ORDER also carries symbol, brokerType, rawStatus, and reason, and dedups on the trade id, separately from AUTO_CLOSE_FAILURE's own dedup window for the same trade.

Order outcome pages

Every order placement opens a guard row: an order.lookup scheduled task in the same transaction as the pending order. When the outcome is unknown, or the process stops before it records one, the task runner looks the order up at the broker: by client_order_id on Alpaca, by broker order id or by its attributes on TradeStation. A broker that cannot find the order never makes it rejected. The order stays unconfirmed, and the lookup tries again 15 seconds, 1 minute, 5 minutes and 15 minutes later, then hourly, for 24 hours.

Event	Type, severity	Dedups on	User notice
The placement gave no clear answer	ORDER_OUTCOME_UNKNOWN, CRITICAL	order id	one "Order status unconfirmed" notice
The lookup found a placement whose outcome was never recorded (the process stopped between writing the pending order and the broker's answer) and flagged it	ORDER_OUTCOME_UNKNOWN, CRITICAL	order id	one "Order status unconfirmed" notice
The lookup found the order and linked it. details.matchedBy is client_order_id, client_order_id_list, broker_order_id or heuristic (a TradeStation attribute match)	ORDER_OUTCOME_RESOLVED, HIGH, resolvedBy: "lookup"	order id	none
The placement's own late answer (its stamp or a rejection) cleared a flag the lookup set. On a stamp folded into a row sync had already created, details.survivorOrderId names that row	ORDER_OUTCOME_RESOLVED, HIGH, resolvedBy: "placement"	order id	none
The placement's own late answer was a refusal after the lookup or sync had already confirmed the order. The order keeps the broker's status, fills and broker order id, the refusal is added to its history, and the trade is not failed. The placement ends as unconfirmed, not rejected, so the order is treated as possibly live	ORDER_OUTCOME_CONFLICT, CRITICAL	order id	none: the order already had its one "Order status unconfirmed" notice
The placement's own late answer was a refusal, and the order the lookup or sync confirmed is itself rejected (TradeStation records a refused placement as a REJ order, and the attribute match can link it). The two answers agree: the refusal is added to the order's history, its rejection reason is recorded, it keeps its broker order id and status, and the placement ends as rejected	none	no page	the placement's usual rejection handling, as for any refused order
The lookup ended at once: two or more broker orders could be the placement, the broker order is already held by another UTM order, or an Alpaca order has no client_order_id. The task's lastError names every id	ORDER_OUTCOME_UNKNOWN, CRITICAL	task id	one "Order status unconfirmed" notice
The lookup gave up after 24 hours (or 60 attempts) without finding the order	ORDER_OUTCOME_UNKNOWN, HIGH	task id	one "Order status unconfirmed" notice, only while the order is still unconfirmed
Alpaca sync matched the order by its key while its lookup was still running	ORDER_OUTCOME_RESOLVED, HIGH, resolvedBy: "sync"	order id	none
Alpaca sync matched the order by its key after its lookup had ended, and attached it	ORDER_OUTCOME_RESOLVED, CRITICAL, resolvedBy: "sync"	order id	none
A sync found a broker order matching an unconfirmed order whose lookup had ended, and did not link or adopt it (TradeStation's attribute match, or an Alpaca key match whose broker order another UTM order holds)	ORDER_OUTCOME_UNKNOWN, CRITICAL	the order and broker order pair; sent once per pair, recorded by a lookup.ended_match audit row	none

Each lookup run writes an order.lookup_run audit row (the outcome, and for each broker list it read, the order count and how fresh the list was), so the daily audit can measure how long each broker takes to show an order. A task run that does not settle within 4 minutes is ended and retried, and logs scheduled_task.run_timed_out.

Configuration

Recipients and the webhook URL live in SystemSetting so they can be changed from the admin Monitoring UI without a redeploy. They are not environment variables.

Three keys drive the system:

Key	Type	Default	Purpose
admin_alert.enabled	boolean	false	Master switch. When false, dispatch is a silent no-op regardless of the values below.
admin_alert.email_recipients	string	""	Comma-separated email addresses. Leave empty to disable email dispatch.
admin_alert.webhook_url	string	""	Single webhook URL. Slack, Discord, and PagerDuty all accept the JSON body that UTM posts. Leave empty to disable webhook dispatch.

Set the recipients first, then flip admin_alert.enabled=true. The default false prevents the system from blasting an unconfigured mailbox the moment the migration runs.

Email dispatch

Email goes through the same notification email path used elsewhere in UTM (SMTP or Resend, whichever is configured under Email Configuration). Each recipient gets one email per alert. Subject prefix is [UTM CRITICAL] or [UTM HIGH] followed by the alert type and title.

Webhook dispatch

Body is a JSON object:

{
  "type": "AUTO_CLOSE_FAILURE",
  "severity": "CRITICAL",
  "title": "Auto-close failed: AAPL",
  "message": "Worker could not place close order for trade ...",
  "details": {
    "tradeId": "...",
    "strategyId": "...",
    "errorMessage": "..."
  },
  "timestamp": "2026-05-19T20:00:00.000Z"
}

A 5xx response triggers one retry. A 4xx response is treated as non-retryable. When both attempts fail (or a 4xx lands), UTM writes an alert.webhook_failed row to the audit log at severity=error with the full payload so the alert is never fully lost.

Send test email

The Admin Alerts card has a "Send test email" button next to the email recipients field. It exercises the same email path a live page uses, so you can confirm deliverability for a specific address without waiting for a real failure to fire.

• Type a single address into the recipients box and click "Send test email" to target just that address. Leave the box on the saved list (or type a comma-separated list and save it first) to test every configured recipient.
• The result shows per recipient: delivered with the provider message id, or failed with the provider error. The message id is the handle you use to look the send up in the Resend dashboard when chasing a provider-side drop.
• Every attempt persists one alert.test_email row to the audit log: severity=info on success, severity=error on failure, with the recipient, message id, and error in metadata. This is the in-product delivery evidence, so a one-off miss is reproducible after the fact.

The endpoint is POST /api/v1/admin/alerts/test and requires the admin:write scope. The body is an optional { "email": "..." }; when omitted the configured admin_alert.email_recipients list is used. The response is a recipients array of per-recipient results. Email delivery uses the active provider (SMTP or Resend) configured under Email Configuration, so configure a provider first.

Dispatched audit row

Every successful dispatch writes one alert.dispatched row to the audit log at severity=error under category=admin. This is the error-tier trail the daily audit grades, so an out-of-band page is no longer invisible to the audit's error sections. The row carries the alert type, severity, title, dedupKey, and full details in its metadata, and links the affected userId when one is present. The write has its own guard, so a logging failure can never break the page itself.

Naming the affected user

When the details carry a userId, UTM resolves it to an email and folds an ownerEmail field into the details before dispatch, so the email body, the webhook payload, and the dispatched audit row all name the person rather than showing a bare UUID. The lookup is best-effort: a missing user or a database blip leaves the alert untouched and the page still fires.

Dedup

The same (type, entityId) is suppressed for 15 minutes after a successful dispatch. Practically, a single stuck trade pages once at 4:01 pm and not 100 times as the auto-close worker keeps retrying.

Dedup state is per-process and in-memory by design. A process restart clears the window, which is desirable: a restart often means an operator is already paying attention, and one extra page on restart-with-still-stuck-state is safer than swallowing the signal.

Disabled or unconfigured

When admin_alert.enabled=false or both email_recipients and webhook_url are empty, dispatch is a no-op. UTM writes one alert.skipped row to the audit log per process so the silence is visible during a triage. Repeated skips in the same process do not re-emit the row.

Reading the audit log

Both the dispatch path and the fallbacks land in the audit log under category=admin. Useful filters:

Action	Meaning
alert.dispatched	An alert was sent. Written at severity=error with the full alert metadata.
alert.skipped	The system was disabled or unconfigured when an alert tried to dispatch.
alert.webhook_failed	The webhook failed both attempts. Body is in metadata.
alert.test_email	A test email was sent from the admin tool. info on success, error on failure; recipient, message id, and error are in metadata.

What this is not

This system is the operator page for trading-critical failures. It is not a replacement for:

• Per-user notifications (those still fire to the affected user via the in-app and push channels).
• Production observability (Sentry, OpenTelemetry traces, log aggregation).
• An on-call rotation. The webhook URL is a single endpoint; PagerDuty or similar handles rotation downstream.