# Why at-least-once delivery is a product decision

Canonical: https://stevenmoriasi.com/blog/event-driven-automation-engine
By [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/).
Source: https://github.com/steven-moriasi/event-driven-automation-engine
Date: 2025-07-29

## Context

Event-driven automation looks elegant until a webhook is accepted and the enqueue fails. The event is gone. The business thinks it happened. A tool-calling agent has the same problem: a tool call is an outbound event.

The [Event-Driven Automation Engine](https://github.com/steven-moriasi/event-driven-automation-engine) closes that window with a relational inbox.

## Decision

A signed webhook becomes a canonical envelope, persisted and fanned out on the same durable path.

signed request → persist envelope → fan-out → leased delivery → success or dead letter

Duplicate detection uses `(source, external_id)` plus a payload hash so a retry is not confused with a conflict. Sequenced events keep a checkpoint per subscription and subject. Retryable failures back off; the rest become dead letters an operator can replay. Provider bodies are not stored.

## What I rejected

Accept-then-enqueue in the HTTP handler, and storing raw provider payloads “for debug.” The first loses events. The second leaks tokens and personal data into the inbox.

## Failure scenario

The process dies after HTTP 200 and before the queue write. On the durable path there is nothing to lose: the envelope already exists. A duplicate POST matches `(source, external_id)` and does not create a second business event.

## Evidence

FastAPI ingestion, PostgreSQL envelope store, delivery worker tests, and operator replay of exhausted letters. GitHub- and Slack-shaped adapters exercise contracts; they are not production tenancy.

Inspect: [Repository](https://github.com/steven-moriasi/event-driven-automation-engine) · [Run locally](https://github.com/steven-moriasi/event-driven-automation-engine#readme)

_—by [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/)_