Webhooks are a mechanism for delivering alerts in which an HTTP request containing a payload representing the alert is sent by the Oxide control plane to a webhook receiver endpoint.
Webhook Receivers
Alert receivers that receive alerts via webhooks are called webhook receivers. A webhook receiver is implemented by an HTTP server which receives alert delivery requests and performs some action in response to the alert.
In order to receive alerts via webhooks, a receiver endpoint must be registered with the Oxide control plane using the webhook_receiver_create API endpoint, creating an alert receiver API resource. Multiple receivers, with different URLs, may be created, and each receiver may be subscribed to different sets of alerts.
For guidance on implementing a reliable webhook receiver, see here.
Creating a webhook receiver requires a name, the URL of the receiver endpoint, and a list of one or more secrets used to sign the alert payloads sent to the receiver. In order to receive alerts, a receiver must also subscribe to one or more alert classes, as discussed here.
Secrets
Webhook payloads are signed with a HMAC digest using secret keys shared between the webhook receiver and the Oxide control plane, to allow the receiver to verify the authenticity of the payload. In a creation request, a user submits a list of secrets that the webhook dispatcher should use for signing payloads. Each secret will be assigned an ID generated by the system that is unique within the webhook it belongs to.
Secrets cannot be changed once created, and can only be deleted. To rotate a secret, a user first creates a new secret, verifies that their application can validate the new signature, and then deletes the old secret. When a secret is deleted, the value of the shared secret is erased.
While users can ignore signature verification in their receivers, a webhook configuration must always have at least one secret.
Webhook Delivery
Webhook alerts are delivered to receiver endpoints as a HTTP POST request with a JSON body containing the alert payload. As discussed in the documentation on alert delivery, failed delivery requests may be retried multiple times, so a receiver may be sent more than one request for a given alert.
Receiver implementations must treat two webhook payloads with the same alert UUID as representing the same alert.
Two payloads with different alert UUIDs should always be treated as representing distinct alerts.
Delivery Requests
All webhook delivery requests are structured as described in this section.
Request Headers
Webhook delivery requests always include the following headers:
| Name | Value |
|---|---|
|
|
| A UUID that uniquely identifies this delivery of the alert. |
| The UUID that identifies the alert receiver resource representing the endpoint to which the alert was delivered. |
| The alert class of this alert. |
| The alert UUIDs of this alert. |
| A HMAC signature of the payload for each secret key assigned to this webhook receiver. See the next section for details on the format of this header. |
| The time at which the delivery request was sent, in RFC3 339 format. Note that this is not the time at which the event that generated the alert occurred, which may be provided in the data payload for the alert. See <<_order_of_delivery,here> for more details. |
Signature Header Format
For each secret key assigned to a webhook receiver, an x-oxide-signature header is added with the HMAC digest of the payload signed with that secret key.
The values of this header includes the algorithm used to generate the HMAC digest, the UUID of the secret key, and the value of the signature for that secret key.
This data is encoded in the following format:
x-oxide-signature: a={algorithm}&id={secret-id}&s={signature}The value of the digest is encoded in hexadecimal.
Receivers may parse this format to extract the algorithm, secret ID, and signature from each header value.
For example, if a receiver has two secrets, with IDs c06487fd-6636-435a-8ade-915b3c3b7ff5 and 72c31177-fb4d-48ad-9a33-714e02f8ab59, and the signatures for both secrets are generated using the SHA256 algorithm, the following headers would be sent:
x-oxide-signature: a=sha256&id=c06487fd-6636-435a-8ade-915b3c3b7ff5&s=d4da173d6473e40e7e0c38b9feb4dc101b362b55e8a2de936e850b8ed307badc
x-oxide-signature: a=sha256&id=72c31177-fb4d-48ad-9a33-714e02f8ab59&s=b437be7187eb33846c7839ab77e9c3ee03c0f4eee0f19aca531a7f6703f63954Currently, only the SHA256 algorithm is supported.
Request Body
All webhook delivery requests include a JSON body representing the alert. This body always contains the following fields:
| Name | Type | Description |
|---|---|---|
|
| The alert class of this alert. This is the same value as the |
|
| The alert UUID that uniquely identifies this alert. This is the same value as the |
|
| Data payload describing the alert. The schema of the data object is specific to the alert class. See here for details. |
| An object containing metadata about this alert delivery attempt. |
| Name | Type | Description |
|---|---|---|
|
| A unique identifier for this delivery attempt. This is the same value as the |
|
| The UUID of the webhook receiver. This is the same value as the |
|
| The timestamp at which the delivery request was sent, represented as a string in the RFC 3339 format. If a failed delivery attempt is retried, this is the time at which the retry attempt was sent, not the time of the first delivery attempt.
This is the same value as the |
| An enumeration describing why this delivery was created. |
| Value | Description |
|---|---|
| This delivery was triggered by a new alert being published. |
| This delivery was triggered by a liveness probe with |
| This delivery is a liveness probe. |
Data Object Schemas
The data object in a webhook delivery request contains data that is specific to the alert class.
Thus, the schema of this object depends on the alert class of the request.
If the data payload is non-empty, it will always contain a "version" field, a positive integer value indicating the version number of the data schema for this alert class.
If backwards-incompatible changes are made to the structure of the data payload for a given alert class, the version number is incremented.
Webhook receivers should be capable of handling previous schema versions.
If an alert occurs prior to an Oxide system software update, but is not successfully delivered until after the update, the data payload for that alert will have been generated before the update. If the schema for that alert class changed in the new software version, the alert’s data payload will still use the previous schema version.
Example Delivery Request
POST https://company.example HTTP/1.1
content-type: application/json
x-oxide-delivery-id: ea94de84-d0b2-4c6a-b2e9-9dde536bc79a
x-oxide-receiver-id: 8a630640-c5cf-441c-9bdd-163f0323bc29
x-oxide-timestamp: 2025-01-01T00:00:00Z
x-oxide-alert-class: project.create
x-oxide-alert-id: 202b22e5-9d4c-4843-a8ec-a20d501d9e4a
x-oxide-alert-version: 1
x-oxide-signature: a=sha256&id=c06487fd-6636-435a-8ade-915b3c3b7ff5&s=d4da173d6473e40e7e0c38b9feb4dc101b362b55e8a2de936e850b8ed307badc
x-oxide-signature: a=sha256&id=72c31177-fb4d-48ad-9a33-714e02f8ab59&s=b437be7187eb33846c7839ab77e9c3ee03c0f4eee0f19aca531a7f6703f63954{
"alert_class": "project.create",
"alert_id": "202b22e5-9d4c-4843-a8ec-a20d501d9e4a",
"data": {
"version": 1,
...
},
"delivery": {
"id": "ea94de84-d0b2-4c6a-b2e9-9dde536bc79a",
"receiver_id": "8a630640-c5cf-441c-9bdd-163f0323bc29",
"sent_at": "2025-01-01T00:00:00Z",
"trigger": "alert",
}
}| 1 | The alert class that this payload represents. |
| 2 | The alert UUID of the alert. This UUID is unique to the alert, and can be used to de-duplicate multiple deliveries of the same alert. |
| 3 | The version of the data payload schema for this alert class. |
| 4 | The data payload of the alert. |
| 5 | Metadata describing this delivery attempt for the alert. |
Failures and Delivery Retry
If a webhook alert is not successfully delivered, the dispatcher will attempt to deliver it again, in order to ensure that the receiver receives the alert.
Success
A successful delivery is one in which the dispatcher receives a response back from the receiver endpoint with a 2xx HTTP status code.
Once an alert is delivered successfully to a receiver, the webhook dispatcher will not attempt to deliver that alert to that receiver again.
Responding to a webhook delivery request with a 2xx HTTP status code acknowledges that alert, and the Oxide control plane will not attempt to deliver that alert to that receiver again.
This means that a receiver implementation should not return a 2xx HTTP status code until it has durably recorded the alert, such as by writing it to persistent storage, or otherwise ensured that any actions in response to that alert will reliably be performed.
If a receiver process receives a webhook delivery request and responds with a a 2xx status code before it has durably recorded the alert, and then crashes or otherwise loses the alert, the Oxide control plane will not attempt to redeliver that alert to that receiver.
This means that the notification is lost (unless it is explicitly resent).
Webhook delivery does not follow HTTP redirects. If a receiver responds with a 3xx HTTP status code, this is considered a failure.
Failure
Delivering a webhook notification can fail for all of the reasons that an HTTPS request to an arbitrary URL may fail. Failures fall into one of three broad categories:
Receiver Endpoint Unreachable: A TCP connection to the receiver endpoint could not be opened.
This may be because the DNS name for the receiver endpoint URL could not be resolved, the connection was refused, or no connection was established within a 10-second timeout.
Response Timeout: A connection was successfully established, but no response was received within a 30-second timeout.
Receiver HTTP Error: A connection was successfully established and a response was received, but the receiver endpoint returned a
3xx,4xx, or5xxHTTP status code.
When a webhook delivery attempt fails, it will be retried up to two times, as described in the documentation on alert delivery retries.
Order of Delivery
There is no guarantee about the order in which messages will be sent to a receiver. At any point in time, a dispatcher may have multiple in-progress requests to a receiver. Each request may succeed, fail, or retry independently, resulting in messages appearing out of order in the context of the receiver.
Webhook requests always include a timestamp header (indicating the time at which the message was sent) to allow the receiver to reconcile the order of receipt with the order of sends. This header only provides the order in which the messages were sent, and does not make any claims about the order of the underlying events. Most alert payloads will additionally include timestamps in the message body describing the time at which the event was observed.
Liveness Probes
As discussed in the alerts documentation, liveness probes are synthetic alerts sent to test whether a receiver is capable of receiving an alert.
A webhook receiver liveness probe takes the form of an alert delivery requests with the "probe" alert class and trigger.
POST https://company.example HTTP/1.1
content-type: application/json
x-oxide-delivery-id: 3b04163c-d7e3-4e1a-bcca-48bd49e502f3
x-oxide-receiver-id: 8a630640-c5cf-441c-9bdd-163f0323bc29
x-oxide-timestamp: 2025-01-01T00:00:00Z
x-oxide-alert-class: probe
x-oxide-alert-id: 001de000-7768-4000-8000-000000000001
x-oxide-alert-version: 1
x-oxide-signature: a=sha256&id=c06487fd-6636-435a-8ade-915b3c3b7ff5&s=d4da173d6473e40e7e0c38b9feb4dc101b362b55e8a2de936e850b8ed307badc
x-oxide-signature: a=sha256&id=72c31177-fb4d-48ad-9a33-714e02f8ab59&s=b437be7187eb33846c7839ab77e9c3ee03c0f4eee0f19aca531a7f6703f63954{
"alert_class": "probe",
"alert_id": "001de000-7768-4000-8000-000000000001",
"data": {},
"delivery": {
"id": "216cd071-0ec2-4b03-977e-7f1858af294d",
"receiver_id": "8a630640-c5cf-441c-9bdd-163f0323bc29",
"sent_at": "2025-01-01T00:00:00Z",
"trigger": "probe"
}
}Liveness probe requests are represented as alert deliveries to ensure that they are handled by the receiver similarly to actual alert.
This ensures that probes are routed to the same API endpoint as actual alerts. It also means that the receiver implementation need not be able to handle a separate JSON body schema, and is only required to parse the JSON body schema of actual alert payloads.
However, receiver implementations must take care to distinguish between actual alerts and probe requests. For instance, a receiver that integrates with an external alerting system should should not alert operators when it receives a probe.
The receiver endpoint must respond with a HTTP 2xx status code to indicate that it is available.
If the webhook dispatcher cannot connect to the receiver endpoint, the request times out, or the receiver responds with a 3xx, 4xx, or 5xx status code, the probe is considered to have failed, similarly to the failure criteria for alert delivery.
If it is possible for a receiver endpoint to be reachable but unable to process alerts, the receiver implementation may respond to probes with a 5xx status code to indicate that it has failed.
This can be used to report conditions where the receiver endpoint being up, but another component, such as a database to which alerts are written, is down.