Skip to content

Webhooks

Whenever the Gateway recieves status callbacks from further within the networks, a webhook event is generated. This event can be either a "Delivery Report" event or a "Mobile Origin Short Message" event, both of these will be processed by the webhook system. Which builds a JSON payload for each event type and calls the receiving API.

The formats these webhooks submit to the receiving API, is documented under "webhooks" in the OpenAPI specification of the API.

The system receiving the events has to be a webserver capable of receiving HTTP requests, and must answer with a status code between 200 and 299. The webhook system only inspects the status code, any custom headers or even the response content for the event request is ignored.

IP Range

Our webhook system sends event requests from the IP range 167.235.80.128/28.

Validation

The contents of the webhook is signed with a validation key, and the receiving API should carefully validate every delivered event.

Key Encoding

Many implementations of HMAC signature signing and signature validation expects the key plain bytes, but plain bytes can be tricky to transport via higher level protocols and even still tricky to insert into system configuration, be it files or environment variables.

Because of that, the key received as along the API credentials is in hexadecimal representation and might need to be decoded into bytes or even some other format depending on your HMAC library of choice.

The signature of the request is in the Signature header, and it contains two pieces of information: the time the request was signed and the signatures of the current webhook signing keys. The header will most often only have a single signature, which leads to the following format:

t={unix_timestamp},v1={hex_signature}

If multiple signing keys are in use the format would look more like this:

t={unix_timestamp},v1={hex_signature},v1={hex_signature}

The most relevant headers of a webhook event request sent at 2023-03-14T10:55:10.402+00:00 could look like:

POST /event/accept http/1.1
Host: example.com
Signature: t=1678787710,v1=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The timestamp and signatures needs to be extracted from the header, and used later on in the request validation. It is recommended to split on , followed by = on each element, to convert the header value into a bunch of key-value pairs, this is more flexible in case we want/need to introduce additional fields to the signature, such as a new signing algorithm.

The signed data is a bytestring with the format {unix_timestamp}.{request.content}. Where the unix timestamp is the same timestamp as was included in the signature. Every time an event is being processed for sending, a new timestamp and signature is created.

Furthermore the timestamp can be used to reject replay attacks, as the timestamp is a part of the signed data, it cannot be altered without invalidating the whole request. If a request with a timestamp too far in the past, it can safely be rejected with a non 2xx status code, which will schedule any legitimate calls for retry later.

Syncronize your clocks, gentlemen!

Like with anything that needs to happen in a coordinated and time sensitive manner, all parties of the arrangement needs to be in agreement on what the time is. And like in the real world, servers have the same problem where clocks can drift or just plainly be incorrectly set. This would lead to valid events being rejected - only because the servers did not agree what time it is.

To avoid this, we recommend to regulary synchronize the time of your servers using an automated system such as NTP, using one of the many daemons that support it.

The specific algorithm used is HMAC, using SHA256. Below is a reduced example of how to validate the signature, but without some error handling to make the important logic as clear as possible.

import hmac
import time

class Request:
    """Barebones request similar to what many web frameworks expose."""

    content: bytes
    headers: dict[str, str]

def extract_timestamp_and_signatures(header: str) -> tuple[int, list[str]]:
    """Slice and dice the header into the fields we need."""
    pairs = [
        pair.split("=", maxsplit=1)
        for pair in header.split(",")
    ]
    timestamps = [item[1] for item in pairs if item[0] == "t"]
    signatures = [item[1] for item in pairs if item[0] == "v1"]

    return int(timestamps[0]), signatures

def validate_request(request: Request, key_as_hex: str) -> None:
    """Raise errors if validation fails."""
    signed_at, signatures = extract_timestamp_and_signatures(
        request.headers["Signature"],
    )

    now = int(time.time())
    if signed_at < now - 60:
        raise ValueError("signed too long ago")

    expected_signature = hmac.digest(
        bytes.fromhex(key_as_hex),
        b"%i.%b" % (signed_at, request.content),
        "SHA256",
    )

    if not any(
        hmac.compare_digest(expected_signature, bytes.fromhex(sig))
        for sig in signatures
    ):
        raise ValueError("no valid signature")

HTTP Version

The webhook system is able to call either HTTP/1.1 or HTTP/2 APIs. This allows the webhook system to benefit from the advantages that HTTP/2 has compared to HTTP/1.1 where available, but still allow delivery of events to systems that favour the reliability and trust HTTP/1.1 has.

Limitations

As the webhook system is designed to allow a large throughput, there are certain requirements to the receiving APIs that they must follow.

Status Codes

The recieving API has to respond with a status code in the successful status code range, between 200 and 299, when accepting the event, any other status code will be treated as a failed delivery and the event will be scheduled for retry at a later time.

Timeouts

The recieving API has 2.5 seconds to respond to the webhook system, otherwise the request times out, and the event will be scheduled for retry at a later time. These 5 seconds is measured from the time the webhook system sends the request, until a response is received - if an event is written, received, processed, but the response is in transit when the 5 seconds have passed, the event is still failed and will be retried later on.

From the point of the recieving API, it is almost impossible to know how long it took to receive the event request and therefore how long it has left to complete the processing. As a result of that, the safest solution is to design the webhook receiving API to do as little work as possible, to widen the timing tolerance and minimize the number of events that fail due to a timeout.

The least work an API can do is often initial validation of the event, but once that is done the event can be submitted for background processing, making the API available to accept new events.

What Queue ?

Depending on the volume of events, an appropriate solution might be anything from a list structure in redis, to a table in a SQL database, and for high volume systems, a dedicated message queue.

If in doubt, we can recommend to measure how long it took to process an event, and evaluate if/how that changes with volume.

Redirects

Redirect responses will not be followed, and any such response will be treated the same as any other status code outside the successful status code range, meaning that it will be interpreted as a failed delivery and the event will be scheduled for retry at a later time.

Retries

It would be unreasonable to expect 100% uptime for the receiving API. Therefore the system includes a retry mechanism, where a failed event delivery attempt can be retried again later.

Webhook events are retried with an exponential but also randomized backoff, to avoid clustering of retries started at around the same time. Retries are scheduled more often in the beginning, but as the number of attempts increase they become less frequent.

After at least a day and at least a certain number of delivery attempts, events can be dropped from further retries.