Blog | Tristan Kernan

“That some of us should venture to embark on a synthesis of facts and theories, albeit with second-hand and incomplete knowledge of some of them – and at the risk of making fools of ourselves” (Erwin Schrödinger)

Django: Simple Streaming / SSE with Mercure

I've written before about streaming responses / sse in Django, but I wanted to share a different, and possibly simpler approach here. I recently launched a live map for Simple Routing, my SaaS product that provides logistics routing APIs, having been inspired by Github's globe visualization of commits from around the world.

Simple Routing is a standard Django application, where I've chosen not to integrate async features. This is primarily for architectural simplicity, because Django is traditionally a sync platform and async features still feel awkward and bolted on.

To build a soft real time service, then, I had a dilemma: build a pseudo-live stream with polling, or integrate some tooling to which to delegate the streaming. I didn't rule out the first, but I wanted to at least try the latter. After researching the ecosystem, I decided on mercure, a self-hosted sse multiplexing server written in Go that much more efficiently handles multiple streaming connections than poor synchronous wsgi + gunicorn. The architecture is thus:

flowchart TD API1[API Client 1] API2[API Client 2] subgraph internal["Intranet"] Django[Django Webserver] Redis[(Redis)] Celery[Celery Worker] Mercure[Mercure Hub] end Client1[Live Map Client 1] Client2[Live Map Client 2] API1 -->|HTTP request| Django API2 -->|HTTP request| Django Django -->|push country - rpush+rtrim| Redis Celery -->|poll - lpop| Redis Celery -->|push events| Mercure Mercure -->|SSE push| Client1 Mercure -->|SSE push| Client2

While the webserver could push events directly to mercure, I decided against that for a couple reasons. One, I wanted to limit the impact to the hot path of API client requests: a redis push to an existing connection is significantly cheaper than an http request. Second, I wanted a middleware to throttle events: in case of high volume, clients should see a reduced set of events.

Having celery already set up, I created a simple polling task that pulls from the redis list and pushes to mercure:

Python
@shared_task(queue="live_map")
def publish_live_map_batch():
    for _ in range(20):
        time.sleep(1)

        if country := live_map.pop_next_country():
            live_map.publish_country(country)

To avoid latency with every-second celery tasks, each task runs for 20 seconds, and is then queued up again by beat.

On the client side, it's just event stream processing: EventSource (or htmx-sse 1).

End to end I feel this is a performant yet simple approach: Django does what it does best with synchronous flow and gets live streaming bolted on top, without the awkwardness of async and function colors.

  1. Mercure does not work out of the box cross-domain with htmx sse in v4. This is due to htmx sending its custom headers in Access-Control-Request-Headers, which will cause mercure to silently reject the request by not sending the appropriate cors headers, i.e. Access-Control-Allow-Origin. I went with EventSource to avoid the issue for now. ↩