Blog | Tristan Kernan

For a recent project, I decided to go all-in on async Django in order to get a feel for its advances in recent years. I've long been inspired by the infamous "Erlang replaces your entire stack" slide:

And for this project, I wanted to see what I could replicate with Django and async Python.

Project

How do I share text snippets (long links, code snippets, log lines, etc.) between phone, laptop, and desktop? To this day, I rely on email and whatsapp, messaging myself. To improve this, I created SimplePasteService (source code) - a website for quickly sharing text content between devices without long random ids nor the email login bootstrapping problem. For fun, I added live updates: two devices on the same paste will see new text updates stream in.

The project requirements demanded two technical solutions. First, as pastes expire, they should be deleted in a recurring background task. Second, cross-client communication is necessary for live updates.

For a typical sync Django project, I'd solve the former with celery and celery beat. This adds two additional processes to run and manage, in addition to the message broker and result store. For the latter, I'd lean on a pubsub server, possibly a SaaS solution. The complexity jumps for each feature are out of proportion, in my view.

What I'd like is to keep complexity low. Could I replace my stack with asyncio?

Architecture

The short answer is yes, but with caveats. The core limitation is that async Python is a single process, and single threaded, as opposed to the Erlang/BEAM, which runs across all cores, and can be extended to run across multiple machines. As this is a hobby project, I made the decision early to accept this limitation: I'd deploy a single process with a single thread to run the webserver, pubsub, cron, and background tasks. This is not a scalable solution, clearly, but for a low-traffic hobby project, the reduced complexity is a compelling benefit.

Before diving into these components, one more architectural decision had to be made. For live streaming updates, I decided on server sent events, a simple push-only protocol that works mostly out of the box. To implement, I leveraged htmx's sse extension for the frontend, and Django's StreamingHttpResponse on the backend.

Webserver

Django comes out of the box with asgi support; all that's needed is a compatible webserver. I chose the popular uvicorn. For development, I used the following invocation:

uvicorn simplepasteservice.asgi:application --port 8000 \
  --reload \
  --reload-include "*.html" \
  --timeout-graceful-shutdown 1

Note --timeout-graceful-shutdown: I noticed when developing with long lived sse connections that the server would not automatically restart until the connections closed. This parameter will close those connections automatically after the grace period. I also added html files to the reloader, so that template updates would automatically refresh.

Background tasks and cron

To replace celery and beat, I decided on apscheduler, which has an asyncio-friendly scheduler, and is easy to set up. I updated asgi.py to add:

scheduler = AsyncIOScheduler()

# also possible to use string "simplepasteservice.tasks:delete_expired_pastes"
scheduler.add_job(delete_expired_pastes, trigger=IntervalTrigger(hours=1))

scheduler.start()

And with just these few lines, I had my desired recurring background task. One-off immediate-run tasks need no special setup when queued from an async view, look at asyncio.create_task() - but I didn't have a need for such tasks in this project.

Pubsub

The basic feature here was: when a paste is updated with new content, ping the clients subscribed. As the application is just a single event loop, I decided to build a trivial pubsub server within the application itself. The core data structure is a dictionary mapping pastes to subscriber queues:

paste_update_listeners = defaultdict(list)

...

async def view_paste_sse(request, key):
    """Creates and listens on an update queue, pinging client on updates"""
    global paste_update_listeners

    # ensure paste exists
    paste = await aget_object_or_404(Paste.objects.unexpired(), key=key)

    q = asyncio.Queue()
    paste_update_listeners[key].append(q)

    async def event_stream():
        try:
            while True:
                await q.get()

                # update occurred!
                yield "event: update\ndata:\n\n"

        except asyncio.CancelledError:
            # connection closed, clean up
            paste_update_listeners[key].remove(q)
            q.shutdown()

    return StreamingHttpResponse(
        event_stream(),
        content_type="text/event-stream",
    )

When pastes are updated, all clients are notified:

async def create_paste_item(request, key):
    global paste_update_listeners

    paste = await aget_object_or_404(Paste.objects.unexpired(), key=key)

    form = CreatePasteItemForm(request.POST, paste=paste)

    if form.is_valid():
        await PasteItem.objects.acreate(paste=paste, text=form.cleaned_data["text"])

        # TODO: serial processing here could be made async
        for listener_q in paste_update_listeners[key]:
            await listener_q.put("bump")

        return redirect("view_paste", paste.key)

    messages.error(request, "Error adding paste item")

    return redirect("index")

Nice. In a few dozen lines, live streaming cross client updates, all in async Django!

Pitfalls

I am overall quite pleased with the final result, having replaced several elements of my typical web application stack with async Django. That said, asyncio feels off, suffering from the function coloring problem that took several headaches to imprint ¹. Django suffers from the same, and it feels like async support is still some distance away from prime time.

Async Django

I set up my project entirely along async lines: async views, async-compatible middleware. This wasn't strictly necessary, but I wanted to get a feel for the state of an all async project. For views, what bit me was that template rendering is not async friendly. It will work, up until async code is called, or in an async view, if any "synchronous only" code is called. The recommended solution is to wrap template rendering in sync_to_async and call it a day; as an async zealot for the day, I refused this approach, and instead accessed my async database lookups in the view in painstaking fashion.

In my research, I discovered Jinja2 does support async rendering, even leveraging incremental html generation to stream the results to the client. I'd take async-compatible Django template rendering without the bells and whistles, though the prospect of colored templates (either async or non-async) makes me nervous.

The middleware story is also sub-par: if any middleware is non-async friendly, then the entire request processing flow becomes non-async, which means there goes my async processing! For this project, I had few third party middleware, but sadly whitenoise tripped the async unfriendly wire, and had to be replaced with blacknoise. This alternative is set up on top of the asgi application, rather than as a django middleware:

application = BlackNoise(get_asgi_application())
application.add(BASE_DIR / "staticfiles", "/static")

I don't know if this is a technical limitation (perhaps due to file streaming in async?), but I'd have much preferred the standard Django middleware integration.

SSE

The server sent events protocol is not perfect, but suffices for a project of this scope. As described above, I leveraged Django's StreamingHttpResponse with an async streaming generator to send events down the wire. In my homelab deployment with Caprover, I noticed that updates were not coming through; this is due to nginx reverse proxying not streaming sse appropriately by default, the following header instructed nginx to play nice:

return StreamingHttpResponse(
    event_stream(),
    content_type="text/event-stream",
    headers={
        # for nginx proxying sse support
        "X-Accel-Buffering": "no",
    },
)

The other major issue I ran into involved stuffing html into sse event payloads. I first attempted to configure htmx to stream the html from sse events directly into the dom, but ran into a problem with the sse protocol: newline sensitivity. As Django renders templates with many newlines, I attempted to simply remove them:

 # rendered html has to be stripped of newlines, sse format limitation
 # see https://github.com/bigskysoftware/htmx/issues/2292
 # ah.. but then the copy-paste wont work with newlines... :(
 html = render_to_string(
     "paste_detail.html#paste-items-list",
     {
         "pasteitems": pasteitems,
     },
 )

 html = html.replace("\n", "").replace("\r", "")

 yield "event: update\n"
 yield f"data: {html}\n\n"

This would have worked, except that I had newline sensitive code in the templates. Suppose a user inputs a text with newlines (eg paragraphs or log lines). To implement a "copy on click" feature to the page, I added a data-copyable="{{ content }}" attribute to relevant elements, and an event listener that copied the attribute content into the clipboard. That inner content could include newlines that could not simply be stripped out without breaking the copy feature ²

Finding myself limited by the protocol, I pursued a simpler approach: on sse message, make a request to a view to refresh the template partial. While an extra round trip to the server, it fit in more cleanly and saved the headache. I'd advocate that htmx add support for a friendlier encoding scheme, to support html over sse.

Final thoughts

While async in Python and Django has come a long way, I would not choose this stack for a production project. For a hobby project, it's impressive how much of a typical web stack can be eliminated; I think this could be a boon for solo and aspiring devs on side projects. I plan to check out some of the newer async-first kids on the Python-web-framework block, but I know I'll miss Django's batteries.