Django Model Bulk Loading Slowness Pt. 2

It turns out that the django-model-changes library had been forked and modified some time ago by engineers at the company. Among the changes was removing the dynamic signal attachments from the ChangesMixin __init__ method:

    def __init__(self, *args, **kwargs):
        super(ChangesMixin, self).__init__(*args, **kwargs)

        self._states = []
        self._save_state(new_instance=True)

        signals.post_save.connect(
            _post_save, sender=self.__class__,
            dispatch_uid='django-changes-%s' % self.__class__.__name__
        )
        signals.post_delete.connect(
            _post_delete, sender=self.__class__,
            dispatch_uid='django-changes-%s' % self.__class__.__name__
        )

By just commenting out the signals code, the improvement is dramatic:

In [1]: %timeit list(Dese.objects.all())
841 ms ± 3.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This brings the slowdown to around 1.7x, still significant but not nearly as brutal as previously thought. It still leaves the question - why do I see such an immense slowdown at work?

One other thought is that the models are simply huge. I cannot overstate the model obesity epidemic: the BMI is off the charts, thousands upon thousands of lines of code for a single model. Let's explore adding many more properties to the model, to see if anything interesting happens.

Adding 26 TextField properties to our model, let's check the speed:

In [1]: %timeit list(Dese.objects.all())
6.85 s ± 218 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Quite the slowdown, but possibly due to the content of the TextFields. Let's only return the object primary keys:

In [2]: %timeit list(Dese.objects.all().only('pk'))
1.81 s ± 24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Aha, adding more model properties does slow down the model loading. Compared to without the ChangesMixin:

In [1]: %timeit list(Dese.objects.all().only('pk'))
700 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

That's a significant difference at 2.5x. Upon further investigation, it becomes clear that the issue is due to the implementation of the ChangesMixin. Upon object creation, every model property is copied into a dictionary. This makes sense - the purpose of the library is to enable figuring out what has changed over an object's lifetime.

However, the implementation is very slow, as it reads and copies properties individually from self.__dict__:

class ChangesMixin:
    def current_state(self):
    local_data = self.__dict__

        fields = {}

        for field in self._meta.local_fields:

            fields[field.name] = local_data.get(field.name)
            fields[field.attname] = local_data.get(field.attname)

basic implementation outlineSome brief internet sleuthing revealed that an update to dict.copy() has vastly improved performance over other approaches. Let's see what happens when we modify the above code:

In [1]: %timeit list(Dese.objects.all().only('pk'))
844 ms ± 11.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Wow, not bad! We recovered roughly half of the performance loss. Let's rack it up a notch and add more properties to the model. Adding 26 more fields to the model, we find:

In [1]: %timeit list(Dese.objects.all().only('pk'))
1.11 s ± 15.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

An incremental slowdown from fewer fields. Compared to the naive copy implementation:

In [1]: %timeit list(Dese.objects.all().only('pk'))
2.93 s ± 15.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

We see even more improvement than at fewer fields - which makes sense to me.

Finally, compared to no ChangesMixin at all:

In [1]: %timeit list(Dese.objects.all().only('pk'))
974 ms ± 24.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

We see that the improved copy implementation continues to shine.

Notably to me there is a slowdown that occurs even at the base Django model as more fields are added. While I may not expect it to be lightning fast, I wouldn't expect almost 50% loss in performance by roughly doubling the number of fields.

I wonder if there is a more efficient way to load Django model objects (i.e. full Django model objects, not just dictionaries). But then again I also appreciate that this is not necessarily the goal of the project - magic always comes at a cost :)

Addendum

The proposed improvement to ChangesMixin would require additional effort to fully realize. The changes() dictionary which is at least our own primary use case would need to iterate over the fields of the model, constructing the diff between the __dict__ copies. However I don't think this would cause necessarily any loss of performance on that front either, and may actually be an improvement, as I believe that defered fields may be inappropriately marked changed in the current implementation.

social