## ~~10~~ **8 Simple** Techniques to Make Your **DRF API Faster** Note: - Today I'm going to speak about 8 relatively simple techniques to make your DRF API faster. - I originally wanted to cover 10, but when my presentation was finished I realized I have too much content and decided to cut it down to 8. - I'm focusing on Django and DRF, but many of this techniques are also applicable to other languages and frameworks.
### The app - Insurance Distribution API - Django 4.2 + gunicorn / DRF 3.14 / Python 3.11 / Postgres 14 - ~6 yo code base, nearly 5y in prod - ~150k LOC - ~15m API calls per day Note: - The things I'm speaking today are based on my experience working on a large and complex Django app. - It is an insurance distribution API, which is a part of a larger insurance platform called XCover. - The code base is approximately 150k lines of code and it's been in production for nearly 5 years. - At the moment, it serves around 15 million API calls per day. - It is impossible to cover all the things we worked on with the team in the last few years in 20 minutes, so today I picked the most simple of them. - But...
### Measure first - You can't improve what you can't measure Note: - Before jumping into the topic, I want to emphasize that it is important to have a reliable way of measuring your app performance before working on the optimizations. - There are many different ways of measuring app performance.
### APM - NewRelic APM - Measure application performance from inside - Base metrics throughput, latency, error rate (golden SRE signals) - Other important performance, e.g. number and duration of DB queries, external API call, etc Note: - But, if you'd have to choose a single tool, it should be an APM tool. - At Cover Genius we use NewRelic APM. But there are many other tools available on the market. - Examples of them are Datadog, AppDynamics, Dynatrace and Sentry. - These tools allow collecting base and advance performance metrics from your application in real time - And this level of observability that these tools provide is hard to replace with anything.
### Synthetic monitoring - NewRelic Synthetics - Measure uptime or estimate network latency - Scheduled calls - Multiple locations Note: - I will briefly speak about other ways of measuring performance too. - Synthetic monitoring is a technique in which some artificial or (synthetic) traffic to your system is generated in a script that runs on scheduler from multiple locations. - It can be used to measure uptime or estimate network latency from different locations. - We use New Relic Synthetics, but other notable providers are Pingdom, Postman Monitors, etc.
### Performance testing - k6 - Simulate various workload scenarios - Distributed runs Note: - There are many different types of performance testing. - These days at Cover Genius we use k6 for all performance testing. - k6 is a modern tool, that is backed by Grafana. - k6 website is a great source of information about performance testing. - Another popular tool is Locust, Locust is written in Python. I like it a lot too as it implements Pythonic approach to performance testing.
### Profiling (in local environment) - `django-debug-toolbar` (DDT) + plugins - when needed we also use `memory-profiler`, `cProfile` - `pyperf` to run benchmarks Note: - Last but not least. Profiling. We use `django-debug-toolbar` with plugins to profile API calls. - But another popular Django profiler is `silk`. - It's also worth mentioning specialized Python profilers. - `memory-profiler` allows you to profile ...memory usage of your Python code. - And `cProfile` is a built-in Python profiler that allows you to profile function calls and their duration. - `pyperf` is the tool that allows your to run benchmarks for Python code in a consistent way. - We use these tools as needed too. - Now, once we have tools in place, lets jump into the optimizations. - I tried to rank all them by complexity, so I will start with the simplest one.
### Use persistent DB connections - `CONN_MAX_AGE` - `CONN_HEALTH_CHECK` Note: - It's also my favorite. As it's only one lien and it always works. - If you are new to Django, you might not be aware that Django opens and closes a new database connection for each request. - Opening DB connections is an expensive operation and it's a good idea to reuse connections once opened. - CONN_MAX_AGE is a setting that allows you to do exactly this. - When it's set to a positive integer, Django will reuse the database connection for up to that many seconds.
```python DATABASES = { 'default': { 'CONN_MAX_AGE': 600, # 10 minutes } } ``` Note: - Easy? Just one line of code in your settings.py and you can observe the positive impact on your response time.
```python DATABASES = { 'default': { 'CONN_MAX_AGE': 600, # 10 minutes 'CONN_HEALTH_CHECK': True, } } ``` Note: - Sometimes your connection might become stale, for example when the database server is restarted or master failover happens. - So when request starts it's usually a good idea to check if it is in a useable state. - CONN_HEALTH_CHECK setting was introduced in Django 4.1 exactly for this purpose. - When it is set to True, Django will run a simple `SELECT 1` query to check if the connection is still alive before using it for the request. - If the query fails, Django will open a new connection instead.
```python # django/django/db/backends/postgresql/base.py class DatabaseWrapper(BaseDatabaseWrapper): ... def is_usable(self): try: # Use a psycopg cursor directly, bypassing # Django's utilities. with self.connection.cursor() as cursor: cursor.execute("SELECT 1") except Database.Error: return False else: return True ```
### Improve DRF serialization performance - `drf-orjson-renderer` Note: - The next technique allows improving the DRF serialization performance. - By default, DRF uses Python's built-in JSON module. - `drf-orjson-renderer` is a third-party package that adds support for a faster renderer and parser using `orjson` library. - `orjson` is a faster alternative to the standard JSON library that is written in Rust, it is widely used by many other projects too. - `drf-orjson-renderer` is a small and simple package, but it is not very popular. - If you don't want to add it as a project dependency, you can easily implement your own parser and renderer using `orjeson` directly. - Once implemented this is how you can use it in the DRF settings.
```python REST_FRAMEWORK = { "DEFAULT_RENDERER_CLASSES": ( "drf_orjson_renderer.renderers.ORJSONRenderer", ), "DEFAULT_PARSER_CLASSES": ( "drf_orjson_renderer.parsers.ORJSONParser", ), } ```
### Upgrade Python and Django - Python 3.11 is 10-60% faster than Python 3.10 - Performance of Python 3.12 seems to be similar to Python 3.11 - `pyupgrade` - `django-upgrade` Note: - Often, new Python and Django versions are coming out with significant performance improvements. - For example, Python 3.11 is 10-60% faster than Python 3.10 as the Python release team claims. - We also observed quite significant response time decrease when we switched to Python 3.11. - Though Python 3.12 has some performance improvements, it's not as significant as the 3.11 release. - It is important to test the performance impact for your specific use case. - `pyupgrade` and `django-upgrade` are simple Python tools that can help you with the upgrade process. - We use them in our pre-commit configurations to automatically upgrade Python and Django code to a newer version.
```yaml - repo: https://github.com/asottile/pyupgrade rev: v3.3.1 hooks: - id: pyupgrade args: [ --py311-plus ] - repo: https://github.com/adamchainz/django-upgrade rev: 1.14.0 hooks: - id: django-upgrade args: [ --target-version, "4.12" ] ```
### Minimize DB writes - `bulk_create`/`bulk_update` (batching operations) - Avoid writes to multiple DB tables if possible - Don't overuse indexes Note: - Django ORM makes it extremely easy synchronizing objects state to the database. - It's so easy that engineers don't need to think how exactly it is done for them. - It is actually quite important to be careful when your application writing to the database. - As it is easy to make mistakes that will impact the performance. - Let's have a look at the following example.
```python # 👎 - multiple queries with transaction.atomic(): for item in items: item.save() ``` Note: - In this code we loop over a list of model instances and save each individual one to the database. - Each call to `save` method results in a separate SQL query. - Though this code is absolutely working, from the performance point of view it's less than optimal.
```python # 👍 - single query Item.objects.bulk_create(items) ``` Note: - To do it more efficiently, Django provides a method called `bulk_create` that allows creating multiple objects in a single query. - Similar method `bulk_update` can be used to batch multiple update operations. - The less round trips to the DB that app makes - the smaller the response time is.
### Skip instance creation in DB queries - `values` - `values_list` Note: - ORM is useful to operate with objects, but sometimes Objects could ba harmful to the performance. - Sometimes, you can get a significant performance improvement by skipping the instance creation. - Especially if you are not planning to change your data. - So `values` QuerySet methods forces it to return dictionaries, rather than model instances. - And `values_list` could be use to return tuples which are even more efficient from the performance point of view. - Let's have a look at this example.
```python # Fetches only `id` field entries = Item.objects.filter(...).values_list('id', flat=True) # output: <QuerySet [1, 2]> ``` Note: - This is how you can fetch only `id` field from the `Item` model, you can use `values_list` method with `flat=True` argument.
```python # offload computations to DB entries = (Item.objects.filter(...) .annotate(discounted_price=models.F('price') * 0.9) .values_list('id', 'discounted_price') ) # output: <QuerySet [(1, 99.9),(2, 299.99)]> ``` Note: - You can also combine it with offloading some of the computations to the database. - In this example discounted_price is calculated in the database and then returned as a tuple which you can easily convert in a dictionary if needed. - The same offloading could also be used with objects when you are not using `values_list` too, which one to use depends on your specific use case.
### Fix N + 1 query problem - `select_related` (use for many-to-one and one-to-one relations) - `prefetch_related` (use for many-to-many and reverse foreign key relations) - Use `Prefetch` objects for precise control - Be cautious of the size of related data Note: - The infamous N+1 problems happens when your app iterates over a list of objects and for each object it fetches a dependency. - In DRF it often happens in nested serializers, e.g. using `many=True` on the related serializer. - `prefetch_related` and `select_related` are the two QuerySet methods that can help developers to solve this problem. - The difference between them is that `select_related` pulls related objects from database in a single query using JOIN, while `prefetch_related` performs an additional query and does the joining in Python. - Both methods should only be used when you know you're going to need the related data and overusing them could also impact negatively the app's performance. - Let's have a look at some examples:
```python # 👎 users = User.objects.all()[0:10] for user in users: for post in user.posts.all(): print(post.id) # 11 queries ``` Note: - In this code, we take 10 users from the database in a single query - Then for each user we are fetching all the posts. - This produces 11 queries in total. - In order to fix this, we can use `prefetch_related` method.
```python [3] # 👍 users = User.objects.prefetch_related('posts')[0:10] for user in users: for post in user.posts.all(): print(post.id) # 2 queries ``` Note: - This code will result in only 2 queries.
```python # Complex Prefetch scenario book_queryset = Book.objects.order_by('-rating')[:3] prefetch = Prefetch( 'books', queryset=book_queryset, to_attr='three_best_rated_books' ) authors = Author.objects.prefetch_related(prefetch) for author in authors: print(author.three_best_rated_books) # [] if the QuerySet is empty # 2 queries ``` Note: - It is important to be aware of the size of the data that is being prefetching. - If your applications is prefetching too much, it could fill all the app memory. - For the precise control on the prefetching process and advanced use cases it is possible to use `Prefetch` objects. - This is an example of a complex prefetch scenario.
### Avoid (over|under)fetching - `only` - `defer` - `django.test.utils.CaptureQueriesContext` Note: - Overfetching is when your app pulls more data from the database than it actually needs. – Overfetching causes wasting resources such as CPU, memory and network. - Undefetching is the opposite problem, when you need to make additional requests to load more data later when your app needs it. - Both are not good for performance, so it's important to control the fields that are loaded from the database. - `only` and `defer` are the two QuerySet methods that exist to control the fields that are loaded from the database. - Let's have a look at some examples.
```python # SELECT id FROM users users = User.objects.only('id').all() for user in users: print(user.id) # this will cause an additional query to the users table print(users[0].username) ```
```python # CaptureQueriesContext with CaptureQueriesContext(connection) as queries: # triggers sql queries item = MyModel.objects.get(name='Test') # you can analyze the queries now print(f"Total queries: {len(queries)}") for query in queries.captured_queries: print(query['sql']) ``` Note: - Once a block of code is optimized - it is typically a good idea to make sure that it stays optimized. - `CaptureQueriesContext` is a context manager that captures the queries executed inside it. - It can be used in your tests to assert that only the expected queries are executed.
### Replace slow libraries - Replacing `arrow` with `rfc3339-validator` for dates/datetimes validation - Replacing `cerberus` with `jsonschema` (`pydantic` is even better) - NewRelic Python Agent Note: - Finally, I want to speak about the performance of Python libraries. - Not all of the libraries come with the optimal performance. - That's why it is important to benchmark your code when adding a new dependency to the project, especially if the performance requirements are high. - For example, we originally used both `arrow` and `cerberus` to implement custom validation rules for JSON fields. - We initially only used them in a couple of places, but over time the usage increased so the performance impact became more significant. - We found out that custom validation rules `arrow` and `cerberus` contributed ~15% to the total API response time for one of the endpoints. - Which surprised the team a lot. - There are some libraries that are especially dangerous, so you should be very careful when using them. - Unfortunately, one kind of such libraries are the APM instrumentation libraries. - For example, last year we had a major performance issue with new version of NewRelic Python Agent. - It spilled to prod and caused a significant performance degradation. - that was the last one topic I wanted to cover today.
### Not covered in this talk - Caching (especially advanced) - DB indexes (beyond basics) - Multiple DBs - Breaking apps into individual services - Offloading operations to background workers (e.g. Celery) - Optimizing network latency using global CDNs - Optimizing logging - DB Replication - Using async views - Alternative Python runtimes (e.g. PyPy) - Data partitioning and DB sharding Note: - And I want to briefly mention a few more advanced, which we also used here at CG. - I won't go through each and every item on this list, but these are the topics I would be interested to discuss in the future. - If anyone is keen to present on any of these topics in the future, you are always welcome. - In my observation in Django and even Python communities performance topics are often neglected. - I believe we can improve the situation if we discuss them more often. - Just before we wrap up, I want to do a quick quiz. - Please, raise your hand if you were able to learn at lest one new thing today. Thanks everyone!