In a previous post I explained how nested serializers could be used to represent relationships between Django models in your Django REST Framework API.
This works well when the number of related items is small, but what if you have related items numbering in the hundreds or thousands? For example, if we wanted to fetch Justin Bieber’s Twitter followers we probably wouldn’t want all 100 million Beliebers at once; rather, we’d probably fetch only a few of them at a time and then paginate through the rest.
Luckily, we can do this pretty easily using a few features built into Django REST Framework.
Prerequisites
If you want to follow along with this tutorial, install Django REST Framework and make sure you’ve added "rest_framework" to your INSTALLED_APPS list.
The setup
Let’s say that we’re building a pretend social network where users can follow each other. The models file could look something like this:
# socialnetwork/models.py
from django.db import models
class User(models.Model):
username = models.SlugField(unique=True)
followers = models.ManyToManyField("User")
def __str__(self):
return self.username
Note that I’m using SlugField for the username field—this makes sure the username can be used in a URL, which might come in handy later.
Given the model above, a first attempt at a serializer might look something like this:
# socialnetwork/serializers.py
from rest_framework.serializers import ModelSerializer
from .models import User
class UserSerializer(ModelSerializer):
class Meta:
model = User
fields = ["username", "followers"]
This will simply output the user’s username and the primary keys of the user’s followers.
Let’s create a ModelViewSet for our users so we can try out our serializer:
# socialnetwork/views.py
from rest_framework.viewsets import ModelViewSet
from .models import User
from .serializers import UserSerializer
class UserViewSet(ModelViewSet):
serializer_class = UserSerializer
queryset = User.objects.all()
lookup_field = "username"
In order to make the URLs a bit more readable I’ve set the lookup_field to "username"; this means users can be looked up by their username, e.g. instead of dull URLs like /users/123 we’ll get something with a bit more pizzazz, like /users/bieber
Finally, let’s wire the view up to a route. Open up the project’s urls.py and extend it with a router and some routes:
# project/urls.py
from django.contrib import admin
from django.urls import path
from rest_framework.routers import DefaultRouter
from socialnetwork.views import UserViewSet
router = DefaultRouter()
router.register("users", UserViewSet)
urlpatterns = [
path('admin/', admin.site.urls),
] + router.urls
To try it out, create a few users and let them follow each other. Now, if you run manage.py runserver and go to http://localhost:8000/users/ you should see something like this:
[
{
"username": "bieber",
"followers": [ 2, 3 ]
},
{
"username": "bieberfever",
"followers": []
},
{
"username": "true_belieber",
"followers": []
}
]
Nested routes
As I mentioned at the start of the post, including a list of each user’s followers in the response is problematic if the number of followers is large. Instead, what if we could have a separate URL just for listing followers, like /users/bieber/followers? This would also make the list easier to paginate, and we would only have to fetch followers when we actually needed them.
Well, I’m convinced—let’s create that followers view! We’ll start by creating a new ListAPIView where we override get_queryset to get the followers for a given user:
from rest_framework.generics import ListAPIView
from rest_framework.viewsets import ModelViewSet
from .models import User
from .serializers import UserSerializer
class UserViewSet(ModelViewSet):
serializer_class = UserSerializer
queryset = User.objects.all()
lookup_field = "username"
class FollowersListView(ListAPIView):
serializer_class = UserSerializer
def get_queryset(self):
username = self.kwargs["username"]
user = get_object_or_404(User, username=username)
return user.followers.all()
We’re assuming that we’ll receive the user’s username in a URL parameter called username, which means we’ll be able to fetch the username via self.kwargs["username"].
Note
In this example I’ve just used the same UserSerializer in both the “users” and “followers” views. However, in a “real” application you might want to use a more “lightweight” serializer for your list views; see my previous post for instructions on how to do this.
Let’s add a route for our “follwers” view, making sure to include the aforementioned username parameter:
from django.contrib import admin
from django.urls import path
from rest_framework.routers import DefaultRouter
from socialnetwork.views import UserViewSet, FollowersListView
router = DefaultRouter()
router.register("users", UserViewSet)
urlpatterns = [
path('admin/', admin.site.urls),
path(
"users/<slug:username>/followers/",
FollowersListView.as_view(),
name="user-followers",
)
] + router.urls
Now, try opening http://localhost:8000/users/bieber/followers; you should see a list of all of the users following bieber, i.e. something like this:
[
{
"username": "bieberfever",
"followers": [],
},
{
"username": "true_belieber",
"followers": [],
}
]
Neat! However, the followers key still contains a list of user IDs; let’s change it to a link that points to the “followers” URL for each user.
Using links in serializers
Django REST Framework has a few built-in serializer fields to represent links and relations.
Since we want to link from an object to one of its “own” URLs we’ll use HyperlinkedIdentityField. Most often you’ll use this field to link to an object’s “detail” URL, but you can actually use it with any route that takes one of the object’s fields as a parameter.
To wit, here’s how we can use it to point to the user’s “followers” URL:
# socialnetwork/serializers.py
from .models import User
from rest_framework.serializers import (
HyperlinkedIdentityField,
ModelSerializer,
)
class UserSerializer(ModelSerializer):
followers = HyperlinkedIdentityField(
view_name="user-followers",
lookup_field="username",
)
class Meta:
model = User
fields = ["username", "followers"]
Note that we have to specify both the view_name and lookup_field. This tells Django REST Framework that:
- We want the link to point to the route named "user-followers"
- The value of the username field on the object should be supplied as the URL parameter.
Note
By default, Django REST Framework assumes that the URL parameter has the same name as the lookup_field—however, if this is not the case we can set the name of the URL parameter explicitly by using the lookup_url_kwarg keyword argument.
If you try to fetch http://localhost:8000/users/ again you should see something like this:
[
{
"username": "bieber",
"followers": "http://localhost:8000/users/bieber/followers",
},
{
"username": "bieberfever",
"followers": "http://localhost:8000/users/bieberfever/followers",
},
{
"username": "true_belieber",
"followers": "http://localhost:8000/users/true_belieber/followers",
}
]
Perfect! Now we can get a list of a user’s followers simply by following a link from their serialized representation. There’s only one thing left to fix: Pagination.
Pagination
Any time you return a potentially unlimited number of items from your API it’s good practice to add some kind of pagination.
Normally, the advice goes that you shouldn’t optimize things prematurely, and—if you squint—isn’t pagination just a kind of optimization? However, since pagination changes the shape of the response adding it after the fact might break applications that are using your API. Luckily, Django REST Framework makes pagination really easy, so it’s usually a safe bet just to enable it from the start.
Let’s enable pagination by opening up settings.py and adding the following lines:
REST_FRAMEWORK = {
"DEFAULT_PAGINATION_CLASS": "rest_framework.pagination.PageNumberPagination",
"PAGE_SIZE": 10
}
When pagination is turned on, Django REST Framework will automatically paginate results returned from generic views or viewsets. You can also change the style of pagination, or toggle pagination on and off for individual views.
If we fetch http://localhost:8000/users/ you should now see something similar to this:
{
"count": 3,
"next": null,
"previous": null,
"results": [
{
"username": "bieber",
"followers": "http://localhost:8000/users/bieber/followers/",
},
{
"username": "bieberfever",
"followers": "http://localhost:8000/users/bieberfever/followers/",
},
{
"username": "true_belieber",
"followers": "http://localhost:8000/users/true_belieber/followers/",
}
]
}
As you can see, the list of users is now wrapped in an object that, in addition to the user list, contains count, next and previous keys. The next and previous keys will contain links to the next and previous pages, but since we would only have three users we don’t hit the configured limit of ten items per page.
Just to make sure pagination works, try changing the value of the PAGE_SIZE setting to 1 and fetch the user list again:
{
"count": 3,
"next": "http://localhost:8000/users/?page=2",
"previous": null,
"results": [
{
"username": "bieber",
"followers": "http://localhost:8000/users/bieber/followers/",
}
]
}
As you can see, the next key now contains a URL we can follow to get to the next page.
Let’s see what http://localhost:8000/users/bieber/followers/ looks like now:
{
"count": 2,
"next": "http://localhost:8000/users/bieber/followers/?page=2",
"previous": null,
"results": [
{
"username": "bieberfever",
"followers": "http://localhost:8000/users/bieberfever/followers/",
}
]
}
Great! This means that no matter how many followers a user has, we’ll never need to fetch more than PAGE_SIZE of them at a time.
Warning
The PageNumberPagination pagination style that we used in this example works well for smaller result sets, but it will slow down significantly as the result set grows. If you need to paginate large amounts of items use the CursorPagination style instead.
Next steps
Good job! We’ve now given each user a paginated list of followers that should be able to handle even the most popular of teen idols.
If you want to keep working on this example here are a few ideas that you could try out:
- Add an endpoint where we can see all users that the given user is following, e.g. /users/<username>/following.
- Add “follower count” and “following count” aggregate fields on UserSerializer so you can see at a glance how many followers a user has / how many users they’re following
- As mentioned earlier, try using a different “lightweight” serializer for list views (as described in my post on nested serializers).
Wrapping up
That’s it for now! If you have any questions or comments I’d love to hear them—I’m on Twitter as @therealchreke if you want to get in touch. Also, don’t forget to subscribe to my email list if you want to be notified whenever I post something new on this blog!
See you later!