In this step-by-step tutorial I’ll show you how to let users create Django model instances in bulk by uploading a CSV file. We will also validate the created model instances using Django Forms so we don’t end up with invalid data in our database.
Create a new project
For this tutorial we’ll pretend that we’re running an ecommerce platform where business owners can upload the products they want to sell in a CSV file.
We’ll start off by creating a new project for our imaginary online store. Given that you have installed Django, run the following command:
django-admin startproject csvupload
This will create a new Django project called csvupload in a directory with the same name.
We’ll also need a Django app—let’s call it shop:
cd csvupload
python manage.py startapp shop
Finally, don’t forget to add shop to the INSTALLED_APPS list in settings.py!
# csvupload/settings.py
INSTALLED_APPS = [
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
"shop",
]
Set up models
Next, we’ll create a model to represent a product. Products should have a name, a SKU, a price and a description.
(A SKU, short for Stock Keeping Unit, is an identifier or code that refers to a particular product.)
Add the following code to shop/models.py:
# shop/models.py
from django.db import models
class Product(models.Model):
name = models.CharField(max_length=100)
sku = models.CharField(max_length=16)
price = models.DecimalField(max_digits=9, decimal_places=2)
description = models.TextField()
def __str__(self):
return f"{self.name} ({self.sku})"
After you’ve edited your models file, run manage.py makemigrations to create the necessary database migrations.
Set up forms
In order to give sensible error messages to our users in case they upload incorrect data we’ll need a Form. Since we won’t be doing any custom validation we can use Django’s built-in ModelForm class.
To create our form, add a forms.py file to the shop directory with the following contents:
# shop/forms.py
from django.forms import FileField, Form, ModelForm
from .models import Product
class ProductForm(ModelForm):
class Meta:
model = Product
fields = ["name", "sku", "price", "description"]
We’ll also need a form for the actual file upload as well. In the same file, add another form that looks like this:
class UploadForm(Form):
products_file = FileField()
Add a view
Almost done! The only things left are displaying our form and processing the CSV file once the user uploads it.
We’ll start by creating a template that will show our form. First, create a templates directory inside the shop directory. Switch to the directory we just created and add an upload.html file with the following content:
<!-- shop/templates/upload.html -->
<html lang="en">
<body>
<form
action="/upload/"
enctype="multipart/form-data"
method="post"
>
{% csrf_token %}
{{ form }}
<br/>
<input type="submit" value="Submit">
</form>
</body>
</html>
Attention!
Note that the form’s encoding type is set to "multipart/form-data"—this is because the default encoding type for HTML forms (application/x-www-form-urlencoded) does not support uploading files!
Having created our template we need a view that can render it. Open up shop/views.py and add the following code:
# shop/views.py
from django.shortcuts import render
from django.views.generic.base import View
from .forms import UploadForm
class UploadView(View):
def get(self, request, *args, **kwargs):
return render(request, "upload.html", {"form": UploadForm()})
Finally, we’ll need to hook our view up to a URL. Open up the csvimport/urls.py file and add an entry to the urlpatterns list containing our UploadView:
# csvupload/urls.py
from shop.views import UploadView
urlpatterns = [
path('admin/', admin.site.urls),
path('upload/', UploadView.as_view()),
]
If you start the development server with python manage.py runserver and open http://localhost:8000/upload/ in your browser you should now see something like this:
Cool! (Did I ever mention I’m not a designer?) Brutalist aesthetics aside, we now have a way to upload a products file to the server. We’re still not doing anything with the uploaded file however, so let’s fix that.
Handling CSV uploads
For the sake of this tutorial we’ll assume that the uploaded CSV files will have an initial “header” row that gives a name to each column. An example CSV file might look like this:
name,sku,price,description
ExpensiBike XL1000,2E50F578,1799.90,A very expensive bike
MudMaster Offroad SE1779,MM00123Q,799.90,A great offroad bike
Bicicletta,MFA342Y0,14590.00,A fancy European bike
To parse the CSV file we’ll use Python’s built-in DictReader class from the csv module. The advantage of using DictReader over the regular csv.reader is that it will parse each row into a dict, which we can pass directly to our ProductForm (as long as the columns names match the attributes on the Product model).
For example, given the CSV above, DictReader would produce a list of dictionaries like this:
[{'name': 'ExpensiBike XL1000',
'sku': '2E50F578',
'price': '1799.90',
'description': 'A very expensive bike'},
{'name': 'MudMaster Offroad SE1779',
'sku': 'MM00123Q',
'price': '799.90',
'description': 'A great offroad bike'},
{'name': 'Bicicletta',
'sku': 'MFA342Y0',
'price': '14590.00',
'description': 'A fancy European bike'}]
To be able to use DictReader in our view we’ll add the following imports at the top of the shop/views.py file:
from csv import DictReader
from io import TextIOWrapper
Also, we need to change our import from the .forms module to include our ProductForm as well:
from .forms import UploadForm, ProductForm
Now we’re ready to handle the file upload. We’ll do this by adding a post method to our UploadView class:
# shop/views.py
class UploadView(View):
# ...
def post(self, request, *args, **kwargs):
products_file = request.FILES["products_file"]
rows = TextIOWrapper(products_file, encoding="utf-8", newline="")
for row in DictReader(rows):
form = ProductForm(row)
form.save()
return render(request, "upload.html", {"form": UploadForm()})
There’s quite a lot going on here, so let’s break it down:
Django puts all uploaded files in the request.FILES dictionary, so the first thing we need to do is to get our uploaded file. Since we named the upload field products_file we can access the uploaded file like this:
products_file = request.FILES["products_file"]
Django doesn’t make any assumptions about what kind of files we’re uploading (maybe we were expecting an image, for example), so the files in request.FILES are opened in binary mode—this means we’ll get a stream of bytes if we try to read from products_file. As the CSV module works exclusively with text streams we need to convert the byte stream to a text stream before we attempt to read it. Otherwise we’ll get an error like this:
iterator should return strings, not bytes (did you open the
file in text mode?)
A convenient way to convert a byte stream to a text stream is to use the TextIOWrapper class:
rows = TextIOWrapper(products_file, encoding="utf-8", newline="")
Note that we’re assuming that the file is encoded with utf-8. Also, the newline option needs to be set to "", otherwise line breaks inside quoted fields won’t be parsed correctly. (Yes, you read that right, newlines can be embedded inside a CSV field.)
Once we have our text stream we can create a new DictReader. The DictReader works like an iterator which lets us loop through all the rows in the CSV field. Given that the columns in the CSV file have the same names as our model fields we can then create a ProductForm instance for each row and save it:
for row in DictReader(rows):
form = ProductForm(row)
form.save()
Once we’ve saved all the rows we render the upload.html template and return a response:
return render(request, "upload.html", {"form": UploadForm()})
You should now be able to upload the CSV by opening http://localhost:8000/upload/ and submitting a file through the form! Cool!
Error and success messages
If you tried to upload a CSV file you might have noticed that the response looks a bit…underwhelming. You just get the same page again, with nothing to indicate whether the upload succeeded or not. Let’s fix that by displaying a celebratory message if the upload succeeded, or an error message if it failed.
We’ll start by counting the number of imported rows—add a counter just above the for loop like this:
row_count = 0
for row in DictReader(rows):
row_count += 1
# ...
Next, we’ll check if each row is a valid Product before saving it; if we found any errors we’ll save them to a form_errors variable and break out of the loop:
row_count = 0
form_errors = []
for row in DictReader(rows):
row_count += 1
form = ProductForm(row)
if not form.is_valid():
form_errors = form.errors
break
form.save()
# ...
Finally, let’s update the render call to include the row_count and errors variables:
return render(
request,
"upload.html",
{
"form": UploadForm(),
"form_errors": form_errors,
"row_count": row_count,
}
)
The complete code for the UploadView.post() method should now look something like this:
def post(self, request, *args, **kwargs):
products_file = request.FILES["products_file"]
rows = TextIOWrapper(products_file, encoding="utf-8", newline="")
row_count = 0
form_errors = []
for row in DictReader(rows):
row_count += 1
form = ProductForm(row)
if not form.is_valid():
form_errors = form.errors
break
form.save()
return render(
request,
"upload.html",
{
"form": UploadForm(),
"form_errors": form_errors,
"row_count": row_count,
}
)
Let’s update the template so it will display an appropriate message depending on how the upload went.
If there was an error we’ll output the list of validation errors along with the current row_count. If there were no errors, but row_count is not zero it means the upload succeeded, which calls for celebration! As luck would have it, Unicode ships with a sufficiently festive emoji we can use: 🥳
To make this happen, open up the upload.html template add the following code right after the <body> tag:
<!-- ... -->
<body>
{% if form_errors %}
<p>Found errors on line {{ row_count }}:<p>
<ul>
{% for field, errors in form_errors.items %}
{% for error in errors %}
<li> {{field}}: {{ error }} </li>
{% endfor %}
{% endfor %}
{% elif row_count %}
<p>Successfully uploaded {{ row_count }} row(s)! 🥳</p>
{% endif %}
<form action="/upload/" enctype="multipart/form-data" method="post">
<!-- ... -->
Tip
If you want to get rid of the ugly “(s)” in “row(s)” and instead display the singular form when there’s only one row, you can use Django’s pluralize filter.
Great! We now have a working CSV upload that also displays an informative error message if the upload failed.
There’s one a tiny snag however: The code doesn’t keep track of which products we’ve already imported, so if we try to upload the same CSV again (e.g. if there was an error) we’ll end up with duplicate products in the database! We’ll fix that in the next section.
Preventing duplicate imports
How can we prevent a product from being imported twice?
First, we need to decide what it actually means for a product to be a duplicate of another product. A reasonable assumption is that two products with the same SKU are actually the same product. Let’s start by capturing this constraint in our model. Open up models.py and add unique=True to the sku field:
sku = models.CharField(max_length=16, unique=True)
Run manage.py makemigrations and then manage.py migrate to apply the model changes.
Next, we need to decide what should happen if the user tries to upload the same product twice. One way to handle this is to update the existing product with the data from the uploaded file. For example, this would allow users to update the prices of existing products by uploading a new CSV file.
To accomplish this we’ll check if there already exists a product with the given SKU. If so, we’ll update that product instead of creating a new one.
First of all, we need to import our Product model; add the following import in the shop/views.py file:
from .models import Product
Next, inside the upload loop, we’ll try to find a product with the same SKU as the current row. Change the lines where we instantiate the ProductForm to look like this:
product = Product.objects.filter(sku=row["sku"]).first()
form = ProductForm(row, instance=product)
Now if we try to upload a product with the same SKU as an existing product the old product will be updated. Great!
Wrapping up
So what did we do here exactly?
- We’ve created a way for users to upload products in a CSV file.
- Products will either get created or updated depending on if the product’s SKU was in the database already.
- If there were any errors we let the user know what went wrong and on which row of the upload.
Not too bad if you ask me!
The complete code can be found at:
https://github.com/chreke/django-examples/tree/main/csvupload
If you have any questions, comments or suggestions please hit me up on Twitter: @therealchreke
Happy hacking!