user

Development, November 9th, 2017

How to configure your Django project for multiple environments?

How to configure your Django project for multiple environments?

Starting a new Django project, you need to think ahead. Your simple project may grow significantly and you’ll have to introduce changes in order to run your app on different environments.

In this article, I will show you how to configure Django project for multiple environments, based on The Twelve-Factor App methodology for building software-as-a-service apps. I have also used Cookiecutter Django framework by Pydanny.

This project uses several third-party tools including PostgreSQL, Sentry, AWS, WhiteNoise, Gunicorn, Redis, Anymail.

TL;DR If you’d like to just take a quick glance at the code, take a look at this SlideShare prezentation. Below, I’ll explain it step by step.

But before we set up the new project, let’s tackle this question:

Why isn’t the default configuration enough?

The no 1 reason to configure Django project for multiple environments is that when you first start a new project, it lacks such arrangement. Not having the bundles split makes it difficult to configure the project later without having to alter the code.

Also, without setting the Django project for multiple environments, there are no dedicated solutions for production like dedicated path for admin panel, logging errors (e.g., Sentry), cache configuration (memcache/redis), saving the data uploaded to cloud by the user (S3), HSTS or secure cookies.

Testing environment, on the other hand, lacks dedicated solutions either, including turning off debugging templates, in-memory caching, sending mails to console, password hasher, storing the templates.

As a result, setting Django project for multiple environments gives you:

  • more manageable code with fewer duplications
  • more accurate settings depending on environment type

Starting a Django project

First, you need to set up our new Django project. To do it, install virtualenv/virtualenvwrapper and Django: pip install Django==1.11.5 or whatever Django version you want to use for your project.

Then, create a new project: django-admin: django-admin startproject djangohotspot.

At this point, your project should look like this:

(djangohotspot) ╭ ~/Workspace/
╰$ tree djangohotspot
djangohotspot
├── djangohotspot
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── manage.py

Now you need to set it up for multiple environments. You do it by splitting the requirements first.

Splitting requirements

Create base.txt that will cover the common requirements. Then write down the environment-specific requirements in separate files:

  • local.txt
  • production.txt
  • test.txt

The structure of the project is now as follows:

(djangohotspot) ╭ ~/Workspace/ 
╰$ tree djangohotspot
djangohotspot
├── djangohotspot
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
└── requirements
	├── base.txt
	├── local.txt
	├── production.txt
	└── test.txt

Base.txt

django==1.11.5
# Configuration
django-environ==0.4.4
whitenoise==3.3.0
# Models
django-model-utils==3.0.0
# Images
Pillow==4.2.1
# Password storage
argon2-cffi==16.3.0
# Python-PostgreSQL Database Adapter
psycopg2==2.7.3.1
# Unicode slugification
awesome-slugify==1.6.5
# Time zones support
pytz==2017.2
# Redis support
django-redis==4.8.0
redis>=2.10.5

Production.txt

-r base.txt

# WSGI Handler
gevent==1.2.2
gunicorn==19.7.1

# Static and Media Storage
boto3==1.4.7
django-storages==1.6.5

# Email backends for Mailgun, Postmark,
# SendGrid and more
django-anymail==0.11.1

# Raven is the Sentry client
raven==6.1.0

Test.txt

-r base.txt

coverage==4.4.1
flake8==3.4.1
factory-boy==2.9.2

# pytest
pytest-cov==2.4.0
pytest-django==3.1.2
pytest-factoryboy==1.3.1
pytest-mock==1.6.0
pytest-sugar==0.9.0

Local.txt

This file combines production and test files:

-r test.txt
-r production.txt

django-extensions==1.9.0
ipdb==0.10.3

It’s time to configure the settings of each environment.

Splitting settings

First, you need to remove settings.py from the main folder of your Django app djangohotspot/djangohotspot/settings.py and create new Python module named config, where you create another module, settings, where all the settings files will be stored.

The structure of your project has changed and should look like this now:

(djangohotspot) ╭ ~/Workspace/  
╰$ tree djangohotspot
djangohotspot
├── config
│   ├── __init__.py
│   └── settings
│   	├── base.py
│   	├── __init__.py
│   	├── local.py
│   	├── production.py
│   	└── test.py
├── djangohotspot
│   ├── __init__.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
└── requirements
	├── base.txt
	├── local.txt
	├── production.txt
	└── test.txt

config.setting.base

To configure settings in base.py in this example, I have used the django-environ library.

ROOT_DIR = environ.Path(__file__) - 3  # djangohotspot/
APPS_DIR = ROOT_DIR.path('djangohotspot')  # path for django apps

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS

AUTOSLUG_SLUGIFY_FUNCTION = 'slugify.slugify' # allows you to define a function for unicode-supported Slug

DATABASES = { 'default': env.db('DATABASE_URL', default='postgres:///djangohotspot'), }
DATABASES['default']['ATOMIC_REQUESTS'] = True # allows you to open and commit transaction when there are no exceptions. This could affect the performance negatively for traffic-heavy apps.

EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.smtp.EmailBackend')
ADMIN_URL = env('DJANGO_ADMIN_URL', default=r'^admin/')

PASSWORD_HASHERS = ['django.contrib.auth.hashers.Argon2PasswordHasher', (...)] # add this object at the beginning of the list

config.settings.local

Configuring local settings, you need to import base settings:

from .base import * 

Now, add debug toolbar:

MIDDLEWARE += ['debug_toolbar.middleware.DebugToolbarMiddleware', ]
INSTALLED_APPS += ['debug_toolbar', ]
DEBUG_TOOLBAR_CONFIG = {  
  'DISABLE_PANELS': [ 'debug_toolbar.panels.redirects.RedirectsPanel', ],
  'SHOW_TEMPLATE_CONTEXT': True,
}

Define allowed IP addresses:

INTERNAL_IPS = ['127.0.0.1']
And add Django extension:
INSTALLED_APPS += ['django_extensions', ]

config.settings.production

In production settings, you should focus on the security for your project.

# security configuration
SECURE_HSTS_SECONDS = 60
SECURE_HSTS_INCLUDE_SUBDOMAINS = env.bool( 'DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS', default=True)
SECURE_CONTENT_TYPE_NOSNIFF = env.bool( 'DJANGO_SECURE_CONTENT_TYPE_NOSNIFF', default=True)
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
SECURE_SSL_REDIRECT = env.bool('DJANGO_SECURE_SSL_REDIRECT', default=True)
CSRF_COOKIE_SECURE = True
CSRF_COOKIE_HTTPONLY = True
X_FRAME_OPTIONS = 'DENY'
ADMIN_URL = env('DJANGO_ADMIN_URL')

At this point it’s worth to add DJANGO_ADMIN_URL to the production settings. Change it from default to avoid attack attempts on the default URL admin panel.

Next, you need to add your domain or domains:

ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS', default=['djangohotspot.pl', ])

And add a Gunicorn:

INSTALLED_APPS += ['gunicorn', ]

Finally, add django-storage for AWS:

INSTALLED_APPS += ['storages', ]
AWS_ACCESS_KEY_ID = env('DJANGO_AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = env('DJANGO_AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = env('DJANGO_AWS_STORAGE_BUCKET_NAME')
AWS_AUTO_CREATE_BUCKET = True
AWS_QUERYSTRING_AUTH = False
AWS_EXPIRY = 60 * 60 * 24 * 7
MEDIA_URL = 'https://s3.amazonaws.com/%s/' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

To efficiently track errors you may add Sentry:

INSTALLED_APPS += ['raven.contrib.django.raven_compat', ]
RAVEN_MIDDLEWARE = ['raven.contrib.django.raven_compat.middleware.SentryResponseErrorIdMiddleware']
MIDDLEWARE = RAVEN_MIDDLEWARE + MIDDLEWARE
SENTRY_DSN = env('DJANGO_SENTRY_DSN')
SENTRY_CLIENT = env('DJANGO_SENTRY_CLIENT', default='raven.contrib.django.raven_compat.DjangoClient')
SENTRY_CELERY_LOGLEVEL = env.int('DJANGO_SENTRY_LOG_LEVEL', logging.INFO)
RAVEN_CONFIG = {
  'CELERY_LOGLEVEL': env.int('DJANGO_SENTRY_LOG_LEVEL', logging.INFO),
  'DSN': SENTRY_DSN,
}

And if you want to serve static files, add WhiteNoise:

WHITENOISE_MIDDLEWARE = ['whitenoise.middleware.WhiteNoiseMiddleware', ]
MIDDLEWARE = WHITENOISE_MIDDLEWARE + MIDDLEWARE
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

config.setting.test

Configuring test settings, start with turning debugging off:

DEBUG = False
TEMPLATES[0]['OPTIONS']['debug'] = False

Store sent mails in memory. They are available in django.core.mail.outbox:

EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'

Set the cache:

CACHES = {    
  'default': {      
     'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',       
     'LOCATION': ''    
  }
}

Set the password hasher to speed up the tests:

PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher', ]

If you use Django templates, you can set them to be stored in memory:

TEMPLATES[0]['OPTIONS']['loaders'] = [
  ['django.template.loaders.cached.Loader', [
     'django.template.loaders.filesystem.Loader',
     'django.template.loaders.app_directories.Loader', ]
  , ]
, ]

uwsgi.py and urls.py files

Because we’ve split the main settings file into dedicated files with configuration for each environment, we need to point a file which will be used by default when it’s not clearly indicated.

In urls.py file we define the 4xx and 5xx pages. We also add debug toolbar here.

Move uwsgi.py and urls.py files from djangohotspot/djangohotspot catalogue to config module and add following changes to config.settings:

WSGI_APPLICATION = 'config.wsgi.application'
ROOT_URLCONF = 'config.urls'

At the end of the config.urls file add the following code to debug 4xx and 5xx pages:

if settings.DEBUG:
     urlpatterns += [
          url(r'^400/$', 
              default_views.bad_request, 
              kwargs={'exception': Exception('Bad Request!')}),
          url(r'^403/$', 
              default_views.permission_denied, 
              kwargs={'exception': Exception('Permission Denied')}),
          url(r'^404/$', 
              default_views.page_not_found, 
              kwargs={'exception': Exception('Page not Found')}),
          url(r'^500/$', 
              default_views.server_error),
     ]
if 'debug_toolbar' in settings.INSTALLED_APPS:
     import debug_toolbar
          urlpatterns = [
          url(r'^__debug__/', include(debug_toolbar.urls)),
] + urlpatterns

In our example, config.uwsgi file will look like this:

import os
import sys 
from django.core.wsgi import get_wsgi_application
app_path = os.path.dirname(os.path.abspath(__file__)).replace('/config', '')
sys.path.append(os.path.join(app_path, 'djangohotspot'))

if os.environ.get('DJANGO_SETTINGS_MODULE') == 'config.settings.production':
  from raven.contrib.django.raven_compat.middleware.wsgi import Sentry

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production")

application = get_wsgi_application()
if os.environ.get('DJANGO_SETTINGS_MODULE') == 'config.settings.production':
  application = Sentry(application)

Summary

As you’ve seen in this article, setting a Django project for multiple environments is a toilsome task. But trust me, it pays off quickly once you start working on the project.

From this point on, you can think of some containerization with Docker, which will give you portability and easiness of setup for your project regardless the environment it will be run on.

Do you have any questions regarding this Django set-up? Or maybe you’d like to join us and work on exciting projects? If yes, check our job offer for Python Developers.

K H