eGSIM deployment

The following cheatsheet covers two main aspect: Maintenance and Deployment (no docker container). It assumes that you already have access to a remote machine. The following instructions have been tested on Ubuntu 18.04 MATE. The procedure to obtain or update an SSL certificate (in order to have an https address) is covered in a separate document

Set the following variables to see meaningful paths (the document will update automatically):

Maintenance

Important files

Common operations

Update the package

Inside $EGSIM_CWD, as root user:

(cd $EGSIM_CWD && git pull)
service $DJANGO_PROJECT restart
service nginx restart

Update the package and the static files

Same as above but we need to run Django collectstatic in the activated virtualenv

source $VENV_ACTIVATE
(cd $EGSIM_CWD && git pull)
(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py collectstatic)
service $DJANGO_PROJECT restart
service nginx restart
deactivate

Update the db schema

IMPORTANT: Execute these command as $DJANGO_USER, thus FIRST type:

su $DJANGO_USER

source $VENV_ACTIVATE

(cd $EGSIM_CWD && git pull)
rm $EGSIM_DATA_DIR/egsim.db
(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py migrate)
(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py egsim_init)
service $DJANGO_PROJECT restart
service nginx restart

Install system fonts (to produce and download the same plots visible on the browser)

cd /usr/share/fonts
mkdir egsim
cd egsim
wget "https://fonts.google.com/download?family=Encode%20Sans%20Semi%20Condensed" -O ./Encode_Sans_Semi_Condensed.zip
unzip -d . ./Encode_Sans_Semi_Condensed.zip
sudo chmod -R --reference=/usr/share/fonts/opentype /usr/share/fonts/egsim
fc-cache -fv
# check if font is installed:
fc-match EncodeSansSemiCondensed

fc-list | grep Encode

Server installation

System requirements

sudo apt-get update && sudo apt-get upgrade

Install required libraries

apt install git nginx-light curl libhdf5-dev gcc python3-rtree

(libhdf5-dev gcc required by openquake, see below. python3-rtree required by gmpe-smtk, see below)

Optional installations

apt install vim

Install Python desired version

sudo add-apt-repository ppa:deadsnakes/ppa

sudo apt-get update
sudo apt-get install $CURRENT_PYTHON $CURRENT_PYTHON-dev $CURRENT_PYTHON-distutils $CURRENT_PYTHON-venv

	adduser --system --home=$EGSIM_DATA_DIR --no-create-home --disabled-password --group --shell=/bin/bash $DJANGO_USER
	

Final note: as always, proceed installing Python packages being aware that in case of problems, the solution after googling is to apt-get install missing packages. For instance, in case of Python 3.9 installation we had to: sudo apt-get install gfortran libopenblas-dev liblapack-dev libfreetype6-dev libgeos-dev

Program requirements

Create project folder

cd /opt
mkdir $DJANGO_PROJECT
cd $DJANGO_PROJECT

Clone package(s)

git clone https://github.com/rizac/eGSIM.git

Create the virtual environment

Installation

$CURRENT_PYTHON -m venv $VENV_PATH

Activation

source $VENV_ACTIVATE

IMPORTANT: from now on, every Python-related operation must be performed within your activated virtual env

Check correct path: activate virtualenv and type

which python
python --version

the Python path should refer to the newly installed Python in the current virtual environment (Python version should be 3.7+)

Install the repositories

For any error in any of the 'pip install' commands below, the most likely cause is some system package missing: we suggest to google the error first, then check if anybody suggests to apt install missing packages and follow the instructions. Ignore at first complicated suggestions, they are rarely the solution to your problem. This doc was created precisely following the pattern above. Eventually, we added in the apt-install command above all packages that we discovered missing, so hopefully a fresh new installation might work fine without any new error

Install eGSIM: activate virtualenv, then

cd ./eGSIM/
pip install -r ./requirements.dev.txt

Create project structure

Data directory with correct ownership

mkdir $EGSIM_DATA_DIR
chown $DJANGO_USER $EGSIM_DATA_DIR

Log directory with correct ownership

mkdir $EGSIM_LOG_DIR
chown $DJANGO_USER $EGSIM_LOG_DIR

Config directory. Change group ownership (with the flag --group added to the command adduser above we created a group with the same name as $DJANGO_USER). Change also permissions (750 means the current user, i.e. root, can read, write, and execute while the group and others cannot write)

mkdir $EGSIM_CONFIG_DIR
chgrp $DJANGO_USER $EGSIM_CONFIG_DIR
chmod 750 $EGSIM_CONFIG_DIR

Setup the project

Create the settings file: Create (or copy from a previous deployment file):

$EGSIM_CONFIG_DIR/settings.py

"""
Django settings for eGSIM project.

Copied from https://djangodeployment.com/wp-content/uploads/2017/03/Django-deployment-cheatsheet.pdf

This file should be copied from previous deployments and modified accordingly
Listed below are the Django variables that are most likely to be edited. Note that this module
inherits from the debug settings under the egsim git repository
"""

from egsim.settings_debug import *

DEBUG = False  # DO NOT CHANGE THIS!
ALLOWED_HOSTS = ['$DOMAIN']
# SECRET_KEY SHOULD BE UNIQUE FOR EACH SETTINGS FILE AND CAN BE GENERATED ON THE TERMINAL (WITHIN THE DJANGO VIRUAL ENV)  WITH THE COMMAND:
# python -c "from django.core.management.utils import get_random_secret_key;print(get_random_secret_key())"
# COPY THE OTUPUT STRING HERE BELOW
SECRET_KEY = ''
DATABASES = {
	'default': {
		'ENGINE': 'django.db.backends.sqlite3',
		'NAME': '$EGSIM_DATA_DIR/egsim.db',
	}
}
# static files root (path on the server)
STATIC_ROOT = '/var/cache/$DJANGO_PROJECT/static/'
# static files url (already set in settings debug, here for ref.):
# STATIC_URL = '/static/'

# media root (path on the server):
MEDIA_ROOT = '$EGSIM_DATA_DIR/media/'
# static files url (already set in settings debug, here for ref.):
# MEDIA_URL = '/media/'

CSRF_TRUSTED_ORIGINS = ["https://$DOMAIN"]  # required in Django 4+. Https or http depends on your SSL certificate

Run collectstatic: copy static files to the new location. Activate virtualenv and:

(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py collectstatic)

(the PYTHONPATH simply makes the option --settings=settings to point to the newly created settings.py). You should see a message like: '175 static files copied to '/var/cache/$DJANGO_PROJECT/static' (the number of files might be huge but there are also Django css and so on

Migration and data initialization scripts

IMPORTANT: Execute these command as $DJANGO_USER, thus FIRST type:

su $DJANGO_USER

source $VENV_ACTIVATE

Django migration

(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py migrate)

Initialize our database (OpenQuake GSIMs IMTs and TRTs)

(cd $EGSIM_CWD; export PYTHONPATH=$EGSIM_CONFIG_DIR DJANGO_SETTINGS_MODULE=settings; python manage.py egsim_init)

Server configuration

NGINX is the main application server, however, it delegates Gunicorn for serving the Python code (everything except static and media files).

IMPORTANT: remember that After any modification to the code or the config below, before checking the results online, is better to restart the services

service $DJANGO_PROJECT restart
service nginx restart

Gunicorn

Become root:

su root

The command above should get you out of the activated virtual environment. (Re)activate virtualenv and install Gunicorn:

pip install gunicorn

OPTIONAL, test: start Gunicorn and go to $DOMAIN (from the server or another external computer) to test:

export PYTHONPATH=$EGSIM_CONFIG_DIR; export DJANGO_SETTINGS_MODULE=settings
gunicorn --workers=4 --log-file=$EGSIM_LOG_DIR/gunicorn.log --bind=127.0.0.1:$GUNICORN_PORT --bind=[::1]:$GUNICORN_PORT egsim.wsgi:application

Create the systemd task to start automatically egsim django server

su $DJANGO_USER

Save the content below in /etc/systemd/system/$DJANGO_PROJECT.service. (or copy a previous service file and use the template below as reference. Remember that $DJANGO_PROJECT cannot contain dots)

Important: in the file below, note the --bind argument: it is typed twice to bind our Django application on localhost both on ipv6 and ipv4 interfaces. We will then tell Nginx (see below) to proxy each request to localhost (see proxy_pass argument) delegating Gunicorn for everything that is not a static file (everything not in media or static directories)

The URL port of the --bind arguments is set as , you can change the value in the field to automatically update the templates below. Changing the port is useful if you want to run several Gunicorn instances from Nginx (e.g., previous versions alongside the current)

[Unit]
Description=$DJANGO_PROJECT

[Service]
User=$DJANGO_USER
Group=$DJANGO_USER
Environment="PYTHONPATH=$EGSIM_CONFIG_DIR:$VENV_PATH"
Environment="DJANGO_SETTINGS_MODULE=settings"
WorkingDirectory=$EGSIM_CWD
ExecStart=$VENV_PATH/bin/gunicorn \
	--workers=4 \
	--log-file=$EGSIM_LOG_DIR/gunicorn.log \
	--bind=127.0.0.1:$GUNICORN_PORT --bind=[::1]:$GUNICORN_PORT \
	egsim.wsgi:application

[Install]
WantedBy=multi-user.target

To restart the service (second line is mandatory only if the script above has been changed):

service $DJANGO_PROJECT restart

To restart the service if the script above has been changed:

service $DJANGO_PROJECT stop
systemctl daemon-reload
service $DJANGO_PROJECT start

Optional. To start the service automatically on boot:

systemctl enable $DJANGO_PROJECT

NGINX

Create /etc/nginx/sites-available/$DJANGO_PROJECT (or edit the existing one). See template below with the relevant variables to be overwritten. NOTE: proxy_pass below must point to the Gunicorn URL just configured (see above).

The line proxy_pass http://localhost:$GUNICORN_PORT; is the line that forwards to Gunicorn all locations starting with '/' and not matching '/static/' or '/media/'

server {
	listen 80;
	listen [::]:80;
	server_name $DOMAIN www.$DOMAIN;
	location / {
		proxy_pass http://localhost:$GUNICORN_PORT;
	}
	location /static/ {
		alias /var/cache/$DJANGO_PROJECT/static/;
	}
	location /media/ {
		alias $EGSIM_DATA_DIR/media/;
	}
}

	cd /etc/nginx/sites-enabled
	ln -s ../sites-available/$DJANGO_PROJECT .
	

	127.0.0.1	$DOMAIN