Installation Instructions for production environments¶

Installation using the Python package manager pip¶

Make sure you have the pip Python package manager installed on your Python 3 environment, and the C library for YAML development. On Debian/Ubuntu, the easiest way to do that is:

apt-get install python3-pip libyaml-dev

Install squad:

pip3 install squad

By default, SQUAD works with sqlite, but if Postgres is required, then specific binaries are needed:

apt-get install python3-pip libyaml-dev libpq-dev

Install squad with Postgres support:

pip3 install squad[postgres]

Message broker¶

In order to SQUAD processes to be able to communicate between each other, you need to install an AMQP server. We recommend RabbitMQ:

apt-get install rabbitmq-server

By default SQUAD will look for an AMQP server running on localhost, listening to the standard port, so for a single-server deployment you don't need to do anything else.

If you have a multi-server setup, then each server needs to be configured with the location of a central AMQP server. See the SQUAD_CELERY_BROKER_URL in the "Further configuration" section below.

Processes¶

SQUAD is composed of 4 different process:

web application server
background worker (celery worker)
periodic task scheduler (celery beat)
CI backend listener

To run the web interface, run as a dedicated user (i.e. don't run the application as root!):

squad

Note: if that doesn't work, ~/.local/bin is probably missing in the $PATH environment variable.

This will make the web UI available at http://localhost:8000/. To serve the UI to external users, you will need to setup a public-facing web server such as Apache or nginx and reverse proxy to localhost on port 8000. You can change the address or port SQUAD listens to by passing the --bind command line option, e.g. to make it listen to port 5000 on the local loopback interface, use:

squad --bind 127.0.0.0:5000

For production usage, you will want to tweak at least the database configuration. Keep reading for more information.

After starting SQUAD, but before acessing it, you will need a user. To create an admin user for yourself, use:

squad-admin createsuperuser

These are the command lines to run the other processes:

Process	Command
worker	celery -A squad worker
scheduler	celery -A squad beat
listener	squad-admin listen

You most probably want all the processes (including the web interface) being managed by a system manager such as systemd, or a process manager such as supervisor.

For an example deployment, check the configuration management repository for Linaro's qa-reports (using ansible).

After having the necessary processes running, there are a few extra setup steps needed:

Create Backend instances for your test execution backends. Go to the administration web UI, and under "CI", choose "Backends".
For each project, create authentication tokens and subscriptions

Worker configuration¶

The worker process handles background tasks, such as submitting CI jobs, fetching the results fo CI jobs, preparing reports that require intensive processing, etc. Some tasks take a lot longer than the others, e.g. submitting a CI job is pretty quick, while fetching CI job results takes some time to fetch all the data, parse it, and store in the database.

To allow for better load balancing, these tasks are split into multiple queues:

ci_fetch
ci_poll
ci_quick
core_notification
core_postprocess
core_quick
core_reporting

ci_fetch and ci_poll can be potentially slow, and if there is a large influx of those types of tasks, your system may display some congestion because all available worker threads are occupied running slow tasks, while several of the quick ones are waiting their turn.

To avoid this, you might want to start a small part of your workers so that they will not pick up any of those slow tasks:

squad worker --exclude-queues ci_fetch,ci_poll

Or you can also give an explicit task list, which is less flexible but might be useful:

squad worker --queues core_quick,ci_quick

By default, workers listen to all queues.

For message brokers that support prefixed-queue names, SQUAD has the optional environment variable SQUAD_CELERY_QUEUE_NAME_PREFIX, that, if set, will prepend it before all queue names. SQUAD also support adding a suffix via the optional environment variable SQUAD_CELERY_QUEUE_NAME_SUFFIX

Further configuration¶

The following environment variables affect the behavior of SQUAD:

DATABASE: controls the database connection parameters. Should contain KEY=VALUE pairs, separated by colons (:).

By default, SQUAD will use a SQLite3 database in its internal data directory.

For example, to use a PostgreSQL database (requires the psycopg2 Python package to be installed):
```
DATABASE=ENGINE=django.db.backends.postgresql_psycopg2:NAME=mydatabase:USER=myuser:HOST=myserver:PASSWORD=mypassword
```
SQUAD_EXTRA_SETTINGS: path to a Python file with extra Django settings.
SQUAD_SITE_NAME: name to be displayed at the page title and navigation bar. Defaults to 'SQUAD'.
XDG_DATA_HOME: parent directory of the SQUAD internal data directory. Defaults to ~/.local/share. The actual data directory will be ${XDG_DATA_HOME}/squad.
SECRET_KEY_FILE: file to store encryption key for user sessions. Defaults to ${XDG_DATA_HOME}/squad/secret.dat
DJANGO_LOG_LEVEL: the logging level used for Django-related logging. Default: INFO.
SQUAD_LOG_LEVEL: the logging level for SQUAD-specific logging. Default: INFO.
SQUAD_HOSTNAME: hostname used to compose links in asynchronous notifications (e.g. emails). Defaults to the FQDN of the host where SQUAD is running.
SQUAD_BASE_URL: Base URL to the SQUAD web interface, used when composing links in notifications (e.g. emails). Defaults to https://$SQUAD_HOSTNAME.
SQUAD_EMAIL_FROM: e-mail used as sender of email notifications. Defaults to noreply@$SQUAD_HOSTNAME.
SQUAD_EMAIL_HOST: hostname to use as e-mail delivery host. Sets Django's EMAIL_HOST setting. See the Django documentation on sending email for more details.

SQUAD_LOGIN_MESSAGE: a message to be displayed to users right above the login form. Use for example to provide instructions on what credentials to use. Defaults no message.
SQUAD_ADMINS: Comma-separated list of administrator email addresses, for use in exception notifications. Each address must be formatted as First Last <first.last@example.com>.
SQUAD_SEND_ADMIN_ERROR_EMAIL: Determines whether or not to send exception notifications to administrators. Defaults to True.
SENTRY_DSN: Defines Sentry's DSN token, if defined SQUAD will attempt to import Sentry SDK and use it. Defaults to None. If Sentry is configured it's recommended to disable sending admin notifications by setting SQUAD_SEND_ADMIN_ERROR_EMAIL = False.
SQUAD_STATIC_DIR: Directory where SQUAD will find it's preprocessed static assets. This usually does not need to be set manually, and exists mostly for use in the Docker image.
SQUAD_CELERY_BROKER_URL: URL to the broker to be used by Celery for background jobs. Defaults to amqp://localhost:5672.
SQUAD_CELERY_QUEUE_NAME_PREFIX: Name to prefix all queues in Celery. Useful when multiple environments share the same broker. Defaults to ''.
SQUAD_CELERY_QUEUE_NAME_SUFFIX: Name to concatenate all queues in Celery. Useful when a queue extension is needed by the broker. Defaults to ''.
SQUAD_CELERY_POLL_INTERVAL: Number of seconds a worker will sleep after an empty answer from SQS before the next polling attempt. Defaults to 1.

User management¶

SQUAD provides 'users' management command that allows to list, add, update and display details about users. This command comes handy when trying to automate SQUAD setup with containers. Details about user management with 'users' command:

list Displays list of all available users with their names (first, last) from database

details <username> Displays details about requested username. Details include:

username

is_active

is_staff

is_superuser

groups

add <username> Adds new user with given 'username'. It also takes additional parameters

--email EMAIL email of the user

--passwd PASSWD Password for this user. If empty, a random password is generated.

--staff Make this user a staff member

--superuser Make this user a super user

update <username> Updates database record of existing user identified with 'username'. It takes additional parameters

--email EMAIL Change email of the user

--active Make this user active

--not-active Make this user inactive

--staff Make this user a staff member

--not-staff Make this user no longer a staff member

--superuser Make this user a superuser

--not-superuser Make this user no longer a superuser

Docker compose setup¶

Alternatively, SQUAD can be run inside a Docker container using Docker Compose. A baseline setup is provided here.

Not only will this run all SQUAD processes, but also automatically pull and start all required services to run a local SQUAD instance with a web interface. All container images are pulled automatically from Dockerhub.

Please be aware, that some of the predefined settings, like SQUAD_BASE_URL or SQUAD_EMAIL_FROM in docker-compose.yml and squad-apache.conf will need to be changed. To use this setup, copy all of the provided files locally and run the docker using:

docker compose up -d

After that, the logs and status of the container can be checked resp. using:

docker compose logs

and:

docker stats

If everything has started successfully, the frontend should be reachable under http://localhost:10080.

Before being able to actually use the frontend of SQUAD, a user that acts as an administrator (with superuser rights) needs to be created. This can be done by entering the container with:

docker compose exec squad-frontend /bin/bash

While being in the container, a new administrator account can be created with the following command:

squad-admin createsuperuser

After this command, enter the required data and exit the container environment by issuing exit.

Now, you should be able to login on the frontend web interface, using the credentials you just entered while creating the administrator account.

To stop the SQUAD instance, issue the following command to stop the container:

docker compose down

Database dump and restore¶

Note that container must be running for the following commands to work.

To dump the database to the file db-dumped, issue the following command:

docker compose exec -t postgres pg_dump -F custom -U postgres -f /db-dumped squad

To restore the database from the file db-to-be-restored, issue the following command:

docker compose exec -t postgres pg_restore -U postgres -c -d squad -j4 /db-to-be-restored