Backend workers

Backend workers are used to process asynchronous tasks. Currently it is used for:

It’s based on a notification queue managed by Redis and a worker that will process the the queue as soon as it’s pushed. Unlike “SystemEvents” there is no delay between the queue and the processing of the job.

Configure backend workers

Install and configure Redis

Note

If redis is already configured, you just need to configure the connection with the server. If redis is installed for several servers, you must setup firewall rules to ensure only granted front-end servers can access it.

You must install redis from EPEL.

You will need to adapt 2 things in the configuration file /etc/redis.conf

  1. You should set a password (at least 30 chars) with requirepass key

  2. You should enable appendonly persistence.

We highly recommend that you read Redis Persistance Guide as well as Redis Security Guide to understand how data are stored and security practices.

Then start the server and make it on at reboot time

$ sudo service redis start
$ sudo chkconfig redis on

And finally set server parameters for Tuleap in your config file /etc/tuleap/conf/redis.inc

<?php

$redis_server   = '127.0.0.1';
$redis_port     = 6379;
$redis_password = '${REDIS_PASSWORD}';

Configure Tuleap

In local.inc you should set $sys_nb_backend_workers to a number greater than or equal to 1. This controls the number of workers to process background jobs. It should be adapted given your server workload. 2 is a good starting value.

You might need to service tuleap restart afterwards.

Troubleshooting

You can track worker activity in /var/log/tuleap/worker_log log file (you might need to change the $sys_logger_level value to make if more verbose).

Backend notifications

When you have a server with large amount of users or a mail system that is not really efficient, you may face troubles at artifact creation with very long creation/update timing.

By profiling your page or by enabling ‘debug’ ($sys_logger_level = 'debug';) you can identify how long the notification is taking.

Look at [Tuleap\Tracker\Artifact\Changeset\Notification\Notifier] string in codendi_syslog and measure how long it takes between Start notification and End notification marker. You can save this amount of time to your end users by switching to backend based notifications.

Configure Tuleap

In local.inc you should add $sys_async_emails variable. It can take following values:

  • '': equivalent to not defining the variable at all: disable backend notifications, the notification will be done inline. Useful to disable the feature if it doesn’t work.

  • 'all': activate the feature for all projects.

  • 'X,Y,Z': activate the feature for projects X, Y and Z (project ids, integers)

Note

$sys_async_emails is independent of $sys_nb_backend_workers configuration. This means that even if $sys_nb_backend_workers is not set (or set to 0), then backend notifications will still be available.

After having set the variable to at least 1 project, the backend worker (/usr/share/tuleap/src/utils/worker.php) will automatically be started by Tuleap and will process jobs and send emails.

Troubleshooting

The front end will log useful information in codendi_syslog with the key Notification.

We also added a double check in SYSTEM_CHECK system event to ensure there is no pending notifications that last forever. If such a situation occurs, the SystemEvent will be marked as Warning, be sure to monitor that.