-
-
Notifications
You must be signed in to change notification settings - Fork 3
multi queue.7
multi-queue - indimail's multi queue architecture
indimail can be configured to use multiple queues for mail deliveries, using the qscheduler(8) daemon. These can be configured by setting few environment variables and by specifying command-line switches to the qscheduler(8) daemon. The command line options can be used either to configure a set of fixed static queues or have the queue increased dynamically with load. The load is the ratio of concurrency used and total concurrency configured for a queue. These are actually two ratios. One ratio for local deliveries and another ratio for remote deliveries.
A queue in indimail is denoted by a number and a base path. The compile time default is /var/indimail/queue, but can be changed by the control file /etc/indimail/control/queue_base or the environment variable QUEUE_BASE. The environment variable takes precedence. The queue number and the base path constitutes the actual path for the queue. e.g.
queue numbered 1, means /var/indimail/queue/queue1
queue numbered 2, means /var/indimail/queue/queue2
queue numbered n, means /var/indimail/queue/queuen
The compile time default for the first queue of the multi-queue environment is queue1. This can be changed by setting QUEUE_START environment variable. The compile time default for the number of queues that indimail uses is 5, which can be changed by setting QUEUE_COUNT environment variable.
The following environment variables are used to configure the queues.
QUEUE_START - defines the first queue that should be used. QUEUE_START=1 implies that /var/indimail/queue/queue1 be the first queue
QUEUE_COUNT - defines the total number of queues that should be used for mail deliveries. This is the fixed upper limit when qscheduler(8) is passed the -s argument
As explained earlier, qscheduler(8) can be used for configuring mail delivery with fixed number of queues or load dependent dynamic queues. The following additional environment variables can be set when dynamic queues are used.
QUEUE_MAX - defines the maximum queues that qscheduler can use for deliveries when dynamic queues is configured (-d argument to qscheduler(8). The compile time default is 20.
QUEUE_LOAD - defines the max queue load after which qscheduler(8) should increment the queue count by creating a new queue and starting an additional qmail-send(8), todo-proc(8) instance. The compile time default is 90.
ERROR_INTERVAL - Used by qscheduler(8) when in dynamic mode. Any failure in accessing message queues, allocating memory, opening control files qscheduler sleeps for ERROR_INTERVAL seconds before continuing. The compile time default is 5 seconds.
qscheduler(8) uses POSIX inter-process communication namely shared memory segments and message queues. The various forms of IPC are described as below
shared memory /qscheduler
This is created by qscheduler(8) using shm_open(3), owned by
root:root with perm 0644.
This segment is used to store two integers
queue count - qcount number of queues in use by qscheduler(8). This value will be the value of QUEUE_COUNT environment variable on qscheduler(8) startup. This is a dynamic number that is incremented when the total concurrency load goes beyond a threshold (defined by QUEUE_LOAD environment variable). qscheduler(8) will increase the queue count upto a maximum defined by the QUEUE_MAX environment variable. qmail-multi(8), qmail-queue(8) cycles through qcount queues to find the queue with the least concurrency and uses that queue to queue the message.
[step]
queue configured - qconf number of existing queues in /var/indimail directory. This number is incremented by qscheduler when it creates a new queue. This number cannot go beyond the value of QUEUE_MAX environment variable.
shared memory /queueN
This is created by qscheduler(8) but used by qmail-send(8) to
write its current concurrency value and the maximum concurrency
configured (for both local and remote deliveries). The concurrency is
incremented whenever a job is opened and decremented whenever a job is
closed. This is owned by qmails:qmail with perm 0644. The queue load is
the ratio of current concurrency and the maximum concurrency.
The program qtop(8) uses the above shared memory segments to display the queues with top load continuously without using file IO. qmail-qread(8) is another program that uses the above shared memory to display the queue status.
message queue /qscheduler
This is created by qscheduler(8) and used by qmail-send(8) to
communicate back, when its queue reaches, the maximum threshold
percentage for concurrency to total concurrency ratio. When the average
concurrency load reaches QUEUE_LOAD, qscheduler(8) starts a new
qmail-send(8), qmail-send(8) instance to install a new queue.
This is owned by root:root with perm 0600. This message queue is also
used by setqload(8) to set queue load by sending a message on
qscheduler message queue to set the queue laod or to increase the
number of queues manually at runtime. setqload(8) can also be used
to disable a queue temporarily.
message queue /queueN
For each queue, qscheduler(8) creates a message queue using
mq_open(3). Here N refers to the queue number). This is owned by
qmailq:qmail with perm 0640.
This message queue is used by qmail-queue(8) to communicate to todo-proc(8) the split dir and the inode number of the message in the outgoing queue. This is more efficient than the trigger method that qmail-queue uses, where the trigger wakes up qmail-send(8) to scan the todo directory. Scanning todo using opendir(3), readdir(2), also introduces what is known as the Silly Qmail Syndrome (SQS), where a single instance of qmail-send(8) cannot keep up with the injection rate from multiple qmail-queue clients. The solution to this problem is to have a seperate process to process todo directory. Having multiple instances of qmail-send can also help avoiding SQS. These instances of qmail-send can be a fixed number (qscheduler -s) or increased dynamically (qscheduler -d).
qscheduler(8), setqload(8), qtop(8) qmail-qread(8), qmail-start(8), qmail-multi(8), qmail-queue(8), qmail-send(8), todo-proc(8), shm_open(3) mq_open(3) ipcs(1) mq_overview(7)