Ubuntu 12.04 – Install Gearman and GearmanManager

Gearman is a wonderful job management system. It allows a web application to pass work to be done on other servers that may be more powerful or more capable in other ways. In addition, multiple jobs can be done asyncronously. But, it’s a bit hard to get it all set up and working. This tutorial will show you how to install the latest version of Gearman, GearmanManager and Gearman Monitor on Ubuntu 12.04. Supervisor is often used to start Gearman workers and keep them running. GearmanManager looks a lot like Supervisor, but is even easier to use for Gearman workers.

Perform all actions as root!

Gearman

Install packages that are required to install the gearmand service.

apt-get install gcc autoconf bison flex libtool make libboost-all-dev libcurl4-openssl-dev curl libevent-dev memcached uuid-dev libsqlite3-dev libmysqlclient-dev gperf libcloog-ppl0

The version of Gearman in the repository of Ubuntu 12.04 is very old, so get the latest Gearman version at: https://launchpad.net/gearmand

cd /opt
wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz
tar xvzf gearmand-1.1.12.tar.gz
cd gearmand-1.1.12
./configure
make
make install
ldconfig

Persistent queue

By default the queue of the Gearman job server exists in memory. So if you restart Gearman or reboot your server, all the jobs in the queue will be gone. It probably depends on your situation, but I prefer a persistent storage. You can choose between MySQL and SQLite3. I prefer MySQL.

Create the MySQL database “gearman”. Gearman will automatically create a table if it doesn’t exist when the gearmand service starts.

Create a startup script for the gearmand service so it will automatically start when the server boots:

nano /etc/init/gearmand.conf

description "Gearmand Server"

start on started mysql
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-user=example \
--mysql-password=example01 \
--mysql-db=gearman \
--mysql-table=gearman_queue \
2>> /var/log/gearmand.log

Make sure you set the correct MySQL settings!

Troubleshooting / running tests

Start the gearmand service now:

service gearmand start

Check if the gearmand service is running:

ps ax | grep gearmand

Check if germand is listening for jobs on TCP port 4730:

lsof -i tcp:4730

Gearman PHP extension

To communicate with the Gearman job server, the Gearman PHP extension is required. Look for the latest version of the extension at: http://pecl.php.net/package/gearman
If you don’t have pecl installed, execute:

apt-get install php-pear php5-dev

If pecl is installed, execute:

pecl install gearman-1.1.2
chmod 644 /usr/lib/php5/<long number like 20121212>/gearman.so

Add the Gearman extension to the PHP configuration:

nano /etc/php5/mods-available/gearman.ini

extension=gearman.so

Enable the Gearman PHP extension:

cd /etc/php5/cli/conf.d
ln -s ../../mods-available/gearman.ini 20-gearman.ini
cd /etc/php5/fpm/conf.d
ln -s ../../mods-available/gearman.ini 20-gearman.ini

Restart PHP:

service php5-fpm restart

GearmanManager

Go to http://brian.moonspot.net/GearmanManager if you like to read some background information about the GearmanManager.

Install GearmanManager:

cd /opt
git clone https://github.com/brianlmoon/GearmanManager.git gearman-manager
cd gearman-manager
./install/install.sh

Choose pecl by typing 1 and press ENTER.

The GearmanManager requires the PHP5 PCNTL extension. There is no debian package for it available that can be installed with apt-get. But it can be installed this way:

mkdir /tmp/phpsource
cd /tmp/phpsource
apt-get source php5
cd /tmp/phpsource/php5-*/ext/pcntl
phpize
./configure
make
cd modules
cp pcntl.so /usr/lib/php5/<long number like 20121212>/
chmod 644 /usr/lib/php5/<long number like 20121212>/pcntl.so

Add the pcntl extension to the PHP configuration:

nano /etc/php5/mods-available/pcntl.ini

extension=pcntl.so

Do NOT create a symlink like you would normally do in /etc/php5/cli/conf.d, because you will get a PHP warning everytime you run PHP: “PHP Warning: Module ‘pcntl’ already loaded in Unknown on line 0″.

We just want to enable pcntl for CLI, so change the disable_functions setting in the php.ini of PHP CLI.

nano /etc/php5/cli/php.ini

;disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
disable_functions =

Test if pcntl extension is active on the command line with:

php -i | grep pcntl

The GearmanManager configuration can be edited in /etc/gearman-manager/config.ini. By default gearman runs under the user gearman. If you want to configure another user, the documentation says add to config.ini:

; The user to run the daemon as
user=example

But that didn’t work. Too bad I had to change the GEARMANUSER variable as workaround in /etc/init.d/gearman-manager:

GEARMANUSER="example"

Start the gearman-manager service on startup:

update-rc.d gearman-manager defaults

Start the gearman-manager service now:

service gearman-manager start

You will probably get an error that no worker scripts are available. They can be put in the /etc/gearman-manager/workers directory. For example add the following worker:

nano /etc/gearman-manager/workers/ExecuteZendFramework2Task.php

<?php

class ExecuteZendFramework2Task
{
public function run($job, &$log)
{
$workload = $job->workload();

// do work on $job here as documented in pecl/gearman docs

// Log is an array that is passed in by reference that can be
// added to for logging data that is not part of the return data
$log[] = "Success";

$output = array();
//exec('/usr/bin/php /vagrant/zf2application/public/index.php cache clear', $output);

$result = 'Executed ZF2 task!' ."\n";
$result .= implode("\n", $output);

// return your result for the client
return $result;
}
}

Try to start the GearmanManager again:

service gearman-manager start

Now create a Gearman client that will add a ExecuteZendFramework2Task job to the queue.

nano /opt/gearman-client.php

<?php
$client= new GearmanClient();
$client->addServer("localhost", 4730);
echo $client->do("ExecuteZendFramework2Task", "Some optional data to pass...");
echo 'Okay, awesome that worked!' . "\n";

And execute it:

php /opt/gearman-client.php

This should be displayed in your terminal:

Executed ZF2 task!
Okay, awesome that worked!

Asynchronous

nano /opt/gearman-client-async.php

<?php
$client= new GearmanClient();
$client->addServer("localhost", 4730);
echo $client->doBackground("ExecuteZendFramework2Task", "Some optional data to pass...");
echo 'Added job to the Gearman Job Server asynchronously.' . "\n";

And execute it:

php /opt/gearman-client-async.php

This should be displayed in your terminal:

H:example.com:8Added job to the Gearman Job Server asynchronously.

If you stop the gearman workers with “service gearman-manager stop”, you will see jobs appearing in the MySQL database. As soon as you start the workers again with “service gearman-manager start” they will pick up the jobs and the list will be empty again.

Gearman Monitor

To list all the jobs that are in the queue you could view the gearman_queue table in the gearman MySQL database. But it’s better to respect that database as an ordinary Gearman storage and not get that information there directly.

It’s much better too use for example the Gearman Monitor. Installation is easy:

cd /opt
git clone https://github.com/yugene/Gearman-Monitor.git gearman-monitor

Make it publicly available by a web server, for example:

ln -s /opt/gearman-monitor /var/www/gearman-monitor

If you open the gearman-monitor in your webbrowser you will probably get an error about Net_Gearman that is not installed. Install the dependency:

pear install Net_Gearman-0.2.3

And configure the correct settings:

cd /opt/gearman-monitor
nano _config.php

$cfgServers[$i]['address'] = '127.0.0.1:4730';
$cfgServers[$i]['name'] = 'Gearman';

Reboot your server now to make sure all services come up again correctly. It all should be working fine, enjoy!