Solved: Sensu won't start after upgrade to 0.23.2-2

After upgrading to Sensu 0.23.2-2 on RHEL 6.8 the Sensu server would not start. The logs looked like this:

{"timestamp":"","level":"debug","message":"connecting to redis","settings":{"host":"localhost","port":6379}}
{"timestamp":"","level":"debug","message":"connecting to transport","name":"rabbitmq","settings":{"ssl":{"private_key_file":"/etc/sensu/ssl/client_key.pem","cert_chain_file":"/etc/sensu/ssl/client_cert.pem"},"host":"localhost","port":5671,"user":"REDACTED","password":"REDACTED","vhost":"/sensu"}}
{"timestamp":"","level":"warn","message":"reconnecting to redis"}
{"timestamp":"","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"","level":"debug","message":"not currently the leader"}
{"timestamp":"","level":"debug","message":"subscribing to keepalives"}
{"timestamp":"","level":"debug","message":"subscribing to results"}
{"timestamp":"","level":"debug","message":"updating client registry","client":{"name":"","address":"","safe_mode":false,"subscriptions":["os"],"version":"0.23.2","timestamp":1463588088}}

The cause of this is that localhost is resolving to ::1 and Redis is not listening with IPv6.

There are two solutions. One is to go through your /etc/sensu configuration files and change localhost to so that you connect via IPv4.

The other solution is to change Redis configuration to bind to and ::1.



Sensu config.json cannot be left blank

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-settings-3.4.0/lib/sensu/settings/loader.rb:292:in `deep_merge': undefined method `each' for nil:NilClass (NoMethodError)
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-settings-3.4.0/lib/sensu/settings/loader.rb:96:in `load_file'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-settings-3.4.0/lib/sensu/settings.rb:17:in `load'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-settings-3.4.0/lib/sensu/settings.rb:38:in `get'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/lib/sensu/daemon.rb:104:in `load_settings'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/lib/sensu/daemon.rb:50:in `initialize'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/lib/sensu/server/process.rb:40:in `initialize'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/lib/sensu/server/process.rb:28:in `new'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/lib/sensu/server/process.rb:28:in `run'
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.23.2/exe/sensu-server:10:in `'
from /opt/sensu/bin/sensu-server:22:in `load'
from /opt/sensu/bin/sensu-server:22:in `'

That's what it looks like when you put all your configuration information for sensu into /etc/sensu/conf.d and leave /etc/sensu/config.json empty.

To solve, you can just put the following into /etc/sensu/config.json and sensu will happily do a deep merge with your other files:


For more on how Sensu starts up, see Sensu Configuration Reference documentation.


Monitoring Drupal with Sensu

I gave a presentation called Monitoring Drupal with Sensu at DrupalCorn 2013.

Annotated slideset is up at the link above.


Installing sensu-admin on RHEL6

Sensu-Admin is a nice GUI for the Sensu monitoring framework. However, installing it on Red Hat Enterprise Linux 6 ends in pain, since RHEL6 comes with an older version of ruby (1.8).

You might be creative and get most of the required gems installed but ultimately there will be gems that simply require ruby 1.9:

Gem::InstallError: nokogiri requires Ruby version >= 1.9.2.
An error occurred while installing nokogiri (1.6.0), and Bundler cannot continue.

Fortunately, Red Hat has created Software Collections which provides a modern distribution of the major languages, including ruby 1.9.3.

I enabled the Red Hat Software Collections 1 Beta channel in Red Hat Network and proceeded to install ruby 1.9.3. Note that the RHEL Server Optional channel is also enabled, as is EPEL.

# yum install ruby193 ruby193-ruby-devel ruby193-rubygem-rake

Then I set paths for ruby 1.9:

# Setup PATH, LD_LIBRARY_PATH and MANPATH for ruby-1.9
ruby19_dir=$(dirname `scl enable ruby193 "which ruby"`)
export PATH=$ruby19_dir:$PATH
ruby19_ld_libs=$(scl enable ruby193 "printenv LD_LIBRARY_PATH")
export LD_LIBRARY_PATH=$ruby19_ld_libs:$LD_LIBRARY_PATH
ruby19_manpath=$(scl enable ruby193 "printenv MANPATH")
export MANPATH=$ruby19_manpath:$MANPATH

Check that the correct ruby is being found:

$ which ruby

Now to install the dependencies needed for the gems needed for sensu-admin:

# yum install make gcc gcc-c++ libxml2-devel libxslt-devel openssl-devel mysql-devel sqlite-devel

Install all the necessary gems:

# cd path/to/sensu-admin
# bundle install

Edit sensu-admin/db/seeds.rb to modify the default username and password. Then use rake to install:

# rake db:migrate
# rake db:seed

Open port 3000 in your firewall:

iptables -I INPUT 4 -p tcp --dport 3000 -m state --state NEW -j ACCEPT

And start the thin webserver which will run on port 3000:

rails s

Point a web browser to http://yourserver:3000 and you'll see the sensu-admin logon screen:

Reference: Puppet template for ruby 1.9.3 environment variables

Subscribe to RSS - Sensu