RHEL6

After upgrading to Sensu 0.23.2-2 on RHEL 6.8 the Sensu server would not start. The logs looked like this:

{"timestamp":"","level":"debug","message":"connecting to redis","settings":{"host":"localhost","port":6379}}
{"timestamp":"","level":"debug","message":"connecting to transport","name":"rabbitmq","settings":{"ssl":{"private_key_file":"/etc/sensu/ssl/client_key.pem","cert_chain_file":"/etc/sensu/ssl/client_cert.pem"},"host":"localhost","port":5671,"user":"REDACTED","password":"REDACTED","vhost":"/sensu"}}
{"timestamp":"","level":"warn","message":"reconnecting to redis"}
{"timestamp":"","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"","level":"debug","message":"not currently the leader"}
{"timestamp":"","level":"debug","message":"subscribing to keepalives"}
{"timestamp":"","level":"debug","message":"subscribing to results"}
{"timestamp":"","level":"debug","message":"{\"name\":\"client.example.com\",\"address\":\"10.1.2.3\",\"safe_mode\":false,\"subscriptions\":[\"os\"],\"version\":\"0.23.2\",\"timestamp\":1463588088}"}
{"timestamp":"","level":"debug","message":"updating client registry","client":{"name":"client.example.com","address":"10.1.2.3","safe_mode":false,"subscriptions":["os"],"version":"0.23.2","timestamp":1463588088}}


The cause of this is that localhost is resolving to ::1 and Redis is not listening with IPv6.

There are two solutions. One is to go through your /etc/sensu configuration files and change localhost to 127.0.0.1 so that you connect via IPv4.

The other solution is to change Redis configuration to bind to 127.0.0.1 and ::1.

Reference: https://github.com/sensu/sensu/issues/1209

Sensu-Admin is a nice GUI for the Sensu monitoring framework. However, installing it on Red Hat Enterprise Linux 6 ends in pain, since RHEL6 comes with an older version of ruby (1.8).

You might be creative and get most of the required gems installed but ultimately there will be gems that simply require ruby 1.9:

Fortunately, Red Hat has created Software Collections which provides a modern distribution of the major languages, including ruby 1.9.3.

I enabled the Red Hat Software Collections 1 Beta channel in Red Hat Network and proceeded to install ruby 1.9.3. Note that the RHEL Server Optional channel is also enabled, as is EPEL.

Then I set paths for ruby 1.9:

Check that the correct ruby is being found:

Now to install the dependencies needed for the gems needed for sensu-admin:

Install all the necessary gems:

Edit sensu-admin/db/seeds.rb to modify the default username and password. Then use rake to install:

Open port 3000 in your firewall:

And start the thin webserver which will run on port 3000:

Point a web browser to http://yourserver:3000 and you'll see the sensu-admin logon screen:

Reference: Puppet template for ruby 1.9.3 environment variables

R from EPEL has been updated to version 3.

We would like to run rpy2 but as of rpy2 version 2.3.x, it requires Python 2.7.

Red Hat Enterprise Linux 6 comes with Python 2.6.

So we need Python 2.7 so we can use rpy2 2.3 so we can use R 3.

The solution is to create a Python virtualenv containing Python 2.7 and install rpy2 in there. That way a user can log in and get Python 2.7 when they type python at the command line.

Take a moment to make sure standard build tools are installed. At a minimum, we need the following since we want numpy to use ATLAS, BLAS, and LAPACK, and not run into problems using SSL from Python:

Python 2.7 can now be installed from source:

Now setuptools:

Finally, virtualenv. This installs the virtual environment at /usr/local/ve but you might want it elsewhere.

That last line activates the virtual environment, so adding it to your .bash_profile might be handy.

Trying to add numpy with easy_install-2.7 didn't work for me:

However, pip worked like a charm:

Likewise rpy2 worked fine:

Reference: Undefined symbol (Rf_PrintWarnings) when using rpy 2.3.1 and R 3.0.1

I had a RHEL6 machine with lots of large MySQL binary log files in /var/log/mysql and was in need of more space on that volume. I am running MySQL 5.5 from IUS.

The system had an unused LVM logical volume. So I proceeded to mount the LVM logical volume at /var/log/mysql so that binary logs would get written to the logical volume.

Step one: unmount the logical volume

Not strictly necessary for a rename but a nice precaution.

Step two: rename the logical volume

The volume was named foo (not really true but it will server for this example) and was in the volume group named vg_db. I renamed it to something obvious, namely lv_mysqllogs.

Step three: create a temporary directory as a mount point

I'll put the new directory right next to the existing /var/log/mysql.

Step four: purge binary logs

In order to have only a small amount of data to move, I checked with our MySQL slave server to make sure it was up-to-date, then logged into MySQL as root and purged the logs:

Step five: mount LVM logical volume

Step six: copy the binary log files

For this, we have to stop MySQL.

And then copy logs:

Step seven: unmount LVM and remount at /var/log/mysql

We want to keep the original directory around in case things go horribly wrong.

And then move our temporary directory to its final destination for use as a mount point:

And remount the LVM logical volume to /var/log/mysql:

This is a good time to update /etc/fstab with your new mount entry so it will survive reboots:

Step eight: start up MySQL

Here we go:

Step nine: PANIC

(Note to my readers. Ach, who am I kidding? Note to my reader: if you follow the steps above you should not end up here. I had forgotten to change ownership of the newly-mounted volume to mysql, so when MySQL tried to start it tried really hard but couldn't -- see Bonus tidbit, below. I'm adding these details for frantic Googlers whose MySQL is down with errno 13).

No, don't panic. Just look in /var/log/mysqld.log to see why it's not starting:

Step ten: breathe. Just breathe.

The errno 13 shown above is a permissions error. Let's review the permissions:

That all looks good. So it must be SELinux. Let's check:

Aha! See how bin-log.000189 is marked var_log_t instead of mysqld_log_t? Let's restore its correct SELinux settings and look again:

Step eleven: start MySQL, for real

Yay!

Bonus tidbit

If you start up MySQL and it doesn't start, and /var/log/mysqld.log says

the problem is with the ownership of /var/log/mysql. Compare what you've got

with what you want to have:

See how . is owned by root:root instead of mysql:mysql? That's why you want to do