RHEL6

Solved: Sensu won't start after upgrade to 0.23.2-2

After upgrading to Sensu 0.23.2-2 on RHEL 6.8 the Sensu server would not start. The logs looked like this:

{"timestamp":"","level":"debug","message":"connecting to redis","settings":{"host":"localhost","port":6379}}
{"timestamp":"","level":"debug","message":"connecting to transport","name":"rabbitmq","settings":{"ssl":{"private_key_file":"/etc/sensu/ssl/client_key.pem","cert_chain_file":"/etc/sensu/ssl/client_cert.pem"},"host":"localhost","port":5671,"user":"REDACTED","password":"REDACTED","vhost":"/sensu"}}
{"timestamp":"","level":"warn","message":"reconnecting to redis"}
{"timestamp":"","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"","level":"debug","message":"not currently the leader"}
{"timestamp":"","level":"debug","message":"subscribing to keepalives"}
{"timestamp":"","level":"debug","message":"subscribing to results"}
{"timestamp":"","level":"debug","message":"{\"name\":\"client.example.com\",\"address\":\"10.1.2.3\",\"safe_mode\":false,\"subscriptions\":[\"os\"],\"version\":\"0.23.2\",\"timestamp\":1463588088}"}
{"timestamp":"","level":"debug","message":"updating client registry","client":{"name":"client.example.com","address":"10.1.2.3","safe_mode":false,"subscriptions":["os"],"version":"0.23.2","timestamp":1463588088}}

The cause of this is that localhost is resolving to ::1 and Redis is not listening with IPv6.

There are two solutions. One is to go through your /etc/sensu configuration files and change localhost to 127.0.0.1 so that you connect via IPv4.

The other solution is to change Redis configuration to bind to 127.0.0.1 and ::1.

Reference: https://github.com/sensu/sensu/issues/1209

Topic: 

Installing sensu-admin on RHEL6

Sensu-Admin is a nice GUI for the Sensu monitoring framework. However, installing it on Red Hat Enterprise Linux 6 ends in pain, since RHEL6 comes with an older version of ruby (1.8).

You might be creative and get most of the required gems installed but ultimately there will be gems that simply require ruby 1.9:

Gem::InstallError: nokogiri requires Ruby version >= 1.9.2.
An error occurred while installing nokogiri (1.6.0), and Bundler cannot continue.

Fortunately, Red Hat has created Software Collections which provides a modern distribution of the major languages, including ruby 1.9.3.

I enabled the Red Hat Software Collections 1 Beta channel in Red Hat Network and proceeded to install ruby 1.9.3. Note that the RHEL Server Optional channel is also enabled, as is EPEL.

# yum install ruby193 ruby193-ruby-devel ruby193-rubygem-rake

Then I set paths for ruby 1.9:

# Setup PATH, LD_LIBRARY_PATH and MANPATH for ruby-1.9
ruby19_dir=$(dirname `scl enable ruby193 "which ruby"`)
export PATH=$ruby19_dir:$PATH
ruby19_ld_libs=$(scl enable ruby193 "printenv LD_LIBRARY_PATH")
export LD_LIBRARY_PATH=$ruby19_ld_libs:$LD_LIBRARY_PATH
ruby19_manpath=$(scl enable ruby193 "printenv MANPATH")
export MANPATH=$ruby19_manpath:$MANPATH

Check that the correct ruby is being found:

$ which ruby
/opt/rh/ruby193/root/usr/bin/ruby

Now to install the dependencies needed for the gems needed for sensu-admin:

# yum install make gcc gcc-c++ libxml2-devel libxslt-devel openssl-devel mysql-devel sqlite-devel

Install all the necessary gems:

# cd path/to/sensu-admin
# bundle install

Edit sensu-admin/db/seeds.rb to modify the default username and password. Then use rake to install:

# rake db:migrate
# rake db:seed

Open port 3000 in your firewall:

iptables -I INPUT 4 -p tcp --dport 3000 -m state --state NEW -j ACCEPT

And start the thin webserver which will run on port 3000:

rails s

Point a web browser to http://yourserver:3000 and you'll see the sensu-admin logon screen:

Reference: Puppet template for ruby 1.9.3 environment variables

Installing rpy2 2.3.x and numpy on RHEL6

R from EPEL has been updated to version 3.

We would like to run rpy2 but as of rpy2 version 2.3.x, it requires Python 2.7.

Red Hat Enterprise Linux 6 comes with Python 2.6.

So we need Python 2.7 so we can use rpy2 2.3 so we can use R 3.

The solution is to create a Python virtualenv containing Python 2.7 and install rpy2 in there. That way a user can log in and get Python 2.7 when they type python at the command line.

Take a moment to make sure standard build tools are installed. At a minimum, we need the following since we want numpy to use ATLAS, BLAS, and LAPACK, and not run into problems using SSL from Python:

yum install -y gcc atlas-devel blas-devel lapack-devel openssl-devel

Python 2.7 can now be installed from source:

curl -O http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
tar xvzf Python-2.7.3.tgz
cd Python-2.7.3
./configure --enable-shared
make altinstall
echo "/usr/local/lib" > /etc/ld.so.conf.d/python2.7.conf
ldconfig

Now setuptools:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-0.9.8.tar.gz
tar xvfz setuptools-0.9.8.tar.gz
cd setuptools-0.9.8
/usr/local/bin/python2.7 setup.py build
/usr/local/bin/python2.7 setup.py install --record installedfiles.txt

Finally, virtualenv. This installs the virtual environment at /usr/local/ve but you might want it elsewhere.

wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.1.tar.gz
tar xvfz virtualenv-1.9.1.tar.gz
cd virtualenv-1.9.1
python2.7 setup.py install --record installedfiles.txt
virtualenv-2.7 /usr/local/ve
source /usr/local/ve/bin/activate

That last line activates the virtual environment, so adding it to your .bash_profile might be handy.

Trying to add numpy with easy_install-2.7 didn't work for me:

(ve)[snorky]# easy_install-2.7 numpy
...
Running numpy-1.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-u0mtJj/numpy-1.7.1/egg-dist-tmp-kGRfag
Running from numpy source directory.
error: SandboxViolation: open('/dev/null', 'w') {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

However, pip worked like a charm:

(ve)[snorky]# pip install numpy
...
(ve)[snorky]$ python
Python 2.7.3 (default, Jul 29 2013, 17:08:17)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy

Likewise rpy2 worked fine:

pip install rpy2
...

(ve)[snorky]# python
Python 2.7.3 (default, Jul 29 2013, 17:08:17)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2
>>> import rpy2.rinterface

Reference: Undefined symbol (Rf_PrintWarnings) when using rpy 2.3.1 and R 3.0.1

Topic: 

Moving MySQL Binary Logs to a New LVM Logical Volume

I had a RHEL6 machine with lots of large MySQL binary log files in /var/log/mysql and was in need of more space on that volume. I am running MySQL 5.5 from IUS.

The system had an unused LVM logical volume. So I proceeded to mount the LVM logical volume at /var/log/mysql so that binary logs would get written to the logical volume.

Step one: unmount the logical volume

Not strictly necessary for a rename but a nice precaution.

# umount /dev/mapper/vg_db-lv_foo

Step two: rename the logical volume

The volume was named foo (not really true but it will server for this example) and was in the volume group named vg_db. I renamed it to something obvious, namely lv_mysqllogs.

# lvrename vg_db lv_foo lv_mysqllogs
  Renamed "lv_foo" to "lv_mysqllogs" in volume group "vg_db"

Step three: create a temporary directory as a mount point

I'll put the new directory right next to the existing /var/log/mysql.

# mkdir /var/log/mysql.new
chown mysql:mysql /var/log/mysql.new

Step four: purge binary logs

In order to have only a small amount of data to move, I checked with our MySQL slave server to make sure it was up-to-date, then logged into MySQL as root and purged the logs:

mysql> PURGE BINARY LOGS BEFORE '2013-06-04 00:00:01';
Query OK, 0 rows affected (3.81 sec)

Step five: mount LVM logical volume

# mount /dev/mapper/vg_db-lv_mysqllogs /var/log/mysql.new
chown mysql:mysql /var/log/mysql.new

Step six: copy the binary log files

For this, we have to stop MySQL.

# service mysqld stop
Stopping mysqld:                                           [  OK  ]

And then copy logs:

# cp -Rp /var/log/mysql/* /var/log/mysql.new/

Step seven: unmount LVM and remount at /var/log/mysql

# umount /dev/mapper/vg_db-lv_mysqllogs

We want to keep the original directory around in case things go horribly wrong.

# mv /var/log/mysql /var/log/mysql.old

And then move our temporary directory to its final destination for use as a mount point:

# mv /var/log/mysql.new /var/log/mysql

And remount the LVM logical volume to /var/log/mysql:

# mount /dev/mapper/vg_db-lv_mysqllogs /var/log/mysql

This is a good time to update /etc/fstab with your new mount entry so it will survive reboots:

/dev/mapper/vg_db-lv_mysqllogs /var/log/mysql                   ext4    defaults        1 2

Step eight: start up MySQL

Here we go:

# service mysqld start
MySQL Daemon failed to start.
Starting mysqld:                                           [FAILED]

Step nine: PANIC

(Note to my readers. Ach, who am I kidding? Note to my reader: if you follow the steps above you should not end up here. I had forgotten to change ownership of the newly-mounted volume to mysql, so when MySQL tried to start it tried really hard but couldn't -- see Bonus tidbit, below. I'm adding these details for frantic Googlers whose MySQL is down with errno 13).

No, don't panic. Just look in /var/log/mysqld.log to see why it's not starting:

mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
...
InnoDB: Completed initialization of buffer pool
InnoDB: highest supported file format is Barracuda.
InnoDB: Waiting for the background threads to start
InnoDB: 5.5.31 started; log sequence number 27306868416/usr/libexec/mysqld: File '/var/log/mysql/bin-log.000189' not found (Errcode: 13)
[ERROR] Failed to open log (file '/var/log/mysql/bin-log.000189', errno 13)
[ERROR] Could not open log file
[ERROR] Can't init tc log
[ERROR] Aborting
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 27306868416
[Note] /usr/libexec/mysqld: Shutdown complete
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Step ten: breathe. Just breathe.

The errno 13 shown above is a permissions error. Let's review the permissions:

# ls -la /var/log/mysql
drwxr-xr-x.  2 mysql mysql       4096 Jun  4 08:47 .
drwxr-xr-x. 16 root  root        4096 Jun  4 08:49 ..
-rw-rw----.  1 mysql mysql 1073978999 Jun  4 03:51 bin-log.000187
-rw-rw----.  1 mysql mysql  737806907 Jun  4 08:48 bin-log.000188
-rw-rw----.  1 mysql mysql         60 Jun  4 08:47 bin-log.index

That all looks good. So it must be SELinux. Let's check:

# ls -laZ /var/log/mysql
drwxr-xr-x. mysql   mysql   system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root    root    system_u:object_r:var_log_t:s0   ..
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000187
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000188
-rw-rw----. mysql   mysql   unconfined_u:object_r:var_log_t:s0 bin-log.000189
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.index

Aha! See how bin-log.000189 is marked var_log_t instead of mysqld_log_t? Let's restore its correct SELinux settings and look again:

# restorecon /var/log/mysql/bin-log.000189

# ls -laZ /var/log/mysql
drwxr-xr-x. mysql   mysql   system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root    root    system_u:object_r:var_log_t:s0   ..
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000187
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000188
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000189
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.index

Step eleven: start MySQL, for real

# service mysqld start
Starting mysqld:                                           [  OK  ]

Yay!

Bonus tidbit

If you start up MySQL and it doesn't start, and /var/log/mysqld.log says

mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/usr/libexec/mysqld: File '/var/log/mysql/bin-log.~rec~' not found (Errcode: 13)
[ERROR] MYSQL_BIN_LOG::open_purge_index_file failed to open register  file.
[ERROR] MYSQL_BIN_LOG::open_index_file failed to sync the index file.
[ERROR] Aborting
[Note] /usr/libexec/mysqld: Shutdown complete
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

the problem is with the ownership of /var/log/mysql. Compare what you've got

# ls -la /var/log/mysql
drwxr-xr-x.  4 root    root          4096 Jun  4 08:48 .
drwxr-xr-x. 16 root    root          4096 Jun  4 08:49 ..

with what you want to have:

# ls -la /var/log/mysql
drwxr-xr-x.  2 mysql mysql       4096 Jun  4 08:47 .
drwxr-xr-x. 16 root  root        4096 Jun  4 08:49 ..
...

See how . is owned by root:root instead of mysql:mysql? That's why you want to do

chown mysql:mysql /var/log/mysql

Pages

Subscribe to RSS - RHEL6