Installing rpy2 2.3.x and numpy on RHEL6

R from EPEL has been updated to version 3.

We would like to run rpy2 but as of rpy2 version 2.3.x, it requires Python 2.7.

Red Hat Enterprise Linux 6 comes with Python 2.6.

So we need Python 2.7 so we can use rpy2 2.3 so we can use R 3.

The solution is to create a Python virtualenv containing Python 2.7 and install rpy2 in there. That way a user can log in and get Python 2.7 when they type python at the command line.

Take a moment to make sure standard build tools are installed. At a minimum, we need the following since we want numpy to use ATLAS, BLAS, and LAPACK, and not run into problems using SSL from Python:

yum install -y gcc atlas-devel blas-devel lapack-devel openssl-devel

Python 2.7 can now be installed from source:

curl -O http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
tar xvzf Python-2.7.3.tgz
cd Python-2.7.3
./configure --enable-shared
make altinstall
echo "/usr/local/lib" > /etc/ld.so.conf.d/python2.7.conf
ldconfig

Now setuptools:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-0.9.8.tar.gz
tar xvfz setuptools-0.9.8.tar.gz
cd setuptools-0.9.8
/usr/local/bin/python2.7 setup.py build
/usr/local/bin/python2.7 setup.py install --record installedfiles.txt

Finally, virtualenv. This installs the virtual environment at /usr/local/ve but you might want it elsewhere.

wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.1.tar.gz
tar xvfz virtualenv-1.9.1.tar.gz
cd virtualenv-1.9.1
python2.7 setup.py install --record installedfiles.txt
virtualenv-2.7 /usr/local/ve
source /usr/local/ve/bin/activate

That last line activates the virtual environment, so adding it to your .bash_profile might be handy.

Trying to add numpy with easy_install-2.7 didn't work for me:

(ve)[snorky]# easy_install-2.7 numpy
...
Running numpy-1.7.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-u0mtJj/numpy-1.7.1/egg-dist-tmp-kGRfag
Running from numpy source directory.
error: SandboxViolation: open('/dev/null', 'w') {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

However, pip worked like a charm:

(ve)[snorky]# pip install numpy
...
(ve)[snorky]$ python
Python 2.7.3 (default, Jul 29 2013, 17:08:17)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy

Likewise rpy2 worked fine:

pip install rpy2
...

(ve)[snorky]# python
Python 2.7.3 (default, Jul 29 2013, 17:08:17)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2
>>> import rpy2.rinterface

Reference: Undefined symbol (Rf_PrintWarnings) when using rpy 2.3.1 and R 3.0.1

Topic: 

Moving MySQL Binary Logs to a New LVM Logical Volume

I had a RHEL6 machine with lots of large MySQL binary log files in /var/log/mysql and was in need of more space on that volume. I am running MySQL 5.5 from IUS.

The system had an unused LVM logical volume. So I proceeded to mount the LVM logical volume at /var/log/mysql so that binary logs would get written to the logical volume.

Step one: unmount the logical volume

Not strictly necessary for a rename but a nice precaution.

# umount /dev/mapper/vg_db-lv_foo

Step two: rename the logical volume

The volume was named foo (not really true but it will server for this example) and was in the volume group named vg_db. I renamed it to something obvious, namely lv_mysqllogs.

# lvrename vg_db lv_foo lv_mysqllogs
  Renamed "lv_foo" to "lv_mysqllogs" in volume group "vg_db"

Step three: create a temporary directory as a mount point

I'll put the new directory right next to the existing /var/log/mysql.

# mkdir /var/log/mysql.new
chown mysql:mysql /var/log/mysql.new

Step four: purge binary logs

In order to have only a small amount of data to move, I checked with our MySQL slave server to make sure it was up-to-date, then logged into MySQL as root and purged the logs:

mysql> PURGE BINARY LOGS BEFORE '2013-06-04 00:00:01';
Query OK, 0 rows affected (3.81 sec)

Step five: mount LVM logical volume

# mount /dev/mapper/vg_db-lv_mysqllogs /var/log/mysql.new
chown mysql:mysql /var/log/mysql.new

Step six: copy the binary log files

For this, we have to stop MySQL.

# service mysqld stop
Stopping mysqld:                                           [  OK  ]

And then copy logs:

# cp -Rp /var/log/mysql/* /var/log/mysql.new/

Step seven: unmount LVM and remount at /var/log/mysql

# umount /dev/mapper/vg_db-lv_mysqllogs

We want to keep the original directory around in case things go horribly wrong.

# mv /var/log/mysql /var/log/mysql.old

And then move our temporary directory to its final destination for use as a mount point:

# mv /var/log/mysql.new /var/log/mysql

And remount the LVM logical volume to /var/log/mysql:

# mount /dev/mapper/vg_db-lv_mysqllogs /var/log/mysql

This is a good time to update /etc/fstab with your new mount entry so it will survive reboots:

/dev/mapper/vg_db-lv_mysqllogs /var/log/mysql                   ext4    defaults        1 2

Step eight: start up MySQL

Here we go:

# service mysqld start
MySQL Daemon failed to start.
Starting mysqld:                                           [FAILED]

Step nine: PANIC

(Note to my readers. Ach, who am I kidding? Note to my reader: if you follow the steps above you should not end up here. I had forgotten to change ownership of the newly-mounted volume to mysql, so when MySQL tried to start it tried really hard but couldn't -- see Bonus tidbit, below. I'm adding these details for frantic Googlers whose MySQL is down with errno 13).

No, don't panic. Just look in /var/log/mysqld.log to see why it's not starting:

mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
...
InnoDB: Completed initialization of buffer pool
InnoDB: highest supported file format is Barracuda.
InnoDB: Waiting for the background threads to start
InnoDB: 5.5.31 started; log sequence number 27306868416/usr/libexec/mysqld: File '/var/log/mysql/bin-log.000189' not found (Errcode: 13)
[ERROR] Failed to open log (file '/var/log/mysql/bin-log.000189', errno 13)
[ERROR] Could not open log file
[ERROR] Can't init tc log
[ERROR] Aborting
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 27306868416
[Note] /usr/libexec/mysqld: Shutdown complete
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Step ten: breathe. Just breathe.

The errno 13 shown above is a permissions error. Let's review the permissions:

# ls -la /var/log/mysql
drwxr-xr-x.  2 mysql mysql       4096 Jun  4 08:47 .
drwxr-xr-x. 16 root  root        4096 Jun  4 08:49 ..
-rw-rw----.  1 mysql mysql 1073978999 Jun  4 03:51 bin-log.000187
-rw-rw----.  1 mysql mysql  737806907 Jun  4 08:48 bin-log.000188
-rw-rw----.  1 mysql mysql         60 Jun  4 08:47 bin-log.index

That all looks good. So it must be SELinux. Let's check:

# ls -laZ /var/log/mysql
drwxr-xr-x. mysql   mysql   system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root    root    system_u:object_r:var_log_t:s0   ..
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000187
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000188
-rw-rw----. mysql   mysql   unconfined_u:object_r:var_log_t:s0 bin-log.000189
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.index

Aha! See how bin-log.000189 is marked var_log_t instead of mysqld_log_t? Let's restore its correct SELinux settings and look again:

# restorecon /var/log/mysql/bin-log.000189

# ls -laZ /var/log/mysql
drwxr-xr-x. mysql   mysql   system_u:object_r:var_log_t:s0   .
drwxr-xr-x. root    root    system_u:object_r:var_log_t:s0   ..
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000187
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000188
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.000189
-rw-rw----. mysql   mysql   unconfined_u:object_r:mysqld_log_t:s0 bin-log.index

Step eleven: start MySQL, for real

# service mysqld start
Starting mysqld:                                           [  OK  ]

Yay!

Bonus tidbit

If you start up MySQL and it doesn't start, and /var/log/mysqld.log says

mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/usr/libexec/mysqld: File '/var/log/mysql/bin-log.~rec~' not found (Errcode: 13)
[ERROR] MYSQL_BIN_LOG::open_purge_index_file failed to open register  file.
[ERROR] MYSQL_BIN_LOG::open_index_file failed to sync the index file.
[ERROR] Aborting
[Note] /usr/libexec/mysqld: Shutdown complete
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

the problem is with the ownership of /var/log/mysql. Compare what you've got

# ls -la /var/log/mysql
drwxr-xr-x.  4 root    root          4096 Jun  4 08:48 .
drwxr-xr-x. 16 root    root          4096 Jun  4 08:49 ..

with what you want to have:

# ls -la /var/log/mysql
drwxr-xr-x.  2 mysql mysql       4096 Jun  4 08:47 .
drwxr-xr-x. 16 root  root        4096 Jun  4 08:49 ..
...

See how . is owned by root:root instead of mysql:mysql? That's why you want to do

chown mysql:mysql /var/log/mysql

How to Change an SVN Commit Message Retroactively

Changing an SVN commit message retroactively involves two steps. First, the repository must have a hook enabled. Then the svn client must issue a propset command.

On the subversion server:

cd /var/www/svn/myrepository/hooks
cp pre-revprop-change.tmpl pre-revprop-change
chmod u+x pre-revprop-change
chown apache:apache pre-revprop-change

Line-by-line explanation:

Change to the subversion repository's hooks directory.
Copy the template file that is already there to a file without the .tmpl extension.
Make the file executable.
Change ownership of the file to the apache user (this is the user that Apache runs as on Fedora-like systems such as RHEL6).

Changes to log messages are now allowed.

Now an svn command can be issued on the client at the command line to revise the log message for a certain revision (in this case, revision 42):

svn propset -r 42 --revprop svn:log 'my new commit message'
property 'svn:log' set on repository revision 42

Drupal 7 and Cannot decode raw data (NSURLErrorDomain:-1015)

This error is related to caching and compression. Wish I had time to dig deeper now, but I don't.

For us, it happened when a database was moved to a new server without turning off anonymous page caching first.

Solved by:

1. Adding the following line to settings.php:

$conf['page_compression'] = FALSE

2. Logging into the site using Firefox (not Safari) and going to http://www.example.com/admin/config/development/performance and hitting the Clear All Caches button.

It's not enough to just truncate the cache tables in your database.

3. Go back and undo your settings.php change.

Topic: 

Pages

Subscribe to SysArchitects RSS