RHEV

Solved: internal error boot orders have to be contiguous and starting from 1

Submitted by John on Wed, 2013-01-30 19:16

I exported a VM from RHEV 3.0 to an export domain. Then I imported the VM into RHEV 3.1.

I was surprised that the VM would not start. Instead, I got this message:

Exit message: internal error boot orders have to be contiguous and starting from 1

Upon further inspection, the boot order information was completely missing from the imported KVM definition:

    <disk device="disk" snapshot="no" type="block">
      <source dev="/rhev/data-center/a1f867a3-237a-4be3-8e07-bf61f5b95280/71051d89-b9bd-4db5-8fe1-7c4499114874/images/d543a66c-4dd6-4d49-a8ea-7d65c137ce41/39bbb768-13d0-4bb8-ae93-a2c8611da655"/>
      <target bus="virtio" dev="vda"/>
      <serial>d543a66c-4ef6-4d49-a8ea-7d65c137ce41</serial>
      <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/>
    </disk>

I solved this by just editing the imported VM's disk in the RHEV UI. I changed the disk's properties from bootable to nonbootable and saved the change. Then I changed it back to bootable again. This wrote in the boot order:

    <disk device="disk" snapshot="no" type="block">
      <source dev="/rhev/data-center/a1f867a3-237a-4be3-8e07-bf61f5b95280/71051d89-b9bd-4db5-8fe1-7c4499114874/images/d543a66c-4dd6-4d49-a8ea-7d65c137ce41/39bbb768-13d0-4bb8-ae93-a2c8611da655"/>
      <target bus="virtio" dev="vda"/>
      <serial>d543a66c-4ef6-4d49-a8ea-7d65c137ce41</serial>
      <boot order="1"/>
      <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/>
    </disk>

I think this happened because the VM had a CD mounted in the RHEV 3.0 data center that was unavailable in the RHEV 3.1 data center.

See also: https://bugzilla.redhat.com/show_bug.cgi?id=874952

Topic: 
RHEV
KVM

Solution to Permission denied to /etc/mtab

Submitted by John on Thu, 2012-07-12 10:58

I was setting up a hypervisor for RHEV running a full RHEL6 install instead of RHEV-H, and I wanted to tune it for better hypervisorness.

So I thought, Hey! I'll install tuned, since I am all excited about tuned after viewing the slides from Mark Wagner's Achieving Top Network Performance (PDF) at the 2012 Red Hat Summit. I was particularly excited about slide 12 entitled "Teaser 3 - latency."

Not to mention that the tuned virtual-host profile is the recommended profile for RHEV hosts.

Anyway, I went ahead and installed tuned and applied the virtual-host profile:

# yum -y install tuned
...
# tuned-adm profile virtual-host
Switching to profile 'virtual-host'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf: [ OK ]
Calling '/etc/ktune.d/tunedadm.sh start': can't rename /etc/mtab.tmp to /etc/mtab: Permission denied
can't rename /etc/mtab.tmp to /etc/mtab: Permission denied

Er...what? Permission denied for root to rename?

It's probably SELinux. It's always SELinux. Sure enough:

# ls -laZ /etc/mtab
-rw-r--r--. root root system_u:object_r:system_conf_t:s0 /etc/mtab
# restorecon /etc/mtab
# ls -laZ /etc/mtab
-rw-r--r--. root root system_u:object_r:etc_runtime_t:s0 /etc/mtab

Now it works fine:

# tuned-adm profile virtual-host
Reverting to saved sysctl settings: [ OK ]
Calling '/etc/ktune.d/tunedadm.sh stop': [ OK ]
Stopping tuned: [ OK ]
Switching to profile 'virtual-host'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf: [ OK ]
Calling '/etc/ktune.d/tunedadm.sh start': [ OK ]
Applying sysctl settings from /etc/sysctl.conf
Starting tuned: [ OK ]

An observation: each time you apply a tuned profile (for example, going back to the default and then to a non-default again), tuned-adm may append options to /etc/mtab. For example, we start with the following line in /etc/mtab:

/dev/mapper/[UUID] /rhev/data-center/mnt/blockSD/[UUID] ext3 rw,barrier 0 0

After doing

# tuned-adm profile virtual-host
# tuned-adm profile default
# tuned-adm profile virtual-host

the line in /etc/mtab looks like this:

/dev/mapper/[UUID] /rhev/data-center/mnt/blockSD/[UUID] ext3 rw,barrier,nobarrier,barrier,nobarrier 0 0

This was a bit confusing, but since mount and umount maintain /etc/mtab I assume they know what they're doing. Interestingly, after a reboot, /proc/mounts says:

/dev/mapper/[UUID] /rhev/data-center/mnt/blockSD/[UUID] ext3 rw,seclabel,relatime,errors=continue,barrier=1,data=ordered 0 0

That makes me sleep a little better, given the dire warnings about running ext3 without barrier=1.

Topic: 
RHEL6
tuned
RHEV

Using RHEL6 to share RAID volume via iSCSI: the Mystery of the Missing LUN

Submitted by John on Mon, 2012-04-16 15:23

My use case was pretty simple. I wanted to share out a raw device via iSCSI to a nearby host on the 172.16.2.x network.

In addition to a minimal Red Hat Enterprise Linux 6 (or equivalent) install, a few packages are needed:

# yum install -y iscsi-initiator-utils scsi-target-utils sg3_utils lsof

I knew the device I wanted to share was /dev/sdb by looking at the output from dmesg:

# dmesg | grep sd
...
sd 0:0:1:0: [sdb] 7812456448 512-byte logical blocks: (3.99 TB/3.63 TiB)
...

The target definition in /etc/tgt/targets.conf was simple as well. Isn't everything simple after you've beaten your head against a wall for hours trying to get it working? The following configuration defines one target with one LUN shared as a raw device which may only be connected to by IP address 172.16.2.2:


Starting up the tgtd service and turning it on permanently led to wonderment and success:

# service tgtd start
# chkconfig tgtd on

Depending on your configuration, you may need to open port 3260 in your firewall.

However, after a reboot, only the controller showed up (as LUN 0). LUN 1 had disappeared!

# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2012-04.com.example:sharename
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
    Account information:
    ACL information:
        172.16.2.2

Why is LUN 1 not showing up? Telling tgt-admin to reparse targets.conf in verbose mode leads to the reason:

# tgt-admin --update ALL -v
# Removing target: iqn.2012-04.com.example:sharename
tgtadm -C 0 --mode target --op delete --tid=1

# Adding target: iqn.2012-04.com.example:sharename
tgtadm -C 0 --lld iscsi --op new --mode target --tid 1 -T iqn.2012-04.com.example:sharename
# Device /dev/sdb is used by the system (mounted, used by swap?).
# Skipping device /dev/sdb - it is in use.
# You can override it with --force or 'allow-in-use yes' config option.
# Note - do so only if you know what you're doing, you may damage your data.

What? I assure you that /dev/sdb is NOT in use by the system. Show me mounts:

# mount

Nope, /dev/sdb does not appear anywhere in the output. Show me a list of open files on /dev/sdb:

# lsof | grep sdb

Nothing. Show me active swap:

# swapon -s

Nothing! Finally, choirboy in the #rhel irc channel pointed me to the answer: you must create a filter in /etc/lvm/lvm.conf so that LVM leaves the device alone. Appropriate section of lvm.conf, showing new filter:

    # By default we accept every block device:
    #filter = [ "a/.*/" ]
    # Every block device except /dev/sdb, that is.
    filter = [ "r|/dev/sdb|" ]

After a restart, LUN 1 persists and the world is once again a happy place to be:

# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2012-04.com.example:sharename
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: 9206CBBG71194900CF07
            Size: 3999978 MB, Block size: 512
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/sdb
            Backing store flags:
    Account information:
    ACL information:
        172.16.2.2

Edit: mutipathd may also be the culprit. Take a look at the world-wide identifier (WWID) of /dev/sdb:

scsi_id -g -u /dev/sdb
36a4badb051c18d0018e462537ac943d6

Is that identifier also listed under /dev/mapper ?

If so, add

find_multipaths         yes

To the defaults section of /etc/multipath.conf, remove the matching line from /etc/multipath/wwids, and restart the multipath daemon:

service multipathd restart
Topic: 
RHEL6
RHEV
iSCSI

RHEV 3.0 Firewall Annotated iptables Configuration for Netfilter

Submitted by John on Fri, 2012-03-02 10:40

When Red Hat Enterprise Virtualization Manager for Servers is installed, it offers to configure iptables for you:

...
Firewall ports need to be opened.
You can let the installer configure iptables automatically overriding the current configuration. The old configuration will be backed up.
Alternately you can configure the firewall later using an example iptables file found under /usr/share/rhevm/conf/iptables.example
...

Here's an annotated version of what the RHEVM installer will give you:

# ssh
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

# XBAP clients for Administration Portal
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8006 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8007 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8008 -j ACCEPT

# Web interface to Administrator Portal
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 8080 -j ACCEPT
# Web interface to Administrator Portal (SSL)
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 8443 -j ACCEPT

# Portmapper (rpcbind on RHEL6)
-A RH-Firewall-1-INPUT -m state --state NEW -p udp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 111 -j ACCEPT

# mountd; NFS MOUNTD_PORT (defined in /etc/sysconfig/nfs)
-A RH-Firewall-1-INPUT -m state --state NEW -p udp --dport 892 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 892 -j ACCEPT

# rquotad; NFS RQUOTAD_PORT (defined in /etc/sysconfig/nfs)
-A RH-Firewall-1-INPUT -m state --state NEW -p udp --dport 875 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 875 -j ACCEPT

# NFS STATD_PORT (defined in /etc/sysconfig/nfs)
-A RH-Firewall-1-INPUT -m state --state NEW -p udp --dport 662 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 662 -j ACCEPT

# nfsd for nfs and nfs_acl
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 2049 -j ACCEPT

# nlockmgr; NFS LOCKD_TCPPORT (defined in /etc/sysconfig/nfs)
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 32803 -j ACCEPT

# NFS LOCKD_UDPPORT (defined in /etc/sysconfig/nfs)
-A RH-Firewall-1-INPUT -m state --state NEW -p udp --dport 32769 -j ACCEPT

Topic: 
RHEL6
RHEV
Subscribe to RSS - RHEV