Solved: iSCSI disconnects and timeouts after successful login

Submitted by John on Mon, 2013-01-21 15:04

Consider the following from /var/log/messages:

iscsid: Connection16:0 to [target: iqn.2012-12.com.example:fooportal, portal: 198.51.100.3,3260] through [iface: default] is operational now
kernel: sde:
kernel: connection16:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4379432620, last ping 4379437620, now 4379442620
kernel: connection16:0: detected conn error (1011)
iscsid: Kernel reported iSCSI connection 16:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)
iscsid: connection16:0 is operational after recovery (1 attempts)
kernel: connection16:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4379445875, last ping 4379450875, now 4379455875
kernel: connection16:0: detected conn error (1011)
iscsid: Kernel reported iSCSI connection 16:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)

As you can see, login to the iSCSI target was successful. But shortly thereafter, the client becomes unhappy and the connection fails out, only to be reinstated and disconnected repeatedly.

In my case, the problem ended up being jumbo frames. I diagnosed it by doing a wireshark capture of the bonded interface on the client, which revealed the following message:

scsi transfer limited due to allocation_length too small

and showed the message

[TCP Retransmission] SCSI: Read(10) LUN: 0x01 (LBA: 0x00000000, Len: 8)

Turning off jumbo frames in the bonded interfaces on both ends of the connection solved the problem.

This was happening because the switch I was going over (HP ProCurve 2910al) does not have jumbo frames enabled for the default VLAN:

# show vlan

Status and Counters - VLAN Information

  Maximum VLANs to support : 256                 

  VLAN ID Name                             | Status     Voice Jumbo
  ------- -------------------------------- + ---------- ----- -----
  1       DEFAULT_VLAN                     | Port-based No    No  

The ultimate solution was to create a separate VLAN on the switch and enable jumbo frames on the new VLAN. After that, everything worked swimmingly.

Topic: 
RHEL6
iSCSI