How to scan a newly assigned LUN in multipathd in CentOS / RHEL
How to scan newly Assigned LUNs in Multipathd under CentOS / RHEL
1. Rescan SCSI hosts:
# for host in 'ls /sys/class/scsi_host'
do
echo ${host}; echo "- - -" > /sys/class/scsi_host/${host}/scan
done
2. Issue LIP to FC hosts:
# for host in 'ls /sys/class/fc_host'
do
echo ${host}; echo "1" > /sys/class/fc_host/${host}/issue_lip;
done
3. Run rescan script from sg3_utils:
# rescan-scsi-bus.sh
# rescan-scsi-bus.sh -i
Reload/Refresh Multipathd deamon
After above scan, the new assigned LUNs are visible in Linux OS level, but still multipathd daemon will show no mappings for new LUNs. You will have to refresh multipathd daemon to map the new LUNs.
Run the below command:
# multipath -v2
This should bring the LUNs in control of device mapper multipathd.
Verify
Verify with “multipath -ll“” that the LUNs are visible now.
# multipath -ll
If partitions created on multipath devices and that is not being listed, kpartx need to be executed on the affected multipath devices.
# kpartx -a -v /dev/mapper/XXXXXXX
Summary: Rescan the new Storage(LUN)in Linux
Description:
In order to get the fiber channel adapters detail to rescan, list the
/sys/class/fc_host directory.
In old RHEL 4 host you will not be getting this listing. In this case you
can use the
/sys/class/scsi_host directory but it will list all internal adapters too.
1. In Order to scan New Lun we need to run LIP (Loop Initialization
Protocol) on the interconnect to
update the scsi layer to reflect the devices currently on the bus. LIP, is
a bus reset which cause
the device addition and removal.
# ls -l /sys/class/fc_host
total 0
drwxr-xr-x 3 root root 0 Feb 27 09:11 host0
drwxr-xr-x 3 root root 0 Feb 27 09:11 host1
2. Make the OS aware of the new storage
#echo "1" > /sys/class/fc_host/host1/issue_lip
#echo "1" > /sys/class/fc_host/host2/issue_lip
#echo "- - -" > /sys/class/scsi_host/host1/scan
#echo "- - -" > /sys/class/scsi_host/host2/scan
OR
for host in `ls /sys/class/scsi_host/`;do
echo "- - -" >/sys/class/scsi_host/${host}/scan
done
Note: Below is the explanation about wild card ..
"- - -" in the above means C T L["Channel on HBA" "Target SCSI id" "Lun"]
"- - -" is a wild card instead of specifying the C T L.
3. After rescanning, confirm whether you are seeing the new storage disks
LUN] by listing the content
under proc
# cat /proc/scsi/scsi | grep scsi | uniq
4. Now we can list the newly scanned LUN using "fdisk -l"command.
# fdisk -l
5. If we have multipath configured then we can see as below:
# multipath
# multipath -ll
Finding informations about SCSI devices
The first problem when working with SCSI devices might be to map informations you got from /proc/scsi/scsi to the device names the kernel uses like /dev/sda and so on.
If you use Novell SuSE Enterprise Server this might not be a problem because SLES comes with a nice little commando called lsscsi. Here's an example:
op710-1-lpar1:/ # lsscsi
[0:0:1:0] disk AIX VDASD /dev/sda
[0:0:2:0] disk AIX VDASD /dev/sdb
[0:0:3:0] cd/dvd AIX VOPTA /dev/sr0
[0:0:4:0] disk AIX VDASD /dev/sdc
[0:0:5:0] disk AIX VDASD /dev/sdd
But assuming you don't use SuSE or using other distributions in addition to SuSE. Well, the first place to look at is the good old /proc filesystem - namely /proc/scsi/scsi.
op710-1-lpar1:/ # cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: AIX Model: VDASD Rev:
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: AIX Model: VDASD Rev:
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 04 Lun: 00
Vendor: AIX Model: VDASD Rev:
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 05 Lun: 00
Vendor: AIX Model: VDASD Rev:
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 03 Lun: 00
Vendor: AIX Model: VOPTA Rev:
Type: CD-ROM ANSI SCSI revision: 04
The first line of a stanza is important. For example Host: scsi0 Channel: 00 Id: 01 Lun: 00 will be 0:0:1:0 if you'll like it short or the first device on the first SCSI channel of the first SCSI adapter. This is the SCSI-ID of the device.
O.k. to find out which device like sda is hiding behind this information you must follow me on a journey to the /sys filesystem. If you'll like or not /sys is very important and our friend.
The information about the device names the Linux kernel uses for a specific SCSI-ID can be found at /sys/bus/scsi/drivers/sd/<SCSI-ID> subdirectory as shown in the following example:
op710-1-lpar1:/ # ll /sys/bus/scsi/drivers/sd/0\:0\:1\:0/
total 0
drwxr-xr-x 2 root root 0 Aug 17 14:38 .
drwxr-xr-x 7 root root 0 Aug 17 16:33 ..
lrwxrwxrwx 1 root root 0 Aug 17 14:38 block -> ../../../../../block/sda
--w------- 1 root root 4096 Aug 17 14:38 delete
-rw-r--r-- 1 root root 4096 Aug 17 14:38 detach_state
-r--r--r-- 1 root root 4096 Aug 17 14:38 device_blocked
lrwxrwxrwx 1 root root 0 Aug 17 14:38 generic -> ../../../../../class/scsi_generic/sg0
-r--r--r-- 1 root root 4096 Aug 17 14:38 model
-rw-r--r-- 1 root root 4096 Aug 17 14:38 online
-r--r--r-- 1 root root 4096 Aug 17 14:38 queue_depth
--w------- 1 root root 4096 Aug 17 14:38 rescan
-r--r--r-- 1 root root 4096 Aug 17 14:38 rev
-r--r--r-- 1 root root 4096 Aug 17 14:38 scsi_level
-rw-r--r-- 1 root root 4096 Aug 17 14:38 timeout
-r--r--r-- 1 root root 4096 Aug 17 14:38 type
-r--r--r-- 1 root root 4096 Aug 17 14:38 vendor
Now we know that the device with the SCSI-ID 0:0:1:0 is known to the kernel as sda. Well done.
Note..
Pls. note that there are some differences between Novell's SuSE Enterprise Linux and Red Hat's Enterprise Linux in the way they'll work with /sys. The mentioned method will work on both distributions though.
Rescan of a SCSI bus
Let's assume you've added a SCSI disk to a SCSI bus (either a physical or virtual device). One possibility to make it known to the system would be to reboot the server or partition. But aaargh no, this is not the prefered way. The easiest way is to rescan the whole SCSI bus which will enable the Linux kernel to detect new devices!
To issue a SCSI bus rescan you must know on which bus you've added the device. If you don't know which bus and if there are mutliple buses on the system you can rescan each bus which will be somehow annoying but will not interrupt the system.
To initiate a SCSI bus rescan type echo "- - -" > /sys/class/scsi_host/hostX/scan where X stands for the SCSI bus you want to scan.
op710-1-lpar1:/ # lsscsi
[0:0:1:0] disk AIX VDASD /dev/sda
[0:0:2:0] disk AIX VDASD /dev/sdb
[0:0:3:0] cd/dvd AIX VOPTA /dev/sr0
[0:0:4:0] disk AIX VDASD /dev/sdc
[0:0:5:0] disk AIX VDASD /dev/sdd
op710-1-lpar1:/ # echo "- - -" > /sys/class/scsi_host/host0/scan
op710-1-lpar1:/ # lsscsi
[0:0:1:0] disk AIX VDASD /dev/sda
[0:0:2:0] disk AIX VDASD /dev/sdb
[0:0:3:0] cd/dvd AIX VOPTA /dev/sr0
[0:0:4:0] disk AIX VDASD /dev/sdc
[0:0:5:0] disk AIX VDASD /dev/sdd
[0:0:6:0] disk AIX VDASD /dev/sde
Well done, here's our new device. By the way this does not only work for disks but also for SCSI CD/DVD devices.
Note...
Please note that a rescan of the SCSI bus will only detect new devices added to the bus. It will not detect the state of an already known device - e.g. if it's failed or online.
Deletion of a SCSI Device
There might be the situation where you must remove a SCSI device from the system. Easy by using
echo 1 > /sys/bus/scsi/drivers/sd/<SCSI-ID>/delete. Here's an example:
op710-1-lpar1:/ # lsscsi
[0:0:1:0] disk AIX VDASD /dev/sda
[0:0:2:0] disk AIX VDASD /dev/sdb
[0:0:3:0] cd/dvd AIX VOPTA /dev/sr0
[0:0:4:0] disk AIX VDASD /dev/sdc
[0:0:5:0] disk AIX VDASD /dev/sdd
[0:0:6:0] disk AIX VDASD /dev/sde
op710-1-lpar1:/ # echo 1 > /sys/bus/scsi/drivers/sd/0\:0\:6\:0/delete
op710-1-lpar1:/ # lsscsi
[0:0:1:0] disk AIX VDASD /dev/sda
[0:0:2:0] disk AIX VDASD /dev/sdb
[0:0:3:0] cd/dvd AIX VOPTA /dev/sr0
[0:0:4:0] disk AIX VDASD /dev/sdc
[0:0:5:0] disk AIX VDASD /dev/sdd
Note...
To readd the device, simply rescan the whole SCSI bus.
Note...
Pls. note that deleting and readding the device will change the device names - i.e. by deleting /dev/sde and readding it using the scsi bus rescan it will become dev/sdf!
Rescan of a SCSI Device
The problem of a SCSI bus rescan is, that it will only detect new devices. But let's assume the following situation.
You'll have a client partition on a system with two VIO server. Each VIO server exports several virtual SCSI devices to the client partition and the client will build several RAID1 arrays on top of those devices. This setup will keep the client up and running even if on VIO server fails.
This szenario is shown in the following example:
570-lpar9:~ # cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc1[0] sde1[1]
52428672 blocks [2/2] [UU]
md2 : active raid1 sdd1[1] sdf1[0]
26214272 blocks [2/2] [UU]
md0 : active raid1 sdm2[2](F) sdb2[1]
62896064 blocks [2/1] [_U]
unused devices: <none>
As you can see the RAID device /dev/md0 has one faulty device. Let's assume you've corrected the problem with this device /dev/sdm and you know that it works correctly. But hot-removing and hot-adding of the device to the RAID array does not work. More worse, the kernel itself will post a lot of errors in /var/log/messages. The device keep staying in a faulty state. One possibility is, of course, to reboot the system but hey, this is our SAP system which cannot rebooted easily.
The solution is to rescan the device itself. This will tell the Linux kernel that this device is up and running. To rescan a particularly device, use the command
echo 1 > /sys/bus/scsi/drivers/sd/<SCSI-ID>/block/device/rescan.
570-lpar9:/sys/bus/scsi/drivers/sd/0:0:1:0/block/device # echo 1 > rescan
570-lpar9:/sys/bus/scsi/drivers/sd/0:0:1:0/block/device # tail -f /var/log/messages
...
Aug 17 16:17:49 570-lpar9 kernel: SCSI device sdm: 125829120 512-byte hdwr sectors (64425 MB)
Aug 17 16:17:49 570-lpar9 kernel: sdm: cache data unavailable
Aug 17 16:17:49 570-lpar9 kernel: sdm: assuming drive cache: write through
...
570-lpar9:/sys/bus/scsi/drivers/sd/0:0:1:0/block/device # cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc1[0] sde1[1]
52428672 blocks [2/2] [UU]
md2 : active raid1 sdd1[1] sdf1[0]
26214272 blocks [2/2] [UU]
md0 : active raid1 sdm2[2](F) sdb2[1]
62896064 blocks [2/1] [_U]
unused devices: <none>
570-lpar9:/sys/bus/scsi/drivers/sd/0:0:1:0/block/device # mdadm /dev/md0 -r /dev/sdm2 -a /dev/sdm2
mdadm: hot removed /dev/sdm2
mdadm: hot added /dev/sdm2
570-lpar9:/sys/bus/scsi/drivers/sd/0:0:1:0/block/device # cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc1[0] sde1[1]
52428672 blocks [2/2] [UU]
md2 : active raid1 sdd1[1] sdf1[0]
26214272 blocks [2/2] [UU]
md0 : active raid1 sdm2[2] sdb2[1]
62896064 blocks [2/1] [_U]
[>....................] recovery = 0.3% (211736/62896064) finish=14.7min speed=70578K/sec
unused devices: <none>
As you can see the rescan has told the kernel that the device is up and running again and now the RAID1 resync works. Well done.
Now let's assume you've created a virtual device with 10GB capacity. But soon after that you're facing the problem, that this space is enough. Assuming that we're in a virtual environment, you can extend the logical volume on the VIO server. And then you must tell the client that it has more space. Ok, simply rescan the device and you can use the additional space. Here's an example:
op710-1-lpar1:/ # fdisk -l
...
Disk /dev/sde: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 10240 10485744 83 Linux
op710-1-lpar1:/ # echo 1 > /sys/bus/scsi/drivers/sd/0\:0\:6\:0/rescan
op710-1-lpar1:/ # fdisk -l
...
Disk /dev/sde: 16.1 GB, 16106127360 bytes
64 heads, 32 sectors/track, 15360 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 10240 10485744 83 Linux
...
DOING SOME FDISK PARTITIONING HERE
...
op710-1-lpar1:/ # fdisk -l
...
Disk /dev/sde: 16.1 GB, 16106127360 bytes
64 heads, 32 sectors/track, 15360 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 10240 10485744 83 Linux
/dev/sde2 10241 15360 5242880 83 Linux
The new capacity will be added at the end of the "old" device. Wow.
Be Careful!
Although it sounds great to resize a device online there are some caveheats to consider:
First resizing of the device will only work when the device is not mounted! Well, not really online anymore...
Second as far as I've tested the online resizing of volumes it will not work with LVM devices.
One exception to the second point (!!!) is using md devices - see blow!
The kernel itself discovers the new size of the device during the rescan but as long it is mounted somehow, somewhere or as long as it has a LVM device on it it will not be able to use the addl. space at all - you must reboot the system or partition!
Nevertheless the resizing works even in a virtualized environment - e.g. resize the LUN on the FC disk subsystem, use cfgdev on the VIOS and then use the mentioned method on the Linux LPAR.
Now for production systems this is not acceptable. The solution should be to use LVM and add addl. LUNs to a Volume Group instead of thinking about resizing an existing LUN. You can save a lot of time and headaches...
Last updated
Was this helpful?