I/O error after removing SAN disks from Volume Group | 1 Simple Solution

Introduction

In this post, we are about to know how to resolve I/O errors while listing the logical volumes or volume groups in Linux. After removing anyone of SAN disk from any Volume Group it will print the I/O error whenever we try to run PVS, VGS, or LVS.

This can be only in case when someone not following the proper procedures to remove the SAN LUN’S from a Linux server.

The Actual I/O Error

Have a read to know about How to remove a SAN disk or Storage device (LUN) from Linux server.

root@PRODUORACLEDB03 ~]# lvs 
/dev/Volgrp_Archive_backup/LogVol_Archive: read failed after 0 of 4096 at 219901: Input/output error
/dev/Volgrp_Archive_backup/LogVol_Archive: read failed after 0 of 4096 at 219902: Input/output error
/dev/Volgrp_Archive_backup/LogVol_Archive: read failed after 0 of 4096 at 0: Input/output error
/dev/Volgrp_Archive_backup/LogVol_Archive: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdcy: read failed after 0 of 4096 at 2199014858752: Input/output error

Now let’s see how to find the stale disks related to this I/O error.

Method 1: Removing Stale Disks

The above I/O error can be resolved by removing the stale disks. It’s not required to unmount the file system in this method, But we need to be very careful while performing the below steps.

List out VG information using vgdisplay

# vgdisplay Volgrp_Archive_backup

Remove all the stale devices by running dmsetup and by running echo by appending 1 into each device path, or device.

# dmsetup remove /dev/Volgrp_Archive_backup/*
# echo 1 > /sys/block/sdcy/device/delete

Understand what you are doing in this step. In case, if you choose the wrong disk the disk will be removed from the operating system and make things complex if you did not take a pre output before starting with the operation.

Method 2: Removing Stale Disk using vgchange

Using this method, we need to unmount the file system.

To find the exact Volume Group where the disk has been removed, do a scan.

# umount /Archive_backup
# vgscan

Before bringing down the volume group offline, we need to bring down the Logical volume.

# lvchange -an LogVol_Archive

Once the above logical volume is offline, Make the volume group offline.

# vgchange -an Volgrp_Archive_backup

Now run “vgscan” to Search for all volume groups.

# vgscan

Now it’s time to bring back the VG and LV online in reverse order. First, make the volume Group online.

# vgchange -ay Volgrp_Archive_backup

By following make the Logical volume online to mount the file system.

# lvchange -ay LogVol_Archive
# mount /dev/mapper/Volgrp_Archive_backup-LogVol_Archive /Archive_backup

Using any one of the above methods we can recover from stale disk issues in Linux.

Conclusion:

In case if someone did not follow the right procedure to remove a SAN disk we may notice an I/O error while trying to list the PV, VG, and LV. This can be cleared using the above two steps. However, we need to avoid such practice while removing SAN LUN’s to be free from any I/O error. Let me catch you in the next troubleshooting guide, till then subscribe to our newsletter and feed to keep you updated.