Tag: disk

Checking NVMe Disk

You can use nvme-cli to check your NVMe disks. To install nvme-cli, run

On Ubuntu/Debian

sudo apt-get install -y nvme-cli

On RHEL

sudo yum install nvme-cli

To list of NVMe drives, use

nvme list

Example

[root@server52 ~]# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S439NA0N900171       SAMSUNG MZQLB1T9HAJR-00007               1         531.97  GB /   1.92  TB    512   B +  0 B   EDA5402Q
/dev/nvme1n1     S439NA0N900057       SAMSUNG MZQLB1T9HAJR-00007               1         532.00  GB /   1.92  TB    512   B +  0 B   EDA5402Q
[root@server52 ~]#

To find smart data, run

nvme smart-log DEVICE_NAME_HERE

Example

[root@server52 ~]# nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 39 C
available_spare                     : 100%
available_spare_threshold           : 10%
percentage_used                     : 0%
data_units_read                     : 130,334,241
data_units_written                  : 69,361,921
host_read_commands                  : 866,547,209
host_write_commands                 : 710,448,251
controller_busy_time                : 998
power_cycles                        : 88
power_on_hours                      : 5,464
unsafe_shutdowns                    : 83
media_errors                        : 0
num_err_log_entries                 : 0
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 39 C
Temperature Sensor 2                : 43 C
Temperature Sensor 3                : 47 C
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0
[root@server52 ~]#

To see the NVMe error log

nvme error-log	DEVICE_NAME_HERE

October 27, 2021

mount: unknown filesystem type ‘LVM2_member’.

When I try to mount a partition, I get the following error

[root@sysresccd ~]# mount /dev/nvme0n1p3 /mnt
mount: /mnt: unknown filesystem type 'LVM2_member'.
[root@sysresccd ~]#

This is because the disk is LVM. First of all identify the name of the volume group and logical volume.

vgs
lvs

Example

[root@sysresccd ~]# vgs
  VG #PV #LV #SN Attr   VSize  VFree
  vg   2   3   0 wz--n- <1.82t    0 
[root@sysresccd ~]# lvs
  LV   VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root vg -wi-a-----   1.80t                                                    
  swap vg -wi-a----- <15.69g                                                    
  tmp  vg -wi-a-----   1.00g                                                    
[root@sysresccd ~]#

From the above result, we found the server has 3 logical volumes. The biggest volume is root. Other volumes swap and tmp we can ignore. So the device name is

/dev/vg/root

Next run

modprobe dm-mod
lvscan
vgchange -ay

Example

[root@sysresccd ~]# lvscan
  ACTIVE            '/dev/vg/swap' [<15.69 GiB] inherit
  ACTIVE            '/dev/vg/tmp' [1.00 GiB] inherit
  ACTIVE            '/dev/vg/root' [1.80 TiB] inherit
[root@sysresccd ~]# vgchange -ay
  3 logical volume(s) in volume group "vg" now active
[root@sysresccd ~]#

Now you should be able to mount with disk using command

mount /dev/vg/root /mnt/

October 26, 2021

Delayed block allocation failed for inode

On a CentOS server, nginx web server stopped serving files. On checking logs, I found the following error message

2021/10/25 19:08:35 [crit] 42762#0: *78977627 pread() "/var/www/html/995.mp4" failed (5: Input/output error) while sending response to client, client: 59.92.71.53, server: 158.58.173.92, request: "GET /995.mp4 HTTP/1.1", host: "domain.tld"

I tried to copy the file, copy filed with error message “Input/output error”

[root@mta1 html]# cp /var/www/html/995.mp4 ~/
cp: error reading '/var/www/html/995.mp4': Input/output error
[root@mta1 html]#

On /var/log/messages, I had the following error message

Oct 25 19:19:03 mta1 kernel: EXT4-fs error (device dm-0): ext4_wait_block_bitmap:520: comm kworker/u32:2: Cannot read block bitmap - block_group = 7705, block_bitmap = 252182537
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): Delayed block allocation failed for inode 39976964 at logical offset 33810 with max blocks 14 with error 5
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): This should not happen!! Data will be lost

The problem is fixed by rebooting the server into rescue mode, then running the fsck command.

[root@sysresccd ~]# fsck -y /dev/vg/root
fsck from util-linux 2.35.2
e2fsck 1.45.6 (20-Mar-2020)
/dev/mapper/vg-root contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Unconnected directory inode 61210647 (/var/cache/dnf/epel-modular-95d9a0c53e492cbd/???)
Connect to /lost+found? yes

Unconnected directory inode 61210656 (/var/cache/dnf/extras-2770d521ba03e231/???)
Connect to /lost+found? yes

Unconnected directory inode 61210668 (/var/cache/dnf/epel-6519ee669354a484/???)
Connect to /lost+found? yes

Pass 4: Checking reference counts
Inode 39845944 ref count is 1, should be 2.  Fix? yes

...

Unattached inode 61210637
Connect to /lost+found? yes

Inode 61210637 ref count is 2, should be 1.  Fix? yes

Inode 61210641 ref count is 3, should be 2.  Fix? yes

Directories count wrong for group #7474 (5, counted=0).
Fix? yes

Free inodes count wrong (120894192, counted=120894225).
Fix? yes

Padding at end of inode bitmap is not set. Fix? yes


/dev/mapper/vg-root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/vg-root: 44271/120938496 files (2.2% non-contiguous), 235802671/483741696 blocks
[root@sysresccd ~]#

See fsck

October 26, 2021