Delayed block allocation failed for inode

On a CentOS server, nginx web server stopped serving files. On checking logs, I found the following error message

2021/10/25 19:08:35 [crit] 42762#0: *78977627 pread() "/var/www/html/995.mp4" failed (5: Input/output error) while sending response to client, client: 59.92.71.53, server: 158.58.173.92, request: "GET /995.mp4 HTTP/1.1", host: "domain.tld"

I tried to copy the file, copy filed with error message “Input/output error”

[root@mta1 html]# cp /var/www/html/995.mp4 ~/
cp: error reading '/var/www/html/995.mp4': Input/output error
[root@mta1 html]# 

On /var/log/messages, I had the following error message

Oct 25 19:19:03 mta1 kernel: EXT4-fs error (device dm-0): ext4_wait_block_bitmap:520: comm kworker/u32:2: Cannot read block bitmap - block_group = 7705, block_bitmap = 252182537
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): Delayed block allocation failed for inode 39976964 at logical offset 33810 with max blocks 14 with error 5
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): This should not happen!! Data will be lost

The problem is fixed by rebooting the server into rescue mode, then running the fsck command.

[root@sysresccd ~]# fsck -y /dev/vg/root
fsck from util-linux 2.35.2
e2fsck 1.45.6 (20-Mar-2020)
/dev/mapper/vg-root contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Unconnected directory inode 61210647 (/var/cache/dnf/epel-modular-95d9a0c53e492cbd/???)
Connect to /lost+found? yes

Unconnected directory inode 61210656 (/var/cache/dnf/extras-2770d521ba03e231/???)
Connect to /lost+found? yes

Unconnected directory inode 61210668 (/var/cache/dnf/epel-6519ee669354a484/???)
Connect to /lost+found? yes

Pass 4: Checking reference counts
Inode 39845944 ref count is 1, should be 2.  Fix? yes

...

Unattached inode 61210637
Connect to /lost+found? yes

Inode 61210637 ref count is 2, should be 1.  Fix? yes

Inode 61210641 ref count is 3, should be 2.  Fix? yes

Directories count wrong for group #7474 (5, counted=0).
Fix? yes

Free inodes count wrong (120894192, counted=120894225).
Fix? yes

Padding at end of inode bitmap is not set. Fix? yes


/dev/mapper/vg-root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/vg-root: 44271/120938496 files (2.2% non-contiguous), 235802671/483741696 blocks
[root@sysresccd ~]# 

See fsck