Category: Linux

Delayed block allocation failed for inode

On a CentOS server, nginx web server stopped serving files. On checking logs, I found the following error message

2021/10/25 19:08:35 [crit] 42762#0: *78977627 pread() "/var/www/html/995.mp4" failed (5: Input/output error) while sending response to client, client: 59.92.71.53, server: 158.58.173.92, request: "GET /995.mp4 HTTP/1.1", host: "domain.tld"

I tried to copy the file, copy filed with error message “Input/output error”

[root@mta1 html]# cp /var/www/html/995.mp4 ~/
cp: error reading '/var/www/html/995.mp4': Input/output error
[root@mta1 html]#

On /var/log/messages, I had the following error message

Oct 25 19:19:03 mta1 kernel: EXT4-fs error (device dm-0): ext4_wait_block_bitmap:520: comm kworker/u32:2: Cannot read block bitmap - block_group = 7705, block_bitmap = 252182537
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): Delayed block allocation failed for inode 39976964 at logical offset 33810 with max blocks 14 with error 5
Oct 25 19:19:03 mta1 kernel: EXT4-fs (dm-0): This should not happen!! Data will be lost

The problem is fixed by rebooting the server into rescue mode, then running the fsck command.

[root@sysresccd ~]# fsck -y /dev/vg/root
fsck from util-linux 2.35.2
e2fsck 1.45.6 (20-Mar-2020)
/dev/mapper/vg-root contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Unconnected directory inode 61210647 (/var/cache/dnf/epel-modular-95d9a0c53e492cbd/???)
Connect to /lost+found? yes

Unconnected directory inode 61210656 (/var/cache/dnf/extras-2770d521ba03e231/???)
Connect to /lost+found? yes

Unconnected directory inode 61210668 (/var/cache/dnf/epel-6519ee669354a484/???)
Connect to /lost+found? yes

Pass 4: Checking reference counts
Inode 39845944 ref count is 1, should be 2.  Fix? yes

...

Unattached inode 61210637
Connect to /lost+found? yes

Inode 61210637 ref count is 2, should be 1.  Fix? yes

Inode 61210641 ref count is 3, should be 2.  Fix? yes

Directories count wrong for group #7474 (5, counted=0).
Fix? yes

Free inodes count wrong (120894192, counted=120894225).
Fix? yes

Padding at end of inode bitmap is not set. Fix? yes


/dev/mapper/vg-root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/vg-root: 44271/120938496 files (2.2% non-contiguous), 235802671/483741696 blocks
[root@sysresccd ~]#

See fsck

October 26, 2021

qdbus: not found

When running a program Ubuntu server with XFCE, I get the error

ubuntu@ubuntu:~$ ~/Downloads/grab-song-master/check-media-players.sh
/home/ubuntu/Downloads/grab-song-master/check-media-players.sh: 3: qdbus: not found
ubuntu@ubuntu:~$

The error is fixed by installing qbus

apt-get install qdbus

On CentOS, you need to install qdbus-qt5 package

yum install qdbus-qt5

October 26, 2021

Set Static IP in RHEL 8 OVH VPS

On OVH VPS running AlmaLinux 8, IPv4 IP address gets dropped. I have to connect to the server using KVM feature available in OVH control panel.

I checked IP of the server with the command

ip a

It did hot show any IPv4 IP address. I checked the network configuration file

vi /etc/sysconfig/network-scripts/ifcfg-eth0

That had the following content

BOOTPROTO=dhcp
DEFROUTE=yes
DEVICE=eth0
DHCLIENT_SET_DEFAULT_ROUTE=yes
HWADDR=fa:16:3e:6b:a6:1a
IPV6ADDR=2607:5300:201:3100::952/56
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6_DEFAULTGW=2607:5300:201:3100::1
IPV6_FORCE_ACCEPT_RA=no
MTU=1500
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

To make the IP static, I made the following changes.

Find

BOOTPROTO=dhcp

Replace with

BOOTPROTO=static

Also added following 2 entry

IPADDR=144.217.13.76
GATEWAY=144.217.12.1

144.217.13.76 is the IP address of the VPS.

144.217.12.1 is the gateway used on the VPS, found using “ip route” or “route -n” command.

[root@vps-eb960963 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         144.217.12.1    0.0.0.0         UG    0      0        0 eth0
144.217.12.1    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
[root@vps-eb960963 network-scripts]#

Create a static route file

vi /etc/sysconfig/network-scripts/route-eth0

Add

144.217.12.1 dev eth0
default via 144.217.12.1 dev eth0

Here 144.217.12.1 is the gateway IP address found in the above step.

See IP

October 22, 2021

Rocky Linux

How to Convert CentOS 8 to Rocky Linux

October 21, 2021
How to Convert CentOS 8 to Rocky Linux
You can convert your remote VPS or Dedicated Server running CentOS 8 to Rocky Linux easily by running the following commands.

Install wget and tmux packages.
```
dnf install wget tmux
```
Next, start a tmux session, so in case you get disconnected from the remote server, you can connect back
```
tmux
```
Now download migration script
```
wget https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rocky/migrate2rocky.sh
```
Now verify your system can be converted with command
```
bash migrate2rocky.sh -V
```
If all is good, you can convert CentOS 8 to Rocky Linux with the command
```
bash migrate2rocky.sh -r
```
October 21, 2021
AlmaLinux

How to Migrate CentOS 8 to AlmaLinux
Install PHP 5.6 on AlmaLinux 8
How to install docker on AlmaLinux 8

October 20, 2021
List contents of jks keystore file
To list the content of jks keystore file used by tomcat web server, run command
```
keytool -list -keystore FILE.jks 
```
It will ask Keystore password. Once you enter the password, it will list the contents of the file.

In the above keystore, there are 5 certificates with names inter, root1, root2, ssl_tomcat2, and tomcat.

To get detailed information on the certificates, use the command
```
keytool -list -v -keystore FILE.jks 
```
See keytool
October 19, 2021
keytool

The keytool command is a key and certificate management utility. It enables users to administer their own public/private key pairs and associated certificates for use in self-authentication (where a user authenticates themselves to other users and services) or data integrity and authentication services, by using digital signatures. The keytool command also enables users to cache the public keys (in the form of certificates) of their communicating peers.

A certificate is a digitally signed statement from one entity (person, company, and so on), which says that the public key (and some other information) of some other entity has a particular value. When data is digitally signed, the signature can be verified to check the data integrity and authenticity. Integrity means that the data hasn’t been modified or tampered with, and authenticity means that the data comes from the individual who claims to have created and signed it.

The keytool command also enables users to administer secret keys and passphrases used in symmetric encryption and decryption (Data Encryption Standard). It can also display other security-related information.

The keytool command stores the keys and certificates in a keystone.

List contents of jks keystore file
Extracting SSL certificate from the Java Keystore (JKS)

October 19, 2021

How to view the contents of tar.gz file

To view the contents of tar.gz file, run

tar tf FILE.tar.gz

-f, --file=ARCHIVE         use archive file or device ARCHIVE
-t, --list                 list the contents of an archive

October 17, 2021

zabbix

How to install zabbix agent on Ubuntu

October 16, 2021
How to install zabbix agent on Ubuntu
To install the Zabbix agent on Ubuntu, run the command
```
apt install zabbix-agent
```
To autostart, Zabbix agent on boot, enable it with
```
systemctl enable zabbix-agent
```
To manage Zabbix, use commands
```
systemctl stop zabbix-agent
systemctl start zabbix-agent
systemctl restart zabbix-agent
```
To see Zabbix listening port
```
netstat -lntp | grep zabbix
```
Edit file
```
vi /etc/zabbix/zabbix_agentd.conf 
```
Find
```
Server=127.0.0.1
```
Replace with
```
Server=127.0.0.1,IP_OF_ZABBIX_SERVER_HERE
```
Restart zabbix agent
```
systemctl restart zabbix-agent
```
After the Zabbix agent is installed, you need to add the sevrer in your Zabbix installation.
October 16, 2021
How to Migrate CentOS 8 to AlmaLinux
CentOS 8 is reaching its End of Life in December 2021. If you are using CentOS 8, you may need to consider migrating to other operating systems.

Other alternatives for CentOS 8 are AlmaLinux, Rocky Linux, and Oracle Linux. All of these are free and are based on RHEL 8.

AlmaLinux is released a migration script

https://github.com/AlmaLinux/almalinux-deploy

This script allows you to convert CentOS 8 to AlmaLinux.

WARNING: Make sure you take a full system backup before migrating.

Before migrating, make sure your system is up to date.
```
dnf update
```
If you get error, Error: Failed to download metadata for repo ‘appstream’, run
```
sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-*
sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-*
dnf update
```
Reboot the server
```
reboot
```
Once the server is back online, we can run the AlmaLinux migration script to start the migration.
```
curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
bash almalinux-deploy.sh
```
After the migration is complete, run
```
dnf distro-sync -y 
```
Now you need to reboot the server
```
reboot
```
October 10, 2021