Category: Linux

OVH CentOS 7 server grub rescue prompt

On an OVH Cpanel server running CentOS 7, the server won’t boot. When accessing the server console using IPMI, I found the following error.

I booted the server into rescue mode, checked the disk partitions with the command parted -l

Disk /dev/nvme0n1: 450GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name     Flags
 1      1049kB  537MB   536MB   fat32           primary  boot, esp
 2      537MB   1073MB  536MB   ext4            primary  raid
 3      1073MB  53.5GB  52.4GB  ext4            primary  raid
 4      53.5GB  450GB   396GB   ext4            primary  raid
 5      450GB   450GB   536MB   linux-swap(v1)  primary


Disk /dev/nvme1n1: 450GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name     Flags
 1      1049kB  537MB   536MB   fat32           primary  boot, esp
 2      537MB   1073MB  536MB   ext4            primary  raid
 3      1073MB  53.5GB  52.4GB  ext4            primary  raid
 4      53.5GB  450GB   396GB   ext4            primary  raid
 5      450GB   450GB   536MB   linux-swap(v1)  primary

The server had 2 NVMe disks configured as RAID 1 mirrors. The first partition with fat32 filesystem is used for boot.

To fix the problem, I chrooted to the server file system with the following command (this may be changed based on your server’s partition scheme).

mount /dev/md3 /mnt
mount /dev/md2 /mnt/boot/
mount /dev/md4 /mnt/home
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mount --bind /dev/pts /mnt/dev/pts/
chroot /mnt

Reinstalled the kernel with

yum reinstall kernel

Then reinstalled grub loader. This server used UEFI, so used following commands

mkdir /nvme0n1p1
mkdir /nvme1n1p1
mount /dev/nvme0n1p1 /nvme0n1p1
mount /dev/nvme1n1p1 /nvme1n1p1
grub2-install --target=x86_64-efi --efi-directory=/nvme0n1p1 --bootloader-id=GRUB
grub2-install --target=x86_64-efi --efi-directory=/nvme1n1p1 --bootloader-id=GRUB

grub boot loaded is installed on both disks, so server will be able to boot when either one of the disk is selected as boot device.

Back to grub

January 4, 2023

How to install docker on AlmaLinux 8

To install docker on AlmaLinux, run the following commands

Install yum-utils

dnf install -y yum-utils

Add docker repository

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install Docker CE with

dnf install docker-ce

Enable docker

systemctl enable docker

Start Docker

systemctl start docker

To verify if docker is working properly, run

docker run hello-world

Example

[root@cloud ~]# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:94ebc7edf3401f299cd3376a1669bc0a49aef92d6d2669005f9bc5ef028dc333
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[root@cloud ~]#

See docker

January 3, 2023

How to Install ffmpeg static build

FFmpeg binary static build available for download from

https://johnvansickle.com/ffmpeg/

To install ffmpeg static build, run

cd /usr/local/src
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
tar xvf ffmpeg-release-amd64-static.tar.xz
cd ffmpeg-*-amd64-static/
cp ffmpeg ffprobe qt-faststart /usr/bin/
cp -r model /usr/local/share/

See ffmpeg

December 29, 2022

grubby
grubby is provided by package grubby on RHEL based systems. To install grubby, run
```
yum install grubby
```
To list all kernels, run
```
grubby --info=ALL
```
To view the default kernel, run
```
grubby --default-kernel
```
To set a kernel default, you need to specify the vmlinux file, example
```
grubby --set-default  /boot/vmlinuz-3.10.0-962.3.2.lve1.5.73.el7.x86_64
```
Or specify the index
```
grubby --set-default-index 2
```
Back to grub
December 13, 2022
How to list iptables rules
To view all rules in iptables, run
```
iptables -L
```
Or
```
iptables --list
```
To list iptables rules without resolving IP to hostname
```
iptables -L -n
```
To show the line number
```
iptables -L -n --line-number
```
To list NAT rules
```
iptables -t nat -L -n --line-number
iptables -t nat -L -n -v
```
If you run iptables-save command, it will list all iptables rules
```
iptables-save
```
You can redirect the result of the iptables-save command to a file. You can restore with iptables-restore command.

Back to iptables
December 13, 2022
How to change varnish port in Ubuntu
To change the varnish port, run the command
```
systemctl edit varnish
```
It opens an editor, in the editor, paste the following
```
[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
```
This will change varnish listening port to 80. If you need to change to another port, change “-a :80” to whatever port you want to use. Save and exit the editot.

Now resart varnish
```
systemctl restart varnish
```
Back to Varnish
December 8, 2022
Mounting partition stored inside Logical Volume
On a KVM Virtual machine using LVM storage, the LVM disk is partitioned as follows.
```
root@mail:~# parted /dev/vg1/iredmail print
Model: Linux device-mapper (linear) (dm)
Disk /dev/dm-1: 64.4GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2097kB  1049kB                     bios_grub
 2      2097kB  1076MB  1074MB  ext4
 3      1076MB  64.4GB  63.3GB

root@mail:~# 
```
I want to mount partition number 2 on the host machine.
```
 2      2097kB  1076MB  1074MB  ext4
```
To make the partition available to mount, you need to use kpartx utility. Install kpartx with
```
apt install kpartx
```
To make the partitions inside the Logical volume available to mount, use the command
```
kpartx -a LVM_DEIVE_NAME
```
In this case, I used
```
kpartx -a /dev/vg1/iredmail
```
Now lvscan command will display the new device
```
root@mail:~# lvscan
  ACTIVE            '/dev/vg1/win10' [100.00 GiB] inherit
  ACTIVE            '/dev/vg1/iredmail' [60.00 GiB] inherit
  ACTIVE            '/dev/ubuntu-vg/ubuntu-lv' [29.50 GiB] inherit
root@mail:~# 
```
To mount the device, you can use the command
```
mount /dev/ubuntu-vg/ubuntu-lv /mnt
```
IMPORTANT: do not mount the device on the host if it is used in the guest VM as it will result in data corruption. I mounted the disk on the host machine and created some files, but it did not show in the guest. After I reboot the guest VM, it did not boot back, I had to do fsck to get the VM to work again.

Back to LVM
December 4, 2022

Configure multiple SSL certificates in nuster

Nuster is a high performance HTTP proxy cache server. It is based on haproxy.

To configure multiple SSL certificates in nuster, create SSL in PEM format.

Edit nuster.cfg, you will see something like the following.

global
    nuster cache on dir /cache
    nuster manager on uri /internal/nuster purge-method PURGEX
frontend fe
    bind *:443 ssl crt /etc/ssl/ssl1.pem alpn h2,http/1.1
    mode http
    default_backend ssl_443
backend ssl_443
    mode http
    nuster cache off
    nuster rule all disk on ttl 7d
    http-request set-header X-Client-IP %[src]
    server s1 128.1.2.9:443 ssl verify none

SSL is configured in the line

    bind *:443 ssl crt /etc/ssl/ssl1.pem alpn h2,http/1.1

To add another domain SSL certificate, modify the line as follows

    bind *:443 ssl crt /etc/ssl/ssl1.pem crt /etc/ssl/ssl2.pem alpn h2,http/1.1

You can add as many SSL certificate in the format

crt /etc/ssl/SSL_FILE_NAME.pem

Back to nuster

November 22, 2022

How to Remove a logical volume

To remove a logical volume, you can use the command

lvremove /dev/vg-name/lv-name

Example

lvremove /dev/vg1/win10

This will remove logical volume with the name “win10” from volume group “vg1”.

[root@Alma-85-amd64-base ~]# lvremove  vg1/data1
Do you really want to remove active logical volume vg1/data1? [y/n]: y
  Logical volume "data1" successfully removed.
[root@Alma-85-amd64-base ~]#

back to LVM

November 21, 2022

systemd-networkd

systemd-networkd configuration files are stored in /etc/systemd/network/

Example

/etc/systemd/network/05-eth0.network

Here is a sample config

[Match]
Name=eth0

[Network]
DHCP=no
DNS=172.105.170.5 172.105.161.5 172.105.171.5
Domains=members.linode.com
IPv6PrivacyExtensions=false

Gateway=194.195.127.1
Address=194.195.127.150/24

To restart systemd-networkd

systemctl restart systemd-networkd

See IP

November 14, 2022

How to disable a rule in ModSecurity Apache

To disable a rule in ModSecurity, edit Apache configuration, add

SecRuleRemoveById RULE_ID_SEPERATED_BY_SPACE

This needed to be added after all rules were loaded.

On Ubuntu, I edited the file

/etc/apache2/mods-enabled/security2.conf

Here is what I have in a server which disables rules 941180 949110 980130

<IfModule security2_module>
	# Default Debian dir for modsecurity's persistent data
	SecDataDir /var/cache/modsecurity

	# Include all the *.conf files in /etc/modsecurity.
	# Keeping your local configuration in that directory
	# will allow for an easy upgrade of THIS file and
	# make your life easier
        IncludeOptional /etc/modsecurity/*.conf

	# Include OWASP ModSecurity CRS rules if installed
	IncludeOptional /usr/share/modsecurity-crs/*.load
        SecRuleRemoveById 941180 949110 980130
</IfModule>

Back to ModSecurity

November 11, 2022

access denied: tty ‘tty1’ is not secure

On a CentOS 7 server, when login as user root on the console, login fails. If I log in as a normal user, then I am able to switch to user root with the command “su – root”.

On checking /var/log/secure, I found the following error.

Nov 10 03:44:42 localhost login: pam_securetty(login:auth): access denied: tty 'tty1' is not secure !
Nov 10 03:44:45 localhost login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
Nov 10 03:44:47 localhost login: FAILED LOGIN 1 FROM tty1 FOR root, Authentication failure

To fix the error, edit the file

vi /etc/securetty

In the file, add

tty1

On CentOS 7 server, the contents of the file are

[root@localhost ~]# cat /etc/securetty 
console
vc/1
vc/2
vc/3
vc/4
vc/5
vc/6
vc/7
vc/8
vc/9
vc/10
vc/11
tty1
tty2
tty3
tty4
tty5
tty6
tty7
tty8
tty9
tty10
tty11
ttyS0
ttysclp0
sclp_line0
3270/tty1
hvc0
hvc1
hvc2
hvc3
hvc4
hvc5
hvc6
hvc7
hvsi0
hvsi1
hvsi2
xvc0
[root@localhost ~]#

Permission for the file is 600, in case you have the wrong permission, change it with

chmod 600 /etc/securetty

Back to CentOS 7

November 10, 2022