Category: Linux
-
basic server utils
On CentOS/RHEL
yum update yum upgrade -y yum -y remove mlocate yum install -y nmap lynx wget curl telnet jwhois yum install -y atop iotop yum install -y strace yum install -y git yum install -y sysstat yum install -y vim yum install -y tmux yum install -y ncurses-devel automake autoconf gcc gmake patch make yum install -y libcpp libgcc libstdc++ gcc4 gcc4-c++ gcc4-gfortran yum install -y dos2unix libtool gcc-c++ gcc-c++ compat-gcc-32 compat-gcc-32-c++
For non cpanel server, install epel
See Server Setup
-
bash: locale-gen: command not found
When I run locale-gen command, i get the following error
root@netmon1:~# locale-gen en_US.UTF-8 bash: locale-gen: command not found root@netmon1:~#
To see which package provide the locale-gen command, run
boby@sok-01:~$ dpkg -S /usr/sbin/locale-gen locales: /usr/sbin/locale-gen boby@sok-01:~$
The command locale-gen is provided by package locales. To fix the error install package “locales” with the command
apt install -y locales
See locale-gen
-
Search & Replace using rpl
rpl is a very useful command used for bulk string replacement in files. Basic usage is to specify two strings and one or more filenames or directories on the command line. The first string is the string to replace, and the second string is the replacement string. Another Linux command used to replace string is sed.
To instal rpl on Ubuntu or Debian, use
apt install rpl
To install from source
cd /usr/local/src wget http://downloads.laffeycomputer.com/current_builds/rpl-1.4.1.tar.gz tar -zxvf rpl-1.4.1.tar.gz cd rpl-1.4.1 ./configure make make install
Here are some of the command-line options for rpl command.
-i, --ignore-case = Ignore the case of old_string. -w, --whole-words = Make old_string match only on word boundaries. -b, --backup = Move the original files to filename~ before replacing them. -q, --quiet = Quiet mode. -v, --verbose = Verbose mode. -s, --dry-run = Simulation mode, no files are changed. -R, --recursive = Recurse into subdirectories. -e, --escape = Expand escape sequences in old_string and new_string. Examples of escape sequences are '\n' (new-line), '\t' (tab), '\x42' (hexadecimal number 42), '\033' (octal number 033). -p, --prompt = Prompt for confirmation before replacing each file. -f, --force = Ignore errors when trying to restore permissions and file ownership. -d, --keep-times = Keep modification times when replacing files. -t, --use-tmpdir = Use a temporary directory for storing temporary files, usually the value of the environment variable TMPDIR. The default is to put temporary files in the same directory as the file being modified. -a, --all = Do not ignore files and directories starting with.
Examples
Replace string in one file.
rpl "OLD_STRING" "NEW_STRING" FILE_NAME
Replace in multiple files
rpl "OLD_STRING" "NEW_STRING" FILE_NAME_1 FILE_NAME_2
To replace all occurrences of CAT with RAT run the following command, it will go through all files and do the replacement.
rpl -R -x .php -x .html -x .htm 'CAT' 'RAT' *
-x specify file extensions you need to replace. In above case, it only replace in files with .php, .html and .htm file extensions.
-R option is used to change recursively.Replace all occurences of ”F” (on word boundaries) with ”A” in all text files under the grades/ directory:
rpl -Rwd -x'.txt' 'F' 'A' grades/
See sed.
-
Display real time statistics with Logtop
Logtop is a real-time log analysis tool. It can be used to understand log files. The developer describes it as “Display real time statistics of whatever you want.”. You can pass any value to it, logtop aggregate the data and show it by the number of times the data appear. Most repeated items shows on top of the list.
To install logtop on Ubuntu/Debian, run
apt install logtop -y
To see all IP address that is accessing your web server, run
tail -f access_log | awk {'print $1; fflush();'} | logtop
To see the web page that gets the most requests, run
tail -f access_log | awk {'print $7; fflush();'} | logtop
See log
-
ISPConfig how to rename web directory
ISPConfig is a free hosting control panel. The DocumentRoot in ISPConfig is /var/www/clients/client0/web1/web. In the path client0 and web1 changes depending on number of customers and web site you have on the server. If you try to rename web foder, you get permission denied error, this is because ISPConfig set chattr +i (immune) to parent folder of document root so users can change it. This is to avoid a user delete DocumentRoot folder.
root@server:/var/www/clients/client0/web1# mv web web-old mv: cannot move 'web' to 'web-old': Operation not permitted root@server:/var/www/clients/client0/web1#
If you need to rename or delete the folder, you can run
chattr -i /var/www/clients/client0/web1
To protect the folder again, run
chattr +i /var/www/clients/client0/web1
See ISPConfig
-
How to open a VHD or VHDx file in Linux
VHD or VHDx (newer version) is a Virtual Hard Disk file format used by Microsoft’s Hyper-V. To Open a VHD disk on Ubuntu, you can use guestmount command.
To install guestmount, run the command
apt-get install libguestfs-tools
To see list of partiions available in a VHD/VHDx file, you can run
virt-list-partitions FILE_NAME.vhdx
Example
root@ip-172-30-1-114:/mnt# virt-list-partitions srv-kirikas-storage.vhdx /dev/sda1 /dev/sda2 root@ip-172-30-1-114:/mnt#
To mount a partition, run
guestmount -a srv-kirikas-storage.vhdx -m DEVICE --ro MOUNT_POINT
Example
root@ip-172-30-1-114:/mnt# mkdir /vhdx/ root@ip-172-30-1-114:/mnt# guestmount -a srv-kirikas-storage.vhdx -m /dev/sda2 --ro /vhdx/ root@ip-172-30-1-114:/mnt#
In the above example, /vhdx is the mount point, you need to create an empty folder before mounting.
See mount
-
Backup Linux Server with rsnapshot
rsnapshot is a backup software based on rsync. It can make an incremental backup. rsnapshot makes an automatic incremental backup using rsync and cronjob. It use linux hardlinks to save disk space. When you make the first backup, all files are copied to the backup location, every subsequent backup makes a copy of the previous backup using Linux hard links, then copies over new and changed files, deletes files that are removed from the source folder. This way, even if you have multiple copies of the files, only one file is stored in your disk. rsnapshot is available for installation from apt repositories in Ubuntu and Debian Linux.
https://rsnapshot.org/ (GitHub)
To install rsnapshot on Ubuntu/Debian, run
apt install rsnapshot -y
Configure rsnapshot
rsnapshot configuration file available in /etc/rsnapshot.conf, you need to edit this file to configure rsnapshot.
vi /etc/rsnapshot.conf
To change the location of backup folder, change
snapshot_root /var/cache/rsnapshot/
I change this to /backup/
snapshot_root /backup/
Below you will find
retain alpha 6 retain beta 7 retain gamma 4
To easily understand the backup schedule, I renamed it as daily/weekly/monthly
retain daily 6 retain weekly 3 retain monthly 3
This will store 6 daily backups, 3 weekly backups, and 3 monthly backups.
Uncomment following lines
cmd_ssh /usr/bin/ssh cmd_du /usr/bin/du
The lines starting with “backup” define the backup. In these lines each parameter must be separated with TAB (space won’t work).
You can verify the configuration with the command
rsnapshot configtest
Test Backup
To see the commands used by rsnapshot to take backup, run
rsnapshot -t daily
Here is rsnapshot -t daily result for the backup command
backup /root/ localhost/
To exclude a folder from the backup, you can use +rsync_long_args=–exclude=/full/path/to/folder
Example
backup root@31.42.184.232:/home/serverok.in/ serverok.in/ +rsync_long_args=--exclude=/home/serverok.in/html/wp-content/cache
If you need to exclude multiple files, you can use exclude-from to specify a file with all paths to exclude.
backup root@137.184.80.235:/home/ 137.184.80.235/ +rsync_long_args=--exclude-from="/usr/serverok/exclude-server5"
NOTE: use TAB to separate each item in the backup line. For arguments, use space to separate.
Bash script to schedule Backups
Here is a backup script, that checks the day of the month and the day of the week to decide what type of backup needs to run today, this avoids running multiple backups same day.
creare file
mkdir /usr/serverok/ vi /usr/serverok/backup
add following content
#!/bin/bash # Author: Yujin Boby # Web: https://serverok.in DAY_OF_WEEK=$(date +%w) DAY_OF_MONTH=$(date +%d) if [[ "$DAY_OF_MONTH" -eq 1 ]]; then echo "Running monthly backup..." /usr/bin/rsnapshot monthly elif [[ "$DAY_OF_WEEK" -eq 0 ]]; then echo "Running weekly backup..." /usr/bin/rsnapshot weekly else echo "Running daily backup..." /usr/bin/rsnapshot daily fi
set a cronjob to execute the file every day.
@daily /usr/serverok/backup > /dev/null 2>&1
Running Daily Backup
To run the daily automated backup, you need to set a cronjob
# Daily backup at midnight (00:00) 0 0 * * * /usr/bin/rsnapshot daily # Weekly backup on Sunday at 01:00 0 1 * * 0 /usr/bin/rsnapshot weekly # Monthly backup on 1st of each month at 02:00 0 2 1 * * /usr/bin/rsnapshot monthly
This will run backup every day at 00:00 hours server time.
MySQL Backup
Create file
vi /usr/serverok/backup-mysql.sh
Add following content
#!/bin/bash # MySQL Backup Scipt # Author: admin@ServerOk.in # @daily /usr/serverok/backup-mysql.sh > /var/log/backup-mysql.log 2>&1 BACKUP_DIR="/backup/tmp/" if [ ! -d $BACKUP_DIR ]; then mkdir -p $BACKUP_DIR fi find "${BACKUP_DIR}" -maxdepth 1 -type f -mtime +30 -name *.sql -delete # Backup MySQL Database mysql -e "show databases" | egrep -v "(-|Database|mysql|information_schema|performance_schema|phpmyadmin)" > /tmp/sok-dbs.txt for db in `cat /tmp/sok-dbs.txt` do /usr/bin/mysqldump --opt --triggers --routines --events $db > "${BACKUP_DIR}/${db}.sql"; done
Make it executable
chmod 755 /usr/serverok/backup-mysql.sh
Edit file
vi /etc/rsnapshot.conf
Add
backup_script /usr/serverok/backup-mysql.sh localhost/mysql/
In the above line, white spaces are tabs, not spaces.
See Backup
-
Install PHP drivers for Microsoft SQL Server on Ubuntu PHP 7.2
On Ubuntu 18.04 server running PHP 7.2, i want to install Microsoft SQL Server module for PHP. You can find PHP module for SQL server at
https://github.com/microsoft/msphpsql
At the time of writing this PHP module only support PHP 7.4 and newer. SO i need to find older version that supported PHP 7.2. On checking release page, i found version 5.8.0 supported PHP 7.2
First install php7.2 dev package with
apt install php7.2-dev
Instal php modules with pcel
pecl install sqlsrv-5.8.0
During install, i got error
configure: creating ./config.status config.status: creating config.h config.status: executing libtool commands running: make /bin/bash /tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/libtool --mode=compile g++ -std=c++11 -I. -I/tmp/pear/temp/sqlsrv -DPHP_ATOM_INC -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/include -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/main -I/tmp/pear/temp/sqlsrv -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/tmp/pear/temp/sqlsrv/shared/ -DHAVE_CONFIG_H -std=c++11 -D_FORTIFY_SOURCE=2 -O2 -fstack-protector -c /tmp/pear/temp/sqlsrv/conn.cpp -o conn.lo libtool: compile: g++ -std=c++11 -I. -I/tmp/pear/temp/sqlsrv -DPHP_ATOM_INC -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/include -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/main -I/tmp/pear/temp/sqlsrv -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/tmp/pear/temp/sqlsrv/shared/ -DHAVE_CONFIG_H -std=c++11 -D_FORTIFY_SOURCE=2 -O2 -fstack-protector -c /tmp/pear/temp/sqlsrv/conn.cpp -fPIC -DPIC -o .libs/conn.o In file included from /tmp/pear/temp/sqlsrv/shared/typedefs_for_linux.h:23:0, from /tmp/pear/temp/sqlsrv/shared/xplat_winnls.h:24, from /tmp/pear/temp/sqlsrv/shared/FormattedPrint.h:24, from /tmp/pear/temp/sqlsrv/shared/core_sqlsrv.h:41, from /tmp/pear/temp/sqlsrv/php_sqlsrv_int.h:25, from /tmp/pear/temp/sqlsrv/conn.cpp:24: /tmp/pear/temp/sqlsrv/shared/xplat.h:30:10: fatal error: sql.h: No such file or directory #include
^~~~~~~ compilation terminated. Makefile:194: recipe for target 'conn.lo' failed make: *** [conn.lo] Error 1 ERROR: `make' failed root@server:~# This is fixed with command
apt-get install unixodbc-dev
install pdo_sqlsrv with
pecl install pdo_sqlsrv-5.8.0
Run
printf "; priority=20\nextension=sqlsrv.so\n" > /etc/php/7.2/mods-available/sqlsrv.ini printf "; priority=30\nextension=pdo_sqlsrv.so\n" > /etc/php/7.2/mods-available/pdo_sqlsrv.ini
Enabe PHP modules with
phpenmod -v 7.2 sqlsrv pdo_sqlsrv
Restart Apache, now phpinfo() shows pdo_sqlsrv
But when accessing PHP script that connect to MS SQL server, i get error
This extension requires the Microsoft ODBC Driver for SQL Server to communicate with SQL Server. Access the following URL to download the ODBC Driver for SQL Server for x64
To fix this, do
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
For Ubuntu 16.04
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
For Ubuntu 18.04
curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
For Ubuntu 20.04
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
Ubuntu 20.10
curl https://packages.microsoft.com/config/ubuntu/20.10/prod.list > /etc/apt/sources.list.d/mssql-release.list
Update apt cahe
apt-get update
Install Microsoft ODBC
apt-get install -y msodbcsql17
Optional: for bcp and sqlcmd
apt-get install -y mssql-tools echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc source ~/.bashrc apt-get install -y unixodbc-dev
Now php MS SQL module will work. You can find sample PHP code at
https://gist.github.com/serverok/456b3d1d7295463df42c9822e8db3e5b
https://github.com/microsoft/msphpsql/blob/master/sample/pdo_sqlsrv_sample.phpHere are microsoft documentation
See PHP
-
Whitelist an IP address in imunify360
To white list an IP address in imunify360 firewall, you can run command
imunify360-agent whitelist ip add 1.2.3.4 --comment "one good ip" --full-access
Another way to white list IP is create a folder
mkdir /etc/imunify360/whitelist/
Inside the folder, create a file with .txt extension, add IPs you need whitelist in CIDR format, one per line. Reoad firewall with command
imunify360-agent reload-lists
See imunify360
-
install php redis extension in aaPanel
To install php redis extension on aaPanel control panel, run
cd /usr/local/src git clone https://github.com/phpredis/phpredis cd /usr/local/src/phpredis phpize ./configure make make install
If you get php-config not found error, you need to create a symlink. This is for server with PHP 7.4 installed, if your server have differnt version of PHP, you may need to change path.
ln -s /www/server/php/74/bin/php-config /usr/bin
Now continue the installation from “./configure” command, it will work.
To activate redis, edit php.ini
vi /www/server/php/74/etc/php.ini
At the end, add
extension=/www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/redis.so
The path to redis.so file (/www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/) may be differnt on your server. This path you will get after you run the command “make install”
[root@server3290 phpredis]# make install Installing shared extensions: /www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/ [root@server3290 phpredis]#
You can verify php redis extension is installed with command
php -m | grep redis
To make redis extension available for web site, you need to restart php-fpm-74 service.
systemctl stop php-fpm-74 systemctl start php-fpm-74
You can find name of service file with command
ls -l /etc/init.d | grep php
To verify web sites have redis extension available, you can create a file with phpinfo() function, that will show following.
See aaPanel