Category: Linux

  • basic server utils

    On CentOS/RHEL

    yum update
    yum upgrade -y
    yum -y remove mlocate
    yum install -y nmap lynx wget curl telnet jwhois
    yum install -y atop iotop
    yum install -y strace
    yum install -y git
    yum install -y sysstat
    yum install -y vim
    yum install -y tmux
    yum install -y ncurses-devel automake autoconf gcc gmake patch make
    yum install -y libcpp libgcc libstdc++ gcc4 gcc4-c++ gcc4-gfortran
    yum install -y dos2unix libtool gcc-c++ gcc-c++ compat-gcc-32 compat-gcc-32-c++
    

    For non cpanel server, install epel

    See Server Setup

  • bash: locale-gen: command not found

    bash: locale-gen: command not found

    When I run locale-gen command, i get the following error

    root@netmon1:~# locale-gen en_US.UTF-8
    bash: locale-gen: command not found
    root@netmon1:~# 
    

    To see which package provide the locale-gen command, run

    boby@sok-01:~$ dpkg -S /usr/sbin/locale-gen
    locales: /usr/sbin/locale-gen
    boby@sok-01:~$ 
    

    The command locale-gen is provided by package locales. To fix the error install package “locales” with the command

    apt install -y locales
    

    See locale-gen

  • Search & Replace using rpl

    rpl is a very useful command used for bulk string replacement in files. Basic usage is to specify two strings and one or more filenames or directories on the command line. The first string is the string to replace, and the second string is the replacement string. Another Linux command used to replace string is sed.

    To instal rpl on Ubuntu or Debian, use

    apt install rpl
    

    To install from source

    cd /usr/local/src
    wget http://downloads.laffeycomputer.com/current_builds/rpl-1.4.1.tar.gz
    tar -zxvf rpl-1.4.1.tar.gz
    cd rpl-1.4.1
    ./configure
    make
    make install
    

    Here are some of the command-line options for rpl command.

    -i, --ignore-case = Ignore the case of old_string.
    -w, --whole-words = Make old_string match only on word boundaries.
    -b, --backup = Move the original files to filename~ before replacing them.
    -q, --quiet = Quiet mode.
    -v, --verbose = Verbose mode.
    -s, --dry-run = Simulation mode, no files are changed.
    -R, --recursive = Recurse into subdirectories.
    -e, --escape = Expand escape sequences in old_string and new_string. Examples of escape sequences are '\n' (new-line), '\t' (tab), '\x42' (hexadecimal number 42), '\033' (octal number 033).
    -p, --prompt = Prompt for confirmation before replacing each file.
    -f, --force = Ignore errors when trying to restore permissions and file ownership.
    -d, --keep-times = Keep modification times when replacing files.
    -t, --use-tmpdir = Use a temporary directory for storing temporary files, usually the value of the environment variable TMPDIR. The default is to put temporary files in the same directory as the file being modified.
    -a, --all = Do not ignore files and directories starting with.
    

    Examples

    Replace string in one file.

    rpl "OLD_STRING" "NEW_STRING" FILE_NAME
    

    Replace in multiple files

    rpl "OLD_STRING" "NEW_STRING" FILE_NAME_1 FILE_NAME_2
    

    To replace all occurrences of CAT with RAT run the following command, it will go through all files and do the replacement.

    rpl -R -x .php -x .html -x .htm 'CAT' 'RAT' *
    

    -x specify file extensions you need to replace. In above case, it only replace in files with .php, .html and .htm file extensions.
    -R option is used to change recursively.

    Replace all occurences of ”F” (on word boundaries) with ”A” in all text files under the grades/ directory:

    rpl -Rwd -x'.txt' 'F' 'A' grades/
    

    See sed.

  • Display real time statistics with Logtop

    Logtop is a real-time log analysis tool. It can be used to understand log files. The developer describes it as “Display real time statistics of whatever you want.”. You can pass any value to it, logtop aggregate the data and show it by the number of times the data appear. Most repeated items shows on top of the list.

    To install logtop on Ubuntu/Debian, run

    apt install logtop -y
    

    To see all IP address that is accessing your web server, run

    tail -f access_log | awk {'print $1; fflush();'} | logtop
    

    To see the web page that gets the most requests, run

    tail -f access_log |  awk {'print $7; fflush();'} | logtop
    

    See log

  • ISPConfig how to rename web directory

    ISPConfig how to rename web directory

    ISPConfig is a free hosting control panel. The DocumentRoot in ISPConfig is /var/www/clients/client0/web1/web. In the path client0 and web1 changes depending on number of customers and web site you have on the server. If you try to rename web foder, you get permission denied error, this is because ISPConfig set chattr +i (immune) to parent folder of document root so users can change it. This is to avoid a user delete DocumentRoot folder.

    root@server:/var/www/clients/client0/web1# mv web web-old
    mv: cannot move 'web' to 'web-old': Operation not permitted
    root@server:/var/www/clients/client0/web1# 
    

    If you need to rename or delete the folder, you can run

    chattr -i /var/www/clients/client0/web1
    

    To protect the folder again, run

    chattr +i /var/www/clients/client0/web1
    

    See ISPConfig

  • How to open a VHD or VHDx file in Linux

    How to open a VHD or VHDx file in Linux

    VHD or VHDx (newer version) is a Virtual Hard Disk file format used by Microsoft’s Hyper-V. To Open a VHD disk on Ubuntu, you can use guestmount command.

    mount vhdx

    To install guestmount, run the command

    apt-get install libguestfs-tools

    To see list of partiions available in a VHD/VHDx file, you can run

    virt-list-partitions FILE_NAME.vhdx

    Example

    root@ip-172-30-1-114:/mnt# virt-list-partitions srv-kirikas-storage.vhdx
    /dev/sda1
    /dev/sda2
    root@ip-172-30-1-114:/mnt# 

    To mount a partition, run

    guestmount -a srv-kirikas-storage.vhdx -m DEVICE --ro MOUNT_POINT

    Example

    root@ip-172-30-1-114:/mnt# mkdir /vhdx/
    root@ip-172-30-1-114:/mnt# guestmount -a srv-kirikas-storage.vhdx -m /dev/sda2 --ro /vhdx/
    root@ip-172-30-1-114:/mnt# 

    In the above example, /vhdx is the mount point, you need to create an empty folder before mounting.

    See mount

  • Backup Linux Server with rsnapshot

    Backup Linux Server with rsnapshot

    rsnapshot is a backup software based on rsync. It can make an incremental backup. rsnapshot makes an automatic incremental backup using rsync and cronjob. It use linux hardlinks to save disk space. When you make the first backup, all files are copied to the backup location, every subsequent backup makes a copy of the previous backup using Linux hard links, then copies over new and changed files, deletes files that are removed from the source folder. This way, even if you have multiple copies of the files, only one file is stored in your disk. rsnapshot is available for installation from apt repositories in Ubuntu and Debian Linux.

    https://rsnapshot.org/ (GitHub)

    To install rsnapshot on Ubuntu/Debian, run

    apt install rsnapshot -y
    

    Configure rsnapshot

    rsnapshot configuration file available in /etc/rsnapshot.conf, you need to edit this file to configure rsnapshot.

    vi /etc/rsnapshot.conf

    To change the location of backup folder, change

    snapshot_root   /var/cache/rsnapshot/

    I change this to /backup/

    snapshot_root   /backup/

    Below you will find

    retain  alpha   6
    retain  beta    7
    retain  gamma   4

    To easily understand the backup schedule, I renamed it as daily/weekly/monthly

    retain  daily   6
    retain  weekly    3
    retain  monthly   3

    This will store 6 daily backups, 3 weekly backups, and 3 monthly backups.

    Uncomment following lines

    cmd_ssh        /usr/bin/ssh
    cmd_du         /usr/bin/du

    The lines starting with “backup” define the backup. In these lines each parameter must be separated with TAB (space won’t work).

    You can verify the configuration with the command

    rsnapshot configtest

    Test Backup

    To see the commands used by rsnapshot to take backup, run

    rsnapshot -t daily

    Here is rsnapshot -t daily result for the backup command

    backup	/root/		localhost/

    To exclude a folder from the backup, you can use +rsync_long_args=–exclude=/full/path/to/folder

    Example

    backup  [email protected]:/home/serverok.in/  serverok.in/  +rsync_long_args=--exclude=/home/serverok.in/html/wp-content/cache

    If you need to exclude multiple files, you can use exclude-from to specify a file with all paths to exclude.

    backup	[email protected]:/home/	137.184.80.235/		+rsync_long_args=--exclude-from="/usr/serverok/exclude-server5"

    NOTE: use TAB to separate each item in the backup line. For arguments, use space to separate.

    Running Daily Backup

    To run the daily automated backup, you need to set a cronjob

    0 0 * * * /usr/bin/rsnapshot daily
    0 6 * * 0 /usr/bin/rsnapshot weekly
    

    This will run backup every day at 00:00 hours server time.

    Bash script to schedule Backups

    Here is a backup script, that checks the day of the month and the day of the week to decide what type of backup needs to run today, this avoids running multiple backups same day.

    #!/bin/bash
    # Author: Yujin Boby
    # Web: https://serverok.in
    
    DAY_OF_WEEK=$(date +%w)
    DAY_OF_MONTH=$(date +%d)
    
    if [[ "$DAY_OF_MONTH" -eq 1 ]]; then
        echo "Running monthly backup..."
        /usr/bin/rsnapshot monthly
    elif [[ "$DAY_OF_WEEK" -eq 0 ]]; then
        echo "Running weekly backup..."
        /usr/bin/rsnapshot weekly
    else
        echo "Running daily backup..."
        /usr/bin/rsnapshot daily
    fi

    Save the content in a file, for example, backup.sh, and set a cronjob to execute the file every day.

    See Backup

  • Install PHP drivers for Microsoft SQL Server on Ubuntu PHP 7.2

    Install PHP drivers for Microsoft SQL Server on Ubuntu PHP 7.2

    On Ubuntu 18.04 server running PHP 7.2, i want to install Microsoft SQL Server module for PHP. You can find PHP module for SQL server at

    https://github.com/microsoft/msphpsql

    At the time of writing this PHP module only support PHP 7.4 and newer. SO i need to find older version that supported PHP 7.2. On checking release page, i found version 5.8.0 supported PHP 7.2

    First install php7.2 dev package with

    apt install php7.2-dev
    

    Instal php modules with pcel

    pecl install sqlsrv-5.8.0
    

    During install, i got error

    configure: creating ./config.status
    config.status: creating config.h
    config.status: executing libtool commands
    running: make
    /bin/bash /tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/libtool --mode=compile g++ -std=c++11 -I. -I/tmp/pear/temp/sqlsrv -DPHP_ATOM_INC -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/include -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/main -I/tmp/pear/temp/sqlsrv -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/tmp/pear/temp/sqlsrv/shared/  -DHAVE_CONFIG_H  -std=c++11 -D_FORTIFY_SOURCE=2 -O2 -fstack-protector   -c /tmp/pear/temp/sqlsrv/conn.cpp -o conn.lo
    libtool: compile:  g++ -std=c++11 -I. -I/tmp/pear/temp/sqlsrv -DPHP_ATOM_INC -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/include -I/tmp/pear/temp/pear-build-rootnuxjAy/sqlsrv-5.8.0/main -I/tmp/pear/temp/sqlsrv -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/tmp/pear/temp/sqlsrv/shared/ -DHAVE_CONFIG_H -std=c++11 -D_FORTIFY_SOURCE=2 -O2 -fstack-protector -c /tmp/pear/temp/sqlsrv/conn.cpp  -fPIC -DPIC -o .libs/conn.o
    In file included from /tmp/pear/temp/sqlsrv/shared/typedefs_for_linux.h:23:0,
                     from /tmp/pear/temp/sqlsrv/shared/xplat_winnls.h:24,
                     from /tmp/pear/temp/sqlsrv/shared/FormattedPrint.h:24,
                     from /tmp/pear/temp/sqlsrv/shared/core_sqlsrv.h:41,
                     from /tmp/pear/temp/sqlsrv/php_sqlsrv_int.h:25,
                     from /tmp/pear/temp/sqlsrv/conn.cpp:24:
    /tmp/pear/temp/sqlsrv/shared/xplat.h:30:10: fatal error: sql.h: No such file or directory
     #include 
              ^~~~~~~
    compilation terminated.
    Makefile:194: recipe for target 'conn.lo' failed
    make: *** [conn.lo] Error 1
    ERROR: `make' failed
    root@server:~# 
    

    This is fixed with command

    apt-get install unixodbc-dev
    

    install pdo_sqlsrv with

    pecl install pdo_sqlsrv-5.8.0
    

    Run

    printf "; priority=20\nextension=sqlsrv.so\n" > /etc/php/7.2/mods-available/sqlsrv.ini
    printf "; priority=30\nextension=pdo_sqlsrv.so\n" > /etc/php/7.2/mods-available/pdo_sqlsrv.ini
    

    Enabe PHP modules with

    phpenmod -v 7.2 sqlsrv pdo_sqlsrv
    

    Restart Apache, now phpinfo() shows pdo_sqlsrv

    But when accessing PHP script that connect to MS SQL server, i get error

    This extension requires the Microsoft ODBC Driver for SQL Server to communicate with SQL Server. Access the following URL to download the ODBC Driver for SQL Server for x64

    To fix this, do

    curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
    

    For Ubuntu 16.04

    curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
    

    For Ubuntu 18.04

    curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
    

    For Ubuntu 20.04

    curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
    

    Ubuntu 20.10

    curl https://packages.microsoft.com/config/ubuntu/20.10/prod.list > /etc/apt/sources.list.d/mssql-release.list
    

    Update apt cahe

    apt-get update
    

    Install Microsoft ODBC

    apt-get install -y msodbcsql17
    

    Optional: for bcp and sqlcmd

    apt-get install -y mssql-tools
    echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
    source ~/.bashrc
    apt-get install -y unixodbc-dev
    

    Now php MS SQL module will work. You can find sample PHP code at

    https://gist.github.com/serverok/456b3d1d7295463df42c9822e8db3e5b
    https://github.com/microsoft/msphpsql/blob/master/sample/pdo_sqlsrv_sample.php

    Here are microsoft documentation

    https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017

    https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15

    See PHP

  • Whitelist an IP address in imunify360

    To white list an IP address in imunify360 firewall, you can run command

    imunify360-agent whitelist ip add 1.2.3.4 --comment "one good ip" --full-access
    

    Another way to white list IP is create a folder

    mkdir /etc/imunify360/whitelist/
    

    Inside the folder, create a file with .txt extension, add IPs you need whitelist in CIDR format, one per line. Reoad firewall with command

    imunify360-agent reload-lists
    

    See imunify360

  • install php redis extension in aaPanel

    install php redis extension in aaPanel

    To install php redis extension on aaPanel control panel, run

    cd /usr/local/src
    git clone https://github.com/phpredis/phpredis
    cd /usr/local/src/phpredis
    phpize
    ./configure
    make
    make install
    

    If you get php-config not found error, you need to create a symlink. This is for server with PHP 7.4 installed, if your server have differnt version of PHP, you may need to change path.

    ln -s /www/server/php/74/bin/php-config /usr/bin
    

    Now continue the installation from “./configure” command, it will work.

    To activate redis, edit php.ini

    vi /www/server/php/74/etc/php.ini
    

    At the end, add

    extension=/www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/redis.so
    

    The path to redis.so file (/www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/) may be differnt on your server. This path you will get after you run the command “make install”

    [root@server3290 phpredis]# make install
    Installing shared extensions:     /www/server/php/74/lib/php/extensions/no-debug-non-zts-20190902/
    [root@server3290 phpredis]# 
    

    You can verify php redis extension is installed with command

    php -m | grep redis
    

    To make redis extension available for web site, you need to restart php-fpm-74 service.

    systemctl stop php-fpm-74
    systemctl start php-fpm-74
    

    You can find name of service file with command

    ls -l /etc/init.d | grep php
    

    To verify web sites have redis extension available, you can create a file with phpinfo() function, that will show following.

    redis php extension aapanel

    See aaPanel

  • Force all SSH connection through Jump Host

    I want all my SSH connection go through a VPS so my IP address stay same even if my ISP changes IP. This help me to only access SSH access from my VPS IP address, this increase security as no one else able to SSH to the server.

    To force all my SSH connection to use ProxyJump, i edit file

    vi ~/.ssh/config
    

    Added

    Host * !ok.serverok.in !51.38.246.115 !192.168.1.?
        ProxyJump [email protected]:3333
    

    In above Host * will match all hosts, so whenever you try to SSH to a server, it will use ProxyJump. I specified some host with !, this will exclude those host from using ProxyJump. One of which is the Jump Host itself, it is important you exclude Jump Host or it will cause a loop and won’t work. You may also need to add your Local IP address.

    See SSH