Category: MySQL

  • How to Backup Large MySQL Database Table by Table

    How to Backup Large MySQL Database Table by Table

    Backing up and restoring large databases can be challenging at times.

    Recently, when I had to back up and restore a large database, I did a backup for each database table and restored them one by one using a bash script. In case of a restore failure, you only need to deal with the specific table backup that caused the failure.

    Create a Backup Script

    First, let’s create a bash script to back up the database table by table. Save the following script as backup_db.sh. This script will back up each table in your database to a separate .sql file.

    #!/bin/bash
    
    MYSQL_DB="wordpress_db"
    
    BACKUP_DIR="/root/mysql/${MYSQL_DB}"
    
    mkdir -p "$BACKUP_DIR"
    
    TABLES=$(mysql -N -e "SHOW TABLES FROM ${MYSQL_DB}")
    
    for TABLE in $TABLES
    do
        echo "Backing up table: $TABLE"
        /usr/bin/mysqldump "${MYSQL_DB}" "$TABLE" > "$BACKUP_DIR/${TABLE}.sql"
    done
    
    echo "Backup completed. Files are stored in $BACKUP_DIR"
    

    Run the Backup Script

    After creating backup_db.sh, you can run it with:

    bash backup_db.sh

    Once the script runs, you’ll have each table from your database saved as a separate .sql file in the specified backup directory.

    Create a Restore Script

    Now that you have a backup, you’ll need a way to restore these tables. The following script will restore each table from its corresponding .sql file.

    #!/bin/bash
    
    MYSQL_DB="wordpress_db"
    
    BACKUP_DIR="/root/mysql/${MYSQL_DB}"
    
    if [ ! -d "$BACKUP_DIR" ]; then
        echo "Error: Backup directory does not exist."
        exit 1
    fi
    
    for SQL_FILE in "$BACKUP_DIR"/*.sql
    do
        if [ -f "$SQL_FILE" ]; then
            TABLE_NAME=$(basename "$SQL_FILE" .sql)
            echo "Restoring table: $TABLE_NAME"
            /usr/bin/mysql "${MYSQL_DB}" < "$SQL_FILE"
            if [ $? -eq 0 ]; then
                echo "Successfully restored $TABLE_NAME"
            else
                echo "Error restoring $TABLE_NAME"
            fi
        fi
    done
    
    echo "Restore process completed."

    MYSQL_DB: Again, replace “wordpress_db” with your actual database name.

    Run the Restore Script

    After creating restore_db.sh, make it executable and run it:

    bash restore_db.sh

    For transferring files between servers directly, see Server-to-Server File Transfer with PHP Script

    Back to MySQL Backup

  • How to Recover MySQL Table Structure from .frm File

    How to Recover MySQL Table Structure from .frm File

    To recover MySQL table structure from .frm file, you can use dbsake. You can find dbsake documentation at http://dbsake.readthedocs.org/en/latest/.

    To install dbsake, run

    curl -s http://get.dbsake.net > dbsake
    chmod u+x dbsake
    mv dbsake /usr/local/bin/

    To find the structure of a table from the .frm file, run

    dbsake frmdump /path/to/table_name.frm
    recover table structure with dbsake

    Back to MySQL

  • Free MySQL Database Hosting

    Free MySQL Database Hosting

    Looking for a free MySQL hosting solution? In this blog post, I will list some of the MySQL hosting providers that offer free and paid MySQL hosting. These MySQL hosting is useful for smaller database work loads. If you have lot of data, you may need local MySQL server or a MySQL server to your web application server.

    Filess

    Filess.io is a free MySQL hosting provider that offers a 100% free plan that allows you to host one MySQL database with up to 100MB of storage space. You can also upgrade to a paid plan if you need more storage space or features.

    It is a good option for people who need a free MySQL hosting solution. It is easy to set up and use.

    https://filess.io

    Free MySQL Hosting

    FreeMySQLHosting.net is a free MySQL hosting provider that offers a generous amount of storage space and bandwidth on their free plan. They also offer a variety of features, including backups, replication, and security.

    One of the main benefits of using FreeMySQLHosting.net is that they offer a free plan with unlimited MySQL databases. This means that you can create as many databases as you need, without having to worry about storage space or bandwidth limits. They also offer a generous amount of storage space and bandwidth on their free plan, so you can easily host small to medium-sized websites or applications.

    Another benefit of using FreeMySQLHosting.net is that they offer a variety of features, including backups, replication, and security. This means that your data is safe and secure, even if something happens to their servers. They also offer 24/7 customer support, so you can always get help if you need it.

    https://www.freemysqlhosting.net/

    Back to MySQL

  • mysqldump Lost connection to MySQL server during query when dumping table

    mysqldump Lost connection to MySQL server during query when dumping table

    When taking backup of a MySQL database, i got error

    root@server1:~# mysqldump --opt serverok_wp > serverok_wp.sql
    mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `wp_options` at row: 60831
    root@server1:~#
    

    This error happens when connection between your MySQL client and server is timing out during the mysqldump process.

    To fix, increase the value of max_allowed_packet in both MySQL server and client configuration files. This parameter controls the maximum size of a single packet. The default value is 16M, but you can increase it up to 1G.

    On Ubuntu MariaDB installation, edit MySQL client configuration file

    vi /etc/mysql/mariadb.conf.d/50-client.cnf
    

    Under [client], add

    max_allowed_packet=1G
    

    set max_allowed_packet for mysql client

    Next you need to update MySQL server configuration file

    vi /etc/mysql/mariadb.conf.d/50-server.cnf
    

    Under [mysqld] section, find entry for max_allowed_packet, update its value to 1G, if not found, add an entry.

    max_allowed_packet=1G
    

    configure mariadb server max_allowed_packet

    Now restart MySQL

    systemctl restart mysql
    

    Back to mysqldump

  • How to Migrate MySQL Users to Another Server

    How to Migrate MySQL Users to Another Server

    I wanted to Migrate MySQL from one server to another. Usually, I could just copy over the MySQL data directory (/var/lib/mysql) to the new server, this will copy over MySQL users and data. On this server, the MySQL data file (ibdata1) was too large, 200 GB approx. If I do a mysqldump, the SQL file was only 10 GB, so I decided not to copy over the data directory to the new server.

    I did a MySQL dump of all databases with the command

    mysqldump --events --routines --triggers --all-databases > all-dbs.sql
    

    Copy the file all-dbs.sql to the new server and tried to restore with the command

    mysql < all-dbs.sql
    

    It failed because MySQL on old and new servers are different, recent MySQL versions have users table changed to view.

    So I backed up individual MySQL databases using MySQL backup all databases, copied over to the new server, and restore the databases. When restoring databases, I get an error saying the database does not exist, I created the databases and restored the SQL files.

    Now we have all databases and data on new server. We need to create all users with correct permissions. To do this, on source server, run

    pt-show-grants
    

    pt-show-grants is part of Percona Toolkit. You can install Percona Toolkit on Ubuntu/Debian with the command

    apt install percona-toolkit
    

    If you have password for root user, you can use

    pt-show-grants --host localhost --user root --ask-pass
    

    The command will show the SQL commands used to create the users with the correct permission. Copy it and run it on new server MySQL command prompt. This will create all users on new server with proper permissions.

    Copy a specific user to new server

    To view commands to create a specific user, you can run the command

    show grants for 'USER_HERE'@'HOST_HERE';
    

    Example

    show grants for 'root'@'localhost';
    

    Copy all MySQL users to the new server

    If you don't have Percona Toolkit installed, you can use the following command

    mysql -u root --silent --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql -u root --skip-column-names -A
    

    It will generate SQL commands to create users, copy and run it on new server to create MySQL users.

    Here is a bash script to do the same

    mysql --silent --skip-column-names --execute "select concat('\'',User,'\'@\'',Host,'\'') as User from mysql.user" | sort | \
    while read u
     do echo "-- $u"; mysql --silent --skip-column-names --execute "show grants for $u" | sed 's/$/;/'
    done
    

    Back to MySQL

  • How to disable MySQL X Protocol (mysqlx_port)

    How to disable MySQL X Protocol (mysqlx_port)

    After installing MySQL 8 and checking listening ports with “netstat -lntp”, you will see MySQL is listening on ports 3306 and 33060.

    root@mysql:~# netstat -lntp | grep mysql
    tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      13643/mysqld        
    tcp        0      0 127.0.0.1:33060         0.0.0.0:*               LISTEN      13643/mysqld        
    root@mysql:~# 

    Port 3306 is used by traditional MySQL protocol (MySQL Classic Protocol). MySQL X Protocol uses port 33060.

    What is MySQL X Protocol?

    MySQL X Protocol is a network protocol that allows communication between MySQL Server and client applications using a modern, asynchronous, and high-performance approach.

    One of the key advantages of MySQL X Protocol is its ability to handle large datasets efficiently, making it ideal for applications that require high-speed data transfer.

    It also supports new features in MySQL 5.7 and later versions, such as document store, which allows users to store and retrieve JSON documents directly in the database.

    How to disable MySQL X Protocol?

    To disable MySQL X protocol, edit MySQL configuration file.

    On Ubuntu/Debian

    vi /etc/mysql/mysql.conf.d/mysqld.cnf

    Find, [mysqld] section, and under it, add

    mysqlx=off

    Restart MySQL service

    systemctl restart mysqld

    How to change MySQL X Protocol port?

    To change the MySQL X Protocol port, change the following variables in your MySQL configuration file and restart MySQL server.

    mysqlx-port=33060
    mysqlx-bind-address=127.0.0.1

    MySQL X Protocol related variables

    To see MySQL 8 related variables, use the SQL command

    show variables like 'mysqlx%';

    Example

    mysql> show variables like 'mysqlx%';
    +---------------------------------------------+----------------------------------------+
    | Variable_name                               | Value                                  |
    +---------------------------------------------+----------------------------------------+
    | mysqlx_bind_address                         | 127.0.0.1                              |
    | mysqlx_compression_algorithms               | DEFLATE_STREAM,LZ4_MESSAGE,ZSTD_STREAM |
    | mysqlx_connect_timeout                      | 30                                     |
    | mysqlx_deflate_default_compression_level    | 3                                      |
    | mysqlx_deflate_max_client_compression_level | 5                                      |
    | mysqlx_document_id_unique_prefix            | 0                                      |
    | mysqlx_enable_hello_notice                  | ON                                     |
    | mysqlx_idle_worker_thread_timeout           | 60                                     |
    | mysqlx_interactive_timeout                  | 28800                                  |
    | mysqlx_lz4_default_compression_level        | 2                                      |
    | mysqlx_lz4_max_client_compression_level     | 8                                      |
    | mysqlx_max_allowed_packet                   | 67108864                               |
    | mysqlx_max_connections                      | 100                                    |
    | mysqlx_min_worker_threads                   | 2                                      |
    | mysqlx_port                                 | 33060                                  |
    | mysqlx_port_open_timeout                    | 0                                      |
    | mysqlx_read_timeout                         | 30                                     |
    | mysqlx_socket                               | /var/run/mysqld/mysqlx.sock            |
    | mysqlx_ssl_ca                               |                                        |
    | mysqlx_ssl_capath                           |                                        |
    | mysqlx_ssl_cert                             |                                        |
    | mysqlx_ssl_cipher                           |                                        |
    | mysqlx_ssl_crl                              |                                        |
    | mysqlx_ssl_crlpath                          |                                        |
    | mysqlx_ssl_key                              |                                        |
    | mysqlx_wait_timeout                         | 28800                                  |
    | mysqlx_write_timeout                        | 60                                     |
    | mysqlx_zstd_default_compression_level       | 3                                      |
    | mysqlx_zstd_max_client_compression_level    | 11                                     |
    +---------------------------------------------+----------------------------------------+
    29 rows in set (0.01 sec)
    
    mysql>

    Back to MySQL

  • Limit disk used by MySQL binary log files

    Limit disk used by MySQL binary log files

    You can use following config to keep 7 days bin log

    [mysqld]
    log_bin = /var/log/mysql/mysql-bin.log
    binlog_format = ROW
    expire_logs_days = 7
    max_binlog_size = 100M
    

    After modifying the configuration file, you will need to restart the MySQL server to apply the changes:

    systemctl restart mysql
    

    You can also manually delete old binary log files to free up disk space:

    PURGE BINARY LOGS BEFORE '2024-02-01 00:00:00';
    

    Replace ‘2024-02-01 00:00:00’ with the date before which you want to delete the binary log files.

    On a server, the disk was almost full. On checking most of the disk space is used in “/var/lib/mysql” folder, which had many MySQL binary logs.

    In the MySQL command prompt, run the command to see all MySQL binary logs

    SHOW BINARY LOGS;
    

    On checking MySQL configuration, I had the line

    max_binlog_size   = 100M
    

    This limited the binlog file size to 100 MB, when the file size reached 100 MB, the file get rotated.

    How long logs are kept is determined by the value of binlog_expire_logs_seconds. The default value for this variable was 2592000, which is approx 30 days.

    To make the MySQL bin log expire after 48 hours, add the following to the MySQL configuration file under the [mysqld] directive.

    [mysqld]
    binlog_expire_logs_seconds=86400
    

    Restart the MySQL server with

    systemctl restart mysql
    

    On RHEL based servers

    systemctl restart mysqld
    

    After restarting I checked the disk space used by folder “/var/lib/mysql”, it changed to 13 GB, which is much lower than the initial MySQL disk usage.

    Disable MySQL binlog

    In my.cnf, add

    [mysqld]
    skip-log-bin
    

    Restart the MySQL server

    Back to MySQL

  • How to Install MySQL 5.7 on Ubuntu 22.04

    How to Install MySQL 5.7 on Ubuntu 22.04

    MySQL 5.7 is not supported on Ubuntu 22.04, but you can download MySQL 5.7 for Ubuntu 18.04 and install it on Ubuntu 20.04 or Ubuntu 22.04

    First, install libtinfo5, this is available for download from https://launchpad.net/ubuntu/bionic/amd64/libtinfo5/6.1-1ubuntu1.18.04

    cd ~/
    wget http://launchpadlibrarian.net/371711898/libtinfo5_6.1-1ubuntu1.18.04_amd64.deb
    dpkg -i libtinfo5_6.1-1ubuntu1.18.04_amd64.deb

    The files are moved from the Offical MySQL site

    https://dev.mysql.com/downloads/mysql/5.7.html

    So we will download it from a mirror site. If the mirror is down, you can search with the .deb file name to find another mirror.

    mkdir ~/mysql57
    cd ~/mysql57
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/libmysqlclient-dev_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/libmysqlclient20_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/libmysqld-dev_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-client_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-common_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-community-client_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-community-server_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-community-source_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-community-test_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-server_5.7.38-1ubuntu18.04_amd64.deb
    wget https://mirror.csclub.uwaterloo.ca/mysql/Downloads/MySQL-5.7/mysql-testsuite_5.7.38-1ubuntu18.04_amd64.deb
    dpkg -i *.deb
    apt install -f

    To start MySQL on boot, run

    systemctl enable mysql

    To start MySQL, run

    systemctl start mysql

    Back to MySQL Installation

  • How to create MySQL data directory with mysql_install_db

    How to create MySQL data directory with mysql_install_db

    On a Debian 7 server, the MySQL database got corrupted, which caused the MySQL database to keep crashing. I was able to take a backup of the database following InnoDB Recovery instructions at

    https://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

    Once I have the Database backup taken, I took a backup of the MySQL data directory /var/lib/mysql and removed it.

    To recreate the MySQL data directory, I run the mysql_install_db command, but it failed with the error message “FATAL ERROR: Could not find my-default.cnf”

    root@srv01 ~ # mysql_install_db --defaults-file=/etc/mysql/my.cnf --user=mysql --datadir=/var/lib/mysql
    FATAL ERROR: Could not find my-default.cnf
    
    If you compiled from source, you need to run 'make install' to
    copy the software into the correct location ready for operation.
    
    If you are using a binary release, you must either be at the top
    level of the extracted archive, or pass the --basedir option
    pointing to that location.
    
    root@srv01 ~ # 
    

    It is fixed by copying /etc/mysql/my.cnf to /usr/share/mysql/my-default.cnf

    cp /etc/mysql/my.cnf /usr/share/mysql/my-default.cnf
    

    Back to MySQL

  • How to Repair MyISAM Tables

    How to Repair MyISAM Tables

    MyISAM tables (.MYI and .MYD) can be repaired using myisamchk command.

    To find list of corrupted tables

    myisamchk /var/lib/mysql/DB_NAME/*.MYI >> /root/1.txt
    

    To rapier a table

    cd  /var/lib/mysql/DB_NAME/
    myisamchk -r TABLE_NAME_HERE.MYI
    

    To check and repair all tables

    myisamchk --silent --force --fast --update-state /var/lib/mysql/DB_NAME/*.MYI
    

    Look like newer version of MySQL, you need to specify just table name, no extension, for example

    cd /var/lib/mysql/DB_NAME/
    myisamchk -r TABLE_NAME
    

    Back to MySQL Repair

  • mysqldump errno: 24 Can’t open file  when using LOCK TABLES

    mysqldump errno: 24 Can’t open file when using LOCK TABLES

    When backing up a MySQL database using mysqldump command, I got the following error.

    root@server12:~# mysqldump --opt DB_NAME > DB_NAME.sql
    mysqldump: Got error: 1016: Can't open file: './DB_NAME/TABLE_NAME.frm' (errno: 24) when using LOCK TABLES
    root@server12:~# 
    

    The error is due to open_files_limit

    mysql> show variables like 'open_files_limit';
    +------------------+-------+
    | Variable_name    | Value |
    +------------------+-------+
    | open_files_limit | 2558  |
    +------------------+-------+
    1 row in set (0.00 sec)
    
    mysql> 
    

    You need to edit mysql configuration file and increase value of open_files_limit.

    Another way to fix the error is to use –lock-tables=false option when taking MySQL backup.

    Example

    mysqldump --opt DB_NAME --lock-tables=false > DB_NAME.sql
    

    If you are using systemd, you may need to edit the service file. Identify which service file you are using with command

    systemctl list-unit-files | egrep  'mysql|mariadb'
    

    Now edit the service file with

    systemctl edit mysqld
    

    In the above command replace mysqld with the name of your service. This can be mariadb or mysql depending on your OS. You will get an editor, in the editor, paste the following, save and exit the editor.

    [Service]
    LimitNOFILE=infinity
    LimitMEMLOCK=infinity
    

    Restart MySQL

    systemctl restart mysqld
    

    To change open_file_limit on MariaDB, see MariaDB Change Open Files Limit

    Back to MySQL Backup

  • How to fix InnoDB: ERROR: the age of the last checkpoint

    How to fix InnoDB: ERROR: the age of the last checkpoint

    On a MySQL server, the error log keeps getting the following error message

    221024 17:50:03  InnoDB: ERROR: the age of the last checkpoint is X,
    InnoDB: which exceeds the log group capacity 9433498.
    InnoDB: If you are using big BLOB or TEXT rows, you must set the
    InnoDB: combined size of log files at least 10 times bigger than the
    InnoDB: largest such row.
    

    The error is because you are trying to insert more data into InnoDB and the InnoDB log is getting full before the data can be written into the data file. You need to set the value of innodb_log_file_size value at least ten times bigger than max_allowed_packet.

    To fix the error, edit your MySQL configuration file

    On Debian/Ubuntu with MairaDB

    vi /etc/mysql/mariadb.conf.d/50-server.cnf
    

    Under [mysqld] section, add

    innodb_buffer_pool_size=3G
    innodb_log_file_size=1G
    innodb_log_buffer_size=32M
    

    To select a proper size for these MySQL variables for your system, use MySQL tuner.

    wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl
    perl mysqltuner.pl
    

    It will suggest you recommended values of innodb_buffer_pool_size and innodb_log_file_size.

    Disable fast MySQL shutdown.

    mysql -u root -p -e "SET GLOBAL innodb_fast_shutdown = 0;"
    

    This will flush all changes done on MySQL from log file to data file so we can safely remove MySQL InnoDB log files. We need to remove existing log files before MySQL can create log files with different size.

    Stop MySQL server.

    systemctl stop mysql

    Move log files out

    mkdir ~/mysql-old-log
    mv /var/lib/mysql/ib_logfile* ~/mysql-old-log
    

    Start MySQL service with

    systemctl start mysql
    

    This will recreate MySQL log files (ib_logfile0 and ib_logfile1) with the new size.

    See MySQL