Category: Python

  • Serve a Django application using Gunicorn

    Serve a Django application using Gunicorn

    Create systemd service file

    vi /usr/lib/systemd/system/backend.service
    

    Add

    [Unit]
    Description=Gunicorn instance to serve Django Application
    After=network.target
    
    [Service]
    User=www-data
    Group=www-data
    Environment="APP=app"
    WorkingDirectory=/path/to/app/backend
    Environment="PATH=/path/to/app/backend/venv/bin"
    ExecStart=/path/to/app/backend/venv/bin/gunicorn -w 3 --timeout 120 --bind 0.0.0.0:8080 backend.wsgi
    
    [Install]
    WantedBy=multi-user.target
    

    On the above, you need to replace “/path/to/app/backend” with the path to your application. In backend.wsgi, backend is the application name here, change it to whatever your Django application name is.

    User=www-data
    Group=www-data
    

    You can change this to whatever user you use to upload your Django application files.

    Enable and start the service

    systemctl enable backend.service
    systemctl start backend.service
    

    Now your Django application will be available on port 8080. You can configure nginx or apache to proxy to this port.

    Here is a sample nginx config

    server {
        listen 443 ssl;
        server_name app.serverok.in;
        ssl_certificate /etc/ssl/app.serverok.in.crt;
        ssl_certificate_key /etc/ssl/app.serverok.in.key;
        root /var/www/app.serverok.in/html/;
        index index.html;
        location /media/ {
            alias /var/www/app.serverok.in/html/media/;
        }
    
        location /api/ {
            include proxy_params;
            proxy_pass http://localhost:8080/api/;
        }
    }
    

    Here Django application is served as /api/ folder, you can change this as required.

    Back to Gunicorn

  • ModuleNotFoundError: No module named ‘MySQLdb’

    ModuleNotFoundError: No module named ‘MySQLdb’

    When running migration for a Django application with “python manage.py migrate”, I got the following error message

    ModuleNotFoundError: No module named 'MySQLdb'
    

    To fix install

    pip install mysqlclient
    

    This will require MySQL development package. On the Ubuntu server running MariaDB, I installed packaged libmariadb-dev

    sudo apt install libmariadb-dev
    

    Back to Errors

  • error: invalid command ‘bdist_wheel’

    error: invalid command ‘bdist_wheel’

    When installing a python module on Ubuntu, I got the following error message

    error: invalid command 'bdist_wheel'
    

    The error is fixed by installing

    apt install -y python3-dev libpq-dev python3-wheel gcc
    pip3 install wheel
    

    Back to Errors

  • How to install Python 3.10 on CentOS 7

    How to install Python 3.10 on CentOS 7

    First, install the dependency

    yum install -y wget zlib-devel
    yum groupinstall -y "Development tools"
    

    Install OpenSSL as the openssl-devel package provided by CentOS is old version. Install OpenSSL from source with

    cd /usr/local/src
    wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
    tar xvf openssl-1.1.1g.tar.gz
    cd openssl-1.1.1g
    ./config --prefix=/usr/serverok/openssl --openssldir=/usr/serverok/openssl no-ssl2
    make
    make install
    

    Run following commands

    export PATH=/usr/serverok/openssl/bin:$PATH
    export LD_LIBRARY_PATH=/usr/serverok/openssl/lib
    export LC_ALL="en_US.UTF-8"
    export LDFLAGS="-L/usr/serverok/openssl/lib -Wl,-rpath,/usr/serverok/openssl/lib"
    

    Download the latest version of Python from

    https://www.python.org/downloads/

    To install, run

    cd /usr/local/src
    wget https://www.python.org/ftp/python/3.10.6/Python-3.10.6.tar.xz
    tar xvf Python-3.10.6.tar.xz
    cd /usr/local/src/Python-3.10.6
    make clean && make distclean
    ./configure --enable-optimizations  --with-openssl=/usr/serverok/openssl/
    make altinstall
    

    Python 10 will be installed in folder

    /usr/local/bin/python3.10
    

    To install pip modules, you can use

    python3.10 -m pip install -r requirements.txt 
    

    Back to Python

  • ModuleNotFoundError: No module named ‘PIL’

    ModuleNotFoundError: No module named ‘PIL’

    When running a python application, I got the error message ModuleNotFoundError: No module named ‘PIL’

    (venv) boby@sok-01:~/work/powder (master)$ python main.py 
    Traceback (most recent call last):
      File "main.py", line 6, in 
        from extras import *
      File "/home/boby/work/powder/extras.py", line 2, in 
        from PIL import Image, ImageDraw, ImageFont
    ModuleNotFoundError: No module named 'PIL'
    (venv) boby@sok-01:~/work/powder (master)$
    

    To fix the error install Pillow with

    pip install Pillow
    
  • This version of ChromeDriver only supports Chrome version

    This version of ChromeDriver only supports Chrome version

    When running a python application, I get the error message

    (venv) boby@sok-01:~/work/powder (master)$ python main.py 
    59 seconds remaining
    Process Process-1:
    Traceback (most recent call last):
      File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "main.py", line 51, in main
        driver = webdriver.Chrome(
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
        super().__init__(DesiredCapabilities.CHROME['browserName'], "goog",
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
        super().__init__(
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 277, in __init__
        self.start_session(capabilities, browser_profile)
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 370, in start_session
        response = self.execute(Command.NEW_SESSION, parameters)
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 435, in execute
        self.error_handler.check_response(response)
      File "/home/boby/work/powder/venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 98
    Current browser version is 103.0.5060.53 with binary path /usr/bin/google-chrome
    

    To fix the error, download the chrome driver for your google chrome browser version from

    https://chromedriver.chromium.org/downloads

    Extract the file. You will get a file with the name “chromedriver”. Replace your chromedriver file with this one.

    sudo mv ~/Downloads/chromedriver_linux64/chromedriver /usr/local/bin/chromedriver
    
  • OSError: mysql_config not found

    OSError: mysql_config not found

    When installing the python mysqlclient module with pip, I got the error “OSError: mysql_config not found”.

    (venv) root@sok:~# pip install mysqlclient
    Collecting mysqlclient
    Downloading mysqlclient-2.1.0.tar.gz (87 kB)
    |████████████████████████████████| 87 kB 1.2 MB/s
    ERROR: Command errored out with exit status 1:
    command: /root/venv/bin/python3 -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘”‘”‘/tmp/pip-install-r5qp0a97/mysqlclient/setup.py’”‘”‘; __file__=’”‘”‘/tmp/pip-install-r5qp0a97/mysqlclient/setup.py’”‘”‘;f=getattr(tokenize, ‘”‘”‘open’”‘”‘, open)(__file__);code=f.read().replace(‘”‘”‘\r\n’”‘”‘, ‘”‘”‘\n’”‘”‘);f.close();exec(compile(code, __file__, ‘”‘”‘exec’”‘”‘))’ egg_info –egg-base /tmp/pip-install-r5qp0a97/mysqlclient/pip-egg-info
    cwd: /tmp/pip-install-r5qp0a97/mysqlclient/
    Complete output (15 lines):
    /bin/sh: 1: mysql_config: not found
    /bin/sh: 1: mariadb_config: not found
    /bin/sh: 1: mysql_config: not found
    Traceback (most recent call last):
    File ““, line 1, in
    File “/tmp/pip-install-r5qp0a97/mysqlclient/setup.py”, line 15, in
    metadata, options = get_config()
    File “/tmp/pip-install-r5qp0a97/mysqlclient/setup_posix.py”, line 70, in get_config
    libs = mysql_config(“libs”)
    File “/tmp/pip-install-r5qp0a97/mysqlclient/setup_posix.py”, line 31, in mysql_config
    raise OSError(“{} not found”.format(_mysql_config_path))
    OSError: mysql_config not found
    mysql_config –version
    mariadb_config –version
    mysql_config –libs
    —————————————-
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    (venv) root@sok:~#

    The error is fixed by installing MySQL development library.

    On Ubuntu server with MariaDB, i installed

    apt install libmariadb-dev
    

    For MySQL, you can use

    apt-get install libmysqlclient-dev
    
  • Uvicorn

    Uvicorn

    Uvicorn is a lightning-fast ASGI server used to serve Python ASGI frameworks like FastAPI, Django Channels, Quart, BlackSheep, etc…

    https://www.uvicorn.org

    To install Uvicorn, run

    pip install uvicorn

    To run, you can use

    uvicorn main:app --host 0.0.0.0 --port 8000 --reload

    See Python

  • Flask Python framework

    Flask is a Python framework. You can find more info at the official website.

    https://flask.palletsprojects.com/

    How to start a Flask application?

    To start flask application, run

    python app.py
    

    How to run flask application on port 80

    To run flask application on port 80, use command

    flask run --host 0.0.0.0 --port 80
    

    Or edit app.py, use the following code

    if __name__ == '__main__':
        app.run(host='0.0.0.0', port=80)
    

    Or you can use the following environment variables

    FLASK_RUN_HOST=0.0.0.0
    FLASK_RUN_PORT=80
    

    Port 80 can only be used by the root user, so you need to run the above command as root. Only do this for testing as it is insecure to run an application as the root user.

    Default flask app name

    When you run

    flash run
    

    It looks for app.py file. If you need to change the file name, you can use the environment variable

    FLASK_APP=application.py
    

    See python

  • Python mechanize HTTP Error 403 request disallowed by robots.txt

    Python mechanize HTTP Error 403 request disallowed by robots.txt

    When running following python script i get error “HTTP Error 403 request disallowed by robots.txt”

    from mechanize import Browser
    
    a = ['https://google.com', 'https://serverok.in', 'https://msn.com']
    
    br = Browser() 
    
    for x in range(len(a)):     
        br.open(a[x]) 
        print("Website title: ")
        print(br.title())
        print("\n")
    

    To fix this, find

    br = Browser() 
    

    Add below

    br.set_handle_robots(False)
    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36')]
    

    set_handle_robots is used to disable robots.txt checking. Second line will set a User-agent, so remote won’t won’t block you as a robot.

    See Python

  • error: Python.h: No such file or directory

    error: Python.h: No such file or directory

    When install dlib python module, i get error

    ERROR: Failed building wheel for dlib fatal 
    error: Python.h: No such file or directory
    

    This is because missing python3-dev package. To fix, install

    apt install python3-dev
    

    If you are using python virtualenv, you will need to recreate the virtualenv after installing dev package.

    virtualenv -p /usr/bin/python3 venv
    
  • ERROR: Failed building wheel for mysqlclient

    When install mysqlclient python module, i get error

    (venv) boby@sok-01:~/Downloads/django-deploy$ pip install mysqlclient==1.4.2.post1
    Collecting mysqlclient==1.4.2.post1
      Using cached https://files.pythonhosted.org/packages/f4/f1/3bb6f64ca7a429729413e6556b7ba5976df06019a5245a43d36032f1061e/mysqlclient-1.4.2.post1.tar.gz
    Building wheels for collected packages: mysqlclient
      Building wheel for mysqlclient (setup.py) ... error
      ERROR: Complete output from command /home/boby/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bbfa8tuo/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-bqlba1jk --python-tag cp36:
      ERROR: running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.6
      creating build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/_exceptions.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/compat.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/connections.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/converters.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/cursors.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/release.py -> build/lib.linux-x86_64-3.6/MySQLdb
      copying MySQLdb/times.py -> build/lib.linux-x86_64-3.6/MySQLdb
      creating build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
      running build_ext
      building 'MySQLdb._mysql' extension
      creating build/temp.linux-x86_64-3.6
      creating build/temp.linux-x86_64-3.6/MySQLdb
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,4,2,'post',1) -D__version__=1.4.2.post1 -I/usr/include/mysql -I/usr/include/python3.6m -I/home/boby/venv/include/python3.6m -c MySQLdb/_mysql.c -o build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o
      MySQLdb/_mysql.c:37:10: fatal error: Python.h: No such file or directory
       #include "Python.h"
                ^~~~~~~~~~
      compilation terminated.
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
      ----------------------------------------
      ERROR: Failed building wheel for mysqlclient
      Running setup.py clean for mysqlclient
    Failed to build mysqlclient
    Installing collected packages: mysqlclient
      Running setup.py install for mysqlclient ... error
        ERROR: Complete output from command /home/boby/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bbfa8tuo/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-t_mzj872/install-record.txt --single-version-externally-managed --compile --install-headers /home/boby/venv/include/site/python3.6/mysqlclient:
        ERROR: running install
        running build
        running build_py
        creating build
        creating build/lib.linux-x86_64-3.6
        creating build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/_exceptions.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/compat.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/connections.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/converters.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/cursors.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/release.py -> build/lib.linux-x86_64-3.6/MySQLdb
        copying MySQLdb/times.py -> build/lib.linux-x86_64-3.6/MySQLdb
        creating build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/__init__.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/CLIENT.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/CR.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/ER.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        copying MySQLdb/constants/FLAG.py -> build/lib.linux-x86_64-3.6/MySQLdb/constants
        running build_ext
        building 'MySQLdb._mysql' extension
        creating build/temp.linux-x86_64-3.6
        creating build/temp.linux-x86_64-3.6/MySQLdb
        x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,4,2,'post',1) -D__version__=1.4.2.post1 -I/usr/include/mysql -I/usr/include/python3.6m -I/home/boby/venv/include/python3.6m -c MySQLdb/_mysql.c -o build/temp.linux-x86_64-3.6/MySQLdb/_mysql.o
        MySQLdb/_mysql.c:37:10: fatal error: Python.h: No such file or directory
         #include "Python.h"
                  ^~~~~~~~~~
        compilation terminated.
        error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
        ----------------------------------------
    ERROR: Command "/home/boby/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bbfa8tuo/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-t_mzj872/install-record.txt --single-version-externally-managed --compile --install-headers /home/boby/venv/include/site/python3.6/mysqlclient" failed with error code 1 in /tmp/pip-install-bbfa8tuo/mysqlclient/
    WARNING: You are using pip version 19.1.1, however version 19.2.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    (venv) boby@sok-01:~/Downloads/django-deploy$
    

    Fixed by installing

    sudo apt-get install python3-dev