Python mechanize HTTP Error 403 request disallowed by robots.txt
When running following python script i get error “HTTP Error 403 request disallowed by robots.txt”
from mechanize import Browser a = ['https://google.com', 'https://serverok.in', 'https://msn.com'] br = Browser() for x in range(len(a)): br.open(a[x]) print("Website title: ") print(br.title()) print("\n")
To fix this, find
br = Browser()
Add below
br.set_handle_robots(False) br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36')]
set_handle_robots is used to disable robots.txt checking. Second line will set a User-agent, so remote won’t won’t block you as a robot.
See Python