I can suggest https://pypi.org/project/reppy/ for robots.txt parsing, it's way better than built in tool.
Very good intension, good luck with sharing it.
By Signing In \ Signing Up, you agree to our privacy policy
We use cookies to offer you a better browsing experience, analyze site traffic, personalize content. Read how we use cookies. By using this site you consent to our use of cookies.
Responsible Web Scraping: Gathering Data Ethically and Legally
I can suggest https://pypi.org/project/reppy/ for robots.txt parsing, it's way better than built in tool. Very good intension, good luck with sharing it.