Python Basics: Check Simple Crawl Rules with urllib.robotparser

Before adding a crawler scheduler, test how your chosen parser interprets a small rule set. This fixture makes no network requests and uses simple disallowed prefixes without wildcard or competing allow rules.

Prepare a scratch directory

Tested on Linux with CPython 3.12.14. You need Bash, Python, a text editor, and the standard mktemp, rm, and rmdir utilities. No external packages are required. Run without -O so assertions remain enabled.

Keep the same terminal open. Stop if setup fails, before saving the source:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Run the example

Save this as robotparser-demo.py:

from urllib.robotparser import RobotFileParser

rules = [
    "User-agent: *",
    "Disallow: /private/",
    "Crawl-delay: 3",
    "Request-rate: 2/10",
    "Sitemap: https://example.com/sitemap.xml",
]
parser = RobotFileParser("https://example.com/robots.txt")
parser.parse(rules)
agent = "InventoryBot"
for path in ["/docs/", "/private/notes", "/private"]:
    allowed = parser.can_fetch(agent, "https://example.com" + path)
    print(f"{path}: {allowed}")

print("Delay seconds:", parser.crawl_delay(agent))
rate = parser.request_rate(agent)
assert rate is not None
print(f"Request rate: {rate.requests} per {rate.seconds} seconds")
print("Sitemaps:", parser.site_maps())

Run it:

python3 -B robotparser-demo.py

Observed output:

/docs/: True
/private/notes: False
/private: True
Delay seconds: 3
Request rate: 2 per 10 seconds
Sitemaps: ['https://example.com/sitemap.xml']

Read the result

The disallowed prefix includes a trailing slash. It matches /private/notes, but not the distinct /private path. The parser returns scheduling metadata; it does not delay requests or enforce a rate budget. Setting the robots URL also does not fetch it: the example supplies lines directly with parse().

Apply it carefully

Robots rules are advisory crawl preferences, not access control or authorization. A real crawler needs explicit policy for fetching, caching, redirects, unavailable robots files, and per-origin scheduling. Do not assume this parser implements every modern crawler matching convention; test wildcard, precedence, and user-agent cases against your requirements before using them.

The official urllib.robotparser reference documents the APIs used here.

Clean up

Remove the saved source, return to the starting directory, and remove the empty practice directory:

rm -- robotparser-demo.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If the directory is not empty, inspect its remaining contents before removing anything else.

Acknowledgements

This post was created with assistance from AI.