Python Basics: Building and Reading URLs with urllib.parse

Python’s standard-library urllib.parse module separates URL components and encodes values containing spaces or special characters.

Use Python 3.12 or newer in Bash on Linux. These independent examples only manipulate strings; they make no network requests.

Build a query string

Save as demo_urllib_parse_1.py:

from urllib.parse import urlencode

params = {"q": "tea & cake", "tag": ["python", "beginner"]}
query = urlencode(params, doseq=True)
print(query)
print("https://example.com/search?" + query)

Run it from the directory where you saved it:

python3 demo_urllib_parse_1.py

Output:

q=tea+%26+cake&tag=python&tag=beginner
https://example.com/search?q=tea+%26+cake&tag=python&tag=beginner

urlencode() encodes the ampersand inside the value, so it does not introduce another parameter. Spaces become +. With doseq=True, each list item becomes a separate pair with the same key.

Read components and repeated parameters

Save as demo_urllib_parse_2.py:

from urllib.parse import parse_qs, urlsplit

url = "https://example.com/search?tag=python&tag=beginner&q=#top"
parts = urlsplit(url)
params = parse_qs(parts.query, keep_blank_values=True)
print("Host:", parts.hostname)
print("Path:", parts.path)
print("Tags:", params["tag"])
print("Query:", params["q"])
print("Fragment:", parts.fragment)

Run it from the directory where you saved it:

python3 demo_urllib_parse_2.py

Output:

Host: example.com
Path: /search
Tags: ['python', 'beginner']
Query: ['']
Fragment: top

parse_qs() returns lists, even for a single value. Blank values are dropped unless you request them. urlsplit() does not validate whether a URL is acceptable for your application.

Encode a path value and understand joining

Save as demo_urllib_parse_3.py:

from urllib.parse import quote, urljoin

name = "notes & tips"
print("Path:", "/posts/" + quote(name, safe=""))
base = "https://example.com/docs/"
print("Relative:", urljoin(base, "intro"))
print("Absolute:", urljoin(base, "https://other.example/help"))

Run it from the directory where you saved it:

python3 demo_urllib_parse_3.py

Output:

Path: /posts/notes%20%26%20tips
Relative: https://example.com/docs/intro
Absolute: https://other.example/help

Encode individual values, rather than the entire URL. safe="" also encodes slashes in a value. Encoding does not make a destination trustworthy. An absolute second argument to urljoin() can replace the host, so a trusted base cannot restrict untrusted input. Validate components against your application’s rules before using a URL.

References

• Python urllib.parse documentation
• URL parsing security

Acknowledgements

This post was created with assistance from AI.