Python Basics: Inspect Comments and Tokens with tokenize

A source-maintenance tool sometimes needs details that a syntax tree leaves out, especially comments and the spelling of literals. tokenize exposes those lexical pieces without importing the inspected module. The examples below inspect or transform tiny, valid Python source strings; none executes the analyzed source.

Prepare a scratch directory

These standalone examples were tested on Linux with Python 3.12.14. Use Python 3.6 or newer for these examples, and tokenize source using a Python version that understands its syntax. Use Bash, a text editor, and the standard mktemp, rm, and rmdir utilities. No third-party packages are required. Run without -O so the assertions remain enabled.

Create a fresh directory and keep using the same terminal:

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

Save each named source below as UTF-8. Avoid filenames that shadow standard-library modules.

1. Find comments without mistaking string content for one

Save as list_source_comments.py:

import io
import tokenize

source = 'label = "# not a comment"\n# review the default\ncount = 3  # retry limit\n'
tokens = list(tokenize.generate_tokens(io.StringIO(source).readline))
comments = [token for token in tokens if token.type == tokenize.COMMENT]
assert [token.string for token in comments] == [
    "# review the default", "# retry limit"
]
for token in comments:
    row, column = token.start
    print(f"line {row}, column {column}: {token.string}")

Run it:

python3 -B list_source_comments.py

Output:

line 2, column 0: # review the default
line 3, column 11: # retry limit

The hash inside the quoted label belongs to a STRING token, not a COMMENT. Each token includes start and end positions; rows start at one and columns at zero. generate_tokens consumes a readline callable returning text. See the tokenize interface.

2. Let the source encoding declaration guide decoding

Save as read_source_encoding_token.py:

import io
import tokenize

source_bytes = b"# coding: latin-1\nlabel = 'caf\xe9'\n"
tokens = list(tokenize.tokenize(io.BytesIO(source_bytes).readline))
encoding = tokens[0]
assert encoding.type == tokenize.ENCODING
strings = [token.string for token in tokens if token.type == tokenize.STRING]
assert strings == ["'café'"]
print("Encoding:", encoding.string)
print("String token:", strings[0])

Run it:

python3 -B read_source_encoding_token.py

Output:

Encoding: iso-8859-1
String token: 'café'

Unlike generate_tokens, tokenize reads bytes and emits an ENCODING token. The fixture deliberately declares Latin-1; blindly decoding it as UTF-8 would fail. This declaration governs the inspected source file, not the encoding of the surrounding script. Python’s encoding-declaration rules restrict where a coding comment can appear.

3. Change a name token while keeping string and comment text

Save as replace_selected_name_token.py:

import ast
import io
import tokenize

source = 'MAX_RETRIES = 3\nlabel = "MAX_RETRIES"\n# MAX_RETRIES is a setting\n'
ast.parse(source)
original = list(tokenize.generate_tokens(io.StringIO(source).readline))
pairs = [
    (token.type, "RETRY_LIMIT")
    if token.type == tokenize.NAME and token.string == "MAX_RETRIES"
    else (token.type, token.string)
    for token in original
]
rebuilt = tokenize.untokenize(pairs)
ast.parse(rebuilt)
after = list(tokenize.generate_tokens(io.StringIO(rebuilt).readline))
assert [(token.type, token.string) for token in after] == pairs
names = [token.string for token in after if token.type == tokenize.NAME]
strings = [token.string for token in after if token.type == tokenize.STRING]
comments = [token.string for token in after if token.type == tokenize.COMMENT]
assert names == ["RETRY_LIMIT", "label"]
assert strings == ['"MAX_RETRIES"']
assert comments == ["# MAX_RETRIES is a setting"]
print("Names:", ", ".join(names))
print("String:", strings[0])
print("Comment:", comments[0])
print("Token round-trip:", True)

Run it:

python3 -B replace_selected_name_token.py

Output:

Names: RETRY_LIMIT, label
String: "MAX_RETRIES"
Comment: # MAX_RETRIES is a setting
Token round-trip: True

Selecting NAME tokens avoids a blind text replacement inside the label and comment. untokenize preserves token types and strings, but spacing can change. The round-trip assertion checks that contract; ast.parse checks syntax, not meaning. See untokenize.

This last example is deliberately not a general symbol-renaming tool: token matching does not understand scopes, imported aliases, or object attributes. Review any generated diff and use syntax-aware refactoring tools when semantics matter. tokenize is documented for syntactically valid input, not as a validator for arbitrary broken code. Limit external input size and do not evaluate it.

Clean up

In the same terminal, remove only the named sources, return to the original directory, and remove the empty scratch directory:

rm -- list_source_comments.py read_source_encoding_token.py replace_selected_name_token.py
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

If rmdir reports remaining files, inspect them instead of deleting unknown contents.

References

• Python 3.12 tokenize
• Python source encoding declarations
• Token constants

Acknowledgements

This post was created with assistance from AI.