Python Basics: Splitting Quoted Text with shlex

A small text command might contain a filename with spaces. Python’s standard-library shlex module helps separate words while respecting shell-style quotes.

Use Python 3.12 or newer in Bash on Linux. Each example runs independently and only processes strings; none executes the commands described by those strings.

Keep quoted words together

Save as demo_shlex_1.py:

import shlex

command_text = 'preview "annual report.txt" --label ""'
arguments = shlex.split(command_text)
print(arguments)
print("Filename:", arguments[1])
print("Empty label:", arguments[-1] == "")

Run it from the directory where you saved it:

python3 demo_shlex_1.py

Output:

['preview', 'annual report.txt', '--label', '']
Filename: annual report.txt
Empty label: True

split() defaults to POSIX-style parsing: surrounding quotes disappear, but the filename stays together. Plain str.split() would break it at the space. A quoted empty string survives as an empty argument.

Choose how to handle comments

Save as demo_shlex_2.py:

import shlex

command_text = 'preview "chapter #1.txt" # review later'
print("Default:", shlex.split(command_text))
print("Comments:", shlex.split(command_text, comments=True))
try:
    shlex.split('preview "unfinished')
except ValueError:
    print("Invalid input: unfinished quote")

Run it from the directory where you saved it:

python3 demo_shlex_2.py

Output:

Default: ['preview', 'chapter #1.txt', '#', 'review', 'later']
Comments: ['preview', 'chapter #1.txt']
Invalid input: unfinished quote

Comments are disabled by default. With comments=True, an unquoted # starts a comment; the quoted one remains part of the filename. Catch ValueError when text may contain unmatched quotes or an unfinished escape.

Turn arguments back into text

Save as demo_shlex_3.py:

import shlex

arguments = ["preview", "annual report.txt", "--label", ""]
command_text = shlex.join(arguments)
print(command_text)
print("Round trip:", shlex.split(command_text) == arguments)
print("Literal tokens:", shlex.split("$HOME *.txt"))

Run it from the directory where you saved it:

python3 demo_shlex_3.py

Output:

preview 'annual report.txt' --label ''
Round trip: True
Literal tokens: ['$HOME', '*.txt']

join() produces POSIX-shell-quoted text. Tokenization does not expand variables or wildcards, execute programs, or validate allowed arguments. It is not universal input sanitization. Its quoting is unsuitable for Windows shells. When launching a program, prefer an argument list with subprocess.run(..., shell=False) and validate the program and its options separately.

References

• Python shlex documentation

Acknowledgements

This post was created with assistance from AI.