Linux Basics: Divide Files and Work Lists with GNU split

A large text export or a list of tasks is easier to handle in smaller pieces. GNU split can create line-based batches, fixed-size byte fragments, or one slice of a round-robin stream. The right choice depends on whether you must preserve records, exact bytes, or a worker assignment.

These examples were tested on Linux with GNU Bash 5.2.37 and GNU coreutils 9.7. Use Bash rather than sh. The commands operate only on the small practice inputs shown here.

Prepare a scratch directory

You need GNU split, Bash, a text editor, and the cat, cmp, mktemp, rm, and rmdir utilities. Keep using the same terminal. Save each named source file in this new directory, then run it once in the order shown. Invoking bash explicitly avoids changing executable permissions. No administrator access is needed.

start_dir="$PWD"
work_dir="$(mktemp -d)" || exit 1
cd -- "$work_dir" || exit 1

1. Create batches of complete lines

Save this as split-lines.sh:

#!/usr/bin/env bash
set -eu
export LC_ALL=C
printf '%s\n' alpha beta gamma delta epsilon > records.txt
split -l 2 -d --additional-suffix=.txt records.txt lines-
for piece in lines-00.txt lines-01.txt lines-02.txt; do
  printf '%s:\n' "$piece"
  cat -- "$piece"
done
cat -- lines-00.txt lines-01.txt lines-02.txt > joined-lines.txt
cmp -s records.txt joined-lines.txt
printf 'Reassembled: identical\n'

Run it:

bash split-lines.sh

Output:

lines-00.txt:
alpha
beta
lines-01.txt:
gamma
delta
lines-02.txt:
epsilon
Reassembled: identical

Five newline-terminated records become batches of two, two, and one. Numeric suffixes make the order visible, while the extra .txt suffix identifies the resulting text files. The last operand, lines-, is the output prefix, not a directory.

The explicit cat order reconstructs the original input. cmp -s checks the bytes without printing them; the success message appears only if that check passes. Keep the original until you have verified the pieces and any intended transfer.

2. Create fixed-size byte fragments

Save this as split-bytes.sh:

#!/usr/bin/env bash
set -eu
printf 'ABCDEFGHIJ' > payload.txt
split -b 4 -d payload.txt bytes-
for piece in bytes-00 bytes-01 bytes-02; do
  printf '%s: ' "$piece"
  cat -- "$piece"
  printf '\n'
done
cat -- bytes-00 bytes-01 bytes-02 > joined-bytes.txt
cmp -s payload.txt joined-bytes.txt
printf 'Reassembled: identical\n'

Run it:

bash split-bytes.sh

Output:

bytes-00: ABCD
bytes-01: EFGH
bytes-02: IJ
Reassembled: identical

The ten-byte input becomes four, four, and two bytes. The script adds display newlines after showing each fragment; those newlines are not written into the fragments themselves. The second comparison confirms that reassembly recovers the original ten bytes.

Use this style when a size ceiling matters more than readable records. A byte boundary can cut through a text line or a multibyte character. Do not assume an individual fragment is independently valid text or a complete structured document.

3. Select one worker’s round-robin records

Save this as split-worker-stream.sh:

#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' task-1 task-2 task-3 task-4 task-5 task-6 |
  split -n r/2/3

Run it:

bash split-worker-stream.sh

Output:

task-2
task-5

Here r/2/3 selects the second of three round-robin streams. The records assigned to that stream are the second and fifth input lines. Because a particular stream is selected, split prints it to standard output instead of creating chunk files.

This only selects input; it does not start workers or execute tasks. It also differs from contiguous batching: concatenating all round-robin streams would group assignments by worker rather than restore the original record order.

Keep in mind

Use a fresh output directory or a carefully chosen unused prefix: existing chunk names can be overwritten. After a failed run, do not assume every expected piece exists. The three examples use tiny inputs so you can inspect both the pieces and the reconstruction.

Line-based splitting understands newline boundaries, not application formats. It does not repeat a header in every piece or recognize a quoted field that contains a newline. Byte sizes also have different suffix conventions: GNU split uses powers of 1024 for K and powers of 1000 for KB. The example uses plain bytes to avoid that ambiguity.

Clean up

In the same terminal, remove only the named practice files, return to your original directory, and remove the now-empty scratch directory:

rm -- records.txt lines-00.txt lines-01.txt lines-02.txt joined-lines.txt payload.txt bytes-00 bytes-01 bytes-02 joined-bytes.txt split-lines.sh split-bytes.sh split-worker-stream.sh
cd -- "$start_dir" || exit 1
rmdir -- "$work_dir"
unset start_dir work_dir

References

• GNU coreutils manual: split

Acknowledgements

This post was created with assistance from AI.