Stop Re-uploading Unchanged Hugo Files to S3 with a SHA-256 Manifest
A clean Hugo build can produce unchanged files with new modification times. S3 sync may upload them again because it compares size and timestamps rather than file contents.
A SHA-256 manifest lets deployment recognize unchanged output and restore its previous modification time before syncing. The procedure below describes how to add that comparison to a Hugo deployment.
Important
A changed hash does not itself guarantee an upload. For same-size files and
metadata-only changes, the local mtime must be newer than S3’s LastModified.
Same-second deployments and clock skew can break this assumption. Upload
changed keys explicitly if you need stronger guarantees.
1. Build a complete site
Build Hugo into a clean output directory. Compare the generated files rather than Markdown sources: one post edit can also change archives, feeds, and search indexes.
Check that expected pages and assets exist before allowing any deletion. An incomplete or empty build must not become the desired state of your bucket.
2. Load the previous deployment state
Keep a manifest in the destination bucket, separate from site content. For each generated file, record its relative path, SHA-256, size, and chosen mtime. Also record a fingerprint of upload settings that affect object metadata.
Download the last successful manifest before preparing the next deployment. A missing manifest is normal on the first run. Permission, network, or invalid state errors should stop the process rather than silently start from scratch.
Use a separate manifest for each deployment destination.
3. Compare content and upload settings
Hash every generated file and compare it with the previous manifest:
- Same hash and upload settings: restore the previous mtime
- New or changed content: assign a fresh mtime
- A path present only in the previous manifest: mark it for deletion
- Changed upload settings: refresh every affected file
Include settings such as Cache-Control and content encoding in the fingerprint.
Their values can change while the file bytes stay identical.
Save the new manifest locally, but do not replace the stored manifest yet.
Write a summary, for example to .deploy/summary.json, with summary.changed
set when files changed, appeared, or were deleted. This will drive the primary
CloudFront invalidation decision.
4. Preview, then sync
Preview the transfer before applying it. Confirm the destination, uploads, and deletions before running the real sync.
Exclude the manifest key from content sync so it cannot be uploaded or deleted with ordinary site files. Only enable deletion for a bucket or prefix managed entirely by this deployment.
Proceed only when sync succeeds. A dry run does not count as a deployment.
5. Invalidate cached content
After a successful sync, request a CloudFront invalidation when
summary.changed is true. Also invalidate if sync reports an unexpected remote
correction, such as restoring an object that was missing from S3.
A wildcard /* invalidation is a simple starting point. Request acceptance does
not mean every edge is already fresh; wait for completion when your release
checks require it.
6. Save the new manifest last
Upload the new manifest only after sync and any required invalidation request
succeed. Use a JSON content type and Cache-Control: no-store.
Keeping the previous manifest after failure preserves the local change signal for a retry. Never advance it after a preview or failed deployment.
Operating limits
- Use a single deployment writer. The manifest compares local builds, not current S3 bytes. Manual same-size overwrites may escape detection. Restrict access to the manifest and managed content.
- Treat recovery as a separate check. Sync is not atomic. A partial deployment followed by rollback can leave remote bytes that no longer match the manifest. Verify the destination and use an explicit rewrite when needed.
- Do not rely on every retry being automatic. If a remote-only repair succeeds but invalidation fails, the next run may have no change signal. Retry that invalidation explicitly.
These failure cases follow from the comparison rules; they have not been tested against S3 for this article.
Verify the result
Using disposable test content, check that:
- An unchanged rebuild skips site-content uploads and invalidation
- A same-size edit reaches S3, including closely spaced deployments
- Removed files disappear and metadata-only changes update object headers
- A failed invalidation leaves the previous manifest, and recovery refreshes the cache
The manifest itself is still uploaded on successful runs. This approach also still hashes every local file and lists S3 objects. Measure the result in your own environment; no upload savings or timing claims are made here.
References
Acknowledgements
This post was created with assistance from AI.