Hash Generator: When sha256sum -c Says FAILED but the Tarball Is Fine

A vendor sent us a signed build of their SDK last month. The build contained a single tarball and a checksums.txt file. When I ran the sha256sum -c command on the checksums.txt file, it reported a FAILED checksum, which raised concerns that the tarball may have been corrupted or tampered with. Initially, I thought the issue was due to a corrupted download or a malicious attack. However, after further investigation, it turned out that neither of these scenarios was the cause.

To verify the integrity of the tarball, I ran several different hashing methods on it. First, I used a Python script that properly hashed the tarball in binary mode, reading it in chunks to avoid using excessive memory for large files. This method produced a valid hash that matched the digest from the checksums.txt file, unlike the FAILED result from sha256sum -c. The discrepancy between the two results led me to dig deeper into the issue.

Next, I examined the checksums.txt file itself using a shell command. I noticed that every line in the file ended with a carriage return character (^M) instead of the expected line feed (LF). Since the vendor generated the checksums.txt file on Windows, this encoding was expected. However, GNU sha256sum on Linux treats a filename with the trailing ^M as an invalid filename, causing it to fail and report a "No such file or directory" error.

This caused the -c option to display a FAILED message, even though the hash of the tarball itself was correct. To resolve this issue, I used the tr command to remove the ^M characters from the checksums.txt file, creating a new file called checksums.fixed.txt. Running sha256sum -c on this corrected file resulted in a successful match, confirming that the tarball was indeed fine.

After these tests, I decided to take a more straightforward approach by using an online SHA-256 hash generator. This browser-based tool allowed me to quickly compare the vendor's provided digest with the hash generated locally. It ran entirely on the client-side, ensuring that the tarball never left my machine. This method proved to be the most reliable and straightforward way to verify the integrity of the tarball when working with different operating systems and tools.

In summary, the issue with the sha256sum -c command was not due to a corrupted tarball or an attack, but rather three common pitfalls: using text mode file reads, which can cause UnicodeDecodeError when dealing with non-UTF-8 bytes; encountering carriage return characters (^M) in the checksum manifest when working on Linux; and case-sensitive hex comparisons, where uppercase and lowercase digests may not match.

By using binary file reads, normalizing the manifest encoding, and performing case-insensitive comparisons, I was able to successfully verify the integrity of the tarball.