tar checksums its headers and never your files
tar is older than most of the people who type it, and the format underneath has barely changed since the tape drives it was named after. I had used it for years without ever looking inside, so I wrote one by hand from the POSIX specification using nothing but Python's struct module, and checked every claim below against GNU tar 1.35 and Python's own tarfile . The format turns out to be simple…
Tar is an ancient archive format that has remained largely unchanged since its inception, even though the computers it was designed for have long since become obsolete. Despite its age, tar remains in use today, often as a simple and straightforward way to store and transfer files.
Writing a tar archive is a straightforward process, requiring only forty lines of code. Each file in the archive is preceded by a header containing information about the file, such as its name, modification time, and a checksum. The header is written in octal format and occupies a fixed number of bytes in the archive. The archive ends with two blocks of zeros.
The checksum in a tar header is a simple sum of the 512-byte header, calculated using the Python struct module. While this checksum protects the header against corruption, it does not safeguard the contents of the files themselves. This asymmetry is intentional, as it allows tape drives to identify the header and data blocks during a read operation, but it does not provide any guarantee about the integrity of the file contents themselves.
When testing the tar format, it was found that modifying a single byte within a file's contents resulted in the extracted file being corrupted, with no warning or error from tar. However, altering a single byte in the file's header caused tar to refuse to extract the archive altogether. This asymmetry highlights the limitations of the tar format and its lack of integrity protection for file contents.
In practice, a .tar archive offers no assurance of data integrity. However, compressing the archive with tools like gzip or xz adds an additional layer of protection, as these compression formats carry their own integrity checks. This means that a .tar.gz archive is more reliable in terms of data integrity compared to an uncompressed .tar archive.
When designing a tar archive, one must be aware of the format's limitations, particularly its inability to handle large files efficiently. The size field in the header can only accommodate eleven octal digits, limiting the total file size to 8 GiB. Furthermore, tar archives lack a table of contents or index, requiring the file system to sequentially read through each header to locate a specific file. This can significantly slow down the extraction process, especially for large archives with many files.
Despite these drawbacks, tar remains a popular choice for archiving and transferring files due to its simplicity and widespread support across various platforms. However, users should be aware of its limitations and consider alternative formats, such as ZIP or tar.gz, when data integrity and efficiency are critical factors.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
