Breaking up (lines) is hard to do

The task of splitting multi-line text into its constituent lines and returning an array of those lines proves to be more complex than initially assumed. William Woodruff's observation highlighted that Python recognizes up to eleven different Unicode code points or code point sequences as indicating a line break. The discussion begins with ASCII, a widely influential character set in computing.

ASCII introduced control characters like LINE FEED (0x0A) and CARRIAGE RETURN (0x0D) to advance the paper and realign the print head, respectively. However, as virtual teletypes emerged, the two-character approach became less necessary, leading to multiple ways of representing line breaks in various systems. This inconsistency posed challenges for portability of "plain text" across systems.

In response to these issues, Python 2.3 introduced universal newline mode to handle different line ending formats. However, even within ASCII, there were other line-breaking characters. FORM FEED (0x0C), for instance, moved the cursor to the next page, while VERTICAL TAB (0x0B) advanced the cursor to the next line, potentially skipping several lines. The C1 control characters, such as NEXT LINE (0x85), were also introduced for compatibility between ASCII and IBM's EBCDIC character set.

In today's Unicode world, the standardization of line breaks became crucial. Unicode's Newline Guidelines specify seven code points to recognize as line breaks, including the CR LF sequence for systems using it. Two new code points were added specifically for Unicode to address the ambiguity caused by word-processing tools using explicit newline characters for paragraph breaks. These new line-breaking code points were introduced in Unicode version 5.0 and have been standardized in Unicode Technical Report #13.