python - Line reading chokes on 0x1A

Question

I have the following file:

abcde
kwakwa
<0x1A>
line3
linllll

Where <0x1A> represents a byte with the hex value of 0x1A. When attempting to read this file in Python as:

for line in open('t.txt'):
    print line,

It only reads the first two lines, and exits the loop.

The solution seems to be to open the file in binary (or universal newline mode) - 'rb' or 'rU'. Can you explain this behavior ?

How do you know that the byte represented <0x1A>. For me it just says 'SUB' in notepadd ++ — Programmer
Another work-around is to use Python 3 or io.open() in Python 2; the io file objects always use the file in binary mode as far as the OS Is concerned and so Windows won't 'end' the file prematurely. — Martijn Pieters♦

Ned Batchelder Ned Batchelder · Accepted Answer · 2009-01-01T15:31:38

0x1A is Ctrl-Z, and DOS historically used that as an end-of-file marker. For example, try using a command prompt, and "type"ing your file. It will only display the content up the Ctrl-Z.

Python uses the Windows CRT function _wfopen, which implements the "Ctrl-Z is EOF" semantics.

python - Line reading chokes on 0x1A

2 Answers