Lesson 9. Files

2023-01-23

Input and Output of Data

Only small amounts of data can be input by typing commands in a shell or a source file. Large amounts of data need to be stored in files. Some programs also require configuration files to provide a large number of settings like font size, window size, background colour etc.

Objectives

The aim of this lesson is to understand how to read from and write to files. At the end of the lesson, you should know how to

create a file object by opening a file for reading or writing;
close a file;
access a file safely via a with-statement;
read lines from a text file;
write lines to a text file;
read an array from a text file;
save binary data to a NumPy .npy or .npz file, and load it again.

Reading from a Text File

The file quote.txt is a text file containing a extract from Progress and Poverty, by Henry George. Download this file to your working folder so that it appears in the ‘Files’ tab in the top right pane of the Spyder IDE. The command

f = open('quote.txt', 'r')

opens the file and returns a file object f used for accessing the file contents. The mode argument 'r' means that the file is opened for reading only.

The readline method reads one line of a text file, so the commands

line1 = f.readline()
line2 = f.readline()
print(line1, line2, end='')

will read the first two lines of the file and print them:

Thus wages and interest do not depend upon the produce of labor and capital, 
 but upon what is left after rent is taken out; or, upon the produce which

The seek method causes text to be read from the offset provided. For example, the commands

f.seek(11)
f.readline()

reads a line of text starting from just after the 11 byte in the file, displaying

'and interest do not depend upon the produce of labor and capital, \n'

(A plain ASCII character takes up 1 byte.) The most common usage is f.seek(0) that effectively rewinds to the start of the file.

An easy way to process all lines in a file is to use a for-loop, with the file object as the iterator:

for line in f:
    <statements(line)>

For example, we could print the first word in each line as follows.

for line in f:
    word_list = line.split()
    if len(word_list) == 0:
        print()
    else:
        print(word_list[0])

Here, line.split() returns the list of words in line, and we have to allow for blank lines which yield an empty list of words.

When finished reading from a file, you must close it with the command

f.close()

There is always a risk that an error of some kind interupts processing so that the file does not get closed properly. The recommended approach is to open a file using a with-statement as follows.

with open('quote.txt', 'r') as f:
    <statements(f)>

In this way, the file is guaranteed to get closed properly.

Two other methods of a file object are read(n), which reads the next n bytes, and readlines(), which reads the whole file and returns a list whose kth member is a string containing the kth line in the file. Thus,

with open('quote.txt', 'r') as f:
    lines = f.readlines()
    print(lines[-1])

prints the last line in the file:

Henry George, Progress and Poverty, Book III, Chapter 3

Writing to a Text File

We can create a new file with the open command by setting the mode argument to 'w' for ‘write’. Try typing the commands

with open('practice.txt', 'w') as f:
    f.write('This is the first line.\n')
    f.write('This is the second line.\n')

You should see a file ‘practice.txt’ appear in your working folder, containing the two lines of text:

This is the first line.
This is the second line.

If lines is a list of strings, each with a newline \n at the end, then f.writelines(lines) writes the list to the file generated by f.

Text Files and Arrays

NumPy provides the loadtxt function to load an array from a text file.
Use the Spyder editor to create the file input_matrix.txt containing

 2.1 0.5 -3.2
 5.4 8.0  9.1
-6.2 7.3  2.1

The command

M = loadtxt('input_matrix.txt')

reads the contents of the file and creates a \(3\times3\) NumPy array M with these entries.

NumPy also provides savetxt, which saves an array to a text file, as in the following example.

A = array([[  pi,  9.5,  -pi],
           [   e, -3.2, -2*e],
           [ 7.3,  3*e,  4.1]])
savetxt('output_matrix.txt', A)

The file output_matrix.txt will then contain

3.141592653589793116e+00 9.500000000000000000e+00 -3.141592653589793116e+00
2.718281828459045091e+00 -3.200000000000000178e+00 -5.436563656918090182e+00
7.299999999999999822e+00 8.154845485377135716e+00 4.099999999999999645e+00

Using NumPy’s Binary File Format

Although loadtxt and savetxt are simple and convenient for dealing with small arrays, they are inefficient for large arrays because each matrix entry has to be converted from a string to a float when loading, and from a float to a string when saving. Also, the text representation of a float takes up to 25 characters and so requires 25 bytes of file space, whereas a float takes up only 8 bytes.

The save command is used to save an array to a binary file with the extension .npy. In the following experiment,

A = randn(100, 100)
save('big_matrix.npy', A)  
savetxt('big_matrix.txt', A)

you should find that the binary file big_matrix.npy is 79kB whereas the text file big_matrix.txt is 249kB, or about three times larger.

The load command is used to read a .npy file and return the array that it holds. After doing

B = load('big_matrix.npy')

you can check that B is equal to the matrix A that was previously saved.

You can also save several arrays to a .npz file using NumPy’s savez function. Doing

A = randn(7, 5)
x = randn(5)
y = randn(7)
savez('matrices.npz', mat1=A, vec1=x, vec2=y)

creates the file ‘matrices.npz’, which is actually just a zip archive.
(Unzipping it creates a folder matrices containing mat1.npy, vec1.npy, vec2.npy.) To load the arrays, use the load command as follows.

data = load('matrices.npz')
mat1 = data['mat1']
vec1 = data['vec1']
vec2 = data['vec2']

You can check that mat1 equals A, vec1 equals x and vec2 equals y. In practice, you would usually retain the original names when using savez. Also, you can load via a with-statement. So after doing

savez('matrices.npz', A=A, x=x, y=y)

you could, in a new IPython session (perhaps at some later date), retrieve the arrays by doing

with load('matrices.npz') as data:
    A = data['A']
    x = data['x']
    y = data['y']

Summary

This lesson introduced techniques for file processing in Python:

reading from a text file;
writing to a text file;
using the NumPy functions loadtxt and savetxt to read and write the text form of an array;
using the NumPy functions load and save to read and write the binary form of an array.