We have already talked about built-in Python types, but there are more types that we did not speak about. One of these is the file() object which can be used to read or write files.
Let's try and get the contents of the file into IPython. We start off by creating a file object:
f = open('data/data.txt', 'r')
The open function is taking the data/data.txt file, opening it, and returning an object (which we call f) that can then be used to access the data.
Note that f is not the data in the file, it is what is called a file handle, which points to the file:
type(f)
Now, simply type:
f.read()
The read() function basically just read the whole file and put the contents inside a string.
Let's try this again:
f.read()
What's happened? We read the file, and the file 'pointer' is now sitting at the end of the file, and there is nothing left to read.
To close the file handle, you can do:
f.close()
Let's now try and do something more useful, and capture the contents of the file in a string:
f = open('data/data.txt', 'r')
data = f.read()
f.close()
Now data should contain a string with the contents of the file:
data
But what we'd really like to do is read the file line by line. There are several ways to do this, the simplest of which is to use a for loop in the following way:
f = open('data/data.txt', 'r')
for line in f:
print(repr(line))
f.close()
Note that we are using repr() to show any invisible characters (this will be useful in a minute). Also note that we are now looping over a file rather than a list, and this automatically reads in the next line at each iteration. Each line is being returned as a string. Notice the \n at the end of each line - this is a line return character, which indicates the end of a line.
Now we're reading in a file line by line, what would be nice would be to get some values out of it. Let's examine the last line in detail. If we just type line we should see the last line that was printed in the loop:
line
We can first get rid of the \n character with:
line = line.strip()
line
Next, we can use what we learned about strings and lists to do:
columns = line.split()
columns
Finally, let's say we care about the object name (the 2MASS column), and the J band magnitude (the Jmag) column:
name = columns[2]
jmag = columns[3]
name
jmag
Note that jmag is a string, but if we want a floating point number, we can instead do:
jmag = float(columns[3])
jmag
One last piece of information we need about files is how we can read a single line. This is done using:
line = f.readline()We can put all this together to write a little script to read the data from the file and display the columns we care about to the screen! Here is is:
f.close()
# Open file
f = open('data/data.txt', 'r')
# Read and ignore header lines
header1 = f.readline()
header2 = f.readline()
header3 = f.readline()
# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
name = columns[2]
jmag = float(columns[3])
print(name, jmag)
Python also provides a way to read each line in a text file as a separate string (returns a list). In the following example, we read the lines in the same command line, immediately after opening the file:
f.close()
# Open the file and read each line separately
mydata = open('data/data.txt','r').readlines()
print (mydata)
print(mydata[4])
Here is a copy of the above code to read in the file. Modify this code so as to create a dictionary which gives Jmag for a given 2MASS name, i.e.
>>> jmag['00424455+4116103']
10.773
Then loop over the items in the dictionary and print out for each the source name and the Jmag value.
# EDIT THE CODE BELOW
# Open file
f = open('data/data.txt', 'r')
# Read and ignore header lines
header1 = f.readline()
header2 = f.readline()
header3 = f.readline()
d = {}
# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
name = columns[2]
jmag = float(columns[3])
d[name] = jmag
f.close()
for key in sorted(d):
print (key, d[key])
Bonus: can you figure out a way to make sure that you loop over the source names in alphabetical order?
To open a file for writing, use:
f = open('data_new.txt', 'w')
Then simply use f.write() to write any content to the file, for example:
f.write("Hello, World!\n")
If you want to write multiple lines, you can either give a list of strings to the writelines() method:
f.writelines(['roof\n', 'tile\n', 'roof\n'])
or you can write them as a single string:
f.write('roof\ntile\nroof\n')
Once you have finished writing data to a file, you need to close it:
f.close()
(this also applies to reading files)
As we have seen above, files must not just be opened but should be properly closed afterwards to make sure they are actually written before using them somewhere else. Sometimes writes to files get cached by Python to minimize actual writing to disk, which is comparably slow. Closing a file ensures that these changes are actually written.
To avoid forgetting to close a file there is the with-statement.
with open('data/data_new.txt', 'w') as f:
f.write('spam\n')
This opens the specified file and holds the file-object within f, as well as closing the file when the with-codeblock ends. Afterwards, the file is properly closed and not available anymore.
What's behind the with is called a “context manager”. This is a far more general concept, which tends to be useful whenever you need to maintain “external invariants” – which is jargon for “cleaning up after yourself”. For instance, you can use context managers to reliably remove temporary files, log out from services that have some session management, stop background jobs started, and even reset things within your program. If you come back here later, you'll understand the next piece of language: Check out the contextlib module.
Work with the file data/autofahrt.txt
Continuing from the example in the 'Reading files' section, read in columns one (=time $t$) and one of columns two to four (=acceleration $\ddot x(t)$, $\ddot y(t)$, $\ddot z(t)$). You choose which one.
Then, write out the two columns to a new file.
# EDIT THE CODE BELOW
# Open file
f = open('data/autofahrt.txt', 'r')
# Read and ignore header lines
header1 = f.readline()
header2 = f.readline()
# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
time = columns[0]
a = float(columns[1])
The above shows you how you can read and write any data file. Of course, there are also functions that exist to help you read in data in certain formats (for example numpy contains a function numpy.loadtxt to read in arrays from files) but the key is that with the above, you can read in any file.