import requests
response = requests.get('http://xkcd.com/353/')
response holds the response now. You can access the content as text via the text-property:
print(response.text[:1000]) # only print the first 1000 characters
You can either just use this information directly, or in some cases you might want to write it to a file. Let's download one of the full resolution files for the Ice coverage data from Problem Set 9:
r2 = requests.get('http://mpia.de/~robitaille/share/ice_data/20060313.npy')
r2.text[:200]
However, this doesn't seem to be actual text. Instead, its a binary format. The binary data of the response can be accessed via
r2.content[:200]
Note the little b at the beginning indicating a binary byte-string.
Now we can open a new (binary) file and download the data to the file.
f = open('20060313.npy', 'wb')
f.write(r2.content)
f.close()
Let's now load and plot the data:
import numpy as np
data = np.load('20060313.npy')
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(12,12))
plt.imshow(data, origin='lower')
APIs¶
Imagine that you want to access some data online. In some cases, you will need to download a web page and search through the HTML to extract what you want. For example:
r = requests.get('http://www.wetteronline.de/wetter/heidelberg')
r.text[:1000]
This is not ideal because it is messy, and also slow if all you want are a couple of values. A number of websites now offer an "Application programming interface" (or API) which is basically a way of accessing data is a machine-readable way. Let's take a look at http://openweathermap.org/ for example, which has an API: http://openweathermap.org/API.
Unfortunately access to this API is no longer possible without registration (it was when this course was designed). But it is not difficult to register. Just sign up at http://openweathermap.org/appid and you will get an account where you will be given a key, which is a string like ab435cdf743df24543d322ac1445dc3a or so.
To access the weather for Heidelberg, you can do:
key = 'ab435cdf743df24543d322ac1445dc3a' # BUT PLEASE REPLACE THIS FAKE KEY WITH YOUR REAL KEY!
r = requests.get('http://api.openweathermap.org/data/2.5/weather?q=Heidelberg,Germany&APPID='+key)
r.text
This is much shorter, but still not ideal for reading into Python as-is. The format above is called JSON, and Python includes a library to easily read in this data:
import json
data = json.loads(r.text)
data
You should now be able to do:
data[u'main'][u'temp']
It looks like the temperature is in K!
Another API example: Astronomy¶
APIs are everywhere... For instance, there is an astronomical database service called "virtual observatory" which allows you to get machine-readable data of numerous astronomical objects. At the Astronomische Rechen-Institut here at Heidelberg University a team of developers continuously improve this service. One of the teachers of this course, Dr. Markus Demleitner, is one of these developers. He provided the following example of obtaining information about stars that are known to have exoplanets around them.
r = requests.get('http://heasarc.gsfc.nasa.gov/cgi-bin/'
'vo/cone/coneGet.pl?table=exoplanets&',
params={"RA": 0, "DEC": 90, "SR": 30})
from io import BytesIO
from astropy.table import Table
t = Table.read(BytesIO(r.content), format="votable")
The object t now contains the coordinates (RA = "right ascension", DEC = "declination") of those stars within a search radius SR away from the north pole (RA=0, DEC=90), as well other information. You can find out which information is there:
print(t.columns)
and you can print out this information:
t.columns["semi_major_axis"]
You can now make a plot of the positions of these stars on the sky, with symbol size representing the exoplanet's semi-major axis (i.e. distance from its host star):
plt.scatter(t.columns["ra"], t.columns["dec"],
s=2+4*np.log(t.columns["semi_major_axis"]))
Exercise¶
You can find over 2000 tiles of the Arctic ice coverage data using the URL with the format:
http://mpia.de/~robitaille/share/ice_data/YYYYMMDD.npy
Write a Python function that takes three arguments - the year, month, and day, as integers, and returns a Numpy array. If the map does not exist, try and return None instead of having an error:
# your solution here
Try using the function to make a plot, as shown above:
# your solution here