As the big observatories of the world observe ever more astronomical objects, their archives become powerful research tools. Finding out whether an object has been observed with a certain instrument is just a few mouse clicks away, if the observatory has a public archive like ESO provides for all VLT instruments.

However, if you would like to perform more complex searches, the web interface may not be enough, although it has become very comfortable to use, especially with Astropython’s astroquery package.

For a research project, I recently needed to find all local AGNs ever observed with a certain instrument (SINFONI at the VLT). Since I didn’t know the target names or programmes, I got all unique observed coordinates, resolved them via Simbad (which also gives the class of an object) and then selected the AGNs among all the targets.

Since ESO unfortunately does not provide direct access to the archive database, a query like “give me all unique observed coordinates” is not possible per se. So I had to download all headers, parse the relevant information and build my own database (SQLite for the moment).

I have a script to collect the metadata, which does this:

• query the ESO archive for all observations of a day
• then parse the resulting html file for the unique identifiers of each dataset (“data product ID” or DPID, e.g. XSHOO.2015-04-13T04:46:11.730)
• download the header for the given DPID
• parse the header for relevant information and construct an SQL insert statement
• insert all into a database

There are also scripts that

• query the database for all programmes and search metadata (PI/CoI names, titles) for them
• get atmospheric data for all observations (querying the ambient conditions server)
• And there is a top-level script that calls all of these scripts in a meaningful way and that I call about once a month or when needed to update the database.

My database consists of one table for each ESO instrument that I am interested in (currently MIDI, SINFONI and X-SHOOTER), a table with programme meta data (PI/CoI names and titles), a table with atmospheric data as well as tables with basic information about calibrators and science objects that I use for matching up observations and building LaTeX tables in a scripted way. This has become quite handy over the recent years and has helped me in building the largest sample of interferometrically observed AGNs with MIDI (Burtscher et al. 2013) as well as the largest sample of local AGNs observed with SINFONI (Burtscher et al. 2015) and a follow-up paper (submitted).

In case you are interested in tables that I have already compiled and am maintaining, please contact me and I will be happy to share the database with you. It is currently about 700 MiB and I update it every month.

Apart from nice science, one can also use this database to create other plots of interest, like a map of the exposure depth of SINFONI for example:

Interestingly, the Galactic Center (at 17:45h, -29 deg) is not the field with the deepest SINFONI integration time (“just” about 400 hours). Instead the Extended Chandra Deep Field South is the deepest SINFONI field with about 600 hours of integration time. Another field with deep coverage is the COSMOS south field (10:00h, +02 deg). About 300 hours of total integration time have been spent on this field.

Update 12 Jan 2016: I have now put my codes and the database online. Please see the github project page for further details on how to use these.