CosmoSim Queries (astroquery.cosmosim)

This module allows the user to query and download from one of three cosmological simulation projects: the MultiDark project, the BolshoiP project, and the CLUES project. For accessing these databases a CosmoSim object must first be instantiated with valid credentials (no public username/password are implemented). Below are a couple of examples of usage.

Requirements

The following packages are required for the use of this module:

  • requests

  • keyring

  • getpass

  • bs4

Getting started

>>> from astroquery.cosmosim import CosmoSim
>>> CS = CosmoSim()

Next, enter your credentials; caching is enabled, so after the initial successful login no further password is required if desired.

>>> CS.login(username="uname")
uname, enter your CosmoSim password:
Authenticating uname on www.cosmosim.org...
Authentication successful!
>>> # If running from a script (rather than an interactive python session):
>>> # CS.login(username="uname",password="password")

To store the password associated with your username in the keychain:

>>> CS.login(username="uname",store_password=True)
WARNING: No password was found in the keychain for the provided username. [astroquery.cosmosim.core]
uname, enter your CosmoSim password:
Authenticating uname on www.cosmosim.org...
Authentication successful!

Logging out is as simple as:

>>> CS.logout(deletepw=True)
Removed password for uname in the keychain.

The deletepw option will undo the storage of any password in the keychain. Checking whether you are successfully logged in (or who is currently logged in):

>>> CS.check_login_status()
Status: You are logged in as uname.

Below is an example of running an SQL query (BDMV mass function of the MDR1 cosmological simulation at a redshift of z=0):

>>> sql_query = "SELECT 0.25*(0.5+FLOOR(LOG10(mass)/0.25)) AS log_mass, COUNT(*) AS num FROM MDR1.FOF WHERE snapnum=85 GROUP BY FLOOR(LOG10(mass)/0.25) ORDER BY log_mass"
>>> CS.run_sql_query(query_string=sql_query)
Job created: 359748449665484 #jobid; note: is unique to each and
every query

Managing CosmoSim Queries

The cosmosim module provides functionality for checking the completion status of queries, in addition to deleting them from the server. Below are a few examples of functions available to the user for these purposes.

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359748449665484 COMPLETED
>>> CS.delete_job(jobid='359748449665484')
Deleted job: 359748449665484
>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------

The above function ‘check_all_jobs’ also supports the usage of a job’s phase status in order to filter through all available CosmoSim jobs.

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359748449665484 COMPLETED
359748449682647 ABORTED
359748449628375 ERROR
>>> CS.check_all_jobs(phase=['Completed','Aborted'])
     JobID        Phase
--------------- ---------
359748449665484 COMPLETED
359748449682647 ABORTED

Additionally, ‘check_all_jobs’ (and ‘delete_all_jobs’) accepts both phase and/or tablename (via a regular expression) as criteria for deletion of all available CosmoSim jobs. But be careful: Leaving both arguments blank will delete ALL jobs!

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359748449665484 COMPLETED
359748449682647 ABORTED
359748449628375 ERROR
>>> CS.table_dict()
{'359748449665484': '2014-09-07T05:01:40:0458'}
{'359748449682647': 'table2'}
{'359748449628375': 'table3'}
>>> CS.delete_all_jobs(phase=['Aborted','error'],regex='[a-z]*[0-9]*')
Deleted job: 359748449682647 (Table: table2)
Deleted job: 359748449628375 (Table: table3)

Note: Arguments for phase are case insensitive. Now, check to see if the jobs have been deleted:

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359748449665484 COMPLETED

Getting rid of this last job can be done by deleting all jobs with phase COMPLETED, or it can be done simply by providing the ‘delete_job’ function with its unique jobid. Lastly, this could be accomplished by matching its tablename to the following regular expression: ‘[0-9]*-[0-9]*-[0-9]*[A-Z]*[0-9]*:[0-9]*:[0-9]*:[0-9]*’. All jobs created without specifying the tablename argument in ‘run_sql_query’ are automatically assigned one based upon the creation date and time of the job, and is therefore the default tablename format.

Deleting all jobs, regardless of tablename, and job phase:

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359748449665484 ABORTED
359748586913123 COMPLETED

>>> CS.delete_all_jobs()
Deleted job: 359748449665484
Deleted job: 359748586913123
>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------

In addition to the phase and regex arguments for ‘check_all_jobs’, selected jobs can be sorted using two properties:

>>> CS.check_all_jobs(phase=['completed'],regex='[a-z]*[0-9]*',sortby='tablename')
     JobID        Phase   Tablename         Starttime
--------------- --------- --------- -------------------------
361298054830707 COMPLETED    table1 2014-09-21T19:28:48+02:00
361298050841687 COMPLETED    table2 2014-09-21T19:20:23+02:00
>>> CS.check_all_jobs(phase=['completed'],regex='[a-z]*[0-9]*',sortby='starttime')
     JobID        Phase   Tablename         Starttime
--------------- --------- --------- -------------------------
361298050841687 COMPLETED    table2 2014-09-21T19:20:23+02:00
361298054830707 COMPLETED    table1 2014-09-21T19:28:48+02:00

Exploring Database Schema

A database exploration tool is available to help the user navigate the structure of any simulation database in the CosmoSim database.

Note: ‘@’ precedes entries which are dictionaries

>>> CS.explore_db()
Must first specify a database.
        Projects         Project Items                                      Information
------------------------ ------------- --------------------------------------------------------------------------------------
               @ Bolshoi      @ tables
                                   id:                                                                                      2
                          description:                                                                  The Bolshoi Database.
------------------------ ------------- --------------------------------------------------------------------------------------
              @ BolshoiP      @ tables
                                   id:                                                                                    119
                          description:                                                              Bolshoi Planck simulation
------------------------ ------------- --------------------------------------------------------------------------------------
           @ Clues3_LGDM      @ tables
                                   id:                                                                                    134
                          description: CLUES simulation, B64, 186592, WMAP3, Local Group resimulation, 4096, Dark Matter only
------------------------ ------------- --------------------------------------------------------------------------------------
          @ Clues3_LGGas      @ tables
                                   id:                                                                                    124
                          description:          CLUES simulation, B64, 186592, WMAP3, Local Group resimulation, 4096, Gas+SFR
------------------------ ------------- --------------------------------------------------------------------------------------
                  @ MDPL      @ tables
                                   id:                                                                                    114
                          description:                                                            The MDR1-Planck simulation.
------------------------ ------------- --------------------------------------------------------------------------------------
                  @ MDR1      @ tables
                                   id:                                                                                      7
                          description:                                                        The MultiDark Run 1 Simulation.
------------------------ ------------- --------------------------------------------------------------------------------------
@ cosmosim_user_username      @ tables
                                   id:                                                                                 userdb
                          description:                                                                 Your personal database
------------------------ ------------- --------------------------------------------------------------------------------------
>>> CS.explore_db(db='MDPL')
  Projects  Project Items     Tables
----------- ------------- -------------
--> @ MDPL: --> @ tables:         @ FOF
                       id        @ FOF5
              description        @ FOF4
                                 @ FOF3
                                 @ FOF2
                                 @ FOF1
                                 @ BDMW
                            @ Redshifts
                           @ LinkLength
                           @ AvailHalos
                          @ Particles88
>>> CS.explore_db(db='MDPL',table='FOF')
  Projects  Project Items     Tables    Table Items  Table Info Columns
----------- ------------- ------------- ------------ ---------- --------
--> @ MDPL: --> @ tables:    --> @ FOF:          id:        934        y
                       id        @ FOF5    @ columns                   x
              description        @ FOF4 description:                   z
                                 @ FOF3                               ix
                                 @ FOF2                               iz
                                 @ FOF1                               vx
                                 @ BDMW                               vy
                            @ Redshifts                               vz
                           @ LinkLength                               iy
                           @ AvailHalos                               np
                          @ Particles88                             disp
                                                                    size
                                                                    spin
                                                                    mass
                                                                   axis1
                                                                   axis2
                                                                   axis3
                                                                   fofId
                                                                   phkey
                                                                   delta
                                                                   level
                                                                  angMom
                                                                  disp_v
                                                                 axis1_z
                                                                 axis1_x
                                                                 axis1_y
                                                                 axis3_x
                                                                 axis3_y
                                                                 axis3_z
                                                                 axis2_y
                                                                 axis2_x
                                                                 NInFile
                                                                 axis2_z
                                                                 snapnum
                                                                angMom_x
                                                                angMom_y
                                                                angMom_z
>>> CS.explore_db(db='MDPL',table='FOF',col='fofId')
  Projects  Project Items     Tables     Table Items     Columns
----------- ------------- ------------- -------------- ------------
--> @ MDPL: --> @ tables:    --> @ FOF: --> @ columns: --> @ fofId:
                       id        @ FOF5             id       @ disp
              description        @ FOF4    description    @ axis1_z
                                 @ FOF3                   @ axis1_x
                                 @ FOF2                   @ axis1_y
                                 @ FOF1                        @ ix
                                 @ BDMW                        @ iz
                            @ Redshifts                   @ axis3_x
                           @ LinkLength                   @ axis3_y
                           @ AvailHalos                   @ axis3_z
                          @ Particles88                        @ vx
                                                               @ vy
                                                               @ vz
                                                          @ axis2_y
                                                          @ axis2_x
                                                             @ size
                                                            @ axis1
                                                            @ axis2
                                                            @ axis3
                                                               @ iy
                                                           @ angMom
                                                          @ NInFile
                                                               @ np
                                                          @ axis2_z
                                                           @ disp_v
                                                            @ phkey
                                                            @ delta
                                                          @ snapnum
                                                             @ spin
                                                            @ level
                                                         @ angMom_x
                                                         @ angMom_y
                                                         @ angMom_z
                                                             @ mass
                                                                @ y
                                                                @ x
                                                                @ z

Downloading data

Query results can be downloaded and used in real-time from the command line, or alternatively they can be stored on your local machine.

>>> CS.check_all_jobs()
     JobID        Phase
--------------- ---------
359750704009965 COMPLETED

>>> data = CS.download(jobid='359750704009965',format='csv')
>>> print(data)
(['row_id', 'log_mass', 'num'],
 [[1, 10.88, 3683],
  [2, 11.12, 452606],
  [3, 11.38, 3024674],
  [4, 11.62, 3828931],
  [5, 11.88, 2638644],
  [6, 12.12, 1572685],
  [7, 12.38, 926764],
  [8, 12.62, 544650],
  [9, 12.88, 312360],
  [10, 13.12, 174164],
  [11, 13.38, 95263],
  [12, 13.62, 50473],
  [13, 13.88, 25157],
  [14, 14.12, 11623],
  [15, 14.38, 4769],
  [16, 14.62, 1672],
  [17, 14.88, 458],
  [18, 15.12, 68],
  [19, 15.38, 4]])

Unless the filename attribute is specified, data is not saved out to file.

>>> data = CS.download(jobid='359750704009965',filename='/Users/uname/Desktop/test.csv',format='csv')
|==========================================================================================================================| 1.5k/1.5k (100.00%)         0s

Other formats include votable, votableb1, and votableb2 (the latter two are binary files, for easier handling of large data sets; these formats can not be used in an interactive python session).

Data can be stored and/or written out as a VOTable.

>>> data = CS.download(jobid='359750704009965',format='votable')
>>> data
<astropy.io.votable.tree.VOTableFile at 0x10b440150>
>>> data = CS.download(jobid='359750704009965',filename='/Users/uname/Desktop/test.xml',format='votable')
>>> |==========================================================================================================================| 4.9k/4.9k (100.00%)         0s

Reference/API

astroquery.cosmosim Package

CosmoSim Database Query Tool

Revision History

Access to all cosmological simulations stored in the CosmoSim database, via the uws service.

https://www.cosmosim.org/uws/query

Author

Austen M. Groener <Austen.M.Groener@drexel.edu>

Classes

CosmoSimClass()

Conf()

Configuration parameters for astroquery.cosmosim.