This note gives a high-level overview of the EARTHSEQUENCING effort to provide a self-consistent database of ocean drilling data and drill-core meta data.
Rationale
As part of the EARTHSEQUENCING project, we are developing a set of databases that facilitate the processing, analysis and linking of various ocean drilling related data and information.
The main aim is to make it possible to quickly and consistently obtain data that are otherwise slow to access, and scattered across various existing databases.
The databases contain
- Information about DSDP/ODP/IODP Expeditions, Sites, Holes, Cores, Sections and related meta-data
- Actual selected, and mainly multi-sensor track related, data aggregated by Hole and Analysis (partly accessible as data grid on https://paloz.marum.de or via a dedicated viewer DBViewer).
- Selected processed digital image data, allowing fast generation of Site overviews, including core-correlation and splice visualisation (IODPCoreImage).
- Information about some publicly available core-offset and splice tables (Core Affine Table, Splice Interval Table).
Eventually, further data, for example biostratigraphic occurrences, magnetostratigraphy, age models etc. can be linked into this structure.Here we give an overview about the Scientific Ocean Drilling core information part of the database, not the specific scientific measurements themselves (for that, see for example database overview).
Most data come from archived DSDP data, the old ODP Janus database, and the current IODP LIMS database, merged into a consistent format.
Overall Technical Infrastructure
The database system is not directly accessible externally due to security and "denial of service" considerations, but hooks are provided for human and machine readable access via web endpoints.
The databases are contained in a relational and filestore database system (Couchbase, Minio and several others). Public access is via an intermediary REST endpoint (Loopback).
Currently (December 2019), the database contains information about all DSDP/ODP/IODP expeditions conducted on the Glomar Challenger or Joides Resolution upto Expedition 376 (post-moratorium). Eventually MSP and Chikyu data might be added.
In detail, the database entities consist of
- (Drilling) Platform, e.g. Glomar Challenger, Joides Resolution, Chikyu, MSP
- Project (e.g. DSDP, ODP, IODP etc.)
- Expedition
- Site
- Hole
- Hole Occupation (e.g. Holes can be re-occupied)
- Core
- Core Section
- Section Half
- GeoLocation (to store co-ordinates of Holes)
- TimeStamp
- Person (e.g. CoChief Scientist, Expedition Project Manager)
- CoreAffineTable
- CoreAffineValue
- SpliceIntervalTable
- SpliceIntervalEntry
For the actual data storage, we use the two entities
- AnalysisMapping (providing meta-information of data formats and import locations)
- AnalysisHeader (per Hole, per Analysis compressed data and meta-data)
Data access
There are various ways to access these data, but as mentioned before, direct access is not available outside the private network for security reasons.
- For an overview of measurement data available, consult https:/paloz.marum.de. This is suitable for human download of text tables for individual holes and measurements.
- Various REST endpoints are available for machine-readable access. Please contact H. Pälike for further information, also to ensure overloading the current server. See below for a few examples of how this could be accessed:
- A current snapshot of the database (for OFFLINE viewing) is available as a macOS Application (see XXX).
- A number of pre-built applications allow access to these data also, for example DBViewer, IODPCoreImage.
Programmatic access is available via REST endpoints, for example
- To get a list of all Expeditions, only returning the database ID and Expedition name, ordered by expedition number:
https://paloz.marum.de/database/api/Expeditions?filter={"fields":{"id":true,"expeditionName":true},"order":"expeditionNumber ASC"} - To then get a specific Expedition by ID:
https://paloz.marum.de/database/api/Expeditions/29D5ACB9-4A40-499F-A20E-B3978C6B1B3B-74499-0002C851BF3992F4 - or to get a list of all Site data for a given Expedition:
https://paloz.marum.de/database/api/Expeditions/29D5ACB9-4A40-499F-A20E-B3978C6B1B3B-74499-0002C851BF3992F4/Sites etc. ..
A full list can be seen here:
Offline Snapshot Viewer
A software for macOS only (sorry!) to view an offline snapshot of the database is available here:
Prior to running, this requires the installation of a database file into "~/Library/Application Support/" by unzipping the ~1GB file stored here: DATABASE.
Example:
Data Model Graph: