Under The Hood - Direct Searches
Introduction to searching the database directly using command-line tools
Overview
This page provides a basic introduction to searching the postgres meta-data database directly. This kind of access will be useful to anyone needing to search in ways that the web interface does not allow.
Prerequisites
In order to succeed with direct search, you'll need to have;
- configured your iVec account appropriately. Steps involved are covered in Preparing Your Account. Please check that you've covered all of these steps.
- a basic understanding of unix command line tools
- a basic understanding of SQL. If you've not used SQL before, then you'll just need to be prepared to learn.
The Postgres Client
- From wastac-4240: psql -d wastac
- From cognac or pbstore: psql -d wastac -h wastac-4240.ivec.org
Wastac Database Basics - Simplified Views
- wastac.v_swath - metadata about swaths in the archive
- wastac.v_archivefile - metadata about archive files in the archive
wastac.v_swath
| Field | Type | Description |
|---|---|---|
| swath_name | string | The swath name. The convention used in the database is that the swath name is filename off the "primary" datafile (ie L1B or L0 as it may be) without any type extensions. Examples are n16.0912111028_47532, MOD01.A2005059.1545.005.2008196220203 and mut01_20091211_1305_MOD. |
| swath_timestamp_start | timestamp | The UTC time of the first observation in the swath |
| swath_timestamp_end | timestamp | The UTC time of the last observation in the swath |
| swath_dayscene | boolean | True if the swath is a day-time scene |
| swath_scanlines | integer | The reported number of scan-lines in the swath; this number is only comparable between like swaths and is intended to give you an idea of the swath size. |
| swath_bounding | geometry | This is a geometry object mapping the bounds of the swath. It is a true perimeter of the scene in lon, lat coordinates (as opposed to rectangular bounding box). |
| pc_day | real | Percentage of day-time observations. For MODIS this information is collected from MOD35 statistics. |
| pc_land | real | Percentage of land observations. For MODIS this information is collected from MOD35 statistics. |
| pc_water | real | Percentage of water observations. For MODIS this information is collected from MOD35 statistics. |
| pc_glint | real | Percentage of sun-glint observations. For MODIS this information is collected from MOD35 statistics. |
| pc_cloud | real | Percentage of cloud observations. For MODIS this information is collected from MOD35 statistics. |
| quicklook | real | True if a quicklook has been created. |
| sensor_name | string | The sensor name (ie. modis, avhrr) |
| satellite_name | string | The satellite name (ie aqua, terra, n18) |
| acquirer_name | string | The data acquirer name (ie wastac, nasa) |
wastac.v_archivefile
Simple Searches
How many swaths are in the database?
wastac=> select count(*) from wastac.v_swath;
count
--------
130250
(1 row)
How many swaths from each acquirer?
wastac=> select acquirer_name, count(swath_name) from wastac.v_swath group by acquirer_name;
acquirer_name | count
---------------+--------
wastac | 26500
nasa | 103750
What MODIS DACC (NASA) granules exist between two dates?
wastac=> select swath_name
from wastac.v_swath
where
swath_timestamp_start > timestamp '2009-08-01'
and swath_timestamp_end < timestamp '2009-08-31'
and acquirer_name = 'nasa'
order by swath_timestamp_start;
Note to save the results of the above query to a file, you could use the output re-direct function of psql as follows;
wastac=> \o ~/results.txt
wastac=> select swath_name
from wastac.v_swath
where
swath_timestamp_start > timestamp '2009-08-01'
and swath_timestamp_end < timestamp '2009-08-31'
and acquirer_name = 'nasa'
order by swath_timestamp_start;
wastac=> \q
hlynch@wastac-4240% less ~/results.txt
How many swaths per acquirer per month over each year?
wastac=> select
date_part('year', swath_timestamp_start) as year,
date_part('month', swath_timestamp_start) as month,
acquirer_name,
count(swath_name)
from wastac.v_swath
group by year, month, acquirer_name
order by year, month, acquirer_name;
Which swaths fall in october of any year?
wastac=> select swath_name
from wastac.v_swath
where
acquirer_name = 'nasa'
and date_part('month', swath_timestamp_start) = '10'
order by swath_timestamp_start;
Which swaths cover a specific area?
wastac=> SELECT swath_name, swath_timestamp_start AT TIME ZONE 'Australia/Perth' AS localtime
FROM wastac.t_swath_metadata
WHERE swath_bounding IS NOT NULL
AND swath_bounding && GeomFromText('POLYGON((121.75 -34.0, 122.0 -34.0, 122.0 -33.75, 121.75 -33.75, 121.75 -34.0))',-1)
ORDER BY swath_timestamp_start;
How can I display times as my local time rather than UTC?
wastac=> SELECT swath_name, swath_timestamp_start AT TIME ZONE 'Australia/Perth' AS localtime;
