NOAA and AVHRR Processing Overview

Summary

• Data is delivered as HRPT. We need to derive Level 1B (or the equivalent of).
  
• There seems to be 2 "options" to do this (see later for comparison) and both are operational on cognac;
  
• CAPS is the one locally traditionally used by WASTAC.
  
• AAPP is distributed by the UK Met Office / EUMetSat.
  
• See NOAA KLM Users Guide for useful information
• Processing to L1B on Cognac using AAPP now works. CAPS still has some problems (no NOAA 18 and no data in HDF)
  

NOAA Series Platforms Overview

The following text is extracted from the AAPP Scientific Description (PDF):

The American polar orbiting weather satellites are controlled by NOAA. The first launch occurred in 1978 with the TIROS N satellite, followed by the NOAAxx series (from NOAA6 to NOAA14), and since 1998 by the NOAA-KLM (NOAA15,16,17) and NOAA-NN’ series. Each satellite has a lifetime of about 4 years. Two satellites must be operational at the same time. The current system records multispectral images (several spectral channels) of the Earth surface, and allows the computation of temperature and humidity profiles of the atmosphere. It is composed of two sun synchronous polar orbiting spacecraft, flying at an altitude close to 835 km. One is planned to fly in the afternoon orbit (PM), and the other in the morning orbit (AM). Due to the Earth rotation, the satellite track is translated westward with each pass. The nodal period is about 102 minutes, which means that each satellite describes 14.2 orbits per day, so the sub satellite track is not daily periodic (the same orbital configuration is achieved every 9 days). The satellites orbital parameters drift slowly, resulting in changes in observation conditions and overpass times.

• NOAA 6-14: TOVS (HIRS/2, MSU, SSU), AVHRR/2, DCS
• NOAA 15 16 and 17: ATOVS (HIRS/3,AMSU-A,AMSU-B), AVHRR/3,DCS
• NOAA 18 and N’: ATOVS (HIRS/4, AMSU-A,MHS),AVHRR/3, DCS

Advanced Very High Resolution Radiometers (AVHRR)

The following text is extracted from the AAPP Scientific Description (PDF):

These [vis and IR imaging] radiometers are designed to study cloud cover and Earth surface properties. Spatial resolution depends on the instrumentation quality and the satellite height from the observation point.

AVHRR/2 is a 5 channel (0.63 to 12 mm) high resolution infrared (IR) imaging radiometer. The first two channels (0.63 and 0,86mm) measure radiance in the visible (VIS) spectral band. For example, using both channels allows the discrimination of vegetation. The three IR channels (3.74, 10.80,12.00 mm) are generally dedicated to surface temperature determination. Using VIS and IR channels together allows determination of cloud presence and to distinguish cloud type and depth. Images from this radiometer are used for many varied applications. The internal rotating mirror scans (±55° about nadir) cross track over a swath of ±1450 km about the sub satellite track. The resolution is 1.1 km at nadir (when the observed point is at the satellite vertical) for an instantaneous field of view of 1.4 mrad. Therefore, in one day, the satellite covers twice the Earth surface, supplying one night-time and one daytime image.

AVHRR/3 has 6 channels (channel 3a: 1.61 mm, the 3.74 mm channel becomes channel 3b) in the VIS and IR spectral bands. The addition of channel 3a improves for example the discrimination between snow and clouds, and provides better aerosol detection. The resolution and the scanning characteristics remain unchanged with respect to the previous AVHRR/2 version. Only 5 channels will be received by the final user: channel 3a (day) or channel 3b (night).

For the NOAA KLM User Guide,

NOAA's data levels are defined as follows:

• Level 0 - Unprocessed telemetry data as received from the observing platform excluding communications artifacts introduced by the ground system.
• Level 1a - Telemetry data that have been extracted but not decommutated from Level 0 and formatted into time-sequenced data sets for easier processing. The Level 1a formats are NOAA's internal formats and are only used for NOAA processing. They only exist briefly for the purpose of creating the Level 1b datasets.
• Level 1b - Discrete, instrument-specific data sets derived from Level 1a containing unprocessed data at full resolution, time-referenced, and annotated with ancillary information including data quality indicators, calibration coefficients and georeferencing parameters.

Wastac Raw HRPT Data Format

This is the raw file format from the satellite receiving station. HRPT is not a rigidly defined format - internal characteristics vary between receiving stations. Typically however, the HRPT format is binary, composed of fixed length "frames", with defined internal structure in the NOAA KLM Users Guide. Without going into detail, it can be said that each frame commences with a pre-defined pre-sync word pattern and terminates with a pre-defined post-sync word pattern. Although it's been found that one can not rely on the pre/post sync alone for determining frame boundaries, as random discrepancies have been found to occur inside at least the pre-sync pattern (although the length is fixed).

The remaining hrpt data format is now documented below. Note the periods of validity.

Receiving Station and Validity	HRPT Format Characteristics
Murdoch 2005 (receiving station upgrade) to present	Files created with Terascan (16 bit, 11264 word frame size) Then these are run through Ron's packing C program to pack to 10-bit and remove the additional 174 words from each frame. Resulting data is 10 bit words, frames 11090 words long and with additional 4 bits appended to each frame so that each frame is guaranteed to start on a byte boundary (8 doesn't equally divide into 11090). Text ASDA header pre-prended to above (see section ASDA Header Removal).
Curtin circa 1996 to present	Data created in 10 bit words,frames 11090 words long and with additional 4 bits appended to each frame so that each frame is guaranteed to start on a byte boundary (8 doesn't equally divide into 11090). Text ASDA header pre-prended to above (see section ASDA Header Removal).
Previous data needed	A long unsupported program called disimp is used to read previous HRPT data formats. Unfortunately no-one has the source code and there existing only a Solaris executable left from days gone by. We need someone to document the formats used.

ASDA Header

WASTAC currently uses HRPT files with an ASDA header pre-pended. The header is text and is a simple parameter-value language.

The ASDA header is documented here.

If you need to remove the ASDA header (eg using AAPP), there is an unpacking script described later that extracts the header for you, but should you really want to remove it yourself, parse the file for the relevant part;

BEGIN_GROUP = Format;
 BEGIN_GROUP = PVL_Header;
  length = 1228800 <bytes>;
 END_GROUP = PVL_Header;

 BEGIN_GROUP = HRPT_Data;
  length = 3578 <records>;
  record_size = 13864 <bytes>;
  record_type = HRPT_Line;
 END_GROUP = HRPT_Data;
END_GROUP = Format;

observing that the 1228800 bytes needs to be removed from the beginning. Therefore,

$ dd bs=1228800 if=[inputfile] skip=1 > [output_hrpt_data]

CAPS and AAPP Comparison

Feature	AAPP	CAPS
Documentation	Very well documented	Minimal
Source Code Distribution	Open source, authenticated FTP by unrestricted prior arrangement	Closed source and available only to research partners.
Source Build	Trivial configure and make. Worked first time.	Extremely problematic Numerous undocumented dependencies; many TCL/TK llibraries, HDF, NetCDF, PVL (again not public release). Auto configure and Makefile currently broken at many levels. Build is possible however, although not without assistance from software author.
Platforms and Input Data formats	Variety of platforms/sensors (European MetOp and NOAA).	AVHRR only
Input format	Will not read local HRPT formats - an unpacking layer is required to get it to the expected NOAA format (16 bit / "unpacked"). For example see here	Allegedly handles most Australian formats around. No need to remove ASDA header or unpack data (can be left in original 10bit format). Read Wastac HRPT files (10bit+ASDA) with no problems.
Output format	Unfortunately no portable format (HDF or NetCDF). Uses HRPT NOAA frames for L1B. This brings back all the old school problems around custom readers, plotters, mappers and all the endian issues for long-term archives.	Has HDF, so portable and hopefully readable by modern cross-platform libraries like GDAL.
Navigation	TLE or TBUS (NORAD etc)	Clift Model (CSIRO Marine)
Algorithms	Need more info. Can generate cloudmask using HIRS.	CLAVR
Interface	Shell scripting to puppet suite of tool-like binaries. Has AAPP_RUN_NOAA script for single pass execution to generate L1B.	CAPS has fully featured GUI. For batch operations, it's TCL. There is a script avhrr2hdf which is an example of how to write your own using CAPS, or CSIRO Marine's collection of top-level TCL scripts.

CAPS Background

From WASTAC Annual Report 2001. Although DLI and maybe others are using CAPS, we're only evaluating the software currently.

CAPS development began in 1997 and the software is now a mature, stable product. The
following describes developments since the last report on CAPS in the 1998 WASTAC Annual
Report.

As well as the previously supported Linux (for PC), Solaris (Sun) and Irix (Silicon Graphics),
CAPS now runs under the Microsoft Windows operating system and on Hewlett Packard Unix
systems. CAPS is now able to read data in all the known formats in which AVHRR data have
been archived in Australia. There is support for the NOAA-16 satellite, which was launched
in September, 2000. A cloud detection module (based on the CLAVR algorithm) has been
developed. There is a new Graphical User Interface (GUI).

CAPS is used by DOLA to calculate and map hot spots (fires), fire scars, grassland curing
index, sea surface temperatures, land surface temperatures and NDVI. CAPS is used by
CSIRO Marine Research in Hobart for ‘warping’ of SeaWiFS data onto a latitude/longitude
grid. Both the input and output files are in HDF format. There have been successful tests
of warping of MODIS data, which is also commonly stored using HDF.

Ancillary data are required for calibration and navigation. Data for calibration of each satellite
are provided by CSIRO Atmospheric Research. CSIRO Marine Research generates navigation
data (position, velocity and attitude of the satellite) soon after each pass is received. This
aspect of the system has worked well over a number of years, although there have been
occasional errors and other problems.

The NAP (N-dimensional Array Processor) package was developed as part of the CAPS
project. NAP provides a powerful and efficient arithmetic and input/output facility in the Tcl/Tk
environment in which CAPS operates. A number of new features have been added
to NAP. There are new unsigned integer data-types, which are needed to read MODIS
HDF files. There is support for netCDF (as well as HDF) input/output. There are facilities to
read and write binary data. These were used by the Bureau of Meteorology to develop an
interface to the McIDAS file format. There is a new library of statistics functions. The new
function for calculating spatial correlations can be used to correct for errors in navigation by
matching small areas (called ‘chips’) whose position is known. There is a new graphing
facility ‘plot_nao’ for the display of data as XY graphs and images. The facility to load modules
compiled from C and Fortran has been used to produce land/sea masks based on data and
a modified version of software obtained from Rutherford Appleton Laboratory.
The CAPS Developers’ Group has been created and includes several members from WASTAC.

Members have signed a legal agreement, which is designed to promote the sharing of software
while protecting Intellectual Property. It is planned to replace the main CAPS
processing script ‘avhrr2hdf’ with a new modular system that will facilitate the integration of
new processes including cloud detection, atmospheric correction and BRDF.