Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

A guest post by Alex Johnson, Software Development Director at Tech Innovations

The SpatioTemporal Asset Catalog (STAC) specification is designed to create a standardized method for how geospatial assets are made accessible online and queried. The China-Brazil Earth Resources Satellites (CBERS) are a result of a collaborative effort between Brazilian and Chinese space agencies (INPE and CAST), initiated in 1988. Over the years, five satellites have been launched (CBERS-1/2/2A/3/4), generating Earth images similar to those produced by the USGS Landsat and ESA’s Sentinel-2 missions. In 2004, INPE made the announcement that all CBERS-2 images would be available for free to the public, marking the first time such a distribution model was applied to medium-resolution satellite imagery. Today, this model is used for all CBERS satellite images.

This article details how data from CBERS-4, hosted on AWS, utilizes STAC for its satellite imagery archive. The archive includes images captured from MUX, AWFI, and PAN cameras, which provide Red, Green, Blue, NIR, and Panchromatic images with resolutions ranging from 64m to 5m, and a revisit cycle of five to 52 days. The architecture discussed here can be adapted to similar data sources that require ongoing metadata generation and updates. It is specifically designed to leverage services provided by CBERS on AWS.

Creating the STAC Item

The STAC Item is a GeoJSON feature that describes each distinct data object for discovery. The GeoJSON specification outlines a standard for encoding geographical data structures as JSON documents. Below is an excerpt from a CBERS-4 MUX scene, showcasing important geographic information that enables location-based searches.

{
  "id": "CBERS_4_MUX_20170618_057_121_L2",
  "type": "Feature",
  "bbox": [
    47.930129,
    -19.401998,
    49.329281,
    -18.16659
  ],
  "geometry": {
    "type": "MultiPolygon",
    "coordinates": []
  },
  "properties": {
    "datetime": "2017-06-18T07:02:39Z",
    "provider": "INPE",
    "eo:sun_azimuth": 31.6748,
    "eo:sun_elevation": 41.0625,
    "eo:epsg": 32651,
    "cbers:data_type": "L2",
    "cbers:path": 57,
    "cbers:row": 121
  },
  "links": {
    "self": {},
    "catalog": {}
  },
  "assets": {
    "thumbnail": {},
    "metadata": {},
    "B5": {}
  }
}

The final step in ingesting each CBERS-4 scene into AWS involves generating a quicklook or thumbnail image. Notification of the thumbnail’s availability is sent to a public Amazon Simple Notification Service (SNS) topic. Each camera has its own dedicated topic so that subscribers can receive notifications relevant only to their chosen sensor.

The process of generating STAC items begins when the New Scenes Queue, which is subscribed to the quicklook SNS topics, receives a message indicating that a new scene has been ingested. An AWS Lambda function that generates STAC items consumes these messages from the queue. This architecture now utilizes the recently introduced feature that allows direct triggering of Lambda functions from Amazon Simple Queue Service (SQS). This method eliminates the need for explicit polling of the queue for new messages, automatically scales the instantiation of functions based on queue length, and includes a robust error handling mechanism – messages that the Lambda function fails to process are automatically returned to the queue.

The STAC item generator Lambda function retrieves the original metadata of the CBERS scene, stored in an XML file within the publicly accessible cbers-pds bucket, generates a corresponding STAC item, and saves this item into another publicly available cbers-stac bucket. Each item is processed independently, enabling parallel execution and leveraging the automatic scaling capabilities of the SQS/Lambda integration.

Every generated STAC item is also published to a public CBERS STAC SNS topic. The SNS message body contains the STAC item JSON, including attributes such as the scene’s datetime and geographic bounding box, which facilitate basic geographic filtering for listeners. This can potentially enhance a geospatial search mechanism.

The Static STAC Catalog

STAC items can be organized into Static STAC catalogs, referred to as catalogs. The “Static STAC Catalog Update” section of the architecture is responsible for generating and refreshing these catalogs.

Each catalog is a JSON file that references STAC items and/or other catalogs, among additional references. Catalogs provide a structured hierarchy for STAC items and can be browsed using standard tools like Radiant Earth Foundation’s STAC browser. A demonstration of the STAC browser for the CBERS on AWS static catalog is available here.

We opted to organize the items using the CBERS-4 Reference Grid System, linking each scene to a path and row number. This reference system ensures that scenes generated during a satellite pass share the same path while rows increase. The reference grid footprint is displayed by Remote Pixel’s viewer.

Choosing this grid system means that the S3 keys for our STAC items are formatted as {CAMERA}/{PATH}/{ROW}/{SCENE_DATE}. Each level features a catalog file referencing the child catalogs or STAC items. Below is an example of a catalog for CBERS-4 MUX path 57, with a child catalog defined for each row:

{
  "name": "CBERS4 MUX 057",
  "description": "CBERS4 MUX camera path 057 catalog",
  "links": [
    {
      "rel": "self",
      "href": "catalog.json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json"
    },
    {
      "rel": "child",
      "href": "120/catalog.json"
    },
    {
      "rel": "child",
      "href": "121/catalog.json"
    },
    {
      "rel": "child",
      "href": "122/catalog.json"
    }
  ]
}

An excerpt from the MUX catalog for MUX path 057 and row 121 is shown below, linking to all available scenes for this path and row:

{
  "name": "CBERS4 MUX 057/121",
  "description": "CBERS4 MUX camera path 057 row 121 catalog",
  "links": [
    {
      "rel": "self",
      "href": "catalog.json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180713_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180808_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20171121_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180617_057_121_L2.json"
    }
  ]
}

The catalogs could be updated by the STAC Item Generator Lambda function, but this might result in unnecessary repeated operations when one would suffice. For instance, if a new STAC item with the key MUX/057/121/CBERS_4_MUX_20180617_057_121_L2.json is created, we would need to verify and update catalogs at three levels:

1. MUX/ level, since 057 may be a new PATH not yet included in the catalog.
2. MUX/057 level, as 121 could be a new ROW under path 057.
3. MUX/057/121 level, since CBERS_4_MUX_20180617_057_121_L2.json is a new scene and must be added to the 057/121 catalog.

Given the way scenes are acquired, there is a high likelihood that CBERS_4_MUX_20180617_057_121_L2 is generated alongside other scenes with different rows but the same path, such as CBERS_4_MUX_20180617_057_122_L2. For more insights on short-term goals, check out this blog post, and for information on non-solicitation agreements, see this resource.