MUR SST

This chapter documents the Cloud Run workflow used to retrieve, process, and publish Multi-scale Ultra-high Resolution (MUR) Sea Surface Temperature (SST) products for ERDDAP-facing datasets.

The workflow replaces legacy server-based processing with a containerized cloud workflow. It stages MUR data from NASA PO.DAAC, detects when available products have been updated from near-real-time to final quality, generates derived MUR v4.1 products, and publishes outputs to configured Google Cloud Storage (GCS) paths used by ERDDAP.

Purpose

The MUR SST workflow supports two related product streams:

  • MUR v4.1 — daily SST retrieval, daily anomaly generation, frontal edge detection, monthly SST composites, and monthly anomaly composites
  • MUR v4.2 — daily SST retrieval and smart synchronization

MUR v4.1 is the primary processing stream for derived products. MUR v4.2 is handled as a daily synchronization stream so the archive can track availability and quality upgrades.

Repository structure

mur-daily-workflows/
├── README.md                       # Operational overview and deployment notes
├── config/
│   ├── config.yml                  # Runtime settings, GCS targets, PO.DAAC collections
│   └── requirements.txt            # Python dependencies
├── deploy_job.sh                   # Build, push, deploy, and schedule Cloud Run jobs
├── Dockerfile                      # Container definition
├── entrypoint.sh                   # Job mode selector
├── scripts/
│   ├── MURanom1day.py              # Daily MUR v4.1 anomaly computation
│   ├── mur_v41_downloader_dailyproc.sh
│   │                               # MUR v4.1 daily retrieval, anomaly, and fronts driver
│   ├── mur_v42_downloader.sh       # MUR v4.2 daily retrieval driver
│   ├── fronts/
│   │   ├── calc_mur_fronts.py      # Regional fronts runner
│   │   ├── canny_lib.py            # Front-detection utilities
│   │   ├── Canny1.py               # West Coast Canny implementation
│   │   └── Canny2.py               # Atlantic Canny implementation
│   └── MonthlyProc/
│       ├── MUR41_MonProc.sh        # Monthly processing driver
│       ├── CompMURmon.py           # Monthly SST composite generation
│       └── CompMurAnomMon.py       # Monthly anomaly composite generation
├── src/
│   └── roylib.py                   # Workflow-specific config and GCS publishing utilities
└── templates/
    ├── murAnomTemplate.nc          # Template for daily anomaly output
    ├── murSSTmday.nc               # Template for monthly SST composite
    └── murSSTmdayAnom.nc           # Template for monthly anomaly composite

Cloud Run jobs

The workflow uses a single container image with multiple Cloud Run job modes. The entrypoint.sh script dispatches to the appropriate processing script based on the JOB_MODE environment variable.

Job mode Script Description
mur41_daily mur_v41_downloader_dailyproc.sh MUR v4.1 daily retrieval, anomaly generation, and fronts processing
mur42_daily mur_v42_downloader.sh MUR v4.2 daily retrieval and quality-aware synchronization
mur41_monthly MUR41_MonProc.sh MUR v4.1 monthly SST and anomaly composites

This separation allows daily and monthly products to run on different schedules while using the same container, configuration file, and deployment pattern.

Processing pipeline

The daily MUR v4.1 workflow follows this pattern:

Cloud Scheduler
  → Cloud Run Job
  → Check existing GCS product
  → Check NASA PO.DAAC / CMR availability
  → Download or refresh source file when needed
  → Publish daily SST
  → Generate and publish daily SST anomaly
  → Generate and publish regional frontal products
  → Remove local temporary files

The MUR v4.2 workflow follows the same synchronization pattern but does not generate anomaly or frontal products.

Quality-aware synchronization

The daily download scripts check both the product already staged in GCS and the product currently available from NASA. This allows the workflow to detect when a near-real-time granule has been replaced or promoted to a final-quality product.

If the existing GCS copy is still near-real-time and NASA now provides the final version, the workflow re-downloads the file and republishes the updated product. This keeps the ERDDAP-facing archive current without requiring full reprocessing of the entire record.

Runtime configuration

Runtime settings are loaded from config/config.yml. The configuration defines:

  • PO.DAAC collection names
  • production bucket and publish root
  • MUR v4.1 and v4.2 destination directories
  • daily and monthly interval conventions
  • local scratch directories used inside the Cloud Run container
  • the MUR climatology location used for anomaly processing
  • optional ERDDAP reload or flag settings

Bucket-name fields are plain bucket names, not gs:// URIs. Full GCS URI settings are only used where the value represents a complete input location, such as a climatology prefix.

Example destination pattern:

gs://<prod_bucket>/<prod_root>/<prod_dst_dir>/<interval>day/<filename>

MUR v4.1 daily products

The MUR v4.1 daily job retrieves daily SST products and publishes them to the configured production path. After the SST product is staged, the workflow runs the daily anomaly script and the regional front-detection scripts.

Daily MUR v4.1 outputs include:

  • daily SST
  • daily SST anomaly
  • West Coast frontal edge product
  • Atlantic frontal edge product

The daily SST anomaly is computed relative to the configured MUR climatology. The anomaly output is written using the NetCDF template in templates/murAnomTemplate.nc.

Frontal edge products

Frontal products are generated from MUR v4.1 SST using Canny edge detection. The workflow currently processes two regional subsets:

Region Code Approximate coverage
U.S. West Coast WC 22°N–51°N, 135°W–105°W
U.S. Atlantic Coast ATL 20°N–50°N, 90°W–60°W

The front-detection output includes frontal edge masks, x/y gradients, and gradient magnitude fields. The workflow uses calc_mur_fronts.py as the regional runner and canny_lib.py as the shared front-processing utility layer.

Monthly composites

Monthly processing is triggered with JOB_MODE=mur41_monthly. The monthly driver runs two compositors:

  1. CompMURmon.py — builds a monthly mean SST composite from daily MUR v4.1 SST files.
  2. CompMurAnomMon.py — builds a monthly mean anomaly composite from daily anomaly files.

The target month is controlled by the MONTH_OFFSET environment variable. The default behavior is to process the last completed month.

The monthly compositors download the relevant daily files from GCS, process them in parallel where appropriate, and write outputs using the monthly NetCDF templates in templates/.

Publishing model

Final products are uploaded to the configured production bucket under the configured root prefix. Product-specific destination directories and interval labels are defined in config/config.yml.

Example output families include:

MUR41/ssta/1day/
MUR41/anom/1day/
MUR41/erdMurFront41USWest/1day/
MUR41/erdMurFront41USAtlantic/1day/
MUR41/ssta/mday/
MUR41/anom/mday/
MUR42/ssta/1day/

The simplified src/roylib.py used by this workflow provides the shared configuration loader and GCS publishing helper used by the monthly processing scripts.

Credentials and runtime files

NASA access and Google Cloud access are handled at runtime. The container uses Cloud Run service account permissions for GCS access, while NASA/Earthdata credentials are provided through the runtime environment.

Credential files such as .netrc and URS cookie files are runtime-only files and are not part of the public documentation copy.

Codebase

The files linked below provide a view of the workflow code used by this processor. Deployment-specific values such as bucket names, project IDs, internal URLs, and credential paths are shown with generic placeholders.

Deployment

Container, runtime configuration, and Cloud Run deployment files.

Daily retrieval

MUR v4.1 and v4.2 daily synchronization and publishing.

Daily derived products

Anomaly and frontal edge products generated from MUR v4.1 SST.

Monthly composites

Monthly SST and anomaly composite generation.

Shared utilities

Workflow-specific configuration loading and GCS publishing helpers.