Project Architecture
This chapter describes a reference architecture for running ERDDAP on Google Cloud. The design separates public data serving, persistent state, source data, processing, and external routing so each component can be secured, scaled, rebuilt, and maintained independently.
Core and optional components
| Component | Requirement | Purpose |
|---|---|---|
| ERDDAP VM | Required | Runs Java, Tomcat, and ERDDAP on a private Compute Engine instance. |
| Persistent data disk | Required | Stores ERDDAP’s working directory, logs, decompressed files, and optional rclone cache independently of the boot disk. |
| Infrastructure as code (Terraform) | Strongly recommended | Makes VM, disk, IAM, bucket, and snapshot configuration reviewable and reproducible. Terraform supports multiple cloud providers; this guide uses its Google Cloud provider and GCP-specific resources. |
| HTTPS load balancer | Required for the public-service design shown here | Performs health checks, terminates HTTPS, and routes requests to the private backend. |
| GCS data bucket and rclone mount | Optional | Provides a read-only filesystem view of file-backed datasets stored in object storage. |
| Processing VM | Optional | Supports interactive development, backfills, validation, and compute-heavy processing without competing with the serving VM. |
| Cloud Run, Scheduler, Artifact Registry, and Secret Manager | Optional | Support scheduled containerized data-processing workflows. |
ERDDAP can serve remote HTTP, THREDDS, or other ERDDAP sources without a local data bucket. Do not add a bucket, large cache disk, or processing environment unless the catalog and workload require them.
Serving patterns
Remote-source ERDDAP
In the simplest pattern, ERDDAP reads from upstream network services. The VM still needs a persistent disk for ERDDAP state and logs, but it may not need a GCS data bucket or rclone.
This pattern has fewer components, but public availability and dataset-load time depend on the upstream services.
File-backed ERDDAP
For large local archives, source files are stored in GCS and mounted read-only into the ERDDAP VM. ERDDAP reads a normal filesystem path while rclone manages object retrieval and a persistent VFS cache.
This pattern provides a durable shared data layer and keeps source data outside the VM, but it requires deliberate cache sizing, mount monitoring, service ordering, and bucket IAM.
Processing and serving
When the team also generates data products, processing systems write production-ready files to storage and ERDDAP serves them independently. Processing may run on a VM, Cloud Run Jobs, or another managed service. A processing failure should not stop the ERDDAP service itself.
Design principles
- Keep the ERDDAP VM private; expose the service through the load balancer rather than a public VM address.
- Store ERDDAP working state on a persistent disk that survives VM replacement.
- Treat the boot disk as replaceable.
- Mount cloud source data read-only on the serving VM.
- Give the ERDDAP service account read-only object access unless it has a separately justified write responsibility.
- Keep scientific processing off the serving VM whenever practical.
- Manage infrastructure and configuration through reviewed source control.
- Test locally before adding load-balancer, DNS, and certificate dependencies.
Decisions before deployment
| Decision | Questions to answer |
|---|---|
| Catalog model | Are datasets remote, file-backed, or mixed? |
| Initial capacity | What memory, boot disk, persistent disk, and cache sizes are justified by the expected catalog? |
| Data storage | Is a GCS bucket required? Who creates it, owns it, and writes to it? |
| Identity | Which service account runs the VM, and what is the minimum required IAM? |
| Networking | Which VPC, subnet, proxy-only subnet, firewall policy, and private address will be used? |
| Public routing | Who owns the load balancer, health check, certificate, DNS, and external routing? |
| Infrastructure ownership | Which Terraform state manages each resource? Are shared platform resources intentionally outside this repository? |
| Recovery | Which disks are protected by snapshot policies, and who tests restoration? |
| Operations | Who receives alerts, reviews failed datasets, manages upgrades, and maintains setup.xml and datasets.xml? |
Resources may share one GCP project without sharing one Terraform state. A team’s Terraform should manage only the resources that team owns. Shared load-balancer components can be managed separately as long as ownership and interfaces are documented.
Architecture diagram
The SWFSC implementation includes both the required serving components and optional processing components:
End Users
Developer
Balancer
with the same Terraform config
triggered by Cloud Scheduler
Registry
Scheduler
Manager
ⓘ Click any component to learn more.
Use the diagram as a reference, not as a requirement to deploy every component.