ERDDAP VM Setup

This chapter configures a dedicated ERDDAP serving VM after the infrastructure has been created. A file-backed deployment may need a high-memory VM, a large persistent cache disk, and a GCS data bucket. An ERDDAP that mainly serves remote sources can often use a smaller VM and omit the bucket mount.

The examples use these neutral local conventions:

Item	Example used in this guide
Shared administration group	erddap-admins
Persistent data disk	/dev/disk/by-id/google-erddap-data
Persistent data mount	/mnt/disks/erddap-data
Optional GCS mount	/mnt/gcs/erddap-data

These names are not GCP requirements. A team may change them, but the Terraform device name, mount configuration, installer variables, file ownership, and systemd units must remain consistent.

1. Configure the Operating System

1.1 Create the shared administration group

ERDDAP_ADMIN_GROUP=erddap-admins

getent group "$ERDDAP_ADMIN_GROUP" >/dev/null || \
  sudo groupadd "$ERDDAP_ADMIN_GROUP"

sudo usermod -aG "$ERDDAP_ADMIN_GROUP" "$USER"

# Optional: add a separate operational or automation account
sudo usermod -aG "$ERDDAP_ADMIN_GROUP" YOUR_OPERATIONAL_USERNAME

Omit the placeholder account command when no separate account is required. Log out and back in after changing group membership.

1.2 Initialize and mount the persistent data disk

Confirm the stable device path before continuing. The following block formats the device only when blkid finds no existing filesystem.

ERDDAP_ADMIN_GROUP=erddap-admins
data_device=/dev/disk/by-id/google-erddap-data
data_mount=/mnt/disks/erddap-data

ls -l /dev/disk/by-id/

if sudo blkid "$data_device" >/dev/null 2>&1; then
  echo "Existing filesystem detected; the disk will not be formatted."
  sudo blkid "$data_device"
else
  echo "No filesystem detected; formatting the new data disk."
  sudo mkfs.ext4 \
    -m 0 \
    -E lazy_itable_init=0,lazy_journal_init=0,discard \
    "$data_device"
fi

sudo mkdir -p "$data_mount"
sudo mount -o discard,defaults "$data_device" "$data_mount"

disk_uuid=$(sudo blkid -s UUID -o value "$data_device")
test -n "$disk_uuid" || {
  echo "Unable to determine the data-disk UUID" >&2
  exit 1
}

sudo cp -a /etc/fstab "/etc/fstab.before-erddap-$(date +%Y%m%d-%H%M%S)"
if ! grep -qE "^[[:space:]]*UUID=${disk_uuid}[[:space:]]" /etc/fstab; then
  echo "UUID=${disk_uuid} ${data_mount} ext4 defaults,discard 0 2" |
    sudo tee -a /etc/fstab
fi

sudo chown root:root /mnt/disks
sudo chmod 0755 /mnt/disks
sudo chown root:"$ERDDAP_ADMIN_GROUP" "$data_mount"
sudo chmod 2775 "$data_mount"

if sudo test -e /data || sudo test -L /data; then
  echo "/data already exists; review it before changing the path."
else
  sudo ln -s "$data_mount" /data
fi

Validate both the mount and /etc/fstab before rebooting:

namei -l /mnt/disks/erddap-data
findmnt /mnt/disks/erddap-data
df -hT /mnt/disks/erddap-data

sudo umount /mnt/disks/erddap-data
sudo mount -a
findmnt /mnt/disks/erddap-data

1.3 Optionally mount a GCS data bucket with rclone

Note

Complete this subsection only when ERDDAP will read file-backed datasets from a GCS bucket. Skip it for an ERDDAP that uses only remote HTTP, THREDDS, other ERDDAP servers, or other network sources.

The VM service account needs permission to list and read objects in the selected bucket. The infrastructure example grants roles/storage.objectViewer at bucket scope.

Install rclone and permit FUSE --allow-other mounts:

curl -fsSL https://rclone.org/install.sh | sudo bash

if ! sudo grep -qx 'user_allow_other' /etc/fuse.conf; then
  echo user_allow_other | sudo tee -a /etc/fuse.conf
fi

This intentionally installs rclone’s current stable release. Environments requiring fully pinned builds should select and verify a specific release.

Create a root-owned configuration for the systemd service:

ERDDAP_ADMIN_GROUP=erddap-admins

sudo install -d -o root -g "$ERDDAP_ADMIN_GROUP" -m 0750 /etc/rclone

sudo tee /etc/rclone/rclone.conf >/dev/null <<'EOF'
[gcs-remote]
type = google cloud storage
env_auth = true
project_number = YOUR_GCP_PROJECT_NUMBER
location = YOUR_GCP_REGION
EOF

sudo chown root:"$ERDDAP_ADMIN_GROUP" /etc/rclone/rclone.conf
sudo chmod 0640 /etc/rclone/rclone.conf

Test identity and IAM before configuring the mount:

sudo rclone lsd \
  --config /etc/rclone/rclone.conf \
  gcs-remote:YOUR_ERDDAP_BUCKET_NAME

Create the mount and cache directories:

ERDDAP_ADMIN_GROUP=erddap-admins

sudo mkdir -p /mnt/gcs/erddap-data
sudo mkdir -p /mnt/disks/erddap-data/rclone-cache
sudo chown root:root /mnt/gcs
sudo chmod 0755 /mnt/gcs
sudo chown root:"$ERDDAP_ADMIN_GROUP" /mnt/gcs/erddap-data
sudo chmod 2775 /mnt/gcs/erddap-data

Create the persistent mount service. Set the rclone cache limit based on the size of the VM’s data disk and expected workload. 100G is an example, not a recommended limit for every deployment. A large ERDDAP installation may use a limit such as 2000G. Leave enough disk space outside the cache for ERDDAP working files, logs, and growth.

ERDDAP_ADMIN_GROUP=erddap-admins
RCLONE_CACHE_MAX_SIZE=100G # Example; adjust for your data disk

FUSERMOUNT_BIN=$(command -v fusermount3 || command -v fusermount)
test -n "$FUSERMOUNT_BIN" || {
  echo "fusermount is not installed" >&2
  exit 1
}

sudo tee /etc/systemd/system/rclone-mount.service >/dev/null <<EOF
[Unit]
Description=Read-only GCS mount for ERDDAP
After=network-online.target
Wants=network-online.target
RequiresMountsFor=/mnt/disks/erddap-data

[Service]
Type=notify
User=root

ExecStart=/usr/bin/rclone mount gcs-remote:YOUR_ERDDAP_BUCKET_NAME /mnt/gcs/erddap-data \
    --config /etc/rclone/rclone.conf \
    --allow-other \
    --read-only \
    --dir-cache-time 1h \
    --attr-timeout 1h \
    --vfs-cache-mode full \
    --vfs-cache-max-age 48h \
    --vfs-cache-max-size ${RCLONE_CACHE_MAX_SIZE} \
    --vfs-cache-poll-interval 1m \
    --cache-dir /mnt/disks/erddap-data/rclone-cache \
    --buffer-size 128M \
    --vfs-read-ahead 256M \
    --vfs-read-chunk-size 64M \
    --vfs-read-chunk-size-limit 2G \
    --dir-perms 0775 \
    --file-perms 0664 \
    --gid $(getent group "$ERDDAP_ADMIN_GROUP" | cut -d: -f3) \
    --umask 002 \
    --rc \
    --rc-no-auth

ExecStop=${FUSERMOUNT_BIN} -u /mnt/gcs/erddap-data
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now rclone-mount.service

The example values are a tested starting point, not universal tuning. In particular, --attr-timeout controls how long the kernel caches file attributes; it is not an interval for polling GCS for replaced objects. Measure performance and freshness requirements before changing these values.

Verify the service and mount:

systemctl is-active rclone-mount.service
findmnt /mnt/gcs/erddap-data
ls -la /mnt/gcs/erddap-data | head
rclone rc vfs/stats

2. Install and Configure ERDDAP

2.1 Run the installation script

The repository script installs pinned Java, Tomcat, ERDDAP, and erddapContent versions. It configures the tomcat-erddap service, persistent working directory, and reverse-proxy handling. It refuses to overwrite an existing Java or Tomcat installation and is therefore intended for a new host, not an in-place upgrade.

Check physical memory before choosing the Java heap:

free -h

Set the heap to approximately 50–70% of physical RAM and never above 80%, leaving memory for the operating system, Tomcat native memory, rclone, and filesystem cache.

wget -O erddap_setup.sh \
  https://raw.githubusercontent.com/erd-coastwatch/swfsc-gcp-migration/main/scripts/erddap/erddap_setup.sh

chmod 700 erddap_setup.sh
bash -n erddap_setup.sh
./erddap_setup.sh --help

TOMCAT_GROUP=erddap-admins \
JAVA_HEAP_MB=32768 \
./erddap_setup.sh

The example heap is 32 GB; smaller VMs must use a smaller value. The installer enables the service but leaves it stopped so configuration can be reviewed first.

Note

The SWFSC script currently pins Tomcat 11 with ERDDAP 2.31.1 because that combination was validated on a new VM. Treat any version change as a planned, tested upgrade.

Make Tomcat require the persistent disk:

sudo mkdir -p /etc/systemd/system/tomcat-erddap.service.d

sudo tee \
  /etc/systemd/system/tomcat-erddap.service.d/storage.conf \
  >/dev/null <<'EOF'
[Unit]
RequiresMountsFor=/mnt/disks/erddap-data
EOF

For a bucket-backed deployment, also require the rclone service:

sudo tee \
  /etc/systemd/system/tomcat-erddap.service.d/rclone.conf \
  >/dev/null <<'EOF'
[Unit]
Requires=rclone-mount.service
After=rclone-mount.service
EOF

sudo systemctl daemon-reload

Skip the rclone override when no bucket is mounted.

2.2 Configure ERDDAP

Follow the official ERDDAP installation documentation and review at least:

• bigParentDirectory;
• baseUrl and baseHttpsUrl;
• administrator and institution information;
• email, subscription, and authentication settings;
• flagKeyKey;
• the initial datasetsRegex.

sudoedit /usr/local/tomcat_erddap/content/erddap/setup.xml

When ERDDAP is behind an HTTPS load balancer, both public URL settings should use the external hostname:

<baseUrl>https://YOUR_ERDDAP_HOSTNAME</baseUrl>
<baseHttpsUrl>https://YOUR_ERDDAP_HOSTNAME</baseHttpsUrl>

The validated installer content includes the small etopo180 test dataset. Confirm that the installed catalog still contains it before using that ID:

catalog=/usr/local/tomcat_erddap/content/erddap/datasets.xml

if grep -q 'datasetID="etopo180"' "$catalog"; then
  echo "etopo180 is available for the initial test."
else
  echo "Select a small dataset ID that is present in the installed catalog."
  grep -oE 'datasetID="[^"]+"' "$catalog" | head
fi

Restrict datasetsRegex to the selected small dataset for the first start. For the validated content package:

<datasetsRegex>etopo180</datasetsRegex>

The installer also adds Tomcat’s RemoteIpValve, allowing Tomcat to honor the load balancer’s forwarded protocol and client-address headers. The user does not need to paste a valve into server.xml; verify it only when troubleshooting proxy behavior:

sudo grep -A4 -B1 'RemoteIpValve' \
  /usr/local/tomcat_erddap/conf/server.xml

Set configuration ownership and validate the XML:

sudo chown tomcat:erddap-admins \
  /usr/local/tomcat_erddap/content/erddap/setup.xml \
  /usr/local/tomcat_erddap/content/erddap/datasets.xml

sudo chmod 0640 \
  /usr/local/tomcat_erddap/content/erddap/setup.xml \
  /usr/local/tomcat_erddap/content/erddap/datasets.xml

sudo xmllint --noout /usr/local/tomcat_erddap/content/erddap/setup.xml
sudo xmllint --noout /usr/local/tomcat_erddap/content/erddap/datasets.xml
sudo xmllint --noout /usr/local/tomcat_erddap/conf/server.xml

3. Validate ERDDAP in Layers

Complete each layer before troubleshooting the next one.

Layer 1: local service

On the ERDDAP VM:

sudo systemctl start tomcat-erddap
sudo systemctl --no-pager --full status tomcat-erddap

curl -fsSI http://127.0.0.1:8080/erddap/status.html
curl -fsSI http://127.0.0.1:8080/erddap/index.html
curl -fsSI http://127.0.0.1:8080/erddap/griddap/etopo180.html

Use the selected test dataset ID in the final request if it is not etopo180. Do not proceed until the local requests succeed.

Layer 2: private network access

From an authorized host on the VPC, or through the SSH forwarding method described in Accessing VMs:

curl -fsSI http://YOUR_ERDDAP_INTERNAL_IP:8080/erddap/status.html

Failure here, after Layer 1 succeeds, points to host or VPC access rather than ERDDAP configuration.

Layer 3: load-balancer backend

Important

This guide assumes the responsible infrastructure team has created or will create the instance group, named port 8080, backend service, /erddap/status.html health check, URL map, HTTPS proxy, certificate, forwarding rule, and DNS or organizational external route. Record which team and Terraform state own each shared component.

The backend must accept TCP 8080 from Google’s health-check ranges and, for a regional Application Load Balancer, the region’s proxy-only subnet. Determine the proxy-only range rather than copying one from another project:

gcloud compute networks subnets list \
  --project=YOUR_GCP_PROJECT_ID \
  --filter='purpose=REGIONAL_MANAGED_PROXY' \
  --format='table(name,region,network,ipCidrRange,purpose,role)'

Create narrowly scoped VPC firewall rules through the Terraform state or administrative process that owns networking. Do not expose Tomcat port 8080 to the public internet.

If the VM image also uses UFW, allow the same health-check and proxy sources locally:

sudo ufw allow proto tcp from 130.211.0.0/22 \
  to YOUR_ERDDAP_INTERNAL_IP port 8080 comment 'Google LB health checks'

sudo ufw allow proto tcp from 35.191.0.0/16 \
  to YOUR_ERDDAP_INTERNAL_IP port 8080 comment 'Google LB health checks'

sudo ufw allow proto tcp from YOUR_PROXY_ONLY_SUBNET \
  to YOUR_ERDDAP_INTERNAL_IP port 8080 comment 'Regional LB proxy subnet'

sudo ufw status numbered

From Cloud Shell or an administrative workstation, confirm backend health:

gcloud compute backend-services get-health YOUR_BACKEND_SERVICE \
  --project=YOUR_GCP_PROJECT_ID \
  --region=YOUR_GCP_REGION

Proceed when the backend reports HEALTHY.

Layer 4: public HTTPS route

curl -fsSI https://YOUR_ERDDAP_HOSTNAME/erddap/status.html

curl -fsS https://YOUR_ERDDAP_HOSTNAME/erddap/index.html |
grep -oE '<link[^>]*erddap2.css[^>]*>'

The status request should return HTTP 200, and generated asset URLs should use the public HTTPS hostname. Production checks should validate the complete certificate chain.

Troubleshooting logs

Consult logs when a validation layer fails rather than following them as an unrelated setup step:

sudo journalctl -u tomcat-erddap --since '15 minutes ago'

sudo tail -n 200 \
  /mnt/disks/erddap-data/erddap/logs/log.txt

4. Perform the Reboot Test

After all four validation layers succeed:

sudo reboot

Reconnect and verify automatic recovery:

findmnt /mnt/disks/erddap-data

# Bucket-backed deployments only
findmnt /mnt/gcs/erddap-data
systemctl is-active rclone-mount.service

systemctl is-active tomcat-erddap.service
curl -fsSI http://127.0.0.1:8080/erddap/status.html

Repeat the backend-health and public-HTTPS checks. The infrastructure setup is complete when the service recovers automatically after reboot.

The ERDDAP administrator can then manage the production datasets.xml using the official datasets.xml documentation and the team’s dataset-specific procedures. Production catalog authoring and dataset migration are intentionally outside this infrastructure guide.

To also use the ERDDAP hostname as an entry point to a GitHub Pages website, continue to Optional Website Redirect with Apache. This integration is optional and follows the validated ERDDAP setup above.

Operational Reference: rclone Cache Refresh

This subsection applies only to bucket-backed deployments after setup is complete.

Objects added or replaced directly in GCS may remain hidden until rclone’s directory or VFS caches expire. Refresh only the affected directory when practical:

rclone rc vfs/refresh \
  dir=PATH/RELATIVE/TO/BUCKET \
  recursive=true

When one same-name object must be reread, forget that cached path:

rclone rc vfs/forget \
  file=PATH/RELATIVE/TO/BUCKET/file.nc

The VFS cache-size limit is a ceiling, not a disk reservation, and open files can temporarily prevent eviction. Monitor both cache statistics and free disk space:

rclone rc vfs/stats
df -h /mnt/disks/erddap-data