Optional Processing VM Setup

A processing VM is optional. Use one when the team needs cloud compute for data generation, backfills, validation, or interactive development. It is not required for an ERDDAP that serves remote sources or files produced elsewhere.

This chapter covers only the cloud and operating-system foundation. Python environments, scientific packages, Earthdata credentials, external software, and project-directory conventions belong in workflow-specific documentation maintained by the processing team.

1. Provision the VM

Define the processing VM with the same reviewed infrastructure-as-code practices used for the ERDDAP VM. Choose independently:

• machine family, CPU, and memory;
• boot and persistent data-disk sizes;
• VPC, subnet, and IAP access;
• service account and least-privilege IAM;
• snapshot policy, labels, and organization-required controls.

Do not copy the ERDDAP VM size automatically. Start with a defensible estimate and resize after observing actual CPU, memory, I/O, and storage use.

2. Configure Shared Access When Needed

If multiple operators or automation accounts need access to the persistent workspace, create a group with a generic name selected by the team:

PROCESSING_GROUP=processing-admins

getent group "$PROCESSING_GROUP" >/dev/null || \
  sudo groupadd "$PROCESSING_GROUP"

sudo usermod -aG "$PROCESSING_GROUP" "$USER"

# Optional: add a separate operational or automation account
sudo usermod -aG "$PROCESSING_GROUP" YOUR_OPERATIONAL_USERNAME

Omit the placeholder account command when it is not needed. Log out and back in after changing group membership.

3. Initialize the Persistent Data Disk

Confirm the stable device path before running this block. It formats the device only when blkid finds no existing filesystem.

PROCESSING_GROUP=processing-admins
data_device=/dev/disk/by-id/google-processing-data
data_mount=/mnt/disks/processing-data

ls -l /dev/disk/by-id/

if sudo blkid "$data_device" >/dev/null 2>&1; then
  echo "Existing filesystem detected; the disk will not be formatted."
  sudo blkid "$data_device"
else
  echo "No filesystem detected; formatting the new data disk."
  sudo mkfs.ext4 \
    -m 0 \
    -E lazy_itable_init=0,lazy_journal_init=0,discard \
    "$data_device"
fi

sudo mkdir -p "$data_mount"
sudo mount -o discard,defaults "$data_device" "$data_mount"

disk_uuid=$(sudo blkid -s UUID -o value "$data_device")
test -n "$disk_uuid" || {
  echo "Unable to determine the data-disk UUID" >&2
  exit 1
}

sudo cp -a /etc/fstab "/etc/fstab.before-processing-$(date +%Y%m%d-%H%M%S)"
if ! grep -qE "^[[:space:]]*UUID=${disk_uuid}[[:space:]]" /etc/fstab; then
  echo "UUID=${disk_uuid} ${data_mount} ext4 defaults,discard,nofail 0 2" |
    sudo tee -a /etc/fstab
fi

sudo chown root:root /mnt/disks
sudo chmod 0755 /mnt/disks
sudo chown root:"$PROCESSING_GROUP" "$data_mount"
sudo chmod 2775 "$data_mount"

if sudo test -e /data || sudo test -L /data; then
  echo "/data already exists; review it before changing the path."
else
  sudo ln -s "$data_mount" /data
fi

Validate the mount and /etc/fstab entry before rebooting:

namei -l /mnt/disks/processing-data
findmnt /mnt/disks/processing-data
df -hT /mnt/disks/processing-data

sudo umount /mnt/disks/processing-data
sudo mount -a
findmnt /mnt/disks/processing-data

4. Verify Cloud Identity and Storage Access

Prefer the VM’s attached service account for unattended cloud access. Do not place service-account keys on the VM when the attached identity can provide the required permissions.

Confirm the active identity and test only the storage operation the workflow requires:

gcloud auth list

# Read test
gcloud storage ls gs://YOUR_PROCESSING_BUCKET_NAME/

# Optional write test; use only when the service account is intended to write
test_object="gs://YOUR_PROCESSING_BUCKET_NAME/access-tests/$(hostname)-$(date -u +%Y%m%dT%H%M%SZ).txt"
printf 'processing VM access test\n' | gcloud storage cp - "$test_object"
gcloud storage rm "$test_object"

Skip the write test for read-only identities. If access fails, correct IAM through the configuration or administrative process that owns it rather than adding broad permissions on the VM.

5. Hand Off to Workflow-Specific Setup

At this point the VM, persistent disk, private access, identity, and cloud storage permissions are ready. The processing team can now apply its established environment-management and workflow instructions. Keep those changing scientific dependencies and credentials out of the general ERDDAP deployment guide.

For a new repository structure, see the optional public processing VM template. It does not configure this VM or its persistent disk.