Sukanta Maikap

An overly simplified introduction to Google Deployment Manager. Part 1: A Simple Configuration

Assumption

This post assumes that you are familiar with the basic concepts of Google Cloud Platform, and you have gcloud configured in your system.

A walk down the memory lane

Remember the olden days of System Engineering? If you are not from the world of operations, you may remember it as following: when your request for a new environment consisting of a few services and a database(s), it took anywhere between a month and a quarter. Ever wondered why it used to take that long? The most probable reason(s) could be any or all of the below:

Need to provision new hardware.
The request is for an environment that was not maintained in-house before, thus requires extra IT resources.
Network capacity needs to be expanded.

etc.

Let’s call this the era of ClassicOps.

The dependency on hardware procurement essentially went away with the introduction of cloud computing. But the old habits took some time to disappear. Along came ClickOps. Manually installing servers in the server racks, cabling etc., was replaced by clicking on cloud provider web UI coupled with dirty scripts to provide a single VM or an entire clusters. The result was not significantly different from the classic style of operation. In the world of ClickOps, when a part of the cluster or the entire cluster required replacement, scaling it up generally took quite long and the process being manual was error-prone.

Introduction of Google Deployment Manager (GDM)

Out of the ashes of ClickOps, came the need for automated, version-controlled testable & consistently repeatable infrastructure that can be run on-demand, with minimal or no manual intervention. Thus, Infrastructure as Code (IaC) was born. AWS introduced CloudFormation, Azure introduce Resource Manager and Google did Google Deployment Manager a.k.a GDM.

Let’s get our hands dirty

GDM enables you to create repeatable configuration - write once and deploy anytime on any GCP project. Any action on GCP can be presented as a deployment manager config. GCP Marketplace uses the Deployment Manager to deploy applications in a project. Here is a generic format:

name: <<name of the GCP resource>>
type: type
properties:
  <<properties from api documentation for the respective type>>

Here is the config file for a VM.

resources:
- type: compute.v1.instance
  name: quickstart-deployment-vm
  properties:
    zone: us-central1-f
    # Replace [MY_PROJECT] with your project ID
    machineType: https://www.googleapis.com/compute/v1/projects/[MY_PROJECT]/zones/us-central1-f/machineTypes/f1-micro
    disks:
    - deviceName: boot
      type: PERSISTENT
      boot: true
      autoDelete: true
      initializeParams:
        # See a full list of image families at https://cloud.google.com/compute/docs/images#os-compute-support
        # The format of the sourceImage URL is: https://www.googleapis.com/compute/v1/projects/[IMAGE_PROJECT]/global/images/family/[FAMILY_NAME]
        sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
    # Replace [MY_PROJECT] with your project ID
    networkInterfaces:
    - network: https://www.googleapis.com/compute/v1/projects/[MY_PROJECT]/global/networks/default
      # Access Config required to give the instance a public IP address
      accessConfigs:
      - name: External NAT
        type: ONE_TO_ONE_NAT

Run the below command after saving the above content in a file named: 1.VM.yaml with correct project id. The below command assumes that your project id is: my-playground. Open terminal, change directory to where you have created the above file and run the below command:

» gcloud deployment-manager deployments create my-first-deployment --config 1.vm.yaml --project my-playground
The fingerprint of the deployment is b'VG5mdLUi9AlPLxUcErU6nw=='
Waiting for create [operation-1605991002319-5b4a3eb34ac10-e7212984-19cd9506]...done.
Create operation operation-1605991002319-5b4a3eb34ac10-e7212984-19cd9506 completed successfully.
NAME                      TYPE                 STATE      ERRORS  INTENT
quickstart-deployment-vm  compute.v1.instance  COMPLETED  []

You just created a deployment. A deployment can hold any number of GCP resources. In this case, we just created a vm as part of this deployment. To inspect the deployment from your terminal, run:

» gcloud deployment-manager deployments describe my-first-deployment                             
fingerprint: VG5mdLUi9AlPLnUcErU5nw==
id: '2003266554560472245'
insertTime: '2020-11-21T12:36:42.363-08:00'
manifest: manifest-1605991002374
name: my-first-deployment
operation:
  endTime: '2020-11-21T12:37:26.939-08:00'
  name: operation-1605991042319-5m4a9eb34ad10-e7232984-19cd9506
  operationType: insert
  progress: 100
  startTime: '2020-11-21T12:36:42.442-08:00'
  status: DONE
  user: sukanta******@****.com
NAME                      TYPE                 STATE      INTENT
quickstart-deployment-vm  compute.v1.instance  COMPLETED

Head over to GCP Console and navigate to Deployment Manager -> Deployments to inspect the deployment you have just completed.

Also, go to Compute Engine -> VM Instances and inspect the vm you have created. Tally the properties you have mentioned in the config file with the the VM.

If you prefer not to leave the terminal, run this command instead: (this assumes that you have gcloud cli installed & configured)

» gcloud compute instances describe quickstart-deployment-vm --project my-playground
No zone specified. Using zone [us-central1-f] for instance: [quickstart-deployment-vm].
cpuPlatform: Intel Haswell
creationTimestamp: '2020-11-21T12:36:57.017-08:00'
deletionProtection: false
disks:
- autoDelete: true
  boot: true
  deviceName: boot
  diskSizeGb: '10'
  guestOsFeatures:
  - type: VIRTIO_SCSI_MULTIQUEUE
  index: 0
  interface: SCSI
  kind: compute#attachedDisk
  licenses:
  - https://www.googleapis.com/compute/v1/projects/debian-cloud/global/licenses/debian-9-stretch
  mode: READ_WRITE
  source: https://www.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f/disks/quickstart-deployment-vm
  type: PERSISTENT
fingerprint: oyylmznUCwk=
id: '4564542966119256994'
kind: compute#instance
labelFingerprint: e6cOOJuM7_0=
labels:
  goog-dm: my-first-deployment
lastStartTimestamp: '2020-11-21T12:37:04.072-08:00'
machineType: https://www.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f/machineTypes/f1-micro
metadata:
  fingerprint: BkFihHjnksU=
  kind: compute#metadata
name: quickstart-deployment-vm
networkInterfaces:
- accessConfigs:
  - kind: compute#accessConfig
    name: External NAT
    natIP: 35.226.139.85
    networkTier: PREMIUM
    type: ONE_TO_ONE_NAT
  fingerprint: fcAV5A3Y4mn=
  kind: compute#networkInterface
  name: nic0
  network: https://www.googleapis.com/compute/v1/projects/my-playground/global/networks/default
  networkIP: 10.128.0.2
  subnetwork: https://www.googleapis.com/compute/v1/projects/my-playground/regions/us-central1/subnetworks/default
scheduling:
  automaticRestart: true
  onHostMaintenance: MIGRATE
  preemptible: false
selfLink: https://www.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f/instances/quickstart-deployment-vm
startRestricted: false
status: RUNNING
tags:
  fingerprint: 42WySvH8rSM=
zone: https://www.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f

To clean up, you need to delete the deployment. This will also delete all the underlying resources:

» gcloud deployment-manager deployments delete my-first-deployment --project my-playground  
The following deployments will be deleted:
- my-first-deployment

Do you want to continue (y/N)?  y

Waiting for delete [operation-1605991002319-5b4a3eb34ac10-e7212984-19cd9506]...done.
Delete operation operation-1605991002319-5b4a3eb34ac10-e7212984-19cd9506 completed successfully.

What just happened?

As the title suggests, I’m going to oversimplify a few concepts for the sake of ease of understanding. So, here it goes. When you deploy a GCP resource via deployment manager using glcoud cli, you are talking to a REST server, and your gcloud cli pointing to your project is the rest client. By supplying glcoud cli a yaml file, all you are doing is, you are defining a payload for the respective endpoint. Need proof? Here it is, do inspect the output of any deployment-manager command when there is an error:

.
.
.
  \ The referenced network resource cannot be found.\",\"statusMessage\":\"Bad Request\"\
    ,\"requestPath\":\"https://compute.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f/instances\"\
    ,\"httpMethod\":\"POST\"}}"

To reproduce this, all you need to do is reference a non-existing GCP resource from your payload. Refer to [Troubleshooting Guide](#Troubleshooting Guide) below for more details.

What’s Next?

This helps us largely automate infrastructure tasks. But only yaml file is relatively rigid. We can do even better, which will be the topic of part 2 of this series.

Troubleshooting Guide

This assumes that you have a GCP project with the default networks intact. If you have or the admin had deleted the default network in your project, you can do one of the below:

Create a VPC network named default with subnets created in automatic mode. This will result in GCP creating a subnet in all the available zones. You don’t need to change anything in the above file.
Create a VPC network in custom mode, and create a subnet in the region where you want to run this VM. It would be best if you modified the below entries from the above block [Let’s assume you want to create your VM in europe-north1 region and you have created a custom VPC network named europe-north1-test-network with a subnet in europe-north1 region]

zone: europe-north1
.
.
.
networkInterfaces:
    - network: https://www.googleapis.com/compute/v1/projects/my-playground/global/networks/europe-north1-test-network

You may not have access to create or update the network setup in your project. In that case, ask your GCP admin to give you the name of the VPC network and the subnet zone(s) under this network and modify the above block as below:

zone: [ALLOWED-ZONE]
.
.
.
networkInterfaces:
    - network: https://www.googleapis.com/compute/v1/projects/my-playground/global/networks/[SUPPLIED-NETWORK-NAME]

If you are wondering, “how would I know if I do not have the default network” or “when should I bother changing the above file”, the answer is if you run gcloud deployment-manager deployments create & get the below error. You can even be proactive and head over to VPC Network → VPC Networks and check if you have a network named default with a subnet in the region you are about to deploy this VM.

» gcloud deployment-manager deployments create my-first-deployment --config 1.vm.yaml --project my-playground
The fingerprint of the deployment is b'YsR5fSElcPzpm9BUOTLjVQ=='
Waiting for create [operation-1505980710779-5b4f3d9d41cfe-c3872e1c-c509e524]...failed.
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1505980710779-5b4f3d9d41cfe-c3872e1c-c509e524]: errors:
- code: RESOURCE_ERROR
  location: /deployments/my-first-deployment/resources/quickstart-deployment-vm
  message: "{\"ResourceType\":\"compute.v1.instance\",\"ResourceErrorCode\":\"400\"\
    ,\"ResourceErrorMessage\":{\"code\":400,\"errors\":[{\"domain\":\"global\",\"\
    message\":\"Invalid value for field 'resource.networkInterfaces[0].network': 'https://www.googleapis.com/compute/v1/projects/my-playground/global/networks/default'.\
    \ The referenced network resource cannot be found.\",\"reason\":\"invalid\"}],\"\
    message\":\"Invalid value for field 'resource.networkInterfaces[0].network': 'https://www.googleapis.com/compute/v1/projects/my-playground/global/networks/default'.\
    \ The referenced network resource cannot be found.\",\"statusMessage\":\"Bad Request\"\
    ,\"requestPath\":\"https://compute.googleapis.com/compute/v1/projects/my-playground/zones/us-central1-f/instances\"\
    ,\"httpMethod\":\"POST\"}}"

The Code

You can find all the code used in this post in GitHub.

Reference

Published Nov 29, 2020

I'm a father, husband, life long learner, maker / hacker, avid reader, traveller, photographer and foodie in this exact order of priority.