[go: up one dir, main page]
Open in app

Sign in

Write

Sign in

Building a serverless application on Google Cloud to accelerate deployment time with reduced cost and complexity

McKinsey Digital
McKinsey Digital Insights
Published in
14 min readFeb 15, 2024

--

By Maksood Mohiuddin — Senior Cloud Engineer II, Alex Dequevedo — Senior DevOps Engineer I, Thomas Neff — Senior DevOps Engineer I, and Marco Marulli — Principal Lead II

Creating a modern, three-tier serverless application on a cloud service provider (CSP) given a containerized application is a common task for many cloud engineers. The challenge lies in being able to efficiently leverage the CSP’s breadth of services for different application tiers while expediting development efforts and reducing costs and complexity. In this article, we provide a step-by-step guide for deploying a modern, three-tier serverless application on Google Cloud that accelerates deployment time while reducing cost and complexity, along with detailed explanations of the services used in this implementation.

Architecture overview

We will use an example application titled “Amazing Employees,” in which we employ a three-tier microservices architecture that comprises the frontend (the presentation tier), backend (application tier), and a Firestore database (data tier). The frontend, which is developed using Angular, interacts with users, while the backend, which is built with Python Flask, manages business logic. “Amazing Employees” consists of a single frontend and backend container. However, microservices architectures are scalable: adding more containers can facilitate more features. Infrastructure for the application is defined through Terraform, and Cloud Build manages the application’s building and deployment.

Figure 1: Google Cloud three-tier serverless architecture.

Why serverless?

Serverless technology offloads much of the complex work of managing infrastructure to CSPs, enabling developers to concentrate on creating and deploying application code. Serverless offerings provide many benefits, including the following:

  • nearly infinite automatic scaling
  • high availability
  • simplified infrastructure management
  • potential cost savings (pay only for what you use)
  • faster development cycles

Serverless technologies help enable several highly scalable architecture types, such as a microservices architecture. Microservices break down applications into smaller, independently deployable units. With the ability to scale individual microservices dynamically and isolate faults, developers can create highly responsive and cost-effective solutions. While serverless technology is not required to create a microservices application, it greatly simplifies the management and scaling aspects, making it an ideal choice for organizations looking to optimize resource utilization and streamline development efforts.

Another common architecture for serverless applications is an event-driven architecture, in which applications respond dynamically to events triggered by various sources, such as user actions, data changes, or external system interactions. It allows developers to design systems that automatically scale in response to incoming events, ensuring optimal performance during peak loads and cost savings during quieter periods.

Application services

For the frontend and backend of our “Amazing Employees” application, we leverage Cloud Run, a fully managed, serverless container environment from Google Cloud. Cloud Run is based on Knative, a serverless container technology that Google Cloud pioneered. Cloud Run is often easier to use than other container orchestration services, such as Google Kubernetes Engine (GKE), because it abstracts the infrastructure layer away and there is no need to manage the underlying Kubernetes clusters. Cloud Run automatically scales containers based on incoming traffic. The latest version of Cloud Functions — another popular choice to build serverless frontends or backends in Google Cloud — uses Cloud Run behind the scenes.

Of course, what you gain in simplicity, you lose in control. This is true for all managed services in the cloud. If your use case requires you have full control over your containers (such as configuring networking, scaling, and resource allocation at a granular level), you may need to use GKE instead of Cloud Run. Whenever possible, we recommend using fully managed services such as Cloud Run because, simply put, they make your life much easier.

API Gateway is used to secure traffic between frontend and backend Cloud Run instances. Generally speaking, using API Gateway is a best practice when building a microservice architecture because it improves security, performance, and flexibility while reducing complexity. Google Cloud’s API Gateway has three main components: the API, the gateway, and the config. The config uses OpenAPI specs and can be viewed in the infra directory. Note that two variables are used, which allows us to deploy the config without making manual changes. The first variable is ${project_id}, which is provided in variables.tf, and the second is ${url}. The ${url} variable value is provided by a Terraform data block that checks the URL of the backend Cloud Run instance and adds it to the config. In both cases, variable substitution is done through Terraform and can be viewed in infra/api-gateway.tf.

For the application database, Firestore was a natural choice due to its seamless integration into our serverless architecture. Firestore, a NoSQL database from Google Cloud’s Firebase service, was selected primarily for its ability to offer automatic scaling, robust querying support, advanced security rules, and dependable application authentication. In the context of our application, we use Firebase Authentication for user logins and employ basic CRUD database operations for managing employee data.

Infrastructure and deployment

For infrastructure as code, we use Terraform, an open-source tool that HashiCorp developed. Terraform allows you to define, manage, and provision infrastructure resources across various cloud providers, including Google Cloud, in a declarative manner.

Optionally, you can use Cloud Storage to store Terraform state. Cloud Storage is an object storage service within Google Cloud that can store unstructured data such as files and media and offers scalability to manage vast amounts of data across multiple storage classes. Cloud Storage offers data durability through replication and provides accessibility with low latency; security features such as access control and encryption; and versioning, life cycle management, and integration with other Google Cloud services.

To build the frontend and backend for the application, we use Cloud Build, a serverless continuous integration (CI) and continuous delivery (CD) platform from Google Cloud that allows you to execute builds in Google Cloud.

To store the container image, we leverage Artifact Registry, a managed service from Google Cloud that provides secure and scalable repositories for software artifacts such as packages for Java, Node, or Python, but in this case, we will use it to store Docker images.

Below is a description of the steps and a high-level diagram showing the flow of the Cloud Build job:

  1. User triggers Cloud Build from local repository using Google Cloud commands.
  2. Docker image is built from the current working directory.
  3. Docker image is pushed to Artifact Registry.
  4. Cloud Run instances are deployed from the Docker image.
Figure 2: Cloud Build execution flow.

A note about cost

This article is tailored to a cloud engineer who likes to explore serverless offerings from Google Cloud. As such, a typical deployment of the application for noncommercial use should be covered under Google Cloud Free Program. You can visit the Google Cloud free-tier products page to learn more: https://cloud.google.com/free. Following are the services and the related cost (as of September 2023) we will use for this tutorial:

  • Cloud Run free tier offers 2 million requests monthly, 360,000 GB-seconds of memory, 180,000 vCPU-seconds of compute time, and 1 GB network egress (North America only).
  • Firestore free tier offers 1 GB storage per project and 50,000 reads, 20,000 writes, and 20,000 deletes per day, per project.
  • API Gateway is not part of the free-tier offering, but it offers up to 2 million API calls per billing account per month for free.
  • Cloud Build free tier offers 120 build minutes per day.
  • Artifact Registry free tier offers 0.5 GB storage per month.

Here, we will explore one example of a cost estimation for our serverless three-tier web application with example numbers. Actual costs will vary depending on usage.

We will use the following assumptions for our cost estimation:

  • Cloud Run is used as a service, with CPU allocated only when running.
  • Cloud Run instances have 1vCPU and 512 MB memory.
  • There are 10,000 monthly users.
  • Each user averages 400 requests per month.
  • Each request takes 1,000 milliseconds to complete.
  • 60 percent of requests are database reads.
  • 40 percent of the requests are database writes.
  • Firestore has less than 15 GB of data.
  • All resources are deployed in the same region.

API Gateway:

2 million to 1 billion requests per month = $3.00

Firestore:

15 GB storage = $2.52 per month

Document reads:

50,000

$0.036 per 100,000 documents

Document writes:

20,000

$0.108 per 100,000 documents

100 users * 40,000 requests per month = 4 million requests per month ≈ 2.4 million reads and 1.6 million writes per month

Read cost: 2.4 million reads / 30 ≈ 80,000 requests per day — 50,000 free requests = 30,000 paid requests per day = $0.036 per day for read requests * 30 = $1.08 per month

Write cost: 1.6 million writes / 30 ≈ 54,000 requests per day — 20,000 free requests = 34,000 paid requests per day = $0.216 per day for write requests * 30 = $6.48 per month

Egress:

Within same region = $0.00

Cloud Run:

CPU allocation time: 200,000 vCPU-second = $0.48

Memory allocation time: 100,000 GiB-second = $0.00

Number of requests: 4,000,000 = $0.80

Cloud Run total: $1.28 * 2 instances = $2.56 per month

Monthly costs:

API Gateway: $3.00

Firestore: $10.08

Egress: $0.00

Cloud Run: $2.56

Total = $15.64 per month

Step-by-step guide

Setting up the Google Cloud project

Projects are required to manage and organize resources in Google Cloud. We recommend creating a new project for this tutorial. This tutorial assumes that you have a Google Cloud project with admin (owner) access and you have access to Google Cloud Shell with either knowledge of command-line editors such as nano or vim or access to the Cloud Shell Editor. Cloud Shell Editor is a text editor built right into Cloud Shell and is a great option if you are not familiar with command-line editors.

To access your project in the Google Cloud console, navigate to the Google Cloud console, log in (or create an account), and select your project. If you do not already have a Google Cloud project, navigate to Menu > IAM & Admin > Create a Project and follow the prompts in the console.

Initial setup

For simplicity, we use a single repository to hold all of the application, infrastructure, and pipeline code. For the initial setup, open a Cloud Shell session in the Google Cloud console for the project. Cloud Shell represents an internet-based platform for development and operational tasks and can be conveniently reached through a web browser. Google Cloud offers this service at no cost. Within this platform, you can oversee your assets using an online terminal that is equipped with various tools. For an in-depth dive of the preinstalled tools, please see https://cloud.google.com/shell/docs/how-cloud-shell-works#tools.

You can open a new Cloud Shell session by visiting https://console.cloud.google.com/?cloudshell=true.

If not already updated, update Google Cloud Shell console to use your project:

gcloud config set project [YOUR GOOGLE CLOUD PROJECT]

gcloud config list

Step 1: Clone the GitHub repository

In the Cloud Shell session, git clone the reference application. You can clone this repository to start. However, we recommend you fork the repository and clone the forked repository. As part of this step-by-step guide, we will provide instructions on how to change the code to deploy the three-tier serverless application in your Google project.

git clone [original https://github.com/maksoodmohiuddin/google-cloud-serverless-app-pattern.git or forked repo] cd google-cloud-serverless-app-pattern

From here you can select “Open Editor” to launch the Cloud Shell editor.

Step 2: Update Google project ID and region

Next, let’s update project_id and location with the Google Cloud project you are using and a target Google Cloud region (such as “us-west2”) where you would like to deploy this application. Our example shows nano as an editor. However, you can choose to use another available editor that you prefer.

cd infra
nano variables.tf

variable "project_id" {
  description = "Project ID of the GCP project where resources will be deployed"
  type        = string
  default     = "PLEASE UPDATE WITH YOUR GOOGLE PROJECT ID"
}

variable "location" {
  description = "Location (region) where resources will be deployed"
  type        = string
  default     = "PLEASE UPDATE YOUR GOOGLE CLOUD REGION"
}

Please choose a region that supports Cloud Build, Cloud Run, Firebase, API Gateway, and Artifact Registry. For personal accounts, Cloud Build is restricted to certain regions. Therefore, if you are using a personal account, we recommend using one of the regions below:

  • us-west2
  • asia-east1
  • australia-southeast1

To learn more about Google Cloud regions and supported regions for the services for this app, please visit the following:

Deploying initial infrastructure

In this section, we will deploy the initial infrastructure with Terraform. Due to dependency mapping that is not known to Terraform, we will need to target specific resources and set up the application sequentially.

By default, Terraform stores state locally in a file named terraform.tfstate. For purposes of this tutorial, you can use the local state. However, if you prefer, state can be stored in remote state using Google Cloud Storage. Remote state is always preferred for projects with multiple developers. If you are especially savvy with Terraform, you may find it easier to create the bucket with Terraform local state, and then migrate the state into the bucket. See the Google Cloud documentation for more details.

Step 1: Initialize Terraform

terraform init

Step 2: Validate Terraform plan

terraform plan

If prompted to authorize Cloud Shell, please select “Accept” to continue. This is because Cloud Shell needs permission to use your credentials for the Google Cloud command, and clicking “Authorize” grants permission to this and future calls.

The output should be the following:

Plan: 18 to add, 0 to change, 0 to destroy

Step 3: Apply Terraform for the initial resources

terraform apply

When prompted, type “yes” to deploy the resources. It should match the plan.

Deploying backend, API Gateway, and frontend

Step 1: Update the backend Python Flask application code to use your Google project

Navigate to the backend folder and open the firestore.py file:

cd ../app/backend/ nano -l firestore.py

Update line 10 client with the Google Cloud project you are using:

client = firestore.Client(project="PLEASE_UPDATE_PROJECT_ID")

Step 2: Trigger Cloud Build to deploy backend

First, you can review the Cloud Build file and use the Google Cloud command line to deploy the backend Cloud Run based on the Cloud Build file:

cat backend-cloudbuild.yaml gcloud builds submit --region=[YOUR GOOGLE CLOUD REGION] --config backend-cloudbuild.yaml e.g. gcloud builds submit --region=us-west2 --config backend-cloudbuild.yaml

Note that the Cloud Build run can take several minutes to finish the run. This is expected. This Cloud Build file builds a Docker image for the backend service, pushes it to Artifact Registry, and then deploys the images from the Artifact Registry on a Cloud Run instance.

Step 3: Review the Swagger specification

We use API Gateway with OpenAPI Swagger specifications to connect the backend Cloud Run to API Gateway and set up the API Gateway configuration. Review the api-gateway--espv2-definition.yml.tmpl file under the infra folder.

First, use cd to go back to the infra directory and enable the enable_api_gateway flag to true:

cd ../../infra cat api-gateway--espv2-definition.yml.tmpl

You can learn more about OpenAPI specifications for API Gateway, Swagger, and Firebase as authentication for API Gateway by visiting the following:

https://cloud.google.com/api-gateway/docs/openapi-overview

https://swagger.io

https://cloud.google.com/api-gateway/docs/authenticating-users-firebase

Step 4: Deploy API Gateway

While still in the infra directory, enable the enable_api_gateway flag to true:

nano variables.tf

variable "enable_api_gateway" { description = "Feature flag to enable/disable API Gateway. Leverage this to deploy infra sequentially." type = bool default = true }

Run the Terraform plan and then apply:

terraform plan
terraform apply

When prompted, type “yes” to deploy the resources. It should deploy three resources. Note that API Gateway typically takes longer to deploy than other resources. This is normal.

Step 5: Configure Firestore database via Firebase console

Navigate to https://console.firebase.google.com/ and Create/Add Project > Select your Google Cloud project from the dropdown > Accept Terms > Continue > Confirm Plan > Choose to enable Google Analytics or not > Continue.

In the landing page of your Firebase project, add your app as a web app to Firebase.

Alternatively, under Project Overview, click on the gear icon, navigate to Project Settings, and register your app as a web app to Firebase.

Register your app with your chosen name (for example, you can use your Google project name).

Once your app is registered, under the SDK setup and configuration section (you can toggle to the configuration section), copy the value for const firebaseConfig and paste the value into app/frontend/src/environments/environments.ts.

Warning: Make sure to not check in the values of environments.ts into a git repository because this is sensitive information.

cd ../app/frontend/src/environments/ nano -l environment.ts

export const environment = { firebase: { apiKey: "PLEASE UPDATE", authDomain: "PLEASE UPDATE", projectId: "PLEASE UPDATE", storageBucket: "PLEASE UPDATE", messagingSenderId: "PLEASE UPDATE", appId: "PLEASE UPDATE", measurementId: "PLEASE UPDATE" // if measurement is enabled }, production: false };

While still in Firebase console, enable Google as an authentication provider for Firebase with the following steps: All Products > Authentication > Get Started > Google > Enable > Add Support Email > Save.

Step 6: Set the frontend API Gateway configuration

First, run the following command to get the api-gateway configuration:

gcloud api-gateway gateways describe employee-gateway --location LOCATION --project PROJECT_ID --format 'value(defaultHostname)'

The output will look like the following: employee-gateway-#.#.gateway.dev

Then update the three URLs (lines 13–15) in app/frontend/src/employee/services/firestore.service.ts; keep the path (for example, employee) as is.

cd ../employee/services

nano -l firestore.service.ts

addEmployeeUrl = 'https://employee-gateway-#.#.gateway.dev/employee'; employeesUrl = 'https://employee-gateway-#.#.gateway.dev/employees'; deleteEmployeeUrl = 'https://employee-gateway-#.ue.gateway.dev/employee';

Step 7: Trigger Cloud Build to deploy the frontend

Navigate to the frontend folder, review the Cloud Build file, and use the Google Cloud CLI to deploy the backend cloud instance.

cd ../../.. cat frontend-cloudbuild.yaml gcloud builds submit --region=[YOUR GOOGLE CLOUD REGION]] --config frontend-cloudbuild.yaml e.g. gcloud builds submit --region=us-west2 --config frontend-cloudbuild.yaml

Step 8: Update Firebase authorized domain with frontend Cloud Run URL

The frontend Cloud Run URL needs to be authorized for OAuth operations in the Firebase app.

You can use Google Cloud to get the Cloud Run URL:

gcloud run services describe SERVICE --region REGION --format 'value(status.url)'

e.g. gcloud run services describe amazing-employees-frontend-service --region us-west2 --format 'value(status.url)'

The output should be the following: https://amazing-employees-frontend-service-###.a.run.app

Copy that URL and add to the OAuth redirect domains list in the Firebase console.

Browse to https://console.firebase.google.com/ and select your project.

Click on Authentication > Settings Tab > Authorized Domains > Add Domain, and paste your frontend Cloud Run URL.

Final step: Validate end-to-end app

Browse to your frontend Cloud Run URL (https://amazing-employees-frontend-service-###.a.run.app), and use a Google account to log in to the app — your Google Cloud serverless application pattern using Cloud Run, API Gateway, and Firebase is up and running.

Clean up

We will not be able to run a Terraform destroy here due to API Gateway configuration dependencies.

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the project.

Deleting a project has the following consequences:

  • If you used an existing project, you’ll also delete any other work you’ve done in the project.
  • You can’t reuse the project ID of a deleted project. If you created a custom project ID that you plan to use in the future, delete the resources inside the project instead. This ensures that URLs that use the project ID, such as an appspot.com URL, remain available.

To delete a project, do the following:

  • In the Cloud console, go to the projects page (https://console.cloud.google.com/iam-admin/projects).
  • In the project list, select the project you want to delete and click “Delete.”
  • In the dialog, type the project ID and click “Shut down” to delete the project.

Summary

This article delved into the deployment of a serverless three-tier application on Google Cloud. We explored how to use Cloud Run for both an Angular containerized frontend and a Python Flask containerized backend while leveraging Firestore for our database needs. Terraform and Cloud Build were highlighted for efficient infrastructure setup and deployments. For those eager to be hands-on, we provided a detailed step-by-step guide to ensure you can replicate and understand each phase of the deployment.

By understanding these key components and technologies, readers should be better poised to tackle serverless projects on Google Cloud.

Google Cloud
Serverless Cloud
Service Provider

--

--

McKinsey Digital
McKinsey Digital Insights

Written by McKinsey Digital

9.1K Followers

Creating value by reinventing the core, together. www.mckinseydigital.com

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams