Deploying with Google Cloud Platform

This document walks you through how to deploy the WhatsApp Business API on Google Cloud Platform (GCP).

Get Started

Step 1: Set up a GCP account and project

Set up a Google Cloud Platform account and create a new project within it.

Step 2: Enable Google Compute Engine API

Enable the Google Compute Engine API for your project in the GCP console. Make sure to select the project you are using to follow this tutorial and click the Enable button as below:

After you enable it, you are be able to see API enabled green check mark like below:

Step 3: Enable Google Kubernetes Engine API

Enable Google Kubernetes Engine APIs for your project in the GCP console. Make sure to select the project you are using to follow this tutorial and click the Enable button.

Step 4: Enable Cloud Resource Manager API

Enable Cloud Resource Manager API for your project in GCP Console and make sure to select the project you are using to follow this tutorial and click the Enable button.

Step 5: Install gcloud CLI

Download and install gcloud CLI by following link.

Step 6: Create service account and key

Create a service account and service account key.

Step 7: Setup gcloud credential

Download the service account key to local and export the local path to GOOGLE_APPLICATION_CREDENTIALS environment variable.

Example command:

 export GOOGLE_APPLICATION_CREDENTIALS=/your_local_path/key.json 

Step 8: Authorize gcloud

Authorize gcloud to use service account and key-file credentials by running command:

gcloud auth activate-service-account SERVICE_ACCOUNT@DOMAIN.COM —key-file=/your_local_path/key.json —project=PROJECT_ID

Step 9: Initialize gcloud CLI with the service account instead of a user account:

• Step 9.1: Run gcloud init on your console

• Step 9.2: Follow the instruction promoted, Select Create a new configuration

• Step 9.3: Choose service account you just created in above step

• Step 9.4: Choose project id you just created in above step

• Step 9.5: Skip Choose a default Compute Engine zone

• Step 9.6: Once complete initialization, it shows "Your Google Cloud SDK is configured and ready to use"

Step 10: Install Terraform

Step 11: Install kubectl and plugin

Install kubectl and gke-gcloud-auth-plugin by running command:

gcloud components install kubectl

Deployment

Step 1: Download template code

Download all GCP Terraform files from GitHub Repo put them in a directory, for example.: ~/biz/gcp.

Step 2: Configure parameters

Configure parameters in the Update the default value in file variables.tf for each parameter in Template Configuration.

Step 3:

Comment / uncomment the corresponding innodb_buffer_pool_size, innodb_buffer_pool_instances, query_cache_size values in scripts/my.cnf file.

For example, if you configure the throughput value to be 300, its DB machine type is n2d-highmem-32 and you should uncomment the settings for throughput [250, 300] and comment settings for other throughput.

#current value is based on throughput [250, 300], map_db_class n2-highmem-32 (32vCPU, 256G)
innodb_buffer_pool_size=206158430208
innodb_buffer_pool_instances=32
query_cache_size = 17179869184

#current value is based on throughput [200], map_db_class n2-highmem-16 (16CPU, 128GB)
#innodb_buffer_pool_size=103079215104
#innodb_buffer_pool_instances=32
#query_cache_size = 8589934592

#current value is based on throughput [20,40,80,120,160], map_db_class n2-highmem-8 (8vCPU, 64G)
#innodb_buffer_pool_size=51539607552
#innodb_buffer_pool_instances=16
#query_cache_size = 4294967296

# current value is based on throughput [10], map_db_class n2-highmem-4(4vCPU,32GB)
# innodb_buffer_pool_size=25769803776
# innodb_buffer_pool_instances=16
# query_cache_size = 2147483648

Step 4: Go to the root directory created above

This is the directory that includes *.tf file, for example: ~/biz/gcp.

Step 5: Run terraform init

Run command terraform init. This command installs external dependencies required by the template. Optionally, run terraform validate to validate if the template syntax is correct.

Step 6:(Optional) Run terraform plan

Run command terraform plan. This command generates a plan and checks if all required variables are properly set. Make sure there is no error shown in this step.

Step 7: Run terraform apply

Run command terraform apply -var="nfs-pvc-creation-complete=false" -auto-approve. The Kubernetes cluster may take about 5-10 minutes to provision. You can check the provisioning status from the GCP console. Check if a green check mark shows under Status to verify the cluster is ready to use.

Also before proceed, please Check MySQL innodb_buffer_pool_size matches your configuration.

Step 8: Kubectl authentication

Either obtain the output of Step 7 or execute the command terraform output to display the cluster authentication output.

Example of Step 7's output:

cluster_authentication = "gcloud container clusters get-credentials YOUR-CLUSTER-NAME —zone ZONE-YOU-SPECIFIED"

Grep the content within the double quotes on the right side of the equation, run as a command to authenticate cluster. Command example:

gcloud container clusters get-credentials project_id-name_prefix-xxx-gke —zone us-west1-b

After authentication, you are able to continue deploy k8s resources and execute all kubectl command.

Step 9: Run kubectl apply

Run command kubectl apply -f storage.yaml. This command creates an NFS PVC and make NFS ready for use. This is a workaround to enable NFS for our use case due to a Terraform known issue.

Step 10: Run terraform apply again

Run command terraform apply -auto-approve. Once this is completed, all the deployments are completed.

Upon successful creation of stack, the following parameters are displayed:

• app-machine-type: coreApp, webApp compute instance type you are using
• cluster_authentication: command to authenticate cluster, the value is determined by cluster name and zone
• db-machine-type: database compute instance type you are using
• webapp_lb_ip: The host name to access the WA Business API endpoints. The API endpoint root URL can be accessed via https://webapp_lb_ip
• monitor_lb_ip: The IP address to access the Grafana dashboard. The Grafana dashboard can be accessed via http://monitor_lb_ip:3000
• number_of_shards: Number of shards to configure for the WhatsApp Business API
• throughput: throughput you configured, based on this value, template determines app-machine-type and db-machine-type and number_of_shards

Template Configuration

General Configuration

Throughput Configuration

WhatsApp Business API Configuration

Database Configuration

Grafana Configuration

WhatsApp Business Platform API Client Configuration

After the WhatsApp Business Platform API client is successfully deployed, it needs to be configured to bring it into operation.

Step 1: Phone Registration

For more in-depth information about phone number registration, see Phone Number guide.

Download the base64-encoded certificate from your WhatsApp account in the Facebook Business Manager under the Phone Numbers tab of the WhatsApp Manager. Once you have the correct phone number selected and have the base64-encoded certificate, you need to register the WhatsApp Business API client through the account node. For more information, see the Registration documentation.

Step 2: Set Shards

You need to use the shards endpoint to increase the number of active Coreapp instances to achieve the desired throughput. The number of shards can be found in the Output section of the stack.

Step 3: Update Application Settings

The configuration of WhatsApp Business Platform API web callbacks and other parameters is described in the Application Settings documentation. To achieve stable throughput, the following application settings are recommended.

{
    "settings": {
        "application": {
            "callback_backoff_delay_ms": 3000,
            "callback_persist": true,
            "db_garbagecollector_enable": false, # change this to true when there are no ongoing messaging campaigns
            "heartbeat_interval": 5,
            "max_callback_backoff_delay_ms": 900000,
            "media": {
                "auto_download": [
                    "document",
                    "image",
                    "video",
                    "voice",
                    "sticker",
                    "audio"
                ]
            },
            "notify_user_change_number": true,
            "pass_through": false,
            "sent_status": true,
            "show_security_notifications": false,
            "skip_referral_media_download": false,
            "unhealthy_interval": 30,
            "wa_id": "12245552741",
            "webhooks": {
                "max_concurrent_requests": 24,
                "message": {
                    "delivered": true,
                    "read": true,
                    "sent": true
                },
                "url": "<YOUR_WEBHOOK_SERVER_URL>"
            }
        }
    },
    "meta": {
        "api_status": "stable",
        "version": "2.41.3"
    }
}
    
    

You will need to restart all CoreApp pods after configuring the highlighted parameters above for them to take effect. You can do this, use the following command:

kubectl rollout restart deployment coreapp

Wait until all CoreApp pods are back in RUNNING status.

Step 4: Validation of the Setup

Once the configuration and registration steps are successful, a message can be sent and received to validate the basic functionality of WhatsApp Business API client. For more information, see Messages.

Upon successful receipt of a message, the WhatsApp Business API client will POST the message status/details to the Webhook configured in Step 3.

If your message was successfully received, you have successfully completed the configuration set up. For more information on the available API endpoints, see the Reference documentation.

Updating

If you need to upgrade to a newer API version or update the template with any adjusted parameters, you need to update the API version as followed in the variables.tf file and run terraform validate to validate if the template syntax is correct.

variable "api-version" {
  default = "v2.45.2"
}

From the root directory, run terraform apply and all changes are updated automatically.

Destroying

You can use terraform destroy to destroy all resources created by the Terraform plan. At the prompt, enter yes.

  Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
    

Troubleshooting

Issue: Error 403: Insufficient regional quota to satisfy request: resource "N2_CPUS"

Solution: Quota is related to your GCP account, it may vary from Big company and small company or individual developer or free trial account. Below is the required quota list under specific region for >=250 mps for image2MB_or_image4MB:

How to increase Quota: for example if you want to IncreaseVM instance N2 CPU Quotas >= 388 for the region each cluster you want to deploy, you need to:

• Go to Quotas
• Check Compute Engine API and N2 CPUs in specific region option
• Click Edit Quotas to modify quotas

Issue: WARNING: cluster is not RUNNING. The kubernetes API may or may not be available. Check the cluster status for more information.

Solution: Wait for 5-15 mins, go to https://console.cloud.google.com/kubernetes/list/overview and check if a green check mark shows under “Status” to verify the cluster is ready to use, make sure there is no “Repairing the cluster” going on like below.

Below is what we expected (all status green, no pending):

Issue: Dial tcpXXX connect: connection refused

Error example:

Error: Get "https://35.223.76.189/api/v1/namespaces/default/secrets/secret-env": dial tcp [35.223.76.189](tel:3522376189):443: connect: connection refused

Solution: Run cluster authentication command:

gcloud container clusters get-credentials ClusterName —zone Zone

You can run command terraform output, and obtain the right part of the "=". Below is a terraform output result example:

cluster_authentication = "gcloud container clusters get-credentials project-id —zone your-zone" 

Then you need to run command:

gcloud container clusters get-credentials project-id —zone your-zone

Terraform preserves already-deployed services even if the plan fails halfway. After you make a change in the same plan, running terraform apply resumes from the failed resources.

Issue: Cannot connect to the Kubernetes cluster created

Solution::

• Step 1: run command terraform output, you can see the cluster_authentication output
• Step 2: Obtain the content within the double quotes on the right side of the equation in step 8, run as a command to authenticate cluster. Example command: gcloud container clusters get-credentials cluster_name —zone your_zone
• Step 3: After authentication, you are able to continue deploy k8s resources and execute all kubectl command
• Step 4: Once you are connected to the cluster, you can run kubectl commands to manage the cluster. For example:

To list out all pods: kubectl get pod; To get logs of a pod: kubectl logs pod_name; To get logs of a container of a pod: kubectl logs pod_name -c container_name

Issue: Machine Overview Grafana dashboard shows “No Data” for all charts

Reason: This is a known issue and further work is required to fix this.

Mitigation: You can check machine specific metrics from the GCP console instead. To check CoreApp / Web / DB machine metrics:

1. You can create dashboar by following https://cloud.google.com/monitoring/charts/dashboards
2. Filter the by cluster name and timespan

Below is example to monitor CPU usage and memory and IO:

Issue: Check MySQL innodb_buffer_pool_size

You can Login to MySQL by running command kubectl exec —stdin —tty db-0 — /bin/bash and in the bash console, run command mysql —user=USER —password=PASSWORD to login MySQL, then run SELECT @@innodb_buffer_pool_size to get the value.

Issue: Deployment cannot reach expected throughput

• Initially, ensure that you only modify the content within the variable.tf file and refrain from altering the template code. The template's performance has been evaluated with regards to the compute instance type, node/pod quantity, disk size, and setup parameters specified in the template. If any modifications are made to the template, official support cannot be guaranteed.

• Next, it is essential to enable"db_garbagecollector_enable“ for a minimum of 6 hours ("db_garbagecollector_enable": true) or call API node: /services/message/gc before executing champion. However, once you initiate champion, it should be disabled ("db_garbagecollector_enable": false). Please note that after configuring the above parameters, you need to restart all CoreApp pods for the changes to take effect. This can be achieved by using the provided command kubectl rollout restart deployment coreapp. And wait until all CoreApp pods are back in RUNNING status.

• Finally, you need to run and call the contacts node on all the recipient numbers before starting the outbound load test. This will ensure the most optimum load test setup.before starting high throughput campaigns.