Developer Setup: Multiconnect on Minikube

This document shows you how to set up a developer machine installation of the WhatsApp Business API client with Multiconnect on Minikube. Before you start any of these set ups, check our list of requirements.

To set up a High Availability cluster, follow these steps:

1. Create a biz directory for the setup scripts
2. Get the WhatsApp Business API Client configuration files
3. Create a MySQL database
4. Create a whatsapp-config secret
5. Create a local media volume
6. Set the $VERSION environment variable
7. Deploy the containers
8. Find the webapp container port
9. Get an authentication token
10. Perform a health check
11. Register the WhatsApp Business API Client
12. Perform a second health check

To set up a Highly Available multiconnect cluster, follow these steps:

1. Set up two shards
2. Perform a health check
3. Start a third coreapp to maintain High Availability
4. Perform a second health check

Once you have completely set up your instance, you can choose to upgrade it. To uninstall the client, follow these steps.

Before You Start

You will need:

• A Kubernetes cluster set up locally using minikube
• The kubectl command line tool
• A locally set up test account in a development environment
  
  • This is for fast development and to test new releases.

It's also highly recommended you read through the Availability and Scaling guide to learn more about High Availability and Multiconnect.

Setup of a High Availability Cluster

Step 1: Create a biz Directory for the Setup Scripts

Run the following code in your preferred location for the WhatsApp Business API client:

mkdir ~/biz; cd ~/biz;

Step 2: Get the WhatsApp Business API Client Configuration Files

Clone all the configuration files from the installation/kubernetes directory of the WhatsApp-Business-API-Setup-Scripts GitHub repository to the ~/biz directory you created in Step 1.

Step 3: Create a MySQL Database

Create a MySQL database with a persistent volume using:

kubectl apply -f mysql.yaml

The resulting output should look like the following:

persistentvolume/mysql-volume created
persistentvolumeclaim/mysql-volume-claim created
service/mysql-service created
deployment.extensions/mysql-deployment created

You can verify MySQL is running with:

kubectl get pods

The resulting output should look like the following:

NAME                               READY   STATUS    RESTARTS   AGE
mysql-deployment-5d4f898-xtkpj     1/1     Running   0          53m

Step 4: Create a whatsapp-config Secret

Create a whatsapp-config secret from the db.env file with the following command:

kubectl create secret generic whatsapp-config --from-env-file=db.env

The resulting output should look like the following:

secret/whatsapp-config created

You can check the secret was created with:

kubectl get secrets

The resulting output should look like the following:

NAME                TYPE        DATA        AGE
whatsapp-config     Opaque      5           52s

You can check the details of the secret with:

kubectl describe secrets/whatsapp-config

The resulting output should look like the following:

Name:         whatsapp-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
wa-db-password:  8 bytes
wa-db-port:      4 bytes
wa-db-username:  4 bytes
wa-db-engine:    5 bytes
wa-db-hostname:  13 bytes

Step 5: Create a Local Media Volume

To send or receive media messages, the WhatsApp Business API client requires a persistent volume shared between the Webapp, Master, and Coreapp deployments. Create a PersistentVolume and a PersistentVolumeClaim for media messages by running:

kubectl apply -f volume.yaml

This command will set up a local media volume mounted at /usr/local/wamedia inside the Kubernetes cluster with a size of 1GB and request 1GB physical size from it through a PersistentVolumeClaim for development and testing purposes.

You can check the status of media volume and media volume claim with:

kubectl get pv media-volume

kubectl get pvc media-volume-claim

Step 6: Set the $VERSION Environment Variable

Replace the $VERSION environment variable with the latest WhatsApp Business API client version (e.g., 2.23.4) in the containers section of the webapp.yaml, master.yaml, and coreapp.yaml files.

webapp.yaml

containers:
- name: whatsapp-web
  image: docker.whatsapp.biz/web:v2.23.4

master.yaml

containers:
- name: whatsapp-master
  image: docker.whatsapp.biz/coreapp:v2.23.4

coreapp.yaml

containers:
- name: whatsapp-coreapp
  image: docker.whatsapp.biz/coreapp:v2.23.4

Step 7: Deploy the Containers

Deploy the Webapp, Master, and Coreapp containers by running the following commands:

Webapp

kubectl apply -f webapp.yaml

The resulting output should look like the following:

horizontalpodautoscaler.autoscaling/whatsapp-web-autoscaler created
service/whatsapp-web-service created
deployment.apps/whatsapp-web-deployment created

Master

kubectl apply -f master.yaml

The resulting output should look like the following:

horizontalpodautoscaler.autoscaling/whatsapp-master-autoscaler created
service/whatsapp-master-service created
deployment.apps/whatsapp-master-deployment created

Coreapp

kubectl apply -f coreapp.yaml

The resulting output should look like the following:

horizontalpodautoscaler.autoscaling/whatsapp-coreapp-autoscaler created
service/whatsapp-coreapp-service created
deployment.apps/whatsapp-coreapp-deployment created

You can check if the Master/Coreapp/Webapp pod is running with:

kubectl get pods

The resulting output should look like the following:

NAME                                           READY   STATUS    RESTARTS   AGE
mysql-deployment-5d4f898-xtkpj                 1/1     Running   0          53m
whatsapp-coreapp-deployment-78c4d987b8-4cpz4   1/1     Running   0          2s
whatsapp-coreapp-deployment-78c4d987b8-l5qjj   1/1     Running   0          2s
whatsapp-master-deployment-8598d7bf6b-56fvn    1/1     Running   7          15m
whatsapp-master-deployment-8598d7bf6b-9r6lc    1/1     Running   7          16m
whatsapp-web-deployment-cd4c5785c-9vn6l        1/1     Running   0          16m
whatsapp-web-deployment-cd4c5785c-mn7kf        1/1     Running   0          16m

Default configurations will create 2 Master pods, 2 Coreapp pods and 2 Webapp pods. Each pod requests 128MB memory and 0.15 CPUs. Make changes to spec.template.spec.containers.resources in the respective YAML files if you want different resource settings.

Step 8: Find the Webapp Container Port

When you run a Kubernetes cluster using Minikube, you need to check the Kubernetes cluster IP by running:

minikube ip

Find which port the Webapp container is running on using:

kubectl get services/whatsapp-web-service

The resulting output should look like the following:

NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
whatsapp-web-service   NodePort   10.101.114.46   <none>        443:32477/TCP   25m

In this example, port 443 inside the Webapp container is mapped to the port 32477 on the Kubernetes cluster.

You need to use https://your-minikube-cluster-ip:your-webapp-service-targetport (e.g., https://10.101.114.46:32477) as the API root URL when using the Postman collection.

Step 9: Get an Authentication Token

Log into the WhatsApp Business API client to get a Bearer Authentication token for making API calls.

Step 10: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node to verify all pods are running properly.

The resulting output should look like the following:

'health': {
    '172.17.0.11:whatsapp-coreapp-deployment-78c4d987b8-l5qjj': {
        'errors': [
            {
                'code': 1011,
                'title': 'Service not ready',
                'details': 'Wacore is not instantiated. Please check wacore log for details.'
            }
        ]
    },
    '172.17.0.6:whatsapp-master-deployment-8598d7bf6b-56fvn': {
        'errors': [
            {
                'code': 1011,
                'title': 'Service not ready',
                'details': 'Wacore is not instantiated. Please check wacore log for details.'
            }
        ]
    },
    '172.17.0.7:whatsapp-coreapp-deployment-78c4d987b8-4cpz4': {
        'errors': [
            {
                'code': 1011,
                'title': 'Service not ready',
                'details': 'Wacore is not instantiated. Please check wacore log for details.'
            }
        ]
    },
    '172.17.0.8:whatsapp-master-deployment-8598d7bf6b-9r6lc': {
        'gateway_status': 'unregistered',
        'role': 'primary_master'
    }
},
'meta': {
    'version': 'v2.23.4',
    'api_status': 'stable'
}

The response shows a gateway_status of unregistered as the gateway_status for the primary Master container because the WhatsApp Business API client is not yet registered.

Step 11: Register the WhatsApp Business API Client

You can register your WhatsApp Business API client using an API call to the account node.

Step 12: Perform a Second Health Check

Perform another health check on the WhatsApp Business API client using an API call to the health node after completing registration and make sure one of the Coreapp containers has a gateway_status of connected.

The resulting output should look like the following:

'health': {
    '172.17.0.11:whatsapp-coreapp-deployment-78c4d987b8-l5qjj': {
        'gateway_status': 'disconnected',
        'role': 'coreapp'
    },
    '172.17.0.6:whatsapp-master-deployment-8598d7bf6b-56fvn': {
        'gateway_status': 'disconnected',
        'role': 'secondary_master'
    },
    '172.17.0.7:whatsapp-coreapp-deployment-78c4d987b8-4cpz4': {
        'gateway_status': 'connected',
        'role': 'coreapp'
    },
    '172.17.0.8:whatsapp-master-deployment-8598d7bf6b-9r6lc': {
        'gateway_status': 'disconnected',
        'role': 'primary_master'
    }
},
'meta': {
    'version': 'v2.23.4',
    'api_status': 'stable'
}

Note: In High Availability mode, only one Coreapp (whatsapp-coreapp-deployment-78c4d987b8-4cpz4 in this example) will be connected to the Whatsapp server; all other pods, including the primary Master, will have a gateway_status of disconnected. If whatsapp-coreapp-deployment-78c4d987b8-4cpz4 goes down, whatsapp-coreapp-deployment-78c4d987b8-l5qjj will replace it and connect to the Whatsapp server to maintain High Availability.

You have now set up the WhatsApp Business API client in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Setup of a highly available Multiconnect cluster section below.

Setup of a Highly Available Multiconnect Cluster

Step 1: Set Up Two Shards

Use the shards endpoint to set up 2 shards. You should see an HTTP response with a 201 Created status.

Step 2: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node to verify all pods are running properly.

The resulting output should look like the following:

'health': {
    '172.17.0.11:whatsapp-coreapp-deployment-78c4d987b8-l5qjj': {
        'gateway_status': 'connected',
        'role': 'coreapp'
    },
    '172.17.0.6:whatsapp-master-deployment-8598d7bf6b-56fvn': {
        'gateway_status': 'disconnected',
        'role': 'secondary_master'
    },
    '172.17.0.7:whatsapp-coreapp-deployment-78c4d987b8-4cpz4': {
        'gateway_status': 'connected',
        'role': 'coreapp'
    },
    '172.17.0.8:whatsapp-master-deployment-8598d7bf6b-9r6lc': {
        'gateway_status': 'connected',
        'role': 'primary_master'
    }
},
'meta': {
    'version': 'v2.23.4',
    'api_status': 'stable'
}

Note: In MultiConnect mode with 2 shards, 2 Coreapps (whatsapp-coreapp-deployment-78c4d987b8-l5qjj and whatsapp-coreapp-deployment-78c4d987b8-4cpz4 in this example) will be connected to the WhatsApp server, and the primary Master (whatsapp-master-deployment-8598d7bf6b-9r6lc in this example) will also connect to the WhatsApp server.

Step 3: Start a Third Coreapp to Maintain High Availability

So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of your message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction.

To start the third Coreapp container, change the number of replicas to 3 in the coreapp.yaml file:

spec:
  replicas: 3

Then, redeploy the Coreapp by running:

kubectl apply -f coreapp.yaml

The resulting output should look like the following:

horizontalpodautoscaler.autoscaling/whatsapp-coreapp-autoscaler unchanged
service/whatsapp-coreapp-service unchanged
deployment.apps/whatsapp-coreapp-deployment configured

Step 4: Perform a Second Health Check

Perform another health check to verify all nodes are running properly using an API call to the health node to verify all pods are running properly.

The resulting output should look like the following:

'health': {
    '172.17.0.10:whatsapp-coreapp-deployment-78c4d987b8-hwqp6': {
        'gateway_status': 'disconnected',
        'role': 'coreapp'
    },
    '172.17.0.11:whatsapp-coreapp-deployment-78c4d987b8-l5qjj': {
        'gateway_status': 'connected',
        'role': 'coreapp'
    },
    '172.17.0.6:whatsapp-master-deployment-8598d7bf6b-56fvn': {
        'gateway_status': 'disconnected',
        'role': 'secondary_master'
    },
    '172.17.0.7:whatsapp-coreapp-deployment-78c4d987b8-4cpz4': {
        'gateway_status': 'connected',
        'role': 'coreapp'
    },
    '172.17.0.8:whatsapp-master-deployment-8598d7bf6b-9r6lc': {
        'gateway_status': 'connected',
        'role': 'primary_master'
    }
},
'meta': {
    'version': 'v2.23.4',
    'api_status': 'stable'
}

The new whatsapp-coreapp-deployment-78c4d987b8-hwqp6 container now acts as a standby container but is not currently connected to the WhatsApp server. If either whatsapp-coreapp-deployment-78c4d987b8-l5qjj or whatsapp-coreapp-deployment-78c4d987b8-4cpz4stops working,whatsapp-coreapp-deployment-78c4d987b8-hwqp6` will connect to the WhatsApp server to maintain an overall shard count of 2.

Upgrading the WhatsApp Business API Client

Step 1: Change the $VERSION Environment Variable to the New Version

Update the $VERSION environment variable to the version you would like to upgrade to (e.g., 2.23.5) in the containers section of the webapp.yaml, master.yaml, and coreapp.yaml files.

webapp.yaml

containers:
- name: whatsapp-web
  image: docker.whatsapp.biz/web:v2.23.5

master.yaml

containers:
- name: whatsapp-master
  image: docker.whatsapp.biz/coreapp:v2.23.5

coreapp.yaml

containers:
- name: whatsapp-coreapp
  image: docker.whatsapp.biz/coreapp:v2.23.5

Step 2: Upgrade the Containers

Upgrade the Webapp, Master, and Coreapp containers, and monitor the rollout status.

Webapp

kubectl apply -f webapp.yaml

The resulting output should look like the following:

kubectl rollout status deployments/whatsapp-web-deployment

Master

kubectl apply -f master.yaml

The resulting output should look like the following:

kubectl rollout status deployments/whatsapp-master-deployment

Coreapp

kubectl apply -f coreapp.yaml

The resulting output should look like the following:

kubectl rollout status deployments/whatsapp-coreapp-deployment

Uninstalling the WhatsApp Business API Client

Step 1: Delete the Deployments

Delete the deployments using:

kubectl delete -f webapp.yaml
kubectl delete -f master.yaml
kubectl delete -f coreapp.yaml
kubectl delete -f mysql.yaml

Step 2: Delete the Volume

Delete the volume using:

kubectl delete -f volume.yaml

Step 3: Delete the Secret

Delete the secret using:

kubectl delete secrets/whatsapp-config

Troubleshooting

To check if all pods are running use:

kubectl get pods

The resulting output should look like the following:

NAME                                           READY   STATUS    RESTARTS   AGE
mysql-deployment-5d4f898-xtkpj                 1/1     Running   0          1h
whatsapp-coreapp-deployment-78c4d987b8-4cpz4   1/1     Running   2          1h
whatsapp-coreapp-deployment-78c4d987b8-hwqp6   1/1     Running   0          24m
whatsapp-coreapp-deployment-78c4d987b8-l5qjj   1/1     Running   2          1h
whatsapp-master-deployment-8598d7bf6b-56fvn    1/1     Running   9          1h
whatsapp-master-deployment-8598d7bf6b-9r6lc    1/1     Running   8          1h
whatsapp-web-deployment-cd4c5785c-f99n4        1/1     Running   0          50m
whatsapp-web-deployment-cd4c5785c-s5phx        1/1     Running   0          50m

If any pod is not ready (i.e., shows 0/1 in the READY column), you can get logs for that particular pod by running the kubectl logs command with the pod name.

Example:

kubectl logs whatsapp-coreapp-deployment-78c4d987b8-4cpz4 > whatsapp-coreapp-deployment-78c4d987b8-4cpz4.txt

You will then find the logs in the whatsapp-coreapp-deployment-78c4d987b8-4cpz4.txt file in the current directory.

To collect logs for a particular deployed service, such as the Webapp, run the kubectl logs command with the service name.

Example:

kubectl logs deployments/whatsapp-web-deployment > whatsapp-web-deployment.txt

You will then find the logs in the whatsapp-web-deployment.txt file in the current directory.