Developer Setup: High Availability & Multiconnect

This document shows you how to set up a High Availability cluster on a developer machine. It also provides guidance on how to enable Multiconnect on top of that as well as the changes required for a highly available Multiconnect cluster. For a production setup, follow the relevant instructions listed in Production Setups.

Before you start any of these set ups, check our list of requirements.

To set up a High Availability cluster follow these steps:

1. Create a biz Directory for the setup scripts
2. Get the WhatsApp Business API Client configuration files
3. Set the WA_API_VERSION environment variable
4. Start the WhatsApp Business API Client with High Availability
5. Verify containers are running
6. Perform a health check
7. Register the WhatsApp Business API Client
8. Perform a second health check

To set up a Multiconnect cluster follow these steps:

1. Set up two shards
2. Perform a health check
3. Start a third coreapp to maintain High Availability
4. Perform a second health check

Once you have completely set up your instance, you can choose to upgrade it. To uninstall the client, follow these steps.

Before You Start

Before you start, you will need to complete the following:

Other prerequisites

Make sure you set up a local test account in a development environment. This is for fast development and to test new releases. It's also highly recommended you read through the Availability and Scaling guide to learn more about High Availability and Multiconnect.

Setup of a High Availability Cluster

Step 1: Create a biz Directory for the Setup Scripts

Run the following code in your preferred location for the WhatsApp Business API client:

mkdir ~/biz; cd ~/biz;

Step 2: Get the WhatsApp Business API Client Configuration Files

The configuration files for the WhatsApp Business API client are located in the WhatsApp-Business-API-Setup-Scripts GitHub repository. You can set up your WhatsApp Business API client with either a MySQL or Postgres database instance.

• For a MySQL database setup, clone the multiconnect-compose.yml and db.env configuration files from the Installation directory to the ~/biz directory you created in Step 1.
• For a Postgres database setup, clone the multiconnect-compose.yml and db.env configuration files from the Postgres Installation directory to the ~/biz directory you created in Step 1.

Step 3: Set the WA_API_VERSION Environment Variable

The WA_API_VERSION environmental variable should be set to the current version using:

export WA_API_VERSION=current-whatsapp-version

Step 4: Start the WhatsApp Business API Client with High Availability

To start up a High Availability cluster with 1 database container, 1 Webapp container, 2 Master containers, and 2 Coreapp containers in the background similar to the diagram shown in the High Availability Introduction, run the following command:

docker-compose -f multiconnect-compose.yml up -d db waweb master1 master2 wacore1 wacore2

The resulting output should look like the following:

Creating network "biz_default" with the default driver
Creating volume "biz_mysqlData" with local driver
Creating volume "biz_whatsappMedia" with local driver
Creating biz_db_1 ... done
Creating biz_waweb_1 ... done
Creating biz_master1_1 ... done
Creating biz_master2_1 ... done
Creating biz_wacore2_1 ... done
Creating biz_wacore1_1 ... done

Step 5: Verify Containers Are Running

You can check that all containers have an UP state by running:

docker-compose -f multiconnect-compose.yml ps

The resulting output should look like the following:

      Name                    Command               State                                                 Ports
--------------------------------------------------------------------------------------------------------------------------------------------------------------
biz_db_1        docker-entrypoint.sh mysqld      Up      0.0.0.0:33060->3306/tcp, 33060/tcp
biz_master1_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32931->6250/tcp, 0.0.0.0:32930->6251/tcp, 0.0.0.0:32928->6252/tcp, 0.0.0.0:32926->6253/tcp
biz_master2_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32929->6250/tcp, 0.0.0.0:32927->6251/tcp, 0.0.0.0:32925->6252/tcp, 0.0.0.0:32924->6253/tcp
biz_wacore1_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32937->6250/tcp, 0.0.0.0:32935->6251/tcp, 0.0.0.0:32933->6252/tcp, 0.0.0.0:32932->6253/tcp
biz_wacore2_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32939->6250/tcp, 0.0.0.0:32938->6251/tcp, 0.0.0.0:32936->6252/tcp, 0.0.0.0:32934->6253/tcp
biz_waweb_1     /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:9090->443/tcp

By default, the Webapp container will be running on port 9090 and the database container will be running on port 33060.

Step 6: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node.

The resulting output should look like the following:

{
    "health": {
      "master1:b28d835cd579": {
          "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
          ]
      },
      "master2:7fe542d305b4": {
          "gateway_status": "unregistered",
          "role": "primary_master"
      },
      "wacore1:35a5fabfc79d": {
          "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
          ]
      },
      "wacore2:05e1a6d70665": {
          "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
          ]
      }
  }
}

The response shows a gateway_status of unregistered as the gateway_status for the primary Master container because the WhatsApp Business API client is not yet registered.

Step 7: Register the WhatsApp Business API Client

You can register your WhatsApp Business API client using an API call to the account node.

Step 8: Perform a Second Health Check

Perform another health check on the WhatsApp Business API client using an API call to the health node after completing registration and make sure one of the Coreapp containers has a gateway_status of connected.

The resulting output should look like the following:

{
    "health": {
        "master1:b28d835cd579": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "master2:7fe542d305b4": {
            "gateway_status": "disconnected",
            "role": "primary_master"
        },
        "wacore1:35a5fabfc79d": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "wacore2:05e1a6d70665": {
            "gateway_status": "disconnected",
            "role": "coreapp"
        }
    }
} 

Note: In High Availability mode, only one Coreapp (wacore1 in this example) will be connected to the Whatsapp server; all other nodes, including the primary Master, will have a gateway_status of disconnected. If wacore1 goes down, wacore2 will replace it and connect to the Whatsapp server to maintain High Availability.

Now, you have set up the WhatsApp Business API client in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Setup of a Highly Availability Multiconnect Cluster section below.

Setup of a Highly Available Multiconnect Cluster

Step 1: Set Up Two Shards

Use the shards endpoint to set up 2 shards. You should see an HTTP response with a 201 Created status.

Step 2: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node.

The resulting output should look like the following:

{
    "health": {
        "master1:b28d835cd579": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "master2:7fe542d305b4": {
            "gateway_status": "connected",
            "role": "primary_master"
        },
        "wacore1:35a5fabfc79d": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "wacore2:05e1a6d70665": {
            "gateway_status": "connected",
            "role": "coreapp"
        }
    }
}          

Note: In Multiconnect mode with 2 shards, 2 Coreapps (wacore1 and wacore2 in this example) will be connected to the WhatsApp server and the primary Master (master2 in this example) will also connect to the WhastsApp server.

Step 3: Start a Third Coreapp to Maintain High Availability

So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of your message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction.

To start the third Coreapp container, run the following command:

docker-compose -f multiconnect-compose.yml up -d wacore3

The resulting output should look like the following:

biz_db_1 is up-to-date
biz_waweb_1 is up-to-date
biz_master1_1 is up-to-date
Creating biz_wacore3_1 ... done

Step 4: Perform a Second Health Check

Perform another health check to verify all nodes are running properly using an API call to the health node.

The resulting output should look like the following:

{
    "health": {
        "master1:b28d835cd579": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "master2:7fe542d305b4": {
            "gateway_status": "connected",
            "role": "primary_master"
        },
        "wacore1:35a5fabfc79d": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "wacore2:05e1a6d70665": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "wacore3:23b50199bec2": {
            "gateway_status": "disconnected",
            "role": "coreapp"
        }
    }
} 

The new Coreapp container (wacore3 in this example) now acts as a standby container but is not currently connected to the WhatsApp server. If either wacore1 or wacore2 stops working, wacore3 will connect to the WhatsApp server to maintain an overall shard count of 2.

Upgrading the WhatsApp Business API Client

Step 1: Change the WA_API_VERSION Environment Variable to the New Version

The WA_API_VERSION environmental variable should be updated to the new version number using:

export WA_API_VERSION=new-whatsapp-version

Step 2: Restart the Docker Containers

Restart the Docker containers by running:

docker-compose -f multiconnect-compose.yml up -d

For MySQL Database Users Upgrading to v2.23.x and Above

You can now make use of a database upgrade service that will let you upgrade your database while your application is still running to avoid downtime.

Step 1: Download the configuration file

The dbupgrade-compose.yml file has fields indicating the container version.

Example:

services:
  dbupgrade:
    image: docker.whatsapp.biz/coreapp:v${WA_API_VERSION:-2.21.3}

Step 2: Start the container

To upgrade an installation, start the dbupgrade-service container with the WA_API_VERSION environment variable set to the latest version:

WA_API_VERSION=new-whatsapp-version docker-compose -f dbupgrade-compose.yml up -d

Note: If you are using an orchestration that restarts the container upon exit irrespective of the exit code, start the service with the EXIT_ON_SUCCESS environment variable set to FALSE in order to avoid exiting the container when the exit code is 0.

Step 3: Allow the upgrade to finish

If the database upgrade is successful, the container will exit with code 0. You can use the following Docker command to track the status:

docker wait your-database-upgrade-container-name

This will output the exit code of the dbupgrade-service container.

Step 4: Restart the Coreapp and Webapp containers

Restart the Coreapp and Webapp Docker containers with the WA_API_VERSION environment variable set to the latest version:

WA_API_VERSION=new-whatsapp-version docker-compose -f multiconnect-compose.yml up -d

Uninstalling the WhatsApp Business API Client

If you need to reset your development environment by removing all containers, run the following command from the directory containing the multiconnect-compose.yml file:

docker-compose -f multiconnect-compose.yml down

To get rid of all volumes defined in the multiconnect-compose.yml file in addition to the containers, run down with the -v parameter:

docker-compose -f multiconnect-compose.yml down -v

Troubleshooting

To collect logs from all containers, run the following command:

docker-compose -f multiconnect-compose.yml logs > debug_output.txt

To collect logs of a specific service, append the service name (e.g.: waweb, master1, wacore1) to the docker-compose logs command:

docker-compose -f multiconnect-compose.yml logs waweb > debug_output.txt

You can find the logs in the debug_output.txt file in the current directory.