Production Setup: High Availability & Multiconnect

This document shows you how to set up a High Availability cluster in production. It also provides guidance on how to enable Multiconnect on top of that as well as the changes required for a highly available Multiconnect cluster.

Before you start any of these set ups, check our requirements.

To set up a Highly Available cluster:

1. Create a biz directory for the setup scripts
2. Get the WhatsApp Business API client configuration files
3. Set the database environment variables
4. Set up the local media volume
5. Start the WhatsApp Business API client
6. Verify the containers are running
7. Perform a health check
8. Register the WhatsApp Business API client
9. Perform a second health check

To set up a Highly Available Multiconnect cluster:

1. Set up two shards
2. Perform a health check
3. Start a third coreapp to maintain High Availability
4. Perform a second health check

Once you have completely set up your instance, you can choose to upgrade it. To uninstall the client, follow these steps.

Before You Start

You will need:

• Multiple compute servers for running the various components of the WhatsApp Business API client (e.g., Webapp, Master, and Coreapp)
  • For system requirements, refer to What are the server system requirements to run the WhatsApp Business API Client? in the FAQ.
  • For a production High Availability setup, we recommend at least 1 machine for the Webapp container, 2 machines for the Coreapp containers, and 2 machines for the Master containers. Loads on Masters are light and you could co-locate the Master and Coreapp containers on the same host.
• Install Docker Desktop and Docker Compose.
• A standalone database server running a MySQL or PostgreSQL process
  • The database server should not be run on any of the other compute servers hosting the WhatsApp Business API client and should only be a few milliseconds of latency away from the other compute servers.

• A shared file system (e.g., NFS) mounted to a local directory on all Webapp, Master, and Coreapp hosts, if you want to support sending and receiving media messages
  • Make sure 755 is set as the file mode on the mount path and all its subdirectories.

mkdir your-local-media-volume-path
mount -t nfs nfs_server_IP_addr:/shared_directory /your-local-media-volume-path

Setup of a Highly Available Cluster

Step 1: Create a biz Directory for the Setup Scripts

Run the following code in your preferred location for the WhatsApp Business API client:

mkdir ~/biz; cd ~/biz;

Step 2: Get the WhatsApp Business API Client Configuration Files

Clone the prod-multiconnect-compose.yml and db.env configuration files from the Installation directory of the WhatsApp-Business-API-Setup-Scripts GitHub repository to the ~/biz directory you created in Step 1.

Step 3: Set the Database Environment Variables

Change the database environment variables in the db.env file under the ~/biz directory to reflect your MySQL/PostgreSQL configuration.

WA_DB_ENGINE=MYSQL | PGSQL
WA_DB_HOSTNAME=your-database-server
WA_DB_PORT=your-database-server-port
WA_DB_USERNAME=your-database-username
WA_DB_PASSWORD=your-database-password

Step 4: Set up the Local Media Volume

The local media volume (whatsappMedia:/usr/local/wamedia by default) defined in prod-docker-compose.yml file is used for storing media files. By default, the volume is mounted to a directory on the Docker machine. Alternatively, you can choose to mount the media volume to a host directory. To change the mount point of the media volume, modify the volume definition inside the services section from whatsappMedia to the path of the host directory you're using.

services:
  waweb:
    ...
    volumes:
      - /your-local-media-volume-path:/usr/local/wamedia
    ...
  wacore:
    ...
    volumes:
      - /your-local-media-volume-path:/usr/local/wamedia
  ...
  master:
    ...
    volumes:
      - /your-local-media-volume-path:/usr/local/wamedia

Step 5: Start the WhatsApp Business API Client

To start the WhatsApp Business API client with 1 Webapp container, 2 Master containers, and 2 Coreapp containers similar to the High Availability Introduction diagram, use the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):

# copy configuration scripts to each Webapp host, ssh to each Webapp host, execute scripts to install Webapp on the host
for host in your-webapp-hostname; do
    scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd waweb
done 
# copy configuration scripts to each Master host, ssh to each Master host, execute scripts to install Master on the host
for host in your-master1-hostname your-master2-hostname; do
    scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd master
done
# copy configuration scripts to each Coreapp host, ssh to each Coreapp host, execute scripts to install Coreapp on the host
for host in your-coreapp1-hostname your-coreapp2-hostname; do
    scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd wacore
done

In summary, the above commands will:

• Copy the prod-multiconnect-compose.yml and db.env configuration files to all Webapp hosts (your-webapp-hostname in this example) and start the waweb service on these hosts;
• Copy the prod-multiconnect-compose.yml and db.env configuration files to all Master hosts (your-master1-hostname and your-master2-hostname in this example) and start the Master service on these hosts;
• Copy the prod-multiconnect-compose.yml and db.env configuration files to all Coreapp hosts (your-coreapp1-hostname and your-coreapp2-hostname in this example) and start the wacore service on these hosts.

Step 6: Verify the Containers Are Running

You can check that all containers have an RUNNING status by running:

EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f prod-multiconnect-compose.yml ps

Step 7: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node.

The resulting output should look like the following:

{
    "health": {
        "your-master1-hostname:85cdd51506fd": {
            "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
            ]
        },
        "your-master2-hostname:8dd3f5bea27d": {
            "gateway_status": "unregistered",
            "role": "primary_master"
        },
        "your-coreapp1-hostname:753efb1cf72c": {
            "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
            ]
        },
        "your-coreapp2-hostname:75d7355eaaaa": {
            "errors": [
              {
                  "code": 1011,
                  "title": "Service not ready",
                  "details": "Wacore is not instantiated. Please check wacore log for details."
              }
            ]
        }
    }
}
200

The response shows a gateway_status of unregistered because the WhatsApp Business API client is not yet registered.

Step 8: Register the WhatsApp Business API Client

You can register your WhatsApp Business API client using an API call to the account node.

Step 9: Perform a Second Health Check

Perform another health check on the WhatsApp Business API client using an API call to the health node after completing registration and make sure one of the Coreapp containers has a gateway_status of connected.

The resulting output should look like the following:

{
    "health": {
        "your-master1-hostname:85cdd51506fd": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "your-master2-hostname:8dd3f5bea27d": {
            "gateway_status": "disconnected",
            "role": "primary_master"
        },
        "your-coreapp1-hostname:753efb1cf72c": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "your-coreapp2-hostname:75d7355eaaaa": {
            "gateway_status": "disconnected",
            "role": "coreapp"
        }
    }
}
200

Note: In High Availability mode, only one Coreapp (your-coreapp1-hostname:753efb1cf72c in this example) will be connected to the Whatsapp server, all other nodes, including the primary Master, will have a gateway_status of disconnected. If your-coreapp1-hostname:753efb1cf72c goes down, your-coreapp2-hostname:75d7355eaaaa will replace it and connect to the Whatsapp server to maintain High Availability.

Now, you have set up the WhatsApp Business API client in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Setup of a Highly Availability Multiconnect Cluster section below.

Setup of a Highly Available Multiconnect Cluster

Step 1: Set Up Two Shards

Use the shards endpoint to set up 2 shards. You should see an HTTP response with a 201 Created status.

Step 2: Perform a Health Check

You can perform a health check on the WhatsApp Business API client using an API call to the health node to verify all nodes are running properly.

The resulting output should look like the following:

{
    "health": {
        "your-master1-hostname:85cdd51506fd": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "your-master2-hostname:8dd3f5bea27d": {
            "gateway_status": "connected",
            "role": "primary_master"
        },
        "your-coreapp1-hostname:753efb1cf72c": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "your-coreapp2-hostname:75d7355eaaaa": {
            "gateway_status": "connected",
            "role": "coreapp"
        }
    }
}      
200    

Note: In Multiconnect mode with 2 shards, 2 Coreapps (your-coreapp1-hostname:753efb1cf72c and your-coreapp2-hostname:75d7355eaaaa in this example) will be connected to the WhatsApp server, and the primary Master (your-master2-hostname:8dd3f5bea27d in this example) will also connect to the server.

Step 3: Start a Third Coreapp to Maintain High Availability

So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of your message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp on a new Coreapp host (

in this example) to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction.

To start the third Coreapp container, run the following command:

# copy configuration scripts to the 3rd Coreapp host, ssh to the Coreapp host, execute scripts to install Coreapp on the host
for host in your-coreapp3-hostname; do
    scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd wacore
done

Step 4: Perform a Second Health Check

Perform another health check to verify all nodes are running properly using an API call to the health node.

The resulting output should look like the following:

{
    "health": {
        "your-master1-hostname:85cdd51506fd": {
            "gateway_status": "disconnected",
            "role": "secondary_master"
        },
        "your-master2-hostname:8dd3f5bea27d": {
            "gateway_status": "disconnected",
            "role": "primary_master"
        },
        "your-coreapp1-hostname:753efb1cf72c": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "your-coreapp2-hostname:75d7355eaaaa": {
            "gateway_status": "connected",
            "role": "coreapp"
        },
        "your-coreapp3-hostname:23b50199bec2": {
            "gateway_status": "disconnected",
            "role": "coreapp"
        }
    }
}      
200 

The new Coreapp container (your-coreapp3-hostname:23b50199bec2 in this example) now acts as a standby container but is not currently connected to the WhatsApp server. If either of the other 2 connected Coreapp containers stops working, the third container will connect to the WhatsApp server to maintain an overall shard count of 2.

Guidelines & Best Practices

Number of Webapps

To support 100-150 messages/sec, at least 2 Webapps are recommended. If you want to achieve High Availability for Webapp, you could run Webapp containers on more than 2 hosts and host them behind a load balancer such as HAProxy, Nginx, or ELB. Instead of accessing the WhatsApp Business API client endpoints through https://your-webapp-hostname:your-webapp-port/, you should use https://your-load-balancer-name:your-load-balancer-port/.

Number of Masters

It's recommended you have 3 Masters running on different hosts. There is no reason to have more than 3 Masters in production no matter how many shards you have. Loads on Masters are light and you could co-locate them with Coreapp containers.

Number of Coreapps

It's recommended you have shard_number + X number of Coreapps running on different hosts in order to tolerate X number of host failures.

Upgrading the WhatsApp Business API Client

The WA_API_VERSION environmental variable should be updated to the new version number. Run the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):

# ssh to each Webapp host, execute scripts with new WA_API_VERSION to upgrade Webapp on the host
for host in your-webapp-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd waweb
done
# ssh to each Master host, execute scripts with new WA_API_VERSION to upgrade Master on the host
for host in your-master1-hostname your-master2-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd master
done
# ssh to each Coreapp host, execute scripts with new WA_API_VERSION to upgrade Coreapp on the host
for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd wacore
done

For MySQL Database Users Upgrading to Version 2.23.x and Above

You can now make use of a database upgrade service that will let you upgrade your database while your application is still running to avoid downtime.

Step 1: Download the configuration file

The dbupgrade-compose.yml file has fields indicating the container version.

Example:

services:
  dbupgrade:
      image: docker.whatsapp.biz/coreapp:v${WA_API_VERSION:-2.21.3}

Step 2: Start the container

To upgrade an installation, start the dbupgrade-service container with the WA_API_VERSION environment variable set to the latest version:

EXTERNAL_HOSTNAME=$host_to_upgradedb WA_API_VERSION=new-whatsapp-version docker-compose -f dbupgrade-compose.yml up -d

Note: If you are using an orchestration that restarts the container upon exit irrespective of the exit code, start the service with the EXIT_ON_SUCCESS environment variable set to FALSE to avoid exiting the container when the exit code is 0.

Step 3: Allow the upgrade to finish

If the database upgrade is successful, the container will exit with code 0. You can use the following Docker command to track the status:

docker wait your-database-upgrade-container-name

This will output the exit code of the dbupgrade-service container.

Step 4: Restart the Coreapp and Webapp containers

Restart the Coreapp and Webapp Docker containers with the WA_API_VERSION environment variable set to the latest version:

EXTERNAL_HOSTNAME=$host_to_upgradedb WA_API_VERSION=new-whatsapp-version docker-compose -f dbupgrade-compose.yml up -d

For WhatsApp Business API Client Users Upgrading to Version 2.29.3 and Above

If you are upgrading from v2.29.1, v2.29.2 or encountered problems during upgrade to those versions and had to roll back for stability, we recommend you upgrade to v2.29.3, then run the following command on the Webapp Docker container:

chown -R root your-media-directory/incoming your-media-directory/outgoing your-media-directory/shared

Unless you have changed it, the default media directory is /usr/local/wamedia.

Note:

• Since this command can take a while to complete based on pre-existing media volume size, we recommend this command be initiated as soon as you upgrade during the maintenance window.
• This command only modifies ownership of media volumes in the background (There is no impact/downtime for media operations or latency/performance while it is running).
• This is a one-time only upgrade path to fix backward compatibility issues in v2.29.1 and v2.29.2.

Uninstalling the WhatsApp Business API Client

If you need to reset your development environment by removing all containers, use the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):

# ssh to each Webapp host, execute scripts to uninstall Webapp on the host
for host in your-webapp-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd waweb
done
# ssh to each Master host, execute scripts to uninstall Master on the host
for host in your-master1-hostname your-master2-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd master
done
# ssh to each Coreapp host, execute scripts to uninstall Coreapp on the host
for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd wacore
done

Troubleshooting

To collect logs from all containers on a host, run the following command:

EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs > debug_output.txt

To collect logs of a specific service, append the service name (e.g., waweb, master1, wacore1) to the docker-compose logs command.

EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs waweb > debug_output.txt

You can find the logs in the debug_output.txt file in the current directory.