We are sunsetting On-Premises API. Refer to our On-Premises API Sunset document for details, and to learn how to migrate to our next-generation Cloud API.
This document shows you how to set up a High Availability cluster in production. It also provides guidance on how to enable Multiconnect on top of that as well as the changes required for a highly available Multiconnect cluster.
If you haven't already done so, it's recommended you set up High Availability/Multiconnect instance of the WhatsApp Business API client on a developer machine following the Developer Setup: High Availability and Multiconnect instructions to test your set up before following this document to set up the WhatsApp Business API client in production.
Before you start any of these set ups, check our requirements.
To set up a Highly Available cluster:
biz
directory for the setup scriptsTo set up a Highly Available Multiconnect cluster:
Once you have completely set up your instance, you can choose to upgrade it. To uninstall the client, follow these steps.
If you have already run a developer setup and want to reuse the phone number in production, please refer to the Migration guide before continuing with the rest of this document.
The content in this document is based on the assumption of a fresh installation using a new phone number.
You will need:
Either MySQL 5.7.xx/8.0.xx or PostgreSQL 13.x/12.x/11.x is required.
Your database password should not contain any of these characters: ?{}&~!()^=
Failing to comply with this will likely cause the setup to fail.
755
is set as the file mode on the mount path and all its subdirectories. mkdir your-local-media-volume-path mount -t nfs nfs_server_IP_addr:/shared_directory /your-local-media-volume-path
To install Docker Desktop on your developer machine:
The remaining steps are based on macOS and should be very similar for Linux or Windows 10.
To install Docker using macOS:
Docker Compose is a plugin that is bundled with Docker Desktop and should have installed automatically. For more information about using or Docker Compose, see Overview of Docker Compose. If for some reason Docker Compose was not installed, you can install it by following the instructions located at Install Docker Compose.
biz
Directory for the Setup ScriptsRun the following code in your preferred location for the WhatsApp Business API client:
mkdir ~/biz; cd ~/biz;
Clone the prod-multiconnect-compose.yml
and db.env
configuration files from the Installation directory of the WhatsApp-Business-API-Setup-Scripts GitHub repository to the ~/biz
directory you created in Step 1.
Change the database environment variables in the db.env
file under the ~/biz
directory to reflect your MySQL/PostgreSQL configuration.
WA_DB_ENGINE=MYSQL | PGSQL WA_DB_HOSTNAME=your-database-server WA_DB_PORT=your-database-server-port WA_DB_USERNAME=your-database-username WA_DB_PASSWORD=your-database-password
The local media volume (whatsappMedia:/usr/local/wamedia
by default) defined in prod-docker-compose.yml
file is used for storing media files. By default, the volume is mounted to a directory on the Docker machine. Alternatively, you can choose to mount the media volume to a host directory. To change the mount point of the media volume, modify the volume definition inside the services
section from whatsappMedia
to the path of the host directory you're using.
services: waweb: ... volumes: - /your-local-media-volume-path:/usr/local/wamedia ... wacore: ... volumes: - /your-local-media-volume-path:/usr/local/wamedia ... master: ... volumes: - /your-local-media-volume-path:/usr/local/wamedia
To start the WhatsApp Business API client with 1 Webapp container, 2 Master containers, and 2 Coreapp containers similar to the High Availability Introduction diagram, use the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):
# copy configuration scripts to each Webapp host, ssh to each Webapp host, execute scripts to install Webapp on the host for host in your-webapp-hostname; do scp db.env prod-multiconnect-compose.yml username@$host:/local/path/ cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd waweb done # copy configuration scripts to each Master host, ssh to each Master host, execute scripts to install Master on the host for host in your-master1-hostname your-master2-hostname; do scp db.env prod-multiconnect-compose.yml username@$host:/local/path/ cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd master done # copy configuration scripts to each Coreapp host, ssh to each Coreapp host, execute scripts to install Coreapp on the host for host in your-coreapp1-hostname your-coreapp2-hostname; do scp db.env prod-multiconnect-compose.yml username@$host:/local/path/ cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd wacore done
In summary, the above commands will:
prod-multiconnect-compose.yml
and db.env
configuration files to all Webapp hosts (your-webapp-hostname
in this example) and start the waweb
service on these hosts;prod-multiconnect-compose.yml
and db.env
configuration files to all Master hosts (your-master1-hostname
and your-master2-hostname
in this example) and start the Master service on these hosts;prod-multiconnect-compose.yml
and db.env
configuration files to all Coreapp hosts (your-coreapp1-hostname
and your-coreapp2-hostname
in this example) and start the wacore
service on these hosts.You can check that all containers have an RUNNING status by running:
EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f prod-multiconnect-compose.yml ps
You can download and configure our Postman Collection for interacting with the WhatsApp Business API if you do not wish to use the command line.
You can perform a health check on the WhatsApp Business API client using an API call to the health
node.
The resulting output should look like the following:
{ "health": { "your-master1-hostname:85cdd51506fd": { "errors": [ { "code": 1011, "title": "Service not ready", "details": "Wacore is not instantiated. Please check wacore log for details." } ] }, "your-master2-hostname:8dd3f5bea27d": { "gateway_status": "unregistered", "role": "primary_master" }, "your-coreapp1-hostname:753efb1cf72c": { "errors": [ { "code": 1011, "title": "Service not ready", "details": "Wacore is not instantiated. Please check wacore log for details." } ] }, "your-coreapp2-hostname:75d7355eaaaa": { "errors": [ { "code": 1011, "title": "Service not ready", "details": "Wacore is not instantiated. Please check wacore log for details." } ] } } } 200
The response shows a gateway_status
of unregistered
because the WhatsApp Business API client is not yet registered.
You can register your WhatsApp Business API client using an API call to the account
node.
Perform another health check on the WhatsApp Business API client using an API call to the health
node after completing registration and make sure one of the Coreapp containers has a gateway_status
of connected
.
The resulting output should look like the following:
{ "health": { "your-master1-hostname:85cdd51506fd": { "gateway_status": "disconnected", "role": "secondary_master" }, "your-master2-hostname:8dd3f5bea27d": { "gateway_status": "disconnected", "role": "primary_master" }, "your-coreapp1-hostname:753efb1cf72c": { "gateway_status": "connected", "role": "coreapp" }, "your-coreapp2-hostname:75d7355eaaaa": { "gateway_status": "disconnected", "role": "coreapp" } } } 200
Note: In High Availability mode, only one Coreapp (your-coreapp1-hostname:753efb1cf72c
in this example) will be connected to the Whatsapp server, all other nodes, including the primary Master, will have a gateway_status
of disconnected
. If your-coreapp1-hostname:753efb1cf72c
goes down, your-coreapp2-hostname:75d7355eaaaa
will replace it and connect to the Whatsapp server to maintain High Availability.
It's recommended you set up monitoring for your production WhatsApp Business API client.
Now, you have set up the WhatsApp Business API client in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Setup of a Highly Availability Multiconnect Cluster section below.
Use the shards endpoint to set up 2 shards. You should see an HTTP response with a 201 Created
status.
You can perform a health check on the WhatsApp Business API client using an API call to the health
node to verify all nodes are running properly.
The resulting output should look like the following:
{ "health": { "your-master1-hostname:85cdd51506fd": { "gateway_status": "disconnected", "role": "secondary_master" }, "your-master2-hostname:8dd3f5bea27d": { "gateway_status": "connected", "role": "primary_master" }, "your-coreapp1-hostname:753efb1cf72c": { "gateway_status": "connected", "role": "coreapp" }, "your-coreapp2-hostname:75d7355eaaaa": { "gateway_status": "connected", "role": "coreapp" } } } 200
Note: In Multiconnect mode with 2 shards, 2 Coreapps (your-coreapp1-hostname:753efb1cf72c
and your-coreapp2-hostname:75d7355eaaaa
in this example) will be connected to the WhatsApp server, and the primary Master (your-master2-hostname:8dd3f5bea27d
in this example) will also connect to the server.
So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of your message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp on a new Coreapp host (
your-coreapp3-hostname
in this example) to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction.
To start the third Coreapp container, run the following command:
# copy configuration scripts to the 3rd Coreapp host, ssh to the Coreapp host, execute scripts to install Coreapp on the host for host in your-coreapp3-hostname; do scp db.env prod-multiconnect-compose.yml username@$host:/local/path/ cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd wacore done
Perform another health check to verify all nodes are running properly using an API call to the health
node.
The resulting output should look like the following:
{ "health": { "your-master1-hostname:85cdd51506fd": { "gateway_status": "disconnected", "role": "secondary_master" }, "your-master2-hostname:8dd3f5bea27d": { "gateway_status": "disconnected", "role": "primary_master" }, "your-coreapp1-hostname:753efb1cf72c": { "gateway_status": "connected", "role": "coreapp" }, "your-coreapp2-hostname:75d7355eaaaa": { "gateway_status": "connected", "role": "coreapp" }, "your-coreapp3-hostname:23b50199bec2": { "gateway_status": "disconnected", "role": "coreapp" } } } 200
The new Coreapp container (your-coreapp3-hostname
:23b50199bec2
in this example) now acts as a standby container but is not currently connected to the WhatsApp server. If either of the other 2 connected Coreapp containers stops working, the third container will connect to the WhatsApp server to maintain an overall shard count of 2.
To support 100-150 messages/sec, at least 2 Webapps are recommended. If you want to achieve High Availability for Webapp, you could run Webapp containers on more than 2 hosts and host them behind a load balancer such as HAProxy, Nginx, or ELB. Instead of accessing the WhatsApp Business API client endpoints through https://your-webapp-hostname:your-webapp-port/
, you should use https://your-load-balancer-name:your-load-balancer-port/
.
It's recommended you have 3 Masters running on different hosts. There is no reason to have more than 3 Masters in production no matter how many shards you have. Loads on Masters are light and you could co-locate them with Coreapp containers.
It's recommended you have shard_number + X
number of Coreapps running on different hosts in order to tolerate X
number of host failures.
There will be downtime during the upgrade process.
Backing up your current application settings before upgrading is highly recommended to ensure you can get back up and running quickly. Please follow the Backup and Restore documentation.
It is always recommended to perform upgrades during your least busiest hours.
The WA_API_VERSION
environmental variable should be updated to the new version number. Run the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):
# ssh to each Webapp host, execute scripts with new WA_API_VERSION to upgrade Webapp on the host for host in your-webapp-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd waweb done # ssh to each Master host, execute scripts with new WA_API_VERSION to upgrade Master on the host for host in your-master1-hostname your-master2-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd master done # ssh to each Coreapp host, execute scripts with new WA_API_VERSION to upgrade Coreapp on the host for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.x docker-compose -f /local/path/prod-multiconnect-compose.yml up -d" ssh username@$host $cmd wacore done
2.23.x
and AboveYou can now make use of a database upgrade service that will let you upgrade your database while your application is still running to avoid downtime.
The dbupgrade-compose.yml file has fields indicating the container version.
Example:
services: dbupgrade: image: docker.whatsapp.biz/coreapp:v${WA_API_VERSION:-2.21.3}
To upgrade an installation, start the dbupgrade-service container with the WA_API_VERSION
environment variable set to the latest version:
EXTERNAL_HOSTNAME=$host_to_upgradedb WA_API_VERSION=new-whatsapp-version docker-compose -f dbupgrade-compose.yml up -d
Note: If you are using an orchestration that restarts the container upon exit irrespective of the exit code, start the service with the EXIT_ON_SUCCESS
environment variable set to FALSE
to avoid exiting the container when the exit code is 0
.
If the database upgrade is successful, the container will exit with code 0
. You can use the following Docker command to track the status:
docker wait your-database-upgrade-container-name
This will output the exit code of the dbupgrade-service container.
Restart the Coreapp and Webapp Docker containers with the WA_API_VERSION
environment variable set to the latest version:
EXTERNAL_HOSTNAME=$host_to_upgradedb WA_API_VERSION=new-whatsapp-version docker-compose -f dbupgrade-compose.yml up -d
2.29.3
and AboveIf you are upgrading from v2.29.1
, v2.29.2
or encountered problems during upgrade to those versions and had to roll back for stability, we recommend you upgrade to v2.29.3
, then run the following command on the Webapp Docker container:
chown -R root your-media-directory/incoming your-media-directory/outgoing your-media-directory/shared
Unless you have changed it, the default media directory is /usr/local/wamedia
.
Note:
v2.29.1
and v2.29.2
.If you need to reset your development environment by removing all containers, use the following commands with changes necessary for your environment (i.e., hostnames, host usernames, and local paths):
# ssh to each Webapp host, execute scripts to uninstall Webapp on the host for host in your-webapp-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down" ssh username@$host $cmd waweb done # ssh to each Master host, execute scripts to uninstall Master on the host for host in your-master1-hostname your-master2-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down" ssh username@$host $cmd master done # ssh to each Coreapp host, execute scripts to uninstall Coreapp on the host for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.23.4 docker-compose -f /local/path/prod-docker-compose.yml down" ssh username@$host $cmd wacore done
To collect logs from all containers on a host, run the following command:
EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs > debug_output.txt
To collect logs of a specific service, append the service name (e.g., waweb
, master1
, wacore1
) to the docker-compose logs
command.
EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs waweb > debug_output.txt
You can find the logs in the debug_output.txt
file in the current directory.
This software uses code of FFmpeg licensed under the LGPLv2.1 and its source can be downloaded here.