Deploying the WSO2 ESB correctly on your infrastructure is a vital task for the success of the usage of the ESB. You should first analyze the requirement and then select the best matching deployment mode. There are different set of possible deployment modes with different configurations.
Standalone Deployment Guide : Describes the default standalone installation and deployment of the ESB
Webapp Deployment Guide : Contains the information on deploying the ESB on an Application Server, this particular guide explains the deployment of an existing Tomcat Servlet Container.
This document describes the steps you need to follow to setup a working WSO2 ESB cluster in your environment.
Clustering the ESB has two different aspects.
The configuration sharing is done using the WSO2 Governance Registry. All the ESB nodes in the cluster are pointed to one instance of WSO2 G-Reg and through this registry configuration is shared.
WSO2 G-Reg consists of three registry spaces.
So, when it comes to ESB clustering, we will be using local registry per each instance, mounting the configuration registry of each ESB instance to the same config registry and sharing the governance space among GReg and 2-ESB instances.
First lets see how to set up ESB instances pointing to a single G-Reg.
WSO2 ESB Cluster Deployment Diagram
Setting UP WSO2 ESB cluster with GReg
(This documentation http://wso2.org/project/registry/3.5.0/docs/user_guide/registry_config.html contains database configurations with different databases and we recommend you to go through it if you are using a different database server other than mysql.)
ESB cluster database configuration
| Instance Name | Registry Space | Instance Name Registry Space Mysql database name/mounted path (x.x.x.x) |
| ESB Master (x.x.x.x) | local | amamaster |
| config | amagregdb /_system/esbConfig | |
| governance | amagregdb /_system/governance | |
| user-manager | amausermgtdb | |
| ESB Slave1 (x.x.x.x) | local | amaslave1 |
| config | amagregdb /_system/esbConfig | |
| governance | amagregdb /_system/governance | |
| user-manager | amausermgtdb | |
| GREG (x.x.x.x) | All | amagregdb |
In the above table there are 5 separate databases with following names
If you are having your database remotely, please configure mysql instance to allow remote connections to accept.
Running Multiple instance of the WSO2 Carbon Servers in the same machine
If you are running multiple instance same or different WSO2 Product Instances, then you need to configure ports for each instance. You can configure $CARBON_HOME/repository/conf/carbon.xml by using portOffset
<Offset>0</Offset>
the define value + Offset. e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445
Let's configure the GREG and if you see the table above you can see we are using a single database for all the things in GREG instance so we need to change the configuration in registry.xml and user-mgt.xml.
Open the GREG_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to change according to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Open the GREG_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to change according to your setup).
<Property name="url">jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
Copy mysql jdbc driver library to GREG_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in GREG.
If we examine the above configuration you can see that we are pointing the database in to same database amagregdb, and now start the GREG instance with -Dsetup. (This will create tables for user manager database and registry database(pick the database configuration from currentDBConfig element in registry.xml since there might be number of database configurations in your registry.xml which we put to perform mounting from other databases but only one database configuration is configured as currentDBConfig)).
After successfully starting GREG instance, create the following directory from GREG UI. This directory will be used by the ESB instances as their mounted registry. /_system/esbConfig. /_system/esbConfig
Now let's configure ESB master node, here we are changing the default database configuration, to point it's local registry to amamaster,point user manager database to amausermgtdb and mount it's configuration and governance registries from GREG database.
Let's first add local registry configuration and user-manager configuration.
Open the ESB_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to change according to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amamaster?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Open the ESB_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to change according to your setup).
<Property name="url">jdbc:mysql://x.x.x.x:3306/amausermgtdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
Now let's configure the mounting configuration, for that we need to add more parameters to registry.xml file.
Open the ESB_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the following database configuration below. (IP addresses and database urls have to change acccording to your setup).
<dbConfig name="esbMountRegistry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://x.x.x.x:9443/registry">
<id>Mount1</id>
<dbConfig>esbMountRegistry</dbConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/esbConfig</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Please change the remoteInstance URL according to the configuration of the GREG running machine. Please note that "InstanceIds", "id" and "dbConfig" elements should be mapped properly if you are using different names for them.
Copy mysql jdbc driver library to ESB_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in ESB.
Now let's configure ESB slave node, here we are changing the default database configuration, to point it's local registry to amaslave1,point user manager database to amausermgtdb and mount it's configuration and governance registries from GREG database.
Let's first add local registry configuration and user-manager configuration.
Open the ESB_SLAVE_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to change acccording to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amaslave1?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<dbConfig name="esbMountRegistry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://x.x.x.x:9443/registry">
<id>Mount1</id>
<dbConfig>esbMountRegistry</dbConfig>
<readOnly>true</readOnly>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/esbConfig</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>Open the ESB_SLAVE_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to change according to your setup). Note : Here we are changing to mode to readOnly in database configuration.
<Configuration>
<AdminRole>admin</AdminRole>
<AdminUser>
<UserName>admin</UserName>
<Password>admin</Password>
</AdminUser>
<EveryOneRoleName>everyone</EveryOneRoleName> <!-- By default users in this role sees the registry root -->
<ReadOnly>true</ReadOnly>
<Property name="url">jdbc:mysql://localhost:3306/amausermgtdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
</Configuration>
Please change the remoteInstance URL according to the configuration of the GREG running machine. Please note that "InstanceIds", "id" and "dbConfig" elements should be mapped properly if you are using different names for them. Note: If you compare the configuration with master node, we are only changing the configuration of local registry and all the other configurations are same with the master node, but the registry access mode in mounting is readonly but the local registry configuration should be read-write since it is specific for each node.
Copy mysql jdbc driver library to ESB_SLAVE_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in ESB.
Before starting the slave ESB node you need to load registry database schema to amaslave1 database. You can find the required SQL script at ESB_SLAVE_HOME/dbscripts directory. After successfully configuring all of this,run the wso2server.sh or wso2server.bat of master node based on your platform with -Dsetup option for the first time start up to create the database tables, and after creating database tables, do not use -Dsetup option in the start.
sh CARBON_HOME/bin/wso2server.sh -Dsetup
You don't need to specify the -Dsetup option for ESB slave node because we have created DB schema in the previous step. Note: -Dsetup doesn't work with slave node because we share a same database for user manager and current -Dsetup option tries to create user manager schema also at the startup if you use -Dsetup.
After setting up the ESB's pointing to a single Registry we have two options for sharing the configuration.
<MediationConfig>
<LoadFromRegistry>false</LoadFromRegistry>
<SaveToFile>false</SaveToFile>
<Persistence>enabled</Persistence>
<RegistryPersistence>enabled</RegistryPersistence>
</MediationConfig>
LoadFromRegistry parameter has to be changed to true in the slave nodes.
Deployment synchronizer can be used to synchronize the main repository of a Carbon server with a collection in the registry. This feature is sometimes also referred to as the registry based repository synchronizer. Deployment synchronizer has a number of use cases.
In this scenario, an ESB Cluster of two nodes are sharing the same configuration registry. With the deployment synchronizer it is possible to maintain all the ESB nodes in the cluster in sync through the shared registry. The master ESB node upload its local repository to the registry using the deployment synchronizer. Slave ESB nodes (slave nodes) can then download the same repository from the registry and deploy locally.
To support this use case, synchronizer has to be run in auto commit mode in the master node. When in auto commit mode, it will periodically upload the changed artifacts in the local repository to the registry.
Similarly slave nodes should run the synchronizer in the auto checkout mode. If needed registry eventing can be employed to run the checkout operations so that a checkout will be made only when some artifact has changed in the shared registry. Note that Carbon repository is located in the repository/deployment/server directory by default. Once enabled, deployment synchronizer uploads the contents of this directory to a collection named repository/deployment/server in the configuration registry (this is configurable).
Runtime State Replication
WSO2 ESB flows are stateless for the most part. But in very rare deployments users have requested to share some of the runtime information among the cluster nodes. For example Caching can be used with runtime state sharing.
Runtime state replication is based on the Tribes group management system. It provides group membership handling and group communication for clustered Carbon Server instances. Although ESB ships with this built in Tribes based implementation, other clustering implementations based on different group management systems can be plugged in easily. The only thing you have to do is to implement a set of clustering interfaces provided by Carbon Server.
ESB clustering currently does not support distributed locking for session data replication. Therefore, we have to deploy primary backup clusters for stateful services. This restriction does not apply to stateless services and you can direct client requests to any node in the cluster for such services.
ESB clustering is configured using the axis2.xml file. So you have to enable clustering in the axis2.xml of each node.
For more details about Carbon Server clustering, please see WSO Carbon Clustering Configuration Language .
NWe can enable clustering by changing the configuration in the axis2.xml. By default, clustering is turned off to avoid additional overhead for individual deployments. Open the axis2.xml and set the value 'true' for the "enable" attribute of the "clustering" element as follows.
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
You may also change clustering properties to suit your deployment as explianed in the above article. However, the default configuration is sufficient for the demonstration.
The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.