[Download] | [Documentation Home] | [Release Note]

Clustering Guide

This document describes the WSAS clustering functionality and demonstrates it using examples.

Content

Why Web Services Clustering?

Any production deployment of a server has to fulfil two basic needs. They are high availability and scalability. High availability refers to the ability to serve client requests by tolerating failures. Scalability is the ability to serve a large number of clients sending a large number of requests without a degradation of the performance. Almost all successful server products support the above two features to some extent. Another requirement that pops up naturally with the above two features is the ability to manage clustered deployments from a single point. There is no exception when it comes to Web services servers. Many large scale enterprises are adapting to Web services as the de facto middleware standard. These enterprises have to process millions of transactions per day, or even more. A large number of clients, both human and computers, connect simultaneously to these systems and initiate transactions. Therefore, the servers hosting the Web services for these enterprises have to support that level of performance and concurrency. In addition, almost all the transactions happening in such enterprise deployments are critical to the business of the organization. This imposes another requirement for production ready Web services servers. That is to maintain very low downtime. It is impossible to support that level of scalability and high availability from a single server despite how powerful the server hardware or how efficient the server software. Web services clustering is needed to solve this, which allows to deploy and manage several instances of identical Web services across multiple Web services servers running on different server machines. Then we can distribute client requests among these machines using a suitable load balancing system to achieve the required level of availability and scalability. WSAS supports clustering out of the box, by providing functionalities like configuring clustered instances from a single point and replication of configurations as well as client sessions. We will dive into WSAS clustering features in the next sections.

WSAS Clustering

WSAS clustering is based on the Tribes group management system. It provides group membership handling and group communication for clustered WSAS instances. Although WSAS ships with this build in Tribes based implementation, other clustering implementations based on different group management systems can be plugged in easily. The only thing you have to do is to implement a set of clustering interfaces provided by WSAS.

Session State Replication

The ability to store session specific data in Web services is an important feature, specially when the business logic is written in the Web services itself. This might not be very useful when Web services are used just as wrappers for existing legacy systems. WSAS has built in support for Web service sessions as the former case is increasingly becoming popular. This introduces another requirement for clustering. That is to replicate session data among all the nodes in the cluster. WSAS stores session data in three levels. Data that should only be available to a service is stored in the ServiceContext. Common data for all the services in a service group is stored in the ServiceGroupContext. Common data for all the service groups is stored in the ConfigurationContext. In clustering deployments, WSAS replicates session data found in all these levels among all the nodes in the cluster. This ensures that all the nodes are identical in terms of session data as well.

High availability





WSAS high availability cluster



Figure 2: WSAS high availability cluster

Session replication in WSAS can be used to achieve high availability for stateful services as shown in figure 2. All the client requests are always directed to Node 1, which is elected as the primary server by the load balancing system. Node 2 and Node 3 are kept as back up servers and none of the requests will be directed to them under normal operations. All three nodes are configured as a WSAS cluster, so that session state changes in Node 1 will be replicated to Node 2 and Node 3.

Now if Node 1 fails, the load balancing system elects Node 2 as the primary server and the other two are considered as backups. From that point, client requests are directed to Node 2, but the client does not notice any change or data loss, as all the session data is available in Node 2 as well.

Scalability





WSAS scalability cluster



Figure 3: WSAS scalability cluster

By combining the WSAS clustering and a smart load balancing system, it is possible to achieve high availability as well as scalability. Such deployment is shown in figure 3. Two WSAS clusters are created in this scenario. Node 1 and Node 2 belong to the first cluster and Node 3 and Node 4 belong to the second cluster. Once a client sends a request through the load balancing system, it binds that client to a particular cluster. From that point onwards, all the requests from that client will always be directed to the same cluster. If a primary node of a cluster failed, the load balancing system should elect a backup node of that cluster as the primary node and direct requests to that. Thus, scalability is achieved as a growing number of clients can be handled simply by introducing new clusters, and the high availability is achieved as each cluster can have backup nodes with an identical configuration and session state.

WSAS clustering currently does not support distributed locking for session data replication. Therefore, we have to deploy primary backup clusters for stateful services as mentioned above. This restriction does not apply to stateless services and you can direct client requests to any node in the cluster for such services.

WSAS Clustering Configuration

WSAS clustering is configured using the axis2.xml file. As all instances of a WSAS cluster can be configured to load this file from the shared repository, initial clustering configuration can be done by editing a single file.

For more details about WSAS clustering, please see WSO Carbon Clustering Configuration Language

Now we have explored enough background information about WSAS clustering. It is time to see it in action.

WSAS Clustering in Action

Configuring the Cluster

Download the WSAS binary distribution by following the installation guide. As we are going to set up a WSAS cluster for the demonstration purpose, we are going to host all the instances in the same computer. Extract the WSAS distribution to two directories, so that we can start two instances. We refer to these two instances as Node 1 and Node 2. The next step is to configure the URL repository for both nodes. Open the server.xml of Node 1 and change the repository location to http://localhost/repository/. Please make sure that the WSAS repository is accessible from the given URL. Then change the configuration file location to point to the axis2.xml of Node 1. The section of the server.xml file containing the changed locations is shown below:

<Axis2Config>
        <!--
             Location of the Axis2 Services & Modules repository

             This can be a directory in the local file system, or a URL.

             e.g.
             1. /home/wso2wsas/repository/ - An absolute path
             2. repository - In this case, the path is relative to CARBON_HOME
             3. file:///home/wso2wsas/repository/
             4. http://wso2wsas/repository/
        -->
        <RepositoryLocation>http://localhost/repository/</RepositoryLocation>

        <!--
            Location of the main Axis2 configuration descriptor file, a.k.a. axis2.xml file

            This can be a file on the local file system, or a URL

            e.g.
            1. /home/wso2wsas/conf/axis2.xml - An absolute path
            2. conf/axis2.xml - In this case, the path is relative to CARBON_HOME
            3. file:///home/wso2wsas/conf/axis2.xml
            4. http://wso2wsas/conf/axis2.xml
        -->
        <ConfigurationFile>/home/wsas/node1/conf/axis2.xml</ConfigurationFile>

        <!--
          ServiceGroupContextIdleTime, which will be set in ConfigurationContex
          for multiple clients which are going to access the same ServiceGroupContext
          Default Value is 30 Sec.
        -->
        <ServiceGroupContextIdleTime>30000</ServiceGroupContextIdleTime>
</Axis2Config>

Now make the same changes for the server.xml file of Node 2. Make sure to set the configuration file location to the axis2.xml file of Node 2. Please note that we are setting different configuration file locations to the nodes to avoid port conflicts mentioned later. In production deployments, all nodes can be pointed to the same axis2.xml file located in the HTTP repository.

Now we are done with configuring the repository. Next we have to enable clustering for both nodes. By default, clustering is turned off to avoid additional overhead for individual deployments. Open the axis2.xml of both nodes and uncomment the clustering section. You may also change clustering properties to suit your deployment. However, the default configuration is sufficient for the demonstration.

There is one more step left as we are trying to run both nodes in the same machine. That is, we have to change the various ports opened by WSAS to avoid conflicts. As some of these ports are configured in the axis2.xml file, we have to use two axis2.xml files for the two WSAS instances, instead of sharing it from the central repository. However, we can share service archives from the central repository between both nodes.

Open the axis2.xml of Node 2 and change the HTTP transport receiver port to 9763. Then change the HTTPS transport receiver port to 9444. A portion of the axis2.xml file after changing the ports is shown below:

<transportReceiver name="http"
                       class="org.wso2.wsas.transport.http.HttpTransportListener">
        <parameter name="port">9763</parameter>
</transportReceiver>

<transportReceiver name="https"
                       class="org.wso2.wsas.transport.http.HttpsTransportListener">
        <parameter name="port">9444</parameter>
        <parameter name="sslProtocol">TLS</parameter>

Now open the server.xml file of Node 2 and change the command listener port to 6667. Portion of the server.xml file containing the command listener port is shown below:

<CommandListener>
       <Port>6667</Port>
</CommandListener>

We have completed configuring the WSAS nodes for clustered deployment. Now start both WSAS nodes using the wso2wsas.sh from the bin directory of Node 1 and Node 2.

Building a Stateful Service

Let's write a small service for the demonstration. This service has two methods called setValue() and getValue(). The setValue() method stores the value passed as the parameter in the service group context. The getValue() method retrieves that value from the service group context and returns it. If this service is deployed in any other scope than the request scope, this value is available between method invocations, making it a stateful service. Code of the service class is listed below:

public class Service1 {

    public OMElement setValue(OMElement param) {

        param.build();
        param.detach();

        String value = param.getText();
        MessageContext.
                getCurrentMessageContext().getServiceGroupContext().setProperty("myValue", value);

        System.out.println("==============================================================");
        System.out.println("Value: " + value + " is set.");
        System.out.println("==============================================================");

        param.setText("Value: " + value + " is set.");       

        return param;
    }

    public OMElement getValue(OMElement param) {

        param.build();
        param.detach();

        String value = (String) MessageContext.
                getCurrentMessageContext().getServiceGroupContext().getProperty("myValue");

        System.out.println("==============================================================");
        System.out.println("Value: " + value + " is retrieved.");
        System.out.println("==============================================================");

        param.setText(value);

        return param;
    }
}

We are going to deploy this in the SOAP session scope. Therefore, set the scope attribute to soapsession in the services.xml file.

<service name="service1" scope="soapsession">
   ...    
</service>

Now compile the service class and make a service archive named service1.aar from it.

Session State Replication Example

Deploy the service1 in the cluster as mentioned in the previous section. Now we have a stateful service deployed in the cluster. We can verify the session state replication by setting some session data in Node 1 and accessing them from Node 2. This is possible as WSAS clustering layer replicates session data added to any node to all other nodes. We have to write a Web services client to set the value in Node 1 and access it from Node 2. This client code is listed below:

public class SessionClient {

    public static void main(String[] args) {
        new SessionClient().invoke();
    }

    private void invoke() {

        String repository = "/home/wso2/products/axis2/repository";
        String axis2xml_path = "/home/wso2/products/axis2/conf/axis2-client.xml";

        OMFactory fac = OMAbstractFactory.getOMFactory();
        OMNamespace ns = fac.createOMNamespace("http://services", "ns");
        OMElement value = fac.createOMElement("Value", ns);
        value.setText("Sample session data");

        try {
            ConfigurationContext configContext = ConfigurationContextFactory.
                    createConfigurationContextFromFileSystem(repository, axis2xml_path);            
            ServiceClient client = new ServiceClient(configContext, null);
            client.engageModule("addressing");

            Options options = new Options();
            options.setTimeOutInMilliSeconds(10000000);
            options.setManageSession(true);
            client.setOptions(options);

            // Set some session data in Node 1
            options.setTo(new EndpointReference("http://192.168.1.3:9762/services/service1"));
            options.setAction("setValue");
            OMElement response1 = client.sendReceive(value);
            System.out.println(
                    "Server response for setting the sample session data in Node 1: " + response1.getText());

            // Access the session data from Node 2
            options.setTo(new EndpointReference("http://192.168.1.3:9763/services/service1"));
            options.setAction("getValue");            
            OMElement response2 = client.sendReceive(value);
            System.out.println("Retrieved sample session data from Node 2: " + response2.getText());

        } catch (AxisFault axisFault) {
            axisFault.printStackTrace();
        }
    }
}

The above code first invokes the setValue() method of our sample service in Node 1 with the string "Sample session data". Then it invokes the getValue() method from Node 2 and displays the retrieved value. Compile and execute this code with required Axis2 jars in the class path. Please make sure that the "repository" string contains a path of a Axis2 repository and "axis2xml_path" contains a path for an axis2.xml file. You can see the following output in the console:

Server response for setting the sample session data in Node 1: Value: Sample session data is set.
Retrieved sample session data from Node 2: Sample session data

The session data "Sample session data" was set in Node 1 and retrieved from Node 2. You may also disable the clustering in the nodes and perform the above test again. You will see that this time session data cannot be accessed from Node 2.

We have completed the WSAS clustering guide. If you have more questions on WSAS clustering functionality, please feel free to ask them on WSAS user list: wsas-java-user@wso2.org.



Also see Introduction to WSO2 Carbon Clustering