Load balancing with Glassfish 3.1 and Apache

Glassfish 3.1 was released on February 28th. This is the first application server that supports Java EE 6 in a clustered and high available environment. We at LodgON have a high interest in this, as Johan Vos already mentioned in his blog and we are eager to find out how easy it is to configure Glassfish to let it run multiple instances using Apache as a load balancer. I'll describe the steps I had to go through for setting up the following configuration:

• machine1: Glassfish DAS, 1 instance, Apache HTTP server with mod_jk enabled


• machine2: 1 instance

Configuring the nodes

Install Glassfish 3.1 on machine1. Once that is completed, we need to setup ssh so we can manage node machine2 centrally from machine1. So on machine1 enter the following command:

$ asadmin setup-ssh machine2

This is basically a frontend for the ssh-keygen tool. It will ask for the password of your user on machine2 and copies the necessary keyfiles over to that machine. We are now ready to install glassfish on machine2 by using the install-node command:

$ asadmin install-node --installdir /opt/glassfish3 machine2

This copies glassfish over to the node with the host name specified in the last parameter and installs it in the specified installation directory. I encountered a problem where it said that the command jar was not found. To resolve this in Ubuntu, make sure you specify java in your PATH in ~/.bashrc before the following lines:

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

Everything after these lines will not be executed when logging in with an non-interactive shell like in our case, via ssh.

Now we can start the DAS on machine1 and begin configuring our instances.

$ asadmin start-domain

Configuring the cluster

We will now create a remote and a local instance into one cluster. We will first let the DAS know that we have another node on machine2 that we want to use by creating an ssh node:

$ asadmin create-node-ssh --installdir /opt/glassfish3 --nodehost machine2 node1

After that we can create the cluster and add two instances to it: one remote instance on node node1 and a local instance on the local node. This local node, with name localhost-domain1, is automatically available when installing glassfish.

$ asadmin create-cluster cluster1
$ asadmin create-instance --cluster cluster1 --node node1 inst1
$ asadmin create-instance --cluster cluster1 --node localhost-domain1 inst2

We can now start the cluster and deploy an application into it. It doesn't really matter which application you want to deploy. I will use a simple application containing only 1 jsp (download the application):

$ asadmin start-cluster cluster1
$ asadmin deploy --target cluster1 --name simple simple-application.war

When deploying, you specify our cluster1 as the target. This will deploy the application into every instance that was added to the cluster. When deployment was successful, you should be able to browse to the application using the following URL: http://machine2:28080/simple. It should also be running on the second instance at the following URL: http://machine1:28081/simple.

Configuring the load balancer

We have both instances running our application, so now we can start configuring Apache to run as the load balancer. To let Apache connect with Glassfish, you'll have to use the mod_jk module. Installing it in Ubuntu is very easy:

$ sudo apt-get install libapache2-mod-jk

Once the module is installed, create a file called jk.conf in the mods-available directory in your apache configuration directory (mine is located in /etc/apache2/mods-available) and link it into the mods-enabled directory.

$ sudo touch /etc/apache2/mods-available/jk.conf
$ sudo ln -s /etc/apache2/mods-available/jk.conf /etc/apache2/mods-enabled/jk.conf

In that file you can copy the following configuration:

JkWorkersFile /etc/apache2/worker.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"

The first line specifies the location of the workers file in which we will configure our Glassfish instances. So, create the file in the directory /etc/apache2 and add the following properties:

worker.list=worker1,worker2,loadbalancer

# default properties for workers
worker.template.type=ajp13
worker.template.port=28009
worker.template.lbfactor=50
worker.template.connection_pool_timeout=600
worker.template.socket_keepalive=1
worker.template.socket_timeout=300

# properties for worker1
worker.worker1.reference=worker.template
worker.worker1.host=machine1

# properties for worker2
worker.worker2.reference=worker.template
worker.worker2.host=machine2

# properties for loadbalancer
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2

Finally, in the default virtual host configuration file (on my machine this is located at /etc/apache2/sites-available/default) we add a line to tell mod_jk which URLs have to be forwarded to the load balancer:

JkMount /simple/* loadbalancer

You can read more about all these configuration directives and properties on the website of the Apache Tomcat Connector:

• the apache directives


• the workers.properties settings

When you are done with this, we restart the Apache HTTP server to let the settings take effect:

$ sudo apache2ctl graceful

Finally, we need to add a HTTP listener in the Glassfish cluster that will listen for the calls from mod_jk. As we specified in the workers.properties file, this listener will be listening on port 28009. Adding this listener works with the following command:

$ asadmin create-network-listener --protocol http-listener-1 --listenerport 28009 --jkenabled true --target cluster1-config jk-connector

Restart the cluster and we should be able to connect to our application through Apache at the following URL: http://machine1/simple.

$ asadmin stop-cluster cluster1
$ asadmin start-cluster cluster1

Conclusion

You can see that it was quite easy to set up a clustered Glassfish with Apache running as the load balancer. You might want to watch the following video about the new features in Glassfish 3.1. It might give you a bit more insight in the way that clustering and load balancing works. It will also show you that you can configure everything by using the admin website (running at port 4848) instead of using the asadmin command as I did in this blog.