March 01, 2011

Load balancing with Glassfish 3.1 and Apache

Glassfish 3.1 was released on February 28th. This is the first application server that supports Java EE 6 in a clustered and high available environment. We at LodgON have a high interest in this, as Johan Vos already mentioned in his blog and we are eager to find out how easy it is to configure Glassfish to let it run multiple instances using Apache as a load balancer. I'll describe the steps I had to go through for setting up the following configuration:

  • machine1: Glassfish DAS, 1 instance, Apache HTTP server with mod_jk enabled

  • machine2: 1 instance


Configuring the nodes



Install Glassfish 3.1 on machine1. Once that is completed, we need to setup ssh so we can manage node machine2 centrally from machine1. So on machine1 enter the following command:

$ asadmin setup-ssh machine2


This is basically a frontend for the ssh-keygen tool. It will ask for the password of your user on machine2 and copies the necessary keyfiles over to that machine. We are now ready to install glassfish on machine2 by using the install-node command:

$ asadmin install-node --installdir /opt/glassfish3 machine2


This copies glassfish over to the node with the host name specified in the last parameter and installs it in the specified installation directory. I encountered a problem where it said that the command jar was not found. To resolve this in Ubuntu, make sure you specify java in your PATH in ~/.bashrc before the following lines:

# If not running interactively, don't do anything
[ -z "$PS1" ] && return


Everything after these lines will not be executed when logging in with an non-interactive shell like in our case, via ssh.

Now we can start the DAS on machine1 and begin configuring our instances.

$ asadmin start-domain


Configuring the cluster



We will now create a remote and a local instance into one cluster. We will first let the DAS know that we have another node on machine2 that we want to use by creating an ssh node:

$ asadmin create-node-ssh --installdir /opt/glassfish3 --nodehost machine2 node1


After that we can create the cluster and add two instances to it: one remote instance on node node1 and a local instance on the local node. This local node, with name localhost-domain1, is automatically available when installing glassfish.

$ asadmin create-cluster cluster1
$ asadmin create-instance --cluster cluster1 --node node1 inst1
$ asadmin create-instance --cluster cluster1 --node localhost-domain1 inst2


We can now start the cluster and deploy an application into it. It doesn't really matter which application you want to deploy. I will use a simple application containing only 1 jsp (download the application):

$ asadmin start-cluster cluster1
$ asadmin deploy --target cluster1 --name simple simple-application.war


When deploying, you specify our cluster1 as the target. This will deploy the application into every instance that was added to the cluster. When deployment was successful, you should be able to browse to the application using the following URL: http://machine2:28080/simple. It should also be running on the second instance at the following URL: http://machine1:28081/simple.

Configuring the load balancer



We have both instances running our application, so now we can start configuring Apache to run as the load balancer. To let Apache connect with Glassfish, you'll have to use the mod_jk module. Installing it in Ubuntu is very easy:

$ sudo apt-get install libapache2-mod-jk


Once the module is installed, create a file called jk.conf in the mods-available directory in your apache configuration directory (mine is located in /etc/apache2/mods-available) and link it into the mods-enabled directory.

$ sudo touch /etc/apache2/mods-available/jk.conf
$ sudo ln -s /etc/apache2/mods-available/jk.conf /etc/apache2/mods-enabled/jk.conf


In that file you can copy the following configuration:

JkWorkersFile /etc/apache2/worker.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"


The first line specifies the location of the workers file in which we will configure our Glassfish instances. So, create the file in the directory /etc/apache2 and add the following properties:

worker.list=worker1,worker2,loadbalancer

# default properties for workers
worker.template.type=ajp13
worker.template.port=28009
worker.template.lbfactor=50
worker.template.connection_pool_timeout=600
worker.template.socket_keepalive=1
worker.template.socket_timeout=300

# properties for worker1
worker.worker1.reference=worker.template
worker.worker1.host=machine1

# properties for worker2
worker.worker2.reference=worker.template
worker.worker2.host=machine2

# properties for loadbalancer
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2


Finally, in the default virtual host configuration file (on my machine this is located at /etc/apache2/sites-available/default) we add a line to tell mod_jk which URLs have to be forwarded to the load balancer:

JkMount /simple/* loadbalancer


You can read more about all these configuration directives and properties on the website of the Apache Tomcat Connector:


When you are done with this, we restart the Apache HTTP server to let the settings take effect:

$ sudo apache2ctl graceful


Finally, we need to add a HTTP listener in the Glassfish cluster that will listen for the calls from mod_jk. As we specified in the workers.properties file, this listener will be listening on port 28009. Adding this listener works with the following command:

$ asadmin create-network-listener --protocol http-listener-1 --listenerport 28009 --jkenabled true --target cluster1-config jk-connector


Restart the cluster and we should be able to connect to our application through Apache at the following URL: http://machine1/simple.

$ asadmin stop-cluster cluster1
$ asadmin start-cluster cluster1


Conclusion



You can see that it was quite easy to set up a clustered Glassfish with Apache running as the load balancer. You might want to watch the following video about the new features in Glassfish 3.1. It might give you a bit more insight in the way that clustering and load balancing works. It will also show you that you can configure everything by using the admin website (running at port 4848) instead of using the asadmin command as I did in this blog.

21 comments:

Todor said...

Thank you very much! I just tested your configuration from the very beginning and I can confirm this works perfectly on RHEL 5.4 except that one has to compile mod-jk from source (just go to the "native" directory and check the file BUILDING.txt). You saved me a lot of efforts :) GBU!

Anonymous said...

Hi, i'm trying to do the same as you with no success.

Not Found

The requested URL /simple/ was not found on this server.

Apache Server at IP.IP.IP.IP Port 80

Maybe apache mod_jk doesn't works?

J R Hamza said...

its a nice tutorial... thank you very much.
I have one issue in get-health cluster1.
it shows instances are not started.
But when listing out the instances, it shows they are running.
validate-multicast also fails with timeout.

please help on this....
thanks

Anonymous said...

if you are using Linux distro, then add the default UDP port (2048) in the firewall for validate-multicast command to execute

Sandy said...

How to force Apache to use sticky session loadbalancer?

Joeri Sykora said...

Hi Sandy,

session stickyness should be enabled by default on apache. There is a property for the loadbalancer worker in the worker.properties file called sticky_session, which has true as the default value (http://tomcat.apache.org/connectors-doc/reference/workers.html).

However, you need to configure an extra step in glassfish to allow session stickyness to work there as well. Every cluster instance needs a system property called jvmRoute and the value of the property should be the name of the worker. From my worker.properties samples file, you'll have inst1 with an extra system property -DjvmRoute=worker2, while inst2 has the system property -DjvmRoute=worker1.

If you don't want to use the name of the worker as the jvmRoute name, then you can specify an extra setting called route in the worker.properties file, eg: worker.worker1.route=sampleroutename and use that as the value for the jvmRoute system property.

You can read more here: http://download.oracle.com/docs/cd/E19798-01/821-1751/gixpx/index.html

Wang Zi said...

Thanks very much, it saves me a lot of time. This tutorial is even better than the offical one.

And here I have a question, how do you make sure there is no downtime during redeployment?

Wang Zi said...

It seems glassfish do not support sticky mode.

Joeri Sykora said...

Glassfish does support sticky sessions. Read my previous comment for more information about how this is done. I really should be writing a follow-up on session stickyness it seems.

You can follow the following procedure to deploy a new version of your application in for example a cluster with two instances without downtime:

- remove one instance from the cluster
- deploy your application in this instance
- add the instance back into the cluster

Repeat these steps again for the second instance.

Wang Zi said...

It's funnny if you do not enable sticky mode in glassfish. The user will be disconnect from your application 50% if one of the inctance is down

Wang Zi said...

Can you explain clearly how to use the worker.worker1.route? You give an example in two inctance. How about 3/4 or 5?

Thank you for your patient.

Wang Zi said...

When you say "remove one instance from the cluster", did you mean delete the instance from the cluster?

Wang Zi said...

eg: worker.worker1.route=sampleroutename

Where do I find sampleroutename?

Wang Zi said...

I found the session is replicated to the other node.

Anonymous said...

Thank you very much!
I've been trying to find a step-by-step tutorial for this issue in the last 3 days.


What's the meaning of the http-listener-1 value?

Joeri Sykora said...

http-listener-1 is the name of the HTTP protocol that you want to associate with the new HTTP connection. Each HTTP connection needs an HTTP protocol and http-listener-1 happens to be one of the default configured protocols available.

Unknown said...

Nice tutorial "oauthprovider" but I miss
"com.lodgon.extras.util.StringUtil"

Joeri Sykora said...
This comment has been removed by the author.
Joeri Sykora said...

The StringUtil class is available through a maven dependency:

<dependency>
  <groupId>com.lodgon.extras</groupId>
  <artifactId>lodgon-util</artifactId>
  <version>1.10</version>
</dependency>

It should be automatically detected by maven, but you can download it manually from our maven repository if you want: lodgon-util-1.10.jar.

lee said...

After struggling for a couple of days trying to follow other like procedures, this one worked. Thank you.

Andrea Negri said...

I followed your instructions and everything worked well, thanks a lot!

Just for the sessions, I've a problem (I also followed this link you provided: http://docs.oracle.com/cd/E19798-01/821-1751/gixpx/index.html): when i try to login in my web app (operation that should insert my username in session), i can't retrive it and it seems as i'm not logged in (no username in session).

While if i try to run it on the single instances, everithing works.

Any idea? I've setted Availability Service: Enabled, and tried with Single-Sign-On State both enabled or disabled, but nothing changes...

Thanks a lot!
Andrea