mod_cluster is an httpd-based load balancer. Like mod_jk and mod_proxy, mod_cluster uses a communication channel to forward requests from httpd to one of a set of application server nodes. Unlike mod_jk and mod_proxy, mod_cluster leverages an additional connection between the application server nodes and httpd. The application server nodes use this connection to transmit server-side load balance factors and lifecycle events back to httpd via a custom set of HTTP methods, affectionately called the Mod-Cluster Management Protocol (MCMP). This additional feedback channel allows mod_cluster to offer a level of intelligence and granularity not found in other load balancing solutions.
Within httpd, mod_cluster is implemented as a set of modules for httpd with mod_proxy enabled. Much of the logic comes from mod_proxy, e.g. mod_proxy_ajp provides all the AJP logic needed by mod_cluster.
JBoss already prepares binary packages with httpd and mod_cluster so you can quickly try mod_cluster on the following platforms:
Linux x86, x64, ia64
Solaris x86, SPARC
Windows x86, x64, ia64
HP-UX PA-RISC, ia64
mod_cluster boasts the following advantages over other httpd-based load balancers:
Traditional httpd-based load balancers require explicit configuration of the workers available to a proxy. In mod_cluster, the bulk of the proxy's configuration resides on the application servers. The set of proxies to which an application server will communicate is determined either by a static list or using dynamic discovery via the advertise mechanism. The application server relays lifecycle events (e.g. server startup/shutdown) to the proxies allowing them to effectively auto-configure themselves. Notably, the graceful shutdown of a server will not result in a failover response by a proxy, as is the case with traditional httpd-based load balancers.
In contrast with traditional httpd-based load balancers, mod_cluster uses load balance factors calculated and provided by the application servers, rather than computing these in the proxy. Consequently, mod_cluster offers a more robust and accurate set of load metrics than is available from the proxy. See the chapter entitled Server-Side Load Metrics for details.
Traditional httpd-based load balancers do not handle web application undeployments particularly well. From the proxy's perspective requests to an undeployed web application are indistinguishable from a request for an non-existent resource, and will result in 404 errors. In mod_cluster, each server forwards any web application context lifecycle events (e.g. web-app deploy/undeploy) to the proxy informing it to start/stop routing requests for a given context to that server.
Unlike mod_jk, mod_cluster does not require AJP. httpd connections to application server nodes can use HTTP, HTTPS, or AJP.
The original concepts are described in a wiki.
httpd-2.2.8+
JBoss AS 5.0.0+ or JBossWeb 2.1.1+
httpd-2.2.8+ is already in the bundles, so if you use the bundle you don't need to download Apache httpd.
mod_cluster uses shared memory to keep the nodes description, the shared memory is created at the start of httpd and the structure of each item is fixed. The following can't be changed by configuration directives.
Max Alias length 40 characters (Host: hostname header, Alias in <Host/>).
Max context length 40 (for example myapp.war deploys in /myapp /myapp is the context).
Max balancer name length 40 (balancer property in mbean).
Max JVMRoute string length 80 (JVMRoute in <Engine/>).
Max load balancing group name length 20 (domain property in mbean).
Max hostname length for a node 64 (address in the <Connector/>).
Max port length for a node 7 (8009 is 4 characters, port in the <Connector/>).
Max scheme length for a node 6 (possible values are http, https, ajp, liked with the protocol of <Connector/>).
Max cookie name 30 (the header cookie name for sessionid default value: JSESSIONID from org.apache.catalina.Globals.SESSION_COOKIE_NAME).
Max path name 30 (the parameter name for the sessionid default value: jsessionid from org.apache.catalina.Globals.SESSION_PARAMETER_NAME).
Max length for a sessionid 120 (something like BE81FAA969BF64C8EC2B6600457EAAAA.node01).
Download the latest mod_cluster release here.
The release is comprised of the following artifacts:
httpd binaries for common platforms
JBoss AS/JBossWeb/Tomcat binary distribution
Alternatively, you can build from source using the Git repository:
https://github.com/modcluster/mod_cluster/tagsrelease/
Building httpd modules
Building server-side components
If you want to skip the details and just set up a minimal working installation of mod_cluster, see the Quick Start Guide.
Configuring httpd
Configuring JBoss AS
Configuring JBoss Web or Tomcat
Migrating from mod_jk or mod_proxy is fairly straight forward. In general, much of the configuration previously found in httpd.conf is now defined in the application server nodes.
Both the request connections between httpd and the application server nodes, and the feedback channel between the nodes and httpd can be secured. The former is achieved via the mod_proxy_https module and a corresponding ssl-enabled HTTP connector in JBoss Web. The latter requires the mod_ssl module and explicit configuration in JBoss AS/Web.
The mod_cluster binary distribution for JBoss AS/JBossWeb/Tomcat includes a demo application that helps demonstrate how different server-side scenarios affect the routing of client requests by the load balancer. The demo application is located in the mod_cluster distribution's demo directory.
mod_cluster contains mod_ssl, therefore the following warning (copied from OpenSSL) applies:
PLEASE REMEMBER THAT EXPORT/IMPORT AND/OR USE OF STRONG CRYPTOGRAPHY SOFTWARE, PROVIDING CRYPTOGRAPHY HOOKS OR EVEN JUST COMMUNICATING TECHNICAL DETAILS ABOUT CRYPTOGRAPHY SOFTWARE IS ILLEGAL IN SOME PARTS OF THE WORLD. SO, WHEN YOU IMPORT THIS PACKAGE TO YOUR COUNTRY, RE-DISTRIBUTE IT FROM THERE OR EVEN JUST EMAIL TECHNICAL SUGGESTIONS OR EVEN SOURCE PATCHES TO THE AUTHOR OR OTHER PEOPLE YOU ARE STRONGLY ADVISED TO PAY CLOSE ATTENTION TO ANY EXPORT/IMPORT AND/OR USE LAWS WHICH APPLY TO YOU. THE AUTHORS OF OPENSSL ARE NOT LIABLE FOR ANY VIOLATIONS YOU MAKE HERE. SO BE CAREFUL, IT IS YOUR RESPONSIBILITY.
The following are the steps to set up a minimal working installation of mod_cluster on a single httpd server and a single back end server, either JBoss AS, JBossWeb or Tomcat. The steps can be repeated to add as many httpd servers or back end servers to your cluster as is desired.
The steps shown here are not intended to demonstrate how to set up a production install of mod_cluster; for example using SSL to secure access to the httpd-side mod_manager component is not covered. See the httpd-side and java-side configuration documentation for the full set of configuration options.
Download the latest httpd and java release bundles. If there is no pre-built httpd bundle appropriate for your OS or system architecture, you can build the binary from source.
The httpd-side bundles are gzipped tars and include a full httpd install. As they contain already an Apache httpd install you don't need to download Apache httpd. Just extract them in root, e.g.
cd / tar xvf mod-cluster-1.0.0-linux2-x86-ssl.tar.gz
That will give you a full httpd install in your /opt/jboss
directory.
If you already have a working httpd install that you would prefer to use, you'll need to download the bundle named mod_cluster httpd dynamic libraries corresponding to you plaform, extract the modules and copy them directory to your httpd install's module directory.
cd /tmp tar xvf mod_cluster-1.x.y.Final-linux2-x86-so.tar.gz
And then you have to copy the files below to you module directory:
mod_slotmem.so
mod_manager.so
mod_proxy_cluster.so
mod_advertise.so
Since 1.1.0.CR2 a script opt/jboss/httpd/sbin/installhome.sh
allows reconfiguration of the bundle installation so that it can run in
user's home directory.
To do that just extract the bundle in your home directory and run the script.
Once that done, httpd will run on port 8000 and will accept MCMP messages on
localhost:6666 and offer /mod_cluster_manager
on the same host and port.
Unzip the bundle corresponding to your architecture.
Change to the bin directory of the subfolder httpd-2.2 where you unzip the bundle and run the installconf.bat shell script.
cd httpd-2.2\bin installconf.bat
You may run httpd directly by using:
httpd.exe
or install Apache httpd as a service:
httpd.exe -k install
and start the service via net start or using httpd.exe:
net start Apache2.2
httpd.exe -k start
Note that in the windows bundles have a flat directory structure, so you have httpd-2.2/modules/ instead of
opt/jboss/httpd/lib/httpd/modules
.
Since 1.1.0.CR2 httpd.conf is preconfigured with the Quick Start values.
You should adapt the default values to your configuration with older mod_cluster we will have to add the following to httpd.conf.
If you extracted the download bundle to root as shown above and are using that extract as your httpd install,
httpd.conf is located in /opt/jboss/httpd/httpd/conf
.
LoadModule proxy_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy.so LoadModule proxy_ajp_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy_ajp.so LoadModule slotmem_module /opt/jboss/httpd/lib/httpd/modules/mod_slotmem.so LoadModule manager_module /opt/jboss/httpd/lib/httpd/modules/mod_manager.so LoadModule proxy_cluster_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy_cluster.so LoadModule advertise_module /opt/jboss/httpd/lib/httpd/modules/mod_advertise.so Listen 10.33.144.3:6666 <VirtualHost 10.33.144.3:6666> <Location /> Order deny,allow Deny from all Allow from 10.33.144. </Location> KeepAliveTimeout 60 MaxKeepAliveRequests 0 ManagerBalancerName mycluster AdvertiseFrequency 5 </VirtualHost>
If you are using your own install of httpd, httpd.conf is found in your install's conf directory. The content to add to httpd.conf is slightly different from the above (different path to the various .so files):
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule slotmem_module modules/mod_slotmem.so LoadModule manager_module modules/mod_manager.so LoadModule proxy_cluster_module modules/mod_proxy_cluster.so LoadModule advertise_module modules/mod_advertise.so Listen 10.33.144.3:6666 <VirtualHost 10.33.144.3:6666> <Location /> Order deny,allow Deny from all Allow from 10.33.144. </Location> KeepAliveTimeout 60 MaxKeepAliveRequests 0 ManagerBalancerName mycluster AdvertiseFrequency 5 </VirtualHost>
First, extract the java-side binary to a temporary directory. The following assumes it was extracted to /tmp/mod-cluster
Your next step depends on whether your target server is JBoss AS or JBoss Web/Tomcat.
You don't need to do anything to install the java-side binaries in AS 6.x; it's part of the AS distribution's default, standard and all configurations.
Assuming $JBOSS_HOME indicates the root of your JBoss AS install and that you want to use mod_cluster in the AS's all config:
cp -r /tmp/mod-cluster/mod-cluster.sar $JBOSS_HOME/server/all/deploy
Assuming $CATALINA_HOME indicates the root of your Tomcat install:
cp -r /tmp/mod-cluster/JBossWeb-Tomcat/lib/* $CATALINA_HOME/lib/Note that you should remove in the $CATALINA_HOME/lib/ directory the mod_cluster-container-tomcat6* file in Tomcat7 and the mod_cluster-container-tomcat7* in Tomcat 6.
Edit the $CATALINA_HOME/conf/server.xml
file, adding the following next to the other <Listener/> elements:
<Listener className="org.jboss.modcluster.container.catalina.standalone.ModClusterListener" advertise="true"/>
Edit the $JBOSS_HOME/server/all/deploy/jboss-web.deployer/server.xml
file, adding the following next to the other <Listener/> elements:
<Listener className="org.jboss.modcluster.container.catalina.standalone.ModClusterListener" advertise="true"/>
Repeat the back-end server install and configuration steps for each server in your cluster.
The load balancing demo application is a good way to learn about mod_cluster's capabilities.
You need to load the modules that are needed for mod_cluster for example:
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule slotmem_module modules/mod_slotmem.so LoadModule manager_module modules/mod_manager.so LoadModule proxy_cluster_module modules/mod_proxy_cluster.so LoadModule advertise_module modules/mod_advertise.so
mod_proxy and mod_proxy_ajp are standard httpd modules. mod_slotmem is a shared slotmem memory provider. mod_manager is the module that reads information from JBoss AS/JBossWeb/Tomcat and updates the shared memory information. mod_proxy_cluster is the module that contains the balancer for mod_proxy. mod_advertise is an additional module that allows httpd to advertise via multicast packets the IP and port where the mod_cluster is listening. This multi-module architecture allows the modules to easily be changed depending on what the customer wants to do. For example when using http instead of AJP, only
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
needs to be changed to:
LoadModule proxy_http_module modules/mod_proxy_http.so
mod_proxy directives like ProxyIOBufferSize could be used to configure mod_cluster. There is no need to use ProxyPass directives because mod_cluster automatically configures which URLs have to be forwarded to JBossWEB.
CreateBalancers: Define how the balancer are created in the httpd VirtualHosts, this is to allow directives like:
ProxyPass / balancer://mycluster1/
0: Create in all VirtualHosts defined in httpd.
1: Don't create balancers (requires at least one ProxyPass/ProxyPassMatch to define the balancer names).
2: Create only the main server.
Default: 2
Note: When using 1 don't forget to configure the balancer in the ProxyPass directive, because the default is empty stickysession and nofailover=Off and the values received via the MCMP CONFIG message are ignored.
UseAlias: Check that the Alias corresponds to the ServerName (See Host Name Aliases)
Off: Don't check (ignore Aliases)
On: Check aliases
Default: Off Ignore the Alias information from the nodes.
Versions older than 1.3.1.Final only accept values 0 and 1 respectively.
LBstatusRecalTime: Time interval in seconds for loadbalancing logic to recalculate the status of a node
Default: 5 seconds
The actual formula to recalculate the status of a node is:
status = lbstatus + (elected - oldelected) * 1000)/lbfactor;
lbfactor is received for the node via STATUS messages.lbstatus is recalculated every LBstatusRecalTime seconds using the formula:
lbstatus = (elected - oldelected) * 1000)/lbfactor;
elected is the number of time the worker was elected.oldelected is elected last time the lbstatus was recalculated.The node with the lowest status is selected. Nodes with lbfactor ≤ 0 are skipped by the both calculation logic.
WaitForRemove: Time in seconds before a removed node is forgotten by httpd
Default: 10 seconds
ProxyPassMatch/ProxyPass: ProxyPassMatch and ProxyPass are mod_proxy directives that when using ! (instead the back-end url) prevent to reverse-proxy in the path. This could be used allow httpd to serve static information like images.
ProxyPassMatch ^(/.*\.gif)$ !
The above for example will allow httpd to server directly the .gif files.
The Context of a mod_manger directive is VirtualHost except mentioned otherwise.
server config
means that it must be outside a VirtualHost configuration.
If not an error message will be displayed and httpd won't start.
EnableMCPMReceive - Allow the VirtualHost to receive MCPM. Allow the VirtualHost to receive the MCPM from the nodes. You need one EnableMCPMReceive in your httpd configuration to allow mod_cluster to work, put it in the VirtualHost where you configure advertise.
MemManagerFile: That is the base name for the names mod_manager will use to store configuration, generate keys for shared memory or lock files. That must be an absolute path name; the directories will created if needed. It is highly recommended that those files are placed on a local drive and not an NFS share. (Context: server config)
Default: $server_root/logs/
Maxcontext: That is the number max of contexts supported by mod_cluster. (Context: server config)
Default: 100
Maxnode: That is the number max nodes supported by mod_cluster. (Context: server config)
Default: 20
Maxhost: That is the number max host (Aliases) supported by mod_cluster. That is also the max number of balancers. (Context: server config)
Default: 20
Maxsessionid: That is the number of active sessionid we store to give number of active sessions in the mod_cluster-manager handler. A session is unactive when mod_cluster doesn't receive any information from the session in 5 minutes. (Context: server config)
Default: 0 (the logic is not activated).
MaxMCMPMaxMessSize: Maximum size of MCMP messages. from other Max directives.
Default: calculated from other Max directives. Min: 1024
ManagerBalancerName: That is the name of balancer to use when the JBoss AS/JBossWeb/Tomcat doesn't provide a balancer name.
Default: mycluster
PersistSlots: Tell mod_slotmem to persist the nodes, Alias and Context in files. (Context: server config)
Default: Off
CheckNonce: Switch check of nonce when using mod_cluster-manager handler on | off Since 1.1.0.CR1
Default: on Nonce checked
AllowDisplay: Switch additional display on mod_cluster-manager main page on | off Since 1.1.0.GA
Default: off Only version displayed
AllowCmd: Allow commands using mod_cluster-manager URL on | off Since 1.1.0.GA
Default: on Commmands allowed
ReduceDisplay - Reduce the information the main mod_cluster-manager page to allow more nodes in the page. on | off
Default: off Full information displayed
SetHandler mod_cluster-manager: That is the handler to display the node mod_cluster sees from the cluster. It displays the information about the nodes like INFO and additionaly counts the number of active sessions.
<Location /mod_cluster_manager> SetHandler mod_cluster-manager Order deny,allow Deny from all Allow from 127.0.0.1 </Location>
When accessing the location you define in httpd.conf you get something like:
Note that:
Transferred: Corresponds to the POST data send to the back-end server.
Connected: Corresponds to the number of requests been processed when the mod_cluster status page was requested.
sessions: Corresponds to the number of sessions mod_cluster report as active (on which there was a request during the past 5 minutes). That field is not present when Maxsessionid is zero.
mod_advertise uses multicast packets to advertise the VirtualHost where it is configured that must be the same VirtualHost where mod_manager is defined. Of course at least one mod_advertise must be in the VirtualHost to allow mod_cluster to find the right IP and port to give to the ClusterListener.
ServerAdvertise On: Use the advertise mechanism to tell the JBoss AS/JBossWeb/Tomcat to whom it should send the cluster information.
ServerAdvertise On http://hostname:port: Tell the hostname and port to use. Only needed if the VirtualHost is not defined correctly, if the VirtualHost is a Name-based Virtual Host or when VirtualHost is not used.
ServerAdvertise Off: Don't use the advertise mechanism.
Default: Off. (Any Advertise directive in a VirtualHost sets it to On in the VirtualHost)
AdvertiseGroup IP:port: That is the multicast address to use (something like 232.0.0.2:8888 for example). IP should correspond to AdvertiseGroupAddress and port to AdvertisePort in the JBoss AS/JBossWeb/Tomcat configuration. Note that if JBoss AS is used and the -u startup switch is included in the AS startup command, the default AdvertiseGroupAddress is the value passed via the -u. If port is missing the default port will be used: 23364.
Default: 224.0.1.105:23364.
AdvertiseFrequency seconds[.miliseconds]: Time between the multicast messages advertising the IP and port.
Default: 10 Ten seconds.
AdvertiseSecurityKey value: key string used to verify advertisements checksums. If configured on either side the verification is required. Both sides must use the same security key.
Default: No default value.
AdvertiseManagerUrl value: Not used in this version (It is sent in the X-Manager-Url: value header). That is the URL that JBoss AS/JBossWeb/Tomcat should use to send information to mod_cluster
Default: No default value. Information not sent.
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule slotmem_module modules/mod_slotmem.so LoadModule manager_module modules/mod_manager.so LoadModule proxy_cluster_module modules/mod_proxy_cluster.so LoadModule advertise_module modules/mod_advertise.so Listen 10.33.144.3:6666 <VirtualHost 10.33.144.3:6666> <Location /> Order deny,allow Deny from all Allow from 10.33.144. </Location> KeepAliveTimeout 60 MaxKeepAliveRequests 0 ManagerBalancerName mycluster ServerAdvertise On EnableMCPMReceive </VirtualHost>
To build httpd-2.2.x from its sources see ASF httpd doc
If needed, patch the httpd-2.2.x sources with (The patch prevents long waiting time when the node IP can't be resolved that should not happen so you can skip the patch part if you don't want to rebuild httpd). mod_proxy_ajp.patch
(cd modules/proxy patch -p0 < $location/mod_proxy_ajp.patch )
Configure httpd with something like:
./configure --prefix=apache_installation_directory \ --with-mpm=worker \ --enable-mods-shared=most \ --enable-maintainer-mode \ --with-expat=builtin \ --enable-ssl \ --enable-proxy \ --enable-proxy-http \ --enable-proxy-ajp \ --disable-proxy-balancer
Rebuild (make) and reinstall (make install) after that.
You need an httpd installation with mod_proxy (--enable-proxy) and ajp protocol (--enable-proxy-ajp) enabled and with dso enabled (--enable-so)
Download the mod_cluster sources:
git clone git://github.com/modcluster/mod_cluster.git
Build the mod_cluster modules components, for each subdirectory advertise, mod_manager, mod_proxy_cluster and mod_slotmem do something like:
sh buildconf ./configure --with-apxs=apxs_file_location make cp *.so apache_installation_directory/modules
Where apache_installation_directory is the location of an installed version of httpd-2-2.x.
NOTE: You can ignore the libtool message on most platform:
libtool: install: warning: remember to run `libtool --finish apache_installation_directory/modules'
Once that is done use Apache httpd configuration to configure mod_cluster.
Several bundles are available at http://www.jboss.org/mod_cluster/downloads.html.
In case you can't find a prepared package of mod_cluser in the download area, it is possible to build mod_cluster for the sources. You need a distribution of httpd (at least 2.2.8) or (better) a source tarball of httpd and the sources of mod_cluster. Building explains how to build mod_cluster for it sources.
A minimal configuration is needed in httpd (See httpd.conf). A listener must be a added in in JBossWEB conf/server.xml (See Configuring JBoss AS/Web).
The bundles are tar.gz on POSIX platforms just extract them in root something like:
cd / tar xvf mod-cluster-1.x.y-linux2-x86-ssl.tar.gz
The httpd.conf is located in /opt/jboss/httpd/httpd/conf to quick test just add something like:
Listen 10.33.144.3:6666 <VirtualHost 10.33.144.3:6666> <Directory /> Order deny,allow Deny from all Allow from 10.33.144. </Directory> KeepAliveTimeout 60 MaxKeepAliveRequests 0 ManagerBalancerName mycluster AdvertiseFrequency 5 </VirtualHost>
To start httpd do the following:
/opt/jboss/httpd/sbin/apachectl start
NOTE: Make sure to use SSL before going in production.
mod_cluster is supported in AS7 via the modcluster subsystem See AS7.
In other AS version mod_cluster's configuration resides within the following file:
$JBOSS_HOME/server/$PROFILE/deploy/mod_cluster.sar/META-INF/mod_cluster-jboss-beans.xml
file.
The entry point for mod_cluster's server-side configuration is the ModClusterListener
bean, which delegates web container (i.e. JBoss Web) specific events to a container agnostic event handler.
In general, the ModClusterListener
bean defines:
A ContainerEventHandler
in which to handle events from the web container.
A reference to the JBoss mbean server.
e.g.
<bean name="ModClusterListener" class="org.jboss.modcluster.container.jbossweb.JBossWebEventHandlerAdapter"> <constructor> <parameter class="org.jboss.modcluster.container.ContainerEventHandler"> <inject bean="ModClusterService"/> </parameter> <parameter class="javax.management.MBeanServer"> <inject bean="JMXKernel" property="mbeanServer"/> </parameter> </constructor> </bean>
In non-clustered mode, each JBoss AS node communicates with the load balancer directly, and do not communicate with each other.
Non-clustered mode is configured via the ModClusterService
bean.
In general, the ModClusterService
bean defines:
An object containing mod_cluster's configuration properties.
An object responsible for calculating the load balance factor for this node. This is described in detail in the chapter entitled Server-Side Load Metrics.
e.g.
<bean name="ModClusterService" class="org.jboss.modcluster.ModClusterService" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=ModCluster",exposedInterface=org.jboss.modcluster.ModClusterServiceMBean.class)</annotation> <constructor> <parameter class="org.jboss.modcluster.config.ModClusterConfig"> <inject bean="ModClusterConfig"/> </parameter> <parameter class="org.jboss.modcluster.load.LoadBalanceFactorProvider"> <inject bean="DynamicLoadBalanceFactorProvider"/> </parameter> </constructor> </bean>
The ModClusterConfig
bean enumerates the configuration properties used by mod_cluster.
For the complete list of configuration properties and their default values, see the chapter entitled Server-side Configuration Properties.
e.g.
<bean name="ModClusterConfig" class="org.jboss.modcluster.config.ModClusterConfig" mode="On Demand"> <!-- Specify configuration properties here --> </bean>
Like mod_jk and mod_proxy_balancer, mod_cluster requires a connector in your server.xml to which to forward web requests. Unlike mod_jk and mod_proxy_balancer, mod_cluster is not confined to AJP, but can use HTTP as well. While AJP is generally faster, an HTTP connector can optionally be secured via SSL. If multiple possible connectors are defined in your server.xml, mod_cluster uses the following algorithm to choose between them:
If an native (APR) AJP connector is available, use it.
If an AJP connector is available, use it.
Otherwise, choose the HTTP connector with the highest max threads.
Like mod_jk and mod_proxy_balancer, mod_cluster identifies each node via a unique jvm route. By default, mod_cluster uses the following algorithm to assign the jvm route for a given node:
Use the value from server.xml
, <Engine jvmRoute="..."/>
, if defined.
Generate a jvm route using the configured ???. The default implementation does the following:
Use the value of the jboss.mod_cluster.jvmRoute
system property, if defined.
Generate a UUID.
While UUIDs are ideal for production systems, in a development or testing environment, it is useful to know which node served a given request just by looking at the jvm route.
In this case, you can utilize the org.jboss.modcluster.SimpleJvmRouteFactory
.
The factory generates jvm routes of the form:
bind-address:port:engine-name
mod_cluster's entire configuration for JBoss Web or Tomcat resides entirely within $CATALINA_HOME/conf/server.xml
.
This limits the adds the following constraints to mod_cluster's feature set:
Only non-clustered mode is supported
Only one load metric can be used to calculate a load factor.
The entry point for JBoss Web and Tomcat configuration is the ModClusterListener.
All mod_cluster configuration properties are defined as attributes of the <Listener/>
element.
For the complete list of configuration properties and their default values, see the chapter entitled Server-side Configuration Properties.
e.g.
<Listener className="org.jboss.modcluster.container.catalina.standalone.ModClusterListener" advertise="true"/>
mod_cluster uses jboss-logging, which exists in JBoss Web, but not in Tomcat. Consequently, to use mod_cluster with Tomcat, it is necessary to add jboss-logging-spi.jar to $CATALINA_HOME/lib.
In mod_cluster 1.2.x the web containers now are separed cleanly, that requires you to change standalone and mbean configuration a little.
The JBoss microcontainer configuration for mod_cluster 1.1.x is not compatible with the configuration for 1.2.x. you need to change the ModClusterListener mbean class to: "org.jboss.modcluster.container.jbossweb.JBossWebEventHandlerAdapter".
In mod_cluster 1.0.x, you were required to make several manual configuration changes to the jbossweb service before mod_cluster would be usable. mod_cluster 1.1.x eliminates much of this hassle - and is designed to be fully functional out of the box.
In 1.0.x, mod_cluster needed to be deployed before JBoss Web. This ensured that mod_cluster was available to handle web application deployment lifecycle events during triggered during server startup. To achieve this, you had to add an explicit dependency on mod_cluster to jbossweb's jboss-beans.xml config file.
mod_cluster 1.1.x reverses this dependency, such that mod_cluster now depends on JBoss Web. Consequently, you no longer need to make any changes to JBoss Web's jboss-beans.xml file.
As an additional consequence of this dependency reversal, mod_cluster can now be hot-deployed or undeployed to a running server without consequence to the JBoss Web service.
mod_cluster 1.0.x required you to add a lifecycle <Listener/> element to server.xml.
The ModClusterListener bean now registers itself with the JBoss Web server upon deploying the mod_cluster service; and deregisters itself upon undeploy. Consequently, you no longer need to manually add a <Listener/> to server.xml.
In mod_cluster 1.0.x, you needed to add a unique jvmRoute to the <Engine/> element within server.xml. In 1.1.x, this is now optional. The auto-assignment of a jvm route is described in the previous section.
In mod_cluster 1.0.x, you had the option of manually setting the connector bind address, to restrict the network interface on which mod_cluster would listen for proxied requests from the load balancer. In mod_cluster 1.1.x, this no longer necessary. mod_cluster will set the bind address automatically, based on the interface on which mod_cluster's internal connection was established.
The JBoss microcontainer configuration for mod_cluster 1.0.x is not compatible with the configuration for 1.1.x. If you had customized your mod_cluster 1.0.x configuration, please start with the default configuration contained in the 1.1.x mod_cluster.sar and reapply any changes.
The mod_cluster integration is done via the modcluster subsystem. In AS7 only 1.1.x and later is supported.
The minimal configuration is having the modcluster schemaLocation
in the schemaLocation
list:
urn:jboss:domain:modcluster:1.0 jboss-mod-cluster.xsd
and the extension module
in the extensions
list:
<extension module="org.jboss.as.modcluster"/>
and subsystem
declaration like:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0"/>
With that configuration modcluster will listen for advertise on 224.0.1.105:23364
use the simple-load-provider
with
a load factor of 1.
The attributs correspond to the properties
Proxy Discovery Configuration
Attribute | Property | Default |
---|---|---|
proxy-list | proxyList | None |
proxy-url | proxyURL | None |
advertise | advertise | true |
advertise-security-key | advertiseSecurityKey | None |
excluded-contexts | excludedContexts | None |
auto-enable-contexts | autoEnableContexts | true |
stop-context-timeout | stopContextTimeout | 10 seconds (in seconds) |
socket-timeout | nodeTimeout | 20 seconds (in milli seconds) |
Proxy Configuration
Attribute | Property | Default |
---|---|---|
sticky-session | stickySession | true |
sticky-session-remove | stickySessionRemove | false |
sticky-session-force | stickySessionForce | true |
node-timeout | workerTimeout | -1 |
max-attempts | maxAttempts | 1 |
flush-packets | flushPackets | false |
flush-wait | flushWait | -1 |
ping | ping | 10 |
smax | smax | -1 it uses default value |
ttl | ttl | -1 it uses default value |
domain | loadBalancingGroup | None |
load-balancing-group | loadBalancingGroup | None |
SSL Configuration
SSL configuration part needs to be added here too
The simple load provider always send the same load factor. That is the default one. Example:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0"> <mod-cluster-config> <simple-load-provider factor="1"/> </mod-cluster-config> </subsystem>
Attribute | Property | Default |
---|---|---|
factor | LoadBalancerFactor | 1 |
The dynamic load provide allows to have load-metric
as well as custom-load-metric
defined. For example:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0"> <mod-cluster-config advertise-socket="mod_cluster"> <dynamic-load-provider history="10" decay="2"> <load-metric type="cpu" weight="2" capacity="1"/> <load-metric type="sessions" weight="1" capacity="512"/> <custom-load-metric class="mypackage.myclass" weight="1" capacity="512"> <property name="myproperty" value="myvalue" /> <property name="otherproperty" value="othervalue" /> </custom-load-metric> </dynamic-load-provider> </mod-cluster-config> </subsystem>
Attribute | Property | Default |
---|---|---|
history | history | 512 |
decay | decayFactor | 512 |
The load-metric are the "classes" collecting information to allow computation of the load factor sent to httpd
Attribute | Property | Default |
---|---|---|
type | A Server-Side Load Metrics | Mandatory |
weight | weight | 9 |
capacity | capacity | 512 |
type | Corresponding Server-Side Load Metric |
---|---|
cpu | AverageSystemLoadMetric |
mem | SystemMemoryUsageLoadMetric |
heap | HeapMemoryUsageLoadMetric |
sessions | ActiveSessionsLoadMetric |
requests | RequestCountLoadMetric |
send-traffic | SendTrafficLoadMetric |
receive-traffic | ReceiveTrafficLoadMetric |
busyness | BusyConnectorsLoadMetric |
connection-pool | ConnectionPoolUsageLoadMetric |
The custom-load-metric are for user defined "classes" collecting information. They are like the load-metric except type
is replaced by class
:
Attribute | Property | Default |
---|---|---|
class | Name of your class | Mandatory |
The load-metric have 4 commands to add / remove metrics
Allows to add a load-metric
to the dynamic-load-provider
For example:
./:add-metric(type=cpu, weight=2, capacity=1)
Allows to remove a load-metric
from the dynamic-load-provider
For example:
./:remove-metric(type=cpu)
Allows to add a load-custom-metric
to the dynamic-load-provider
For example:
./:add-custom-metric(class=myclass, weight=2, capacity=1, property=[("pool" => "mypool"), ("var" => "myvariable")])
Building mod_cluster's server-side components from source requires the following tools:
JDK 5.0+
Maven 2.0+
Steps to build:
Download the mod_cluster sources
git clone git://github.com/modcluster/mod_cluster.git
Use maven "dist" profile to build:
cd mod_cluster mvn -P dist package
Some unit tests require UDP port 23365. Make sure your local firewall allows the port.
The build produces the following output in the target directory:
Exploded format sar to copy to the deploy dir in your JBoss AS install
Jar files to copy to the lib directory in your JBossWeb or Tomcat install to support use of mod_cluster
The load balancing demo application
The full distribution tarball; includes the above elements
The tables below enumerate the configuration properties available to an application server node. The location for these properties depends on how mod_cluster is configured.
The list of proxies from which an application expects to receive AJP connections is either defined statically, via the addresses defined in the proxyList configuration property; or discovered dynamically via the advertise mechanism. Using a special mod_advertise module, proxies can advertise their existence by periodically broadcasting a multicast message containing its address/port. This functionality is enabled via the advertise configuration property. If configured to listen, a server can learn of the proxy's existence, then notify that proxy of its own existence, and update its configuration accordingly. This frees both the proxy and the server from having to define static, environment-specific configuration values.
Attribute | AS7 Attribute | Default | Scope | Description |
---|---|---|---|---|
proxyList | proxy-list | None | Configuration |
Defines a comma delimited list of httpd proxies with which this node will initially communicate. Value should be of the form: address1:port1,address2:port2 Using the default configuration, this property can by manipulated via the jboss.mod_cluster.proxyList system property. |
excludedContexts | excluded-contexts | ROOT, admin-console, invoker, bossws, jmx-console, juddi, web-console | Configuration |
List of contexts to exclude from httpd registration, of the form: host1:context1,host2:context2,host3:context3 If no host is indicated, it is assumed to be the default host of the server (e.g. localhost). "ROOT" indicates the root context. Using the default configuration, this property can by manipulated via the jboss.mod_cluster.excludedContexts system property. |
autoEnableContexts | auto-enable-contexts | true | Configuration |
If false the contexts are registered disabled in httpd, they need to be enabled via the enable() mbean method or via mod_cluster_manager. |
stopContextTimeout | stop-context-timeout | 10 | Configuration |
The amount of time, measure in units specified by stopContextTimeoutUnit, for which to wait for clean shutdown of a context (completion of pending requests for a distributable context; or destruction/expiration of active sessions for a non-distributable context). |
stopContextTimeoutUnit | None | TimeUnit.SECONDS | Configuration |
The unit of time for use with stopContextTimeout |
sessionDrainingStrategy | None | org.jboss.modcluster.SessionDrainingStrategyEnum.DEFAULT | Configuration |
Indicates the session draining strategy used during undeployment of a web application. There are three possible values:
|
proxyURL | proxy-url | None | Apache HTTPD |
If defined, this value will be prepended to the URL of MCMP commands. |
socketTimeout | socket-timeout | 20000 | Configuration |
Number of milliseconds to wait for a response from an httpd proxy to MCMP commands before timing out, and flagging the proxy as in error. |
advertise | advertise | true, if proxyList is undefined, false otherwise | Configuration |
If enabled, httpd proxies will be auto-discovered via multicast announcements. This can be used either in concert or in place of a static proxyList. |
advertiseGroupAddress | advertise-socket | 224.0.1.105 | Apache HTTPD |
UDP address on which to listen for httpd proxy multicast advertisements In AS7 that corresponding to the name of a socket-binding |
advertisePort | See advertise-socket | 23364 | Apache HTTPD |
UDP port on which to listen for httpd proxy multicast advertisements |
advertiseSecurityKey | advertise-security-key | None | Apache HTTPD |
If specified, httpd proxy advertisements checksums (using this value as a salt) will be required to be verified on the server side |
advertiseThreadFactory | None | Executors.defaultThreadFactory() | Configuration |
The thread factory used to create the background advertisement listener. |
jvmRouteFactory | None | new SystemPropertyJvmRouteFactory(new UUIDJvmRouteFactory(), "jboss.mod_cluster.jvmRoute") | Configuration |
Defines the strategy for determining the jvm route of a node, if none was specified in server.xml.
The default factory first consults the jboss.mod_cluster.jvmRoute system property.
If this system property is undefined, the jvm route is assigned a UUID.
|
The following configuration values are sent to proxies during server startup, when a proxy is detected via the advertise mechanism, or during the resetting of a proxy's configuration during error recovery.
Attribute | AS7 Attribute | Default | Scope | Description |
---|---|---|---|---|
stickySession | sticky-session | true | Balancer |
Indicates whether subsequent requests for a given session should be routed to the same node, if possible. |
stickySessionRemove | sticky-session-remove | false | Balancer |
Indicates whether the httpd proxy should remove session stickiness in the event that the balancer is unable to route a request to the node to which it is stuck. This property is ignored if stickySession is false. |
stickySessionForce | sticky-session-force | false | Balancer |
Indicates whether the httpd proxy should return an error in the event that the balancer is unable to route a request to the node to which it is stuck. This property is ignored if stickySession is false. |
workerTimeout | worker-timeout | -1 | Balancer |
Number of seconds to wait for a worker to become available to handle a request. When no workers of a balancer are usable, mod_cluster will retry after a while (workerTimeout/100). That is timeout in the balancer mod_proxy documentation. A value of -1 indicates that the httpd will not wait for a worker to be available and will return an error if none is available. |
maxAttempts | max-attempts | 1 | Balancer |
Number of times an httpd proxy will attempt to send a given request to a worker before giving up. The minimum value is 1, meaning try only once. (Note that mod_proxy default is also 1: no retry). |
flushPackets | flush-packets | false | Node |
Enables/disables packet flushing |
flushWait | flush-wait | -1 | Node |
Time to wait before flushing packets in milliseconds. A value of -1 means wait forever. |
ping | ping | 10 | Node |
Time (in seconds) in which to wait for a pong answer to a ping |
smax | smax | Determined by httpd configuration | Node |
Soft maximum idle connection count (that is the smax in worker mod_proxy documentation). The maximum value depends on the httpd thread configuration (ThreadsPerChild or 1). |
ttl | ttl | 60 | Node |
Time to live (in seconds) for idle connections above smax |
nodeTimeout | node-timeout | -1 | Node |
Timeout (in seconds) for proxy connections to a node. That is the time mod_cluster will wait for the back-end response before returning error. That corresponds to timeout in the worker mod_proxy documentation. A value of -1 indicates no timeout. Note that mod_cluster always uses a cping/cpong before forwarding a request and the connectiontimeout value used by mod_cluster is the ping value. |
balancer | balancer | mycluster | Node |
The balancer name |
loadBalancingGroup | domain load-balancing-group | None | Node |
If specified, load will be balanced among jvmRoutes withing the same load balancing group. A loadBalancingGroup is conceptually equivalent to a mod_jk domain directive. This is primarily used in conjunction with partitioned session replication (e.g. buddy replication). |
When nodeTimeout is not defined the ProxyTimeout directive Proxy is used. If ProxyTimeout is not defined the server timeout (Timeout) is used (default 300 seconds). nodeTimeout, ProxyTimeout or Timeout is set at the socket level.
The communication channel between application servers and httpd proxies uses HTTP by default. This channel can be secured using HTTPS by setting the ssl property to true.
This HTTP/HTTPS channel should not be confused with the processing of HTTP/HTTPS client requests.
Additional configuration properties used when mod_cluster is configured in JBoss Web standalone or Tomcat.
Attribute | Default | Description |
---|---|---|
loadMetricClass | org.jboss.modcluster.load.metric.impl.BusyConnectorsLoadMetric |
Class name of an object implementing org.jboss.load.metric.LoadMetric |
loadMetricCapacity | 1 |
The capacity of the load metric defined via the loadMetricClass property |
loadHistory | 9 |
The number of historic load values to consider in the load balance factor computation. |
loadDecayFactor | 2 |
The factor by which a historic load values should degrade in significance. |
A major feature of mod_cluster is the ability to use server-side load metrics to determine how best to balance requests.
The DynamicLoadBalanceFactorProvider
bean computes the load balance factor of a node from a defined set of load metrics.
<bean name="DynamicLoadBalanceFactorProvider" class="org.jboss.modcluster.load.impl.DynamicLoadBalanceFactorProvider" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=LoadBalanceFactorProvider",exposedInterface=org.jboss.modcluster.load.impl.DynamicLoadBalanceFactorProviderMBean.class)</annotation> <constructor> <parameter> <set elementClass="org.jboss.modcluster.load.metric.LoadMetric"> <inject bean="BusyConnectorsLoadMetric"/> <inject bean="HeapMemoryUsageLoadMetric"/> </set> </parameter> </constructor> <property name="history">9</property> <property name="decayFactor">2</property> </bean>
Load metrics can be configured with an associated weight and capacity.
The weight (default is 1) indicates the significance of a metric with respect to the other metrics. For example, a metric of weight 2 will have twice the impact on the overall load factor than a metric of weight 1.
The capacity of a metric serves 2 functions:
To normalize the load values from each metric. In some load metrics, capacity is already reflected in the load values. The capacity of a metric should be configured such that 0 <= (load / capacity) >= 1.
To favor some nodes over others. By setting the metric capacities to different values on each node, proxies will effectively favor nodes with higher capacities, since they will return smaller load values. This adds an interesting level of granularity to node weighting. Consider a cluster of two nodes, one with more memory, and a second with a faster CPU; and two metrics, one memory-based and the other CPU-based. In the memory-based metric, the first node would be given a higher load capacity than the second node. In a CPU-based metric, the second node would be given a higher load capacity than the first node.
Each load metric contributes a value to the overall load factor of a node. The load factors from each metric are aggregated according to their weights.
In general, the load factor contribution of given metric is: (load / capacity) * weight / total weight.
The DynamicLoadBalanceFactorProvider applies a time decay function to the loads returned by each metric. The aggregate load, with respect to previous load values, can be expressed by the following formula:
... or more concisely as:
... where D = decayFactor, and H = history.
Setting history = 0 effectively disables the time decay function and only the current load for each metric will be considered in the load balance factor computation.
The mod_cluster load balancer expects the load factor to be an integer between 0 and 100, where 0 indicates max load and 100 indicates zero load. Therefore, the final load factor sent to the load balancer
While you are free to write your own load metrics, the following LoadMetrics are available out of the box:
Requires an explicit capacity
Uses SessionLoadMetricSource
to query session managers
Analogous to method=S in mod_jk
e.g.
<bean name="ActiveSessionsLoadMetric" class="org.jboss.modcluster.load.metric.impl.ActiveSessionsLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=ActiveSessionsLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter><inject bean="SessionLoadMetricSource"/></parameter> </constructor> <property name="capacity">1000</property> </bean> <bean name="SessionLoadMetricSource" class="org.jboss.modcluster.load.metric.impl.SessionLoadMetricSource" mode="On Demand"> <constructor> <parameter class="javax.management.MBeanServer"> <inject bean="JMXKernel" property="mbeanServer"/> </parameter> </constructor> </bean>
Returns the percentage of connector threads from the thread pool that are busy servicing requests
Uses ThreadPoolLoadMetricSource
to query connector thread
Analogous to method=B in mod_jk
e.g.
<bean name="BusyConnectorsLoadMetric" class="org.jboss.modcluster.load.metric.impl.BusyConnectorsLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=BusyConnectorsLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter><inject bean="ThreadPoolLoadMetricSource"/></parameter> </constructor> </bean> <bean name="ThreadPoolLoadMetricSource" class="org.jboss.modcluster.load.metric.impl.ThreadPoolLoadMetricSource" mode="On Demand"> <constructor> <parameter class="javax.management.MBeanServer"> <inject bean="JMXKernel" property="mbeanServer"/> </parameter> </constructor> </bean>
Returns the incoming request POST traffic in KB/sec (the application needs to read POST data)
Requires an explicit capacity
Uses RequestProcessorLoadMetricSource
to query request processors
Analogous to method=T in mod_jk
e.g.
<bean name="ReceiveTrafficLoadMetric" class="org.jboss.modcluster.load.metric.impl.ReceiveTrafficLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=ReceiveTrafficLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter class="org.jboss.modcluster.load.metric.impl.RequestProcessorLoadMetricSource"> <inject bean="RequestProcessorLoadMetricSource"/> </parameter> </constructor> <property name="capacity">1024</property> </bean> <bean name="RequestProcessorLoadMetricSource" class="org.jboss.modcluster.load.metric.impl.RequestProcessorLoadMetricSource" mode="On Demand"> <constructor> <parameter class="javax.management.MBeanServer"> <inject bean="JMXKernel" property="mbeanServer"/> </parameter> </constructor> </bean>
Returns the outgoing request traffic in KB/sec
Requires an explicit capacity
Uses RequestProcessorLoadMetricSource
to query request processors
Analogous to method=T in mod_jk
<bean name="SendTrafficLoadMetric" class="org.jboss.modcluster.load.metric.impl.SendTrafficLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=SendTrafficLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter class="org.jboss.modcluster.load.metric.impl.RequestProcessorLoadMetricSource"> <inject bean="RequestProcessorLoadMetricSource"/> </parameter> </constructor> <property name="capacity">512</property> </bean>
Returns the number of requests/sec
Requires an explicit capacity
Uses RequestProcessorLoadMetricSource
to query request processors
Analogous to method=R in mod_jk
e.g.
<bean name="RequestCountLoadMetric" class="org.jboss.modcluster.load.metric.impl.RequestCountLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=RequestCountLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter class="org.jboss.modcluster.load.metric.impl.RequestProcessorLoadMetricSource"> <inject bean="RequestProcessorLoadMetricSource"/> </parameter> </constructor> <property name="capacity">1000</property> </bean>
Returns CPU load
Requires Java 1.6+
Uses OperatingSystemLoadMetricSource
to generically read attributes
e.g.
<bean name="AverageSystemLoadMetric" class="org.jboss.modcluster.load.metric.impl.AverageSystemLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=AverageSystemLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter><inject bean="OperatingSystemLoadMetricSource"/></parameter> </constructor> </bean> <bean name="OperatingSystemLoadMetricSource" class="org.jboss.modcluster.load.metric.impl.OperatingSystemLoadMetricSource" mode="On Demand"> </bean>
Returns the heap memory usage as a percentage of max heap size
e.g.
<bean name="HeapMemoryUsageLoadMetric" class="org.jboss.modcluster.load.metric.impl.HeapMemoryUsageLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=HeapMemoryUsageLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> </bean>
Returns the percentage of connections from a connection pool that are in use
Uses ConnectionPoolLoadMetricSource to query JCA connection pools
e.g.
<bean name="ConnectionPoolUsageMetric" class="org.jboss.modcluster.load.metric.impl.ConnectionPoolUsageLoadMetric" mode="On Demand"> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.web:service=ConnectionPoolUsageLoadMetric",exposedInterface=org.jboss.modcluster.load.metric.LoadMetricMBean.class)</annotation> <constructor> <parameter><inject bean="ConnectionPoolLoadMetricSource"/></parameter> </constructor> </bean> <bean name="ConnectionPoolLoadMetricSource" class="org.jboss.modcluster.load.metric.impl.ConnectionPoolLoadMetricSource" mode="On Demand"> <constructor> <parameter class="javax.management.MBeanServer"> <inject bean="JMXKernel" property="mbeanServer"/> </parameter> </constructor> </bean>
First, extract the server-side binary to a temporary directory. The following assumes it was extracted to /tmp/mod_cluster
Your next step depends on whether your target server is JBoss AS or JBossWeb/Tomcat.
You don't need to do anything to install the java-side binaries in AS 6.x; it's part of the AS distribution's default, standard and all configurations.
Assuming $JBOSS_HOME indicates the root of your JBoss AS install and that you want to use mod_cluster in the AS's all config:
cp -r /tmp/mod_cluster/mod_cluster.sar $JBOSS_HOME/server/all/deploy
Assuming $CATALINA_HOME indicates the root of your Tomcat install:
cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/jboss-logging.jar $CATALINA_HOME/lib/ cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/mod_cluster-container-catalina* $CATALINA_HOME/lib/ cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/mod_cluster-container-spi* $CATALINA_HOME/lib/ cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/mod_cluster-core* $CATALINA_HOME/lib/and additionally for Tomcat6:
cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/mod_cluster-container-tomcat6* $CATALINA_HOME/liband additionally for Tomcat7:
cp /tmp/mod_cluster/JBossWeb-Tomcat/lib/mod_cluster-container-tomcat7* $CATALINA_HOME/lib
There are 2 connections between the cluster and the front-end. Both could be encrypted. That chapter describes how to encrypt both connections.
As the ClusterListener allows to configure httpd it is adviced to use SSL for that connection. The most easy is to use a virtual host that will only be used to receive information from JBossWEB. Both side need configuration.
mod_ssl of httpd is using to do that. See in one example how easy the configuration is:
Listen 6666 <VirtualHost 10.33.144.3:6666> SSLEngine on SSLCipherSuite AES128-SHA:ALL:!ADH:!LOW:!MD5:!SSLV2:!NULL SSLCertificateFile conf/server.crt SSLCertificateKeyFile conf/server.key SSLCACertificateFile conf/server-ca.crt SSLVerifyClient require SSLVerifyDepth 10 </VirtualHost>
The conf/server.crt file is the PEM-encoded Certificate file for the VirtualHost it must be signed by a Certificate Authority (CA) whose certificate is stored in the sslTrustStore of the ClusterListener parameter.
The conf/server.key file is the file containing the private key.
The conf/server-ca.crt file is the file containing the certicate of the CA that have signed the client certificate JBossWEB is using. That is the CA that have signed the certificate corresponding to the sslKeyAlias stored in the sslKeyStore of the ClusterListener parameters.
There is a wiki describing the SSL parameters of the ClusterListener. See in one example how easy the configuration is:
<Listener className="org.jboss.web.cluster.ClusterListener" ssl="true" sslKeyStorePass="changeit" sslKeyStore="/home/jfclere/CERTS/CA/test.p12" sslKeyStoreType="PKCS12" sslTrustStore="/home/jfclere/CERTS/CA/ca.p12" sslTrustStoreType="PKCS12" sslTrustStorePassword="changeit" />
The sslKeyStore file contains the private key and the signed certificate of the client certificate JBossWEB uses to connect to httpd. The certificate must be signed by a Cerficate Authority (CA) who certificate is in the conf/server-ca.crt file of the httpd
The sslTrustStore file contains the CA certificate of the CA that signed the certificate contained in conf/server.crt file.
The mod-cluster-jboss-beans.xml in $JBOSS_HOME/server/profile/deploy/mod-cluster.sar/META-INF in the ClusterConfig you are using you should have something like:
<property name="ssl">true</property> <property name="sslKeyStorePass">changeit</property> <property name="sslKeyStore">/home/jfclere/CERTS/test.p12</property> <property name="sslKeyStoreType">pkcs12</property> <property name="sslTrustStore">/home/jfclere/CERTS/ca.p12</property> <property name="sslTrustStoreType">pkcs12</property> <property name="sslTrustStorePassword">changeit</property>
The files were created using OpenSSL utilities see OpenSSL CA.pl (/etc/pki/tls/misc/CA for example) has been used to create the test Certificate authority, the certicate requests and private keys as well as signing the certicate requests.
Create a work directory and work for there:
mkdir -p CERTS/Server cd CERTS/Server
Create a new CA:
/etc/pki/tls/misc/CA -newca
That creates a directory for example ../../CA that contains a cacert.pem file which content have to be added to the conf/server-ca.crt described above.
Export the CA certificate to a .p12 file:
openssl pkcs12 -export -nokeys -in ../../CA/cacert.pem -out ca.p12
That reads the file cacert.pem that was created in the previous step and convert it into a pkcs12 file the JVM is able to read.
That is the ca.p12 file used in the sslTrustStore parameter above.
Create a new request:
/etc/pki/tls/misc/CA -newreq
That creates 2 files named newreq.pem and newkey.pem. newkey.pem is the file conf/server.key described above.
Sign the request:
/etc/pki/tls/misc/CA -signreq
That creates a file named newcert.pem. newcert.pem is the file conf/server.crt described above. At that point you have created the SSL stuff needed for the VirtualHost in httpd. You should use a browser to test it after importing in the browser the content of the cacert.pem file.
Create a work directory and work for there:
mkdir -p CERTS/Client cd CERTS/Client
Create request and key for the JBossWEB part.
/etc/pki/tls/misc/CA -newreq
That creates 2 files: Request is in newreq.pem, private key is in newkey.pem
Sign the request.
/etc/pki/tls/misc/CA -signreq
That creates a file: newcert.pem
Don't use a passphrase when creating the client certicate or remove it before exporting:
openssl rsa -in newkey.pem -out key.txt.pem mv key.txt.pem newkey.pem
Export the client certificate and key into a p12 file.
openssl pkcs12 -export -inkey newkey.pem -in newcert.pem -out test.p12
That is the sslKeyStore file described above (/home/jfclere/CERTS/CA/test.p12)
Using https allows to encrypt communications betwen httpd and JBossWEB. But due to the ressources it needs that no advised to use it in high load configuration.
(See Encrypting connection between httpd and TC for detailed instructions).
httpd is configured to be a client for AS/TC so it should provide a certificate AS/TC will accept and have a private key to encrypt the data, it also needs a CA certificate to valid the certificate AS/TC will use for the connection.
SSLProxyEngine On SSLProxyVerify require SSLProxyCACertificateFile conf/cacert.pem SSLProxyMachineCertificateFile conf/proxy.pem
conf/proxy.pem should contain both key and certificate. The certificate must be trusted by Tomcat via the CA in truststoreFile of <connector/>.
conf/cacert.pem must contain the certificat of the CA that signed the AS/TC certificate. The correspond key and certificate are the pair specificed by keyAlias and truststoreFile of the <connector/>. Of course the <connector/> must be the https one (normally on port 8443).
The files were created using OpenSSL utilities see OpenSSL CA.pl (/etc/pki/tls/misc/CA for example) has been used to create the test Certificate authority, the certicate requests and private keys as well as signing the certicate requests.
The certificate and key need to be imported into the java keystore using keytool
make sure you don't use a passphare for the key (don't forget to clean the file when done)
openssl pkcs12 -export -inkey key.pem -in newcert.pem -out test.p12make sure you use the keystore password as Export passphrase.
keytool -importkeystore -srckeystore test.p12 -srcstoretype PKCS12
keytool -import -trustcacerts -alias "caname" \ -file ../../CA/cacert.pem -keystore /etc/pki/java/cacerts
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" keyAlias="1" truststoreFile="/etc/pki/java/cacerts" maxThreads="150" scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" />
openssl s_client -CAfile /home/jfclere/CA/cacert.pem -cert newcert.pem -key newkey.pem \ -host localhost -port 8443There shouldn't be any error and you should be able to see your CA in the "Acceptable client certificate CA names".
When using http or https beween httpd and JBossWEB you need to use the SSLValve and export the SSL variable as header in the request in httpd. If you are using AJP, mod_proxy_ajp will read the SSL variables and forward them to JBossWEB automaticaly.
(See Forwarding SSL environment when using http/https proxy for detailed instructions).
The SSL variable used by mod_proxy_ajp are the following:
"HTTPS" SSL indicateur.
"SSL_CLIENT_CERT" Chain of client certificates.
"SSL_CIPHER" Cipher used.
"SSL_SESSION_ID" Id of the session.
"SSL_CIPHER_USEKEYSIZE" Size of the key used.
Mod_cluster only support Apache httpd, there are no plan to support IIS or Iplanet.
The migration from mod_jk to mod_cluster is not very complex. Only very few worker properties can't be mapped to mod_cluster parameters.
Here is the table of worker properties and how to transfer them in the ClusterListener parameters.
mod_jk worker property | ClusterListener parameter | Remarks |
---|---|---|
host | - | It is read from the <Connector/> Address information |
port | - | It is read from the <Connector/> Port information |
type | - | It is read from the <Connector/> Protocol information |
route | - | It is read from the <Engine/> JVMRoute information |
domain | domain | That is not supported in this version |
redirect | - | The nodes with loadfactor = 0 are standby nodes they will be used no other nodes are available |
socket_timeout | nodeTimeout | Default 10 seconds |
socket_keepalive | - | KEEP_ALIVE os is always on in mod_cluster |
connection_pool_size | - | The max size is calculated to be AP_MPMQ_MAX_THREADS+1 (max) |
connection_pool_minsize | smax | The defaut is max |
connection_pool_timeout | ttl | Time to live when over smax connections. The defaut is 60 seconds |
- | workerTimeout | Max time to wait for a free worker default 1 second |
retries | maxAttempts | Max retries before returning an error Default: 3 |
recovery_options | - | mod_cluster behave like mod_jk with value 7 |
fail_on_status | - | Not supported |
max_packet_size | iobuffersize/receivebuffersize | Not supported in this version. Use ProxyIOBufferSize |
max_reply_timeouts | - | Not supported |
recovert_time | - | The ClusterListener will tell (via a STATUS message) mod_cluster that the node is up again |
activation | - | mod_cluster receives this information via ENABLE/DISABLE/STOP messages |
distance | - | mod_cluster handles this via the loadfactor logic |
mount | - | The context "mounted" automaticly via the ENABLE-APP messages. ProxyPass could be used too |
secret | - | Not supported |
connect_timeout | - | Not supported. Use ProxyTimeout or server TimeOut (Default 300 seconds) |
prepost_timeout | ping | Default 10 seconds |
reply_timeout | - | Not supported. Use ProxyTimeout or server TimeOut? directive (Default 300 seconds) |
As mod_cluster is a sophisticated balancer migration from mod_proxy to mod_cluster is strait forward. mod_cluster replaces a reverse proxy with loadbalancing. A reversed proxy is configured like:
ProxyRequests Off <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /foo http://foo.example.com/bar ProxyPassReverse /foo http://foo.example.com/bar
All the general proxy parameters could be used in mod_cluster they work like in mod_proxy, only the balancers and the workers definitions are slightly different.
Mod_proxy Parameter | ClusterListener parameter | Remarks |
---|---|---|
min | - | Not supported in this version |
max | - | mod_cluster uses mod_proxy default value |
smax | smax | Same as mod_proxy |
ttl | ttl | Same as mod_proxy |
acquire | workerTimeout | Same as mod_proxy acquire but in seconds |
disablereuse | - | mod_cluster will disable the node in case of error and the ClusterListener will for the reuse via the STATUS message |
flushPackets | flushPackets | Same as mod_proxy |
flushwait | flushwait | Same as mod_proxy |
keepalive | - | Always on: OS KEEP_ALIVE is always used. Use connectionTimeout in the <Connector> if needed |
lbset | - | Not supported |
ping | ping | Same as mod_proxy Default value 10 seconds |
lbfactor | - | The load factor is received by mod_cluster from a calculated value in the ClusterListener |
redirect | - | Not supported lbfactor sent to 0 makes a standby node |
retry | - | ClusterListener will test when the node is back online |
route | JVMRoute | In fact JBossWEB via the JVMRoute in the Engine will add it |
status | - | mod_cluster has a finer status handling: by context via the ENABLE/STOP/DISABLE/REMOVE application messages. hot-standby is done by lbfactor = 0 and Error by lbfactor = 1 both values are sent in STATUS message by the ClusterListener |
timeout | nodeTimeout | Default wait for ever (http://httpd.apache.org/docs/2.2/mod/mod_proxy.html is wrong there) |
ttl | ttl | Default 60 seconds |
Mod_proxy Parameter | ClusterListener parameter | Remarks |
---|---|---|
lbmethod | - | There is only one load balancing method in mod_cluster "cluster_byrequests" |
maxattempts | maxAttempts | Default 1 |
nofailover | stickySessionForce | Same as in mod_proxy |
stickysession | StickySessionCookie/StickySessionPath | The 2 parameters in the ClusterListener are combined in one that behaves like in mod_proxy |
timeout | workerTimeout | Default 1 seconds |
The mod_cluster distribution includes a demo application that helps demonstrate how different server-side scenarios affect the routing of client requests by the load balancer. The demo application is located in the mod_cluster distribution's demo directory.
The demo application consists of two components:
The first component is a war file that needs to be deployed in JBossWeb/Tomcat/JBoss AS. The war includes a number of servlets.
The second component is a GUI application that allows a user to launch a pool of threads that repeatedly make requests to the load balancer. The requests are ultimately routed to the demo war's primary servlet. The application tracks which servers are handling the requests and displays this information in a chart.
The application can also send separate requests to the demo war's load generation servlets, allowing the user to see how different load conditions affect the balancing of requests.
Note that the demo application does not actually depend on mod_cluster in any way. Its only dependency is on JBossWeb/Tomcat. [1] Consequently, the demo can be used to demonstrate the effect of different server-side scenarios on the routing decisions made by any load balancer, including mod_jk, mod_proxy or the various hardware load balancers.
Note also that this demo application is not intended to be used as a load testing tool; i.e. something that can demonstrate the maximum load a cluster configuration can handle. Using it as such has a good chance of showing you the maximum load the client can generate rather than the maximum load your cluster can handle.
To run the demo application:
Unpack the mod_cluster distribution on your filesystem. Here we assume it has been unzipped to ~/mod_cluster or C:\mod_cluster.
Install mod_cluster into your httpd server as described at Installation of the httpd part
Install mod_cluster into your JBossAS/JBossWeb/Tomcat servers as described at Installation on the Java side
In AS7 you have to set org.apache.tomcat.util.ENABLE_MODELER to true, Something like:
<system-properties> <property name="org.apache.tomcat.util.ENABLE_MODELER" value="true"/> </system-properties>
Start httpd and your JBossAS/JBossWeb/Tomcat servers
Deploy the load-demo.war found in the distribution's demo/server folder to your JBossAS/JBossWeb/Tomcat servers.
Start the demo application:
On *nix:
cd ~/mod_cluster/demo/client ./run-demo.sh
On Windows:
C:\>cd mod_cluster\demo\client C:\mod_cluster\demo\client>run-demo
Configure the hostname and address of the httpd server, the number of client threads, etc and click the "Start" button. See Client Driver Configuration Options for details on the configuration options.
Switch to the "Request Balancing" tab to see how many requests are going to each of your JBossAS/JBossWeb/Tomcat servers.
Switch to the "Session Balancing" tab to see how many active sessions [2] are being hosted by each of your JBossAS/JBossWeb/Tomcat servers.
Stop some of your JBossAS/JBossWeb/Tomcat servers and/or undeploy the load-demo.war from some of the servers and see the effect this has on load balancing.
Restart some of your JBossAS/JBossWeb/Tomcat servers and/or re-deploy the load-demo.war to some of the servers and see the effect this has on load balancing.
Experiment with adding artificial load to one or more servers to see what effect that has on load balancing. See Load Generation Scenarios for details.
Most of the various panels in application interface also present information on the current status on any client threads. "Total Clients" is the number of client threads created since the last time the "Start" button was pushed. "Live Clients" is the number of threads currently running. "Failed Clients" is the number of clients that terminated abnormally; i.e. made a request that resulted in something other than an HTTP 200 response.
The configuration of the client is driver is done via the application's "Clent Control" tab.
The panel includes the following options:
Proxy Hostname: Hostname of the load balancer or the IP address on which it is listening for requests [3]
Proxy Port: Port on which the load balancer is listening for requests [4]
Context Path: Portion of the request URL that specifies the request is for the load-demo.war
Session Life: Number of seconds a client thread should use a session before invalidating or abandoning it. Generally it is good to keep this to a small value; otherwise the use of session stickiness will prevent changes in server load from affecting the load balancer's routing decisions. With sticky sessions enabled (strongly recommended), it is the creation of a new session that allows the load balancer to try to balance load.
Invalidate: Controls what the client thread should do when it stops using a session because Session Life has passed. If checked, the driver will send a request that results in the session being invalidated. If unchecked, the session will just be abandoned, and will continue to exist on the server until Session Timeout seconds have passed. In the future this will likely be changed to a percentage input, so X% can be invalidated, the rest abandoned.
Session Timeout: Number of seconds a session can remain unused before the server is free to expire it. Unchecking Invalidate and setting a high value relative to Session Life allows a significant number of unused sessions to accumulate on the server.
Num Threads: Number of client threads to launch. Each thread repeatedly makes requests until the "Stop" button is pushed or a request receives a response other than HTTP 200.
Sleep Time: Number of ms the client threads should sleep between requests.
Startup Time: Number of seconds over which the application should stagger the start of the client threads. Staggering the start advised as it avoids the unnatural situation where for the life of the demonstation all sessions start at about the same time and then are invalidated or abandoned at the same time. Staggering the start allows the load balancer to continually see new sessions and decide how to route them.
You can use the application's GUI to instruct individual servers to artificially generate various types of load, and then track how that load affects request and session balancing. Load generation is controlled via the application's "Server Load Control" tab.
The panel includes the following options:
Target Hostname and Target Port: The hostname or IP address of the server on which you want load generated. There are two strategies for setting these:
You can use the hostname and port of the load balancer, in which case the load balancer will pick a backend server and route the request to it. Note the client application does not maintain a session cookie for these requests, so if you invoke another server load generation request, you shouldn't expect the same server to handle it.
If the JBoss AS/JBossWeb/Tomcat servers are running the HttpConnector as well as the AJP connector, you can specify the address and port on which a particular server's HttpConnector is listening. The standard port is 8080.
Load Creation Action: Specifies the type of load the target server should generate. See below for details on the available load types.
Params: Zero or more parameters to pass to the specified load creation servlet. For example, in the screenshot above, Number of Connections and Duration. How many parameters are displayed, their name and their meaning depend on the selected Load Creation Action. The label for each parameter includes a tooltip that explains its use.
The available Load Creation Actions are as follows:
Generates server load by causing session creation on the target server.
Generates server load by taking connections from the java:DefaultDS datasource for a period
Generates server load by tieing up threads in the webserver connections pool for a period
Generates server load by filling 50% of free heap memory for a period
Generates server CPU load by initiating a tight loop in a thread
Generates server traffic receipt load by POSTing a large byte array to the server once per second for a period
Generates server traffic send load by making a request once per second to which the server responds with a large byte array
Generates server load by making numerous requests, increasing the request count on the target server
[1] The demo's "Datasource Use" load generation scenario requires the use of JBoss Application Server.
[2] For purposes of this chart, a session is considered "active" if a client thread will ever again send a request associated with the session. When client threads stop using a session, they can either send a request to invalidate it or just abandon it by no longer including its session cookie in requests. After a session is abandoned, it will not be reflected in the "Session Balancing" chart, but it will continue to exist on the JBossWeb/Tomcat/JBoss AS server until it is removed due to timeout.
[3]
The default value for this field is controlled by the mod_cluster.proxy.host
system property, or localhost if not set.
Editing the run-demo.sh or run-demo.bat file to change the -Dmod_cluster.proxy.host=localhost
passed to java will allow you to avoid re-typing this value every time you launch the demo application.
[4]
The default value for this field is controlled by the mod_cluster.proxy.port
system property, or 8000 if not set.
Editing the run-demo.sh or run-demo.bat file to change the -Dmod_cluster.proxy.port=8000
passed to java will allow you to avoid re-typing this value every time you launch the demo application.
![]() |
MODCLUSTER-344 Pings for HTTP/HTTPS connections are now sent by default, can be disabled by "EnableOptions Off" |
![]() |
MODCLUSTER-360 mod_proxy_cluster.h doesn't contain the right information |
![]() |
MODCLUSTER-334 mod_cluster core when use ProxyPass / balancer://bal and CreateBalancers 1 |
![]() |
MODCLUSTER-335 Regression in ProxyPass integration |
![]() |
MODCLUSTER-339 "proxy: DNS lookup failure" with IPv6 on Solaris |
![]() |
MODCLUSTER-345 http_handle_cping_cpong with EnableOptions and HTTPS connector |
![]() |
MODCLUSTER-347 mod_cluster-manager may break up aliases from a single VirtualHost, causing a messy page |
![]() |
MODCLUSTER-332 Add Engine.getExecutor() to container SPI |
![]() |
HTTP httpd-2.4.x doesn't like the extra CRLF. |
![]() |
MODCLUSTER-349 mod_manager truncates ENABLE-APP messages. |
![]() |
Fix ap_proxy_make_fake_req in 2.4.x |
![]() |
MODCLUSTER-315 Convert project to use i18n logging and exceptions |
![]() |
MODCLUSTER-328 mod_cluster doesn't recognize ? as a proper context delimiter causing 503s on requests with query strings |
![]() |
JBPAPP-9493 mod_cluster returns "Bad Gateway" HTTP ErrorCode 502 with https |
![]() |
MODCLUSTER-326 rename NMAKEmakefile to NMAKEmakefile.example |
![]() |
JBPAPP-9788 Tomcat with mod_cluster refuses to shut down |
![]() |
JBPAPP-10168 Option sessionDrainingStrategy in mod_cluster listener in tomcat causes an exception |
![]() |
JBPAPP-10147 The shutdown port of tomcat is released before performing actual shut down |
![]() |
JBPAPP-9493 mod_cluster returns "Bad Gateway" HTTP ErrorCode 502 with https |
![]() |
JBPAPP-9791 mod_rewrite + mod_cluster: one kind of "proxying" RewriteRules don't work on Windows |
![]() |
MODCLUSTER-224 when using tomcat manager webapp to stop/start an application the start is ignored by mod_cluster |
![]() |
MODCLUSTER-322 Using AverageSystemLoadMetric can improperly cause a Load Factor of 0 |
![]() |
MODCLUSTER-312 port JBPAPP-8956 to 1.2.x |
![]() |
MODCLUSTER-311 mod_manager doesn't handle multiple virtualhosts per node |
![]() |
MODCLUSTER-309 mod_proxy_cluster not checking all available balancers (but only the first one) for an available route |
![]() |
JBPAPP-8957 Segmentation Faut when you run two or more JBoss Server Cluster behind *ONE* Apache Server on RHEL 6 |
![]() |
MODCLUSTER-304 Port to httpd 2.4 |
![]() |
MODCLUSTER-305 ProxyPass can break StickySession |
![]() |
MODCLUSTER-297 Warnings when compiling mod_cluster |
![]() |
MODCLUSTER-192 Add SSL stuff in the FAQ |
![]() |
MODCLUSTER-298 mod_cluster doesn't work with IPv6. |
![]() |
MODCLUSTER-289 MemManagerFile creates directory but put files in .. |
![]() |
MODCLUSTER-290 mod_cluster's mod_advertise can not start on IPv6-only box |
![]() |
MODCLUSTER-279 mod_cluster returns 503s after STATUS |
![]() |
MODCLUSTER-281 Make stickySessionForce default to false in native part from 1.2.0 release |
![]() |
Fix CVE-2011-4608: Add EnableMCPMReceive in configuration. (JBPAPP-7708) |
![]() |
mod_proxy code for old httpd versions needs to be updated. (MODCLUSTER-227) |
![]() |
Missing Documentation For mod_cluster with JBoss AS5. (MODCLUSTER-248) |
![]() |
sticky-session-force change requires httpd restart. (MODCLUSTER-273) |
![]() |
mod_cluster management console/Status does not show exact mod_cluster version. (MODCLUSTER-256) |
![]() |
read-attribute and :write-attribute Error in AS7. (MODCLUSTER-246) |
![]() |
routing table lookup creates performance issues.. (MODCLUSTER-252) |
![]() |
Error 503 using mod_cluster: CVE-2011-3348. (MODCLUSTER-258) |
![]() |
warning while compiling on some lab boxes. (MODCLUSTER-266) |
![]() |
String index out of range: -1 with a lot of nodes/deploymentswarning while compiling on some lab boxes. (MODCLUSTER-266) |
![]() |
404 while/just after removing a node. (MODCLUSTER-272) |
![]() |
ProxyPass doesn't work with mod_cluster whne mixing mod_cluster and other scheme handlers404 while/just after removing a node. (MODCLUSTER-274) |
![]() |
mod_cluster forgets removed node too fast. (MODCLUSTER-231) |
![]() |
Allow context lengths greater than 40 (MODCLUSTER-236) |
![]() |
Increase the MAXMESSSIZE and allow it to be configurable via a parameter. (MODCLUSTER-244) |
![]() |
Calculate the MAXMESSSIZE dynamically. (MODCLUSTER-245) |
![]() |
Load-demo application does not work with AS7. (MODCLUSTER-259) |
![]() |
Modify jbossweb metrics to use service provider spi, instead of jmx. (MODCLUSTER-151) |
![]() |
mod_cluster throws warnings at shutdown in AS7. (MODCLUSTER-143) |
![]() |
Refactor mod_cluster to allow implementations for other application servers. (MODCLUSTER-105) |
![]() |
AS7 integration. (MODCLUSTER-219) |
![]() |
Refactor project into modules for HA, catalina, core, etc. (MODCLUSTER-112) |
![]() |
request hang with a node is stopped in EC2 (MODCLUSTER-217) (jfclere) |
![]() |
Extra ENABLE-APP/REMOVE-APP event after failover (MODCLUSTER-220) (jfclere) |
![]() |
mod_cluster does not work with Tomcat 7 due to API Change in Connector (MODCLUSTER-240) (jfclere) |
![]() |
kill -HUP httpd process increase the number of open locks (MODCLUSTER-241) (jfclere) |
![]() |
mod_cluster failover does not work for a /webappcontext when the / root context exists (MODCLUSTER-188) (jfclere) |
![]() |
UseAlias broken (MODCLUSTER-212) (jfclere) |
![]() |
mod_rewrite PT doesn't work (MODCLUSTER-213) (jfclere) |
![]() |
memory usage growning in httpd (MODCLUSTER-214) (jfclere) |
![]() |
Second attempt to connect from Jboss to apache module sends incomplete Host Header (MODCLUSTER-216) (jfclere) |
![]() |
when using tomcat manager webapp to stop/start an application the start is ignored by mod_cluster (MODCLUSTER-224) (jfclere) |
![]() |
Unable to override load properties when running in Tomcat 6 (MODCLUSTER-232) (jfclere) |
![]() |
Documentation contains wrong default loadMetric class name (MODCLUSTER-233) (jfclere) |
![]() |
mod_cluster should issue a warning when Maxcontext is reached and no more context will be taken (MODCLUSTER-223) (jfclere) |
![]() |
add a note explaining where Maxcontext must be put in httpd.conf and issue error if in wrong location (MODCLUSTER-225) (jfclere) |
![]() |
NPE when overriding default load metric in ModClusterListener (MODCLUSTER-183) (pferraro) |
![]() |
mod_cluster failover does not work for a /webappcontext when the / root context exists (MODCLUSTER-188) (jfclere) |
![]() |
mod_cluster issues an ENABLE-APP too early in the webapp lifecycle (MODCLUSTER-190) (jfclere) |
![]() |
mod_cluster 1.1.0 docs step 11.1.4.3 is wrong (MODCLUSTER-193) (jfclere) |
![]() |
Incorrect routing of requests when one context root is the prefix of another (MODCLUSTER-196) (jfclere) |
![]() |
Can only rewrite from the root context in httpd if there is a root context deployed in JBoss (MODCLUSTER-198) (jfclere) |
![]() |
the STATUS MCMP message is send before the connector is started (MODCLUSTER-202) (jfclere) |
![]() |
The windoze bundle don't have the default configuration (MODCLUSTER-205) (jfclere) |
![]() |
httpd cores after graceful restart after the mod_cluster configuration is added (MODCLUSTER-206) (jfclere) |
![]() |
Make logging on Tomcat use JDK14LoggerPlugin by default (MODCLUSTER-185) (pferraro) |
![]() |
update mod_cluster to use HTTP/1.1 (MODCLUSTER-201) (jfclere) |
![]() |
Add support for system properties used in 1.0.x (MODCLUSTER-207) (pferraro) |
![]() |
Demo servlets throw InstanceNotFoundException against EAP5 (MODCLUSTER-170) (pferraro) |
![]() |
mod_advertise: Invalid ServerAdvertise Address too often (MODCLUSTER-172) (jfclere) |
![]() |
Fix class name of MBeanAttributeRatioLoadMetric in MC config (MODCLUSTER-174) (pferraro) |
![]() |
Wrong configuration could cause an httpd core (MODCLUSTER-175) (jfclere) |
![]() |
Advertise not configured error message in log is actually a warning (MODCLUSTER-176) (jfclere) |
![]() |
Better no servers connected message (MODCLUSTER-165) (jfclere) |
![]() |
How does the UI deal with 20 or more servers (MODCLUSTER-165) (jfclere) |
![]() |
mod_cluster should use hostname provided in address instead a IP address (MODCLUSTER-168) (pferraro) |
![]() |
Read only view of mod_cluster-manager (MODCLUSTER-181) (jfclere) |
![]() |
Demo client throws Bus Error when run with JDK 1.6 on OSX (MODCLUSTER-169) (jfclere) |
![]() |
Use versioned docs (MODCLUSTER-141) (jfclere, pferraro) |
![]() |
Deprecate use of term "domain" (MODCLUSTER-177) (jfclere, pferraro) |
![]() |
Rpc failure can lead to failure to deploy a webapp (MODCLUSTER-140) (pferraro) |
![]() |
Quotes in jsessionId causing sticky sessions to fail (MODCLUSTER-146) (jfclere) |
![]() |
ManagerBalancerName doesn't work (MODCLUSTER-153) (jfclere) |
![]() |
Parsing of IPv6 loopback address fails (MODCLUSTER-156) (pferraro) |
![]() |
SystemMemoryUsageLoadMetric returns wrong load metric (MODCLUSTER-157) (pferraro) |
![]() |
Clean shutdown logic can still inadvertently kill requests for non-distributed contexts. (MODCLUSTER-159) (pferraro) |
![]() |
NoClassDefFoundError running demo app against AS6 (MODCLUSTER-161) (pferraro) |
![]() |
Allow override of default clean shutdown behavior (MODCLUSTER-139) (pferraro) |
![]() |
Avoid unnecessary open sockets for non-master nodes (MODCLUSTER-158) (pferraro) |
![]() |
Add the lifecycle listener dynamically (MODCLUSTER-20) (pferraro) |
![]() |
Use UUID for auto-generated jvmRoute (MODCLUSTER-142) (pferraro) |
![]() |
improve packaging so that the bundle can run in ~ too (MODCLUSTER-150) (jfclere) |
![]() |
jboss.mod_cluster.proxyList: invalid hosts cause mod-cluster startup to be delayed (MODCLUSTER-155) (pferraro) |
![]() |
mod_cluster 1.1.0.CR1 doesn't work with Tomcat (MODCLUSTER-143) (jfclere/pferraro) |
![]() |
Allow configuration of stopContextTimeout units (MODCLUSTER-138) (pferraro) |
![]() |
Add a solaris10 64 bits sparc in the bundles (MODCLUSTER-137) (jfclere) |
![]() |
INFO and mod_cluster_manager/ displays milliseconds and DUMP second (MODCLUSTER-128) (jfclere) |
![]() |
Make mod_cluster manager tolerant to F5 page refresh when disabled context (MODCLUSTER-124) (jfclere) |
![]() |
Update httpd to 2.2.25. (MODCLUSTER-134) (jfclere) |
![]() |
Apache with mod_cluster refuses to start at first, but after 7 retries it starts up (MODCLUSTER-120) (jfclere) |
![]() |
Add getLoad() to load metric mbean interface (MODCLUSTER-130) (pferraro) |
![]() |
Disable "cnone" request parameter to ease remote invocation on mod_cluster-manager (MODCLUSTER-127) (jfclere) |
![]() |
Microcontainer does not always choose the right constructor when creating ModClusterService (MODCLUSTER-116) (pferraro) |
![]() |
Microcontainer does not choose the right constructor when creating RequestCountLoadMetric (MODCLUSTER-126) (pferraro) |
![]() |
Mod_cluster does support more that 3 Alias in <Host/> (MODCLUSTER-121) (jfclere) |
![]() |
Allow toggling of context auto-enable during mod_cluster startup. (MODCLUSTER-125) (pferraro) |
![]() |
STATUS should retry the worker even if there was an error before. (MODCLUSTER-133) (jfclere) |
![]() |
Split ModClusterServiceMBean.ping(String) into 3 methods (MODCLUSTER-110) (pferraro/jfclere) |
![]() |
Use clean shutdown by default, leveraging STOP-APP-RSP for <distributable/> contexts and session draining for non-distributable contexts. mod_cluster shutdown now triggered earlier via Connector JMX notification. (MODCLUSTER-131) (pferraro/jfclere) |
![]() |
move the web site to magnolia (MODCLUSTER-114) (.org team) |
![]() |
ping and nodeTimeout interact. (MODCLUSTER-132) (jfclere) |
![]() |
update mod_jk to 1.2.30 (MODCLUSTER-138) (jfclere) |
![]() |
query string is truncated to (MODCLUSTER-118) (jfclere) |
![]() |
AdvertiseBindAddress does not default to the 23364 port (MODCLUSTER-119) (jfclere) |
![]() |
Skip load balance factor calculation if there are no proxies to receive status message (MODCLUSTER-103) (pferraro) |
![]() |
Disabling contexts does not work (MODCLUSTER-123) (jfclere) |
![]() |
advertise doesn't use new AdvertiseSecurityKey on graceful restarts. (MODCLUSTER-129) (jfclere) |
![]() |
Load-demo.war specifies obsolete servlet in web.xml (MODCLUSTER-113) (pferraro) |
![]() |
Interaction with mod_rewrite looks weird for end-users. (MODCLUSTER-86) (jfclere) |
![]() |
admin-console should be in the excludedContexts. (MODCLUSTER-87) (pferraro) |
![]() |
ClassCastException upon redeploy after mod-cluster-jboss-beans.xml modification. (MODCLUSTER-88) (pferraro) |
![]() |
Alias from webapps/jboss-web.xml are not handled correctly in mod_cluster. (MODCLUSTER-89) (jfclere) |
![]() |
Display version. (MODCLUSTER-90) (jfclere) |
![]() |
Connector bind address of 0.0.0.0 propagated to proxy. (MODCLUSTER-91) (pferraro) |
![]() |
Display status of the worker. (MODCLUSTER-92) (jfclere) |
![]() |
Update httpd to lastest version. (MODCLUSTER-93) (jfclere) |
![]() |
getProxyInfo failed when there are too many nodes. (MODCLUSTER-94) (jfclere) |
![]() |
mod_cluster-manager display corrupted with jboss starting. (MODCLUSTER-95) (jfclere) |
![]() |
DISABLE application active as STOPPED. (MODCLUSTER-96) (jfclere) |
![]() |
Httpd should remove workers it can't ping. (MODCLUSTER-97) (jfclere) |
![]() |
Linux mod_cluster_manager display zero instead values. (MODCLUSTER-98) (jfclere) |
![]() |
mod_cluster_manager doesn't seem to ENABLE/DISABLE the right context. (MODCLUSTER-99) (jfclere) |
![]() |
load balancing logic doesn't allow manual demo of load-balancing. (MODCLUSTER-100) (jfclere) |
![]() |
404 errors when load is increasing. (MODCLUSTER-102) (jfclere) |
![]() |
Advertise security key verification does not work. (MODCLUSTER-104) (jfclere) |
![]() |
Allow advertise listener to listen on a specific network interface. (MODCLUSTER-106) (pferraro) |
![]() |
Allow thread factory injection for advertise listener. (MODCLUSTER-108) (pferraro) |
![]() |
Create SPI and isolate tomcat/jbossweb usage into service provider implementation. (MODCLUSTER-111) (pferraro) |
Advertise allows autodiscovery of httpd proxies by the cluster nodes. It is done by sending multicast messages from httpd to the cluster. The httpd specialized module: mod_advertise sends UDP message on a multicast group, both mod_advertise and the cluster listener joined the multicast group and the cluster receives the messages.
Example for mod_advertise message:
HTTP/1.0 200 OK Date: Wed, 08 Apr 2009 12:26:32 GMT Sequence: 16 Digest: f2d5f806a53effa6c67973d2ddcdd233 Server: 1b60092e-76f3-49fd-9f99-a51c69c89e2d X-Manager-Address: 127.0.0.1:6666 X-Manager-Url: /bla X-Manager-Protocol: http X-Manager-Host: 10.33.144.3
The X-Manager-Address header value is used by the cluster logic to send information about the cluster to the proxy. It is the IP and port of the VirtualHost where mod_advertise is configured or URL parameter of the ServerAdvertise directive.
See Proxy Discovery in Configuration Properties and mod_advertise in Apache httpd configuration.
In the VirtualHost receiving the MCPM of httpd.conf don't use any Advertise directive or use:
ServerAdvertise Off
In mod_cluster-jboss-beans.xml add the addresses and ports of the VirtualHost to the proxyList property and set advertise to false, for example:
<property name="proxyList">10.33.144.3:6666,10.33.144.1:6666</property> <property name="advertise">false</property>
In server.xml (with JBossweb/Tomcat)
<Listener className="org.jboss.modcluster.container.catalina.standalone.ModClusterListener" advertise="true"/>
See at the end of java configuration. You can't use the mod_cluster clustered mode with Tomcat so you get a loadbalancing logic similar to mod_jk but with a dynamic configuration.
Most likely you have a configuration problem. Check the log of the cluster nodes and error_log of httpd.
That happens when Advertise is not working: The nodes don't get the advertise messages from httpd.
Check the modules are loaded and Advertise is started. In httpd.conf activate extended information display, add:
AllowDisplay On
When accessing to the mod_cluster_manager you should get something like:
If not, go to the Minimal Example and add the missing directive(s).
Check that Advertise message are received on the cluster node. A small Java utility could be used to check Advertise. It is in the mod_cluster repository and can be compiled using javac. A compiled version can be found under in /opt/jboss/httpd/tools in the bundles. Run it using java Advertise multicastaddress port. The output should be something like:
[jfclere@jfcpc java]$ java Advertize 224.0.1.105 23364 ready waiting... received: HTTP/1.0 200 OK Date: Mon, 28 Jun 2010 07:30:31 GMT Sequence: 1 Digest: df8a4321fa99e5098174634f2fe2f87c Server: 1403c3be-837a-4e76-85b1-9dfe5ddb4378 X-Manager-Address: test.example.com:6666 X-Manager-Url: /1403c3be-837a-4e76-85b1-9dfe5ddb4378 X-Manager-Protocol: http X-Manager-Host: test.example.com
No Advertise messages
Check firewall (don't forget the boxes firewall). Advertise uses UDP port 23364 and multicast addresse 224.0.1.105
Can't get Advertise messages
Use ProxyList property. In case Advertise can't work you put the address and port of the VirtualHost used in httpd to receive the MCMP. In server/profile/deploy/mod_cluster.sar/META-INF/mod_cluster-jboss-beans.xml
<property name="proxyList">test.example.com:6666</property>
or in server.xml:
<Listener className="org.jboss.modcluster.container.catalina.standalone.ModClusterListener" proxyList="test.example.com:6666"/>
"IO error sending command ":
Check firewall and error_log if there is nothing error_log then it is a firewall problem. If you have something like:
18:36:14,533 INFO [DefaultMCMPHandler] IO error sending command INFO to proxy jfcpc/10.33.144.3:8888
You can use telnet hostname/address port to check by hands that it is OK for example:
[jfclere@jfcpc docs]$ telnet 10.33.144.3 8888 Trying 10.33.144.3... Connected to jfcpc. Escape character is '^]'. GET / <html><body><h1>It works!</h1></body></html>Connection closed by foreign host.
Check that the address and port are the expected ones you may use ServerAdvertise On http://hostname:port, like:
ServerAdvertise On http://localhost:6666
Or use the servlet testhttpd in the testhttpd.war of the bundle (in /opt/jboss/httpd/tools). Install/deploy it in in the node and start the AS/Tomcat on the node, than access to the node directly and call /testhttpd/testhttpd
If you don't get a similar page the output should help to find that is wrong.
"client denied by server configuration":
The location in the VirtualHost is not allowed for the client. If you have something like:
Mon Jun 28 18:08:47 2010] [error] [client 10.33.144.3] client denied by server configuration: /
You need to have something like:
<Location /> Order deny,allow Deny from all Allow from 10.33.144.3 </Location>
You need to configure EnableMCPMReceive in the VirtualHost where you received the MCMP elements in the Apache httpd configuration. Something like:
<VirtualHost localhost:6666> <Location /> Order deny,allow Deny from all Allow from 127.0.0.1 </Location> ServerAdvertise on http://localhost:6666 EnableMCPMReceive </VirtualHost>
When using http/https instead AJP proxyname, proxyhost and redirect must be configured in the Connector. Something like:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" proxyName="httpd_host_name" proxyPort="8000" redirectPort="443" />
Mod_cluster tries to use the first AJP connector configured. If there isn't any AJP connector it uses the http or https that has the biggest maxthreads value. That is "maxThreads" in Tomcat and JBossAS4/5/6:
<Connector port="8080" protocol="HTTP/1.1" maxThreads="201" />
Or "max-connections" in AS7: (32 * processor + 1 for native and 512 * processor + 1 for JIO).
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="513" />
When using chrome with mod_cluster_manager the page is not displayed and the following error is displayed:
Error 312 (net::ERR_UNSAFE_PORT): Unknown error.
you can change the port of the VirtualHost to 7777 or any value chrome accepts or add:
--explicitly-allowed-ports=6666
to the start parameters of chrome.
Mod_cluster needs to open port and create shared memory and files, therefore some permissions have to be added, you need to configure something like:
policy_module(mod_cluster, 1.0) require { type unconfined_java_t; type httpd_log_t; type httpd_t; type http_port_t; class udp_socket node_bind; class file write; } #============= httpd_t ============== allow httpd_t httpd_log_t:file write; corenet_tcp_bind_generic_port(httpd_t) corenet_tcp_bind_soundd_port(httpd_t) corenet_udp_bind_generic_port(httpd_t) corenet_udp_bind_http_port(httpd_t) #============= unconfined_java_t ============== allow unconfined_java_t http_port_t:udp_socket node_bind;
Put the above in a file for example mod_cluster.te and generate the mod_cluster.pp file (for example in Fedora 16):
[jfclere@jfcpc docs]$ make -f /usr/share/selinux/devel/Makefile Compiling targeted mod_cluster module /usr/bin/checkmodule: loading policy configuration from tmp/mod_cluster.tmp /usr/bin/checkmodule: policy configuration loaded /usr/bin/checkmodule: writing binary representation (version 14) to tmp/mod_cluster.mod Creating targeted mod_cluster.pp policy package rm tmp/mod_cluster.mod.fc tmp/mod_cluster.mod
The mod_cluster.pp file should be proceeded by semodule as root:
root@jfcpc docs]# semodule -i mod_cluster.pp [root@jfcpc docs]#