Cluster Overview and PLF Cluster Basic notions, configuration, common errors and current status March 2012 TRAN Trung Thanh, PLF/MQ
Nov 01, 2014
Cluster Overview and PLF ClusterBasic notions, configuration, common errors and current status
March 2012
TRAN Trung Thanh, PLF/MQ
www.exoplatform.com - Copyright 2012 eXo Platform 2
1
2
3
4
Agenda (1)
PLF cluster current status
- Story
- Bugs relating to configuration
- Bugs in the implementation
Cluster Introduction
- Cluster, What's it?
- Why cluster?Basic configuration
- Basic cluster infrastructure with load balancing
- Two-node PLF cluster
Advanced configuration
- Hot spare server
- Asymmetric load
www.exoplatform.com - Copyright 2012 eXo Platform 3
Agenda (2)
Conclusion and Perspective
Q & A
www.exoplatform.com - Copyright 2012 eXo Platform 4
1
2
3
4
Agenda
PLF cluster current status
- Story
- Bugs relating to configuration
- Bugs in the implementation
Cluster Introduction
- Cluster, What's it?
- Why cluster?Basic configuration
- Basic cluster infrastructure with load balancing
- Two-node PLF cluster
Advanced configuration
- Hot spare server
- Asymmetric load
www.exoplatform.com - Copyright 2012 eXo Platform 5
Cluster, What is it?
− High scalable and high available virtual server− Fully transparent to end user
Basic Cluster Infrastructure
www.exoplatform.com - Copyright 2012 eXo Platform 6
Why cluster? (1)
− Challenges− Increase dramatically the traffic of Internet ( annual rate > 100%)− Increase rapidly workload of servers− Enhance the availability of servers
− Solutions− Upgrade the software/hardware of servers− Multi-server
Cluster
Grid
Cloud
www.exoplatform.com - Copyright 2012 eXo Platform 7
Why cluster? (2)
− Cluster Notions− Scalability− Load Balancing
Load Balancing with an appliance
Load Balancing with a software/hardware combination mod_proxy, mod_jk, mod_cluster
− High Availability
www.exoplatform.com - Copyright 2012 eXo Platform 8
1
2
3
4
Agenda
PLF cluster current status
- Story
- Bugs relating to configuration
- Bugs in the implementation
Cluster Introduction
- Cluster, What's it?
- Why cluster?Basic configuration
- Basic cluster infrastructure with load balancing
- Two-node PLF cluster
Advanced configuration
- Hot spare server
- Asymmetric load
www.exoplatform.com - Copyright 2012 eXo Platform 9
Basic Cluster Infrastructure with load balancing
www.exoplatform.com - Copyright 2012 eXo Platform 10
Two-node PLF cluster
− Load Balancer: mod_jk− Server cluster: 2 nodes ( 2
instances of PLF in same machine or in 2 distinguished machines)
− Shared Storage:− NFS: indexing data, key
for gadget
− DB server: shared PLF data
www.exoplatform.com - Copyright 2012 eXo Platform 11
Mod_jk(1)
− Connector used to connect the Tomcat Servlet Container with Web Servers
− Use AJP (Apache Jserv Protocol)− Roles
− Load the servlet container adapter library and initialize it (prior to serving requests)
− Check and redirect requests to servlet to handle it
www.exoplatform.com - Copyright 2012 eXo Platform 12
Mod_jk(2)
− mod_jk configuration step-by-step − Install apache− Add mod_jk modules to apache− Configure mod_jk parameters in apache configuration (httpd.conf)− Assign URL to apache
JkMount <URL prefix> <Worker name>
− Configure worker properties: IP, port for AJP, hostname
www.exoplatform.com - Copyright 2012 eXo Platform 13
PLF cluster configuration(1)
− Notes− Use profile all to deploy PLF server in cluster mode− PLF user external connector to connect DB
Need to add a external connector if necessary
− Configure datasource
− Config target
$TOMCAT_HOME/conf/server.xml (tomcat server)
$JBOSS_HOME/server/all/gatein-ds.xml (jboss server).
− Parameters
User name/password, connector, database address
www.exoplatform.com - Copyright 2012 eXo Platform 14
PLF cluster configuration(2)
− Configure NFS Config target
$TOMCAT_HOME/conf/server.xml (tomcat server)
$JBOSS_HOME/server/all/deploy/jbossweb.sar/server.xml (jboss server)
− Configure AJP route
− Config target $TOMCAT_HOME/gatein/conf/configuration.properties (tomcat server)
$JBOSS_HOME/server/all/conf/gatein/configuration.properties (jboss server)
− Parameters Add JVM Route to mark the name of server used by mod_jk
www.exoplatform.com - Copyright 2012 eXo Platform 15
PLF cluster configuration(3)
− Activate PLF configuration Tomcat
Switch jcr mode to cluster
Add profile cluster to eXo Profile
Add preferIPv4Stack parameter to route message for synchronized
Jboss
Switch jcr mode to cluster
Add profile cluster to eXo Profile
Add preferIPv4Stack parameter to route message for synchronized
Activate Cluster Single On
www.exoplatform.com - Copyright 2012 eXo Platform 16
PLF cluster configuration(4)
− Start PLF Tomcat
./start_eXo.sh
Jboss
bin/run.sh -g MyClusterPartition -b IP_NODE -c all
www.exoplatform.com - Copyright 2012 eXo Platform 17
PLF cluster configuration(5)
− Verify cluster In console, when a new member is added or any downs
INFO [MyClusterPartition] Dead members: 0 ([])
INFO [MyClusterPartition] New Members : 1 ([127.0.0.1:1099])
INFO [MyClusterPartition] All Members : 2 ([127.0.0.1:1199, 127.0.0.1:1099])
www.exoplatform.com - Copyright 2012 eXo Platform 18
PLF cluster configuration(6)
− Common errors in configuring and use cluster mode Error on connection to DB server
Lack of connector in lib Add connector
Wrong database address parameters Verify database parameters
DB isn't set to accept remote access Configure to accept remote access
Indexing data and database aren't matched
Remove indexing data and database if possible
www.exoplatform.com - Copyright 2012 eXo Platform 19
1
2
3
4
Agenda
PLF cluster current status
- Story
- Bugs relating to configuration
- Bugs in the implementation
Cluster Introduction
- Cluster, What's it?
- Why cluster?Basic configuration
- Basic cluster infrastructure with load balancing
- Two-node PLF cluster
Advanced configuration
- Hot spare server
- Asymmetric load
www.exoplatform.com - Copyright 2012 eXo Platform 20
Advanced Configuration(1)
−Hot Spare serverl Failover mechanism to provide reliabilityl A key component fails, the hot spare is switched into
operation
www.exoplatform.com - Copyright 2012 eXo Platform 21
Advanced configuration(2)
− Hot spare configuration− Set hot spare server for main server
# Define preferred failover node for plfnode1
worker.plfnode1.redirect=plfnode2
− Deactivate hot spare server # Disable plfnode2 for all requests except failover
worker.plfnode2.activation=disabled
www.exoplatform.com - Copyright 2012 eXo Platform 22
Advanced Configuration(3)
•Asymmetric loadl User to divide the workload among serversl Use lbfactor for each server worker.plfnode1.lbfactor=2 worker.plfnode2.lbfactor=8
www.exoplatform.com - Copyright 2012 eXo Platform 23
1
2
3
4
Agenda
PLF cluster current status
- Story
- Bugs relating to configuration
- Bugs in the implementation
Cluster Introduction
- Cluster, What's it?
- Why cluster?Basic configuration
- Basic cluster infrastructure with load balancing
- Two-node PLF cluster
Advanced configuration
- Hot spare server
- Asymmetric load
www.exoplatform.com - Copyright 2012 eXo Platform 24
Story
− Support cluster from PLF 3.0.x− EPP SP 5.1.x in cluster mode with RedHat− Lack of test and of evaluation in PLF 3.0.x
− Support fully cluster mode from PLF 3.5.1
− FQA test campaign for cluster
− Cluster configuration changes from PLF 3.0.x to 3.5.x
− Use shared file-system for security token store (EXOGTN-237)
− Reconfigure the parameters for cluster mode (PLF-630)
www.exoplatform.com - Copyright 2012 eXo Platform 25
Bugs relating to configuration
− Too many warnings on console− Related issue: PLF-2652− Description: Too many warning on console repeatedly− Workaround: Hide log
Tomcat: Add to conf/logging.properties
org.jgroups.level = ERROR
org.jgroups.handlers=
java.util.logging.ConsoleHandler,6gatein.org.apache.juli.FileHandler
- Jboss: Add to server/all/conf/jboss-log4j.xml
<category name="org.jgroups">
<priority value="ERROR"/>
</category>
www.exoplatform.com - Copyright 2012 eXo Platform 26
Bugs relating to configuration(2)
− Replication time out− Related issue: PLF-2752− Description: Replication Timeout console on running PLF in
cluster mode with multi instances− Workaround: Increase Replication Timeout
www.exoplatform.com - Copyright 2012 eXo Platform 27
Bugs in implementation(1)
− Lack of Synchronized function− Related issue: ECMS-2324− Description: Job runs only on a server
→ Another job try to access the resource using by another job
− Suggested Solution: Synchronize functions. At one time, only a thread access a resource
www.exoplatform.com - Copyright 2012 eXo Platform 28
Bugs in implementation(2)
− Cache problem− Related issue: ECMS-3337, SOC-1368, EXOGTN-978− Description: To increase performance, the local cache is
implemented. Cache among another nodes isn't synchronized → Cannot see update in other nodes or some objects are
unavailable.
− Suggested: Solution: Only use cache for necessary cases. Evict cache immediately if the changes have to take effect on cluster nodes.
www.exoplatform.com - Copyright 2012 eXo Platform 29
Bugs in implementation(3)
− Problem with login session− Related issue: EXOGTN-943, PLF-2730− Description: The user login session isn't well shared among
servers in cluster. → Problem 1: Cannot relogin after a login failure
→ Problem 2: When a instance fails, all clients using it have to relogin.
− Suggested Solution: Broadcast user login session among servers in cluster
www.exoplatform.com - Copyright 2012 eXo Platform 30
Agenda
Conclusion and Perspective
Q & A
www.exoplatform.com - Copyright 2012 eXo Platform 31
Conclusion and Perspective
− Conclusion− PLF supports quite well in cluster mode− There are some problems in cluster mode
− Perspective
− Use mod_cluster for cluster configuration
− Organize server in many groups. Among in a group, all server shared the same session
www.exoplatform.com - Copyright 2012 eXo Platform 32
References
Admin guide for PLF 3.5
Install cluster mode by Nghi TQA
http://www.mulesoft.com/tomcat-cluster
http://msdn.microsoft.com/en-us/library/ff649250.aspx
http://tldp.org/HOWTO/openMosix-HOWTO/x135.html
http://www.linuxvirtualserver.org/whatis.html
http://www.sql-server-performance.com/2002/clustering-intro/
http://docs.oracle.com/cd/E14049_01/doc.9101/e14046/jboss_cluster.htm#BGBGDCHD
http://oddthesis.org/posts/2008-12-jboss-and-mod_cluster/
http://vinaytechs.blogspot.com/2009/11/clustering-in-jboss-application-server.html
http://www.webopedia.com/TERM/C/clustering.html
www.exoplatform.com - Copyright 2012 eXo Platform 33
Q & A