Reducing the Operations Burden and Increasing QoS with WSO2 Platform Management and Administration Tactics Sanjaya Ratnaweera Amani Soysa
Reducing the Operations Burden andIncreasing QoS with
WSO2 Platform Managementand Administration Tactics
Sanjaya RatnaweeraAmani Soysa
Agenda
● Common Deployment Patterns● Deployment Synchronization● Registry Mounting● Setting up a worker-manager separated Appserver cluster -
Demo/Hands-on● Deployment Automation● Server Monitoring● Production Deployment Guidelines● Common issues in Production Setups
WSO2 Product deployment patterns
● Standalone
● Clustered
● Cloud based
Standalone Deployment
● User engages with one or many independent nodes
Clustered Deployment
● User engages with the load balancer● Highly available● Concept of worker and manager nodes
Worker-Manager Separation ?
➔ Proper separation of concerns◆ worker node - can ONLY deploy artifacts and read
configurations◆ manager node - authorized to add new artifacts and
make configuration changes➔ Lower memory foot in the worker nodes
◆ lesser OSGi bundles➔ Improved security
◆ management nodes can be behind the internal firewall & be exposed to clients running within the organization only, while worker nodes can be exposed to external clients.
Cloud Based Deployment
● Extension of the Clustered deployment
● Highly scalable
● Span across multiple IaaSs
Why Deployment synchronization ?
• Artifact distribution and deployment should be transparent to the end-users
• Manual artifact copying is acceptable for some extent in standalone product clusters.
• Will that work in an elastically scaling cloud environment ??
Deployment Synchronization
● Distribute deployment artifacts and related meta-data across the cluster
● Make the cluster nodes consistent● Automated synchronization based on SVN ● Management nodes commits and worker nodes check-out
Deployment Synchronization<--Manager Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>true</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>
<-- Worker Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>false</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>
Registry Mounting
● An approach to share registry space across cluster nodes● Local Data repository
○ Should not be shared● Configuration registry
○ Shared across multiple nodes of the same product● Governance registry
○ Shared across product platform
Registry Mounting
Worker-Manager Separated Cluster - Demo/Hands-On
● Setting up user management and registry databases● Configuring the Elastic loadbalancer● Configuring WSO2 AS as a management node● Configuring WSO2 AS as a worker node
Deployment Automation
Configuration Management
No matter how small you are
adapt yourself to configuration management
Abstracted Configurationnode001 { appserver { version => "5.2.1", target => "/opt/appserver", cluster_domain => "mgt", }}
node002 { elb { version => "2.1.0", offset => "0", target => "/opt/elb", user => "root", } }
Patch Management
Orchestration
Powered by Puppet
https://github.com/wso2/Puppet-Modules
Server and Health Monitoring
● JMX based monitoring○ JConsole, Graphite
● Message Tracing Tools○ Wireshark, tcpmon
● System monitoring tools○ Ganglia, Cacti, Nagios
● Alerting○ Email, SMS, Phone-call alerts
Production Deployment Guidelines
● Changing the default administrator credentials● Replacing SSL certificates/keys● Hardening the Operating System● Turning off unused services/ports● Switching the default H2 registry database to a production
ready database● Using secure vault to encrypt passwords in configuration files● Tune database connection pools
Production Deployment Guidelines contd..
● Disabling HTTP access logs● Tuning heap memory● Tuning GC● Running servers as background processes
○ nohup, YAJSW
Patching
● Follow the recommended procedure● Automate the patch applying process● Test the patch in dev and staging environments
○ Automated process is a must● Update production servers and restart in a RR manner● Maintain a log for patches
Common errors in production
● IO errors - Connection reset by peer, Broken pipe, Too many open files, I/O reactor has been shut down
● Database connection pool errors● OutOfMemoryErrors
Production Troubleshooting Tips
● carbondump○ WSO2_HOME/bin/carbondump.{sh,bat}
● Replicate the issue in staging● Report JIRAs with detailed information
○ Upload carbondump output to the provided FTP server○ Attach the relevant deployment artifacts and logs
Sanjaya Ratnaweera
Blog: http://samudura.org
Follow: @sanjayar
Amani Soysa
Blog: http://sparkletechthoughts.blogspot.com/
Follow: @poohdedoo
Thank You