Architecting extremelylarge scale web applications

ARCHITECTING EXTREMELY LARGE SCALE WEB

APPLICATIONSA MUST read for every architect

[ #Prashanth B Panduranga ]

An overview of the much needed

vicissitude in the architectural

thought transformation from

monolithic to microservices

architecture

New York times auto scaled to 500,000 users

HipChat has about 1.2 billion messages/documents stored

Sales force deals with 1,300,000,000 daily transactions with over 24,000 database

transactions per second and over 22PB of raw SAN storage capacity

CinchCast has over 50 million page views a month

Pinterest has over 18 million visitors with a 10X growth rate

Amazon has over 55 million active customer accounts

Flickr has over 4 billion queries per day

Netflix has 48 million members with over 50,000 requests per second and the list goes

on. .

Thanks to HighScalability, where the above statistics are derived from, you get a

picture of the websites and the scale that I am referring to

I have elaborated on SAAS requirements and general architectural requirements in my previous blog.

https://prashanthpanduranga.wordpress.com/2015/03/25/inevitability-of-multi-tenancy-saas-in-product-

engineering/

The requirements which apply the most to large scale web applications:

Performance: There are lot of statistics published relating to performance implications of web applications. One

such statistic, Users abandon website even if there is a 2 second delay during a transaction. Large scale web

applications obviously have very high performance requirements. A very large percentage of applications gets

redesigned primarily for a better user experience. Average Page Load Time as a fraction of Server and client time,

Network time, Page views, Bounce Rate, Percentage Exit, Average Redirection Time, Average Domain Lookup Time,

Average Server Connection Time, Average Server Response Time, Average Page Download Time, Average content

load time, Average session time, DNS resolution time, TCP connection time, Time to first byte, Full page object load

time, Requests per second, error rates, Peak response time, Uptime, CPU utilization, Memory Utilization are just a

few metrics to look for

Availability:

Business continuity is of utmost importance. Various availability techniques can be applied on every layer.

AlwaysOn Failover Cluster Instances, AlwaysOn Availability Groups, Database mirroring, Log shipping,

Redundancy models - Active-Active, Active-Passive, Redundancy Methods - Hot Standby, Warm Standby, Cold

Standby, and the measurement of the same expressed as mean time to failure, mean time to repair, Eliminating

single points of failure, Accelerating fault detection, isolation and resolution, hot spares, warm spares, cold

spares, clustering, RAID, redundancy, Heart beats, watermarking resources, check pointing, Watch dogs and

more

Monitoring and Diagnostics:

26 front end proxy serves. Double that in backend app servers [Atlassian HipChat]

15,000+ hardware systems – [Salesforce]

100 hardware nodes in production – [CinchCast]

180 Web Engines + 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, 200

Memcache Instances, 4 Redis Task Manager + 80 Task Processors, Sharded Solr – [Pinterest]

1000+ supported devices – [Netflix]

Imagine maintaining those, when there are innumerable servers involved, failure of system components is

common, needless to say, to take appropriate timely actions monitoring and diagnostics plays a very important

role.

Scalability:

Capability of supporting and optimizing resource utilization on increasing workloads on various dimensions such

as memory, cores, data structures, throughout and more. Goes without doubt that the application need to scale,

in order for the application to perform well, and without automating it, the application cannot stay inexpensive.

Automation:

Identifying failures and automating re-provisioning of those components/servers is extremely important

Architecture has significantly emerged from a monolithic architecture to microservices architecture

APPLICATION

DATA

PRESENTATION

SERVICE/BUSINESS LOGIC

DATA

PRESENTATION

SERVICE/BUSINESS LOGIC

DATA

SECU

RITY

ANAL

YTIC

S

MO

NITO

RIN

G &

DIA

GNO

STIC

S

STORAGE

EVEN

TS

NO

TIFI

CATI

ON

VIRTUALIZATION

NETWORKCOMPUTE

HARDWARE LAYER

CLOUD

CLOUD ADAPTER

AUTO

MAT

ION/

BATC

H

INTEGRATION

PROCESSING ENGINE

CACHEM

ANAG

EMEN

T

AD-H

OC

LAY

ER

AUDI

TING

CON

FIGU

RATI

ON

DISTRIBUTED PROCESSING

INGESTION

DATA MANAGEMENT DATA RULES META DATA DATA QUALITY

CLICKSTREAM DATA SOURCES

CHAT

SENSOR

SOCIAL

LOGS

CRM

ERP

APPLICATION DATA

CHANNELS

EDW

METADATA MANAGEMENT MULTI TENANT PROCESSING PARALLEL COMPUTE

PARALLEL OPERATIONS COMPLEX EVENT PROCESSING GOVERNANCE

WORKFLOW

REPO

RTIN

G

STREAM PROCESSING

IN-MEMORY PROCESSING

MESSAGE TRANSPORTSMESSAGE QUEUESMESSAGE BROKERS

INTEGRATION FRAMEWORKSENTERPRISE SERVICE BUS

INTEGRATION SUITES

OPERATING SYSTEM

Tools/software Usage/Description TOP few similar tools/software to consider Comments

RabbitMQ Message broker

system

Message oriented

middleware

Queuing software

ESB

ActiveMQ, Amazon SQS, HornetQ, HiveMQ,

JMS, Kafka, ZeroMQ, MSMQ, NServiceBus, Azure

Service Bus, OpenMQ, Redis, Storm, Akka,

Apache Camel and Spring, OFM, Fuse ESB,

WebsphereMQ, Windows Service Bus, BizTalk,

WSO2, Mule, Talend ESB, Gearman, JBoss,

ServiceMix, OpenESB, Apache QPID

Look out for AMQP

compliance

Some of the tools

referred aren’t

Message brokers but

are used in

conjunction to perform

the same

Other AMQP

PIKA

Shovel

Kinesis Real time data

processing

Kafka, Storm

Tornado Web Server

HTTP Server

Nginx, Apache, IIS, lighttpd, haproxy, varnish,

glassfish, Jetty, Geronimo, Tomcat,

Some are even used

as reverse proxy,

proxies,

Cassandra Distributed database

management system

Mongodb, aerospike, accumulo, azure table storage,

bigtable, couchbase, couchdb, dynamodb, datastax,

ElasticSearch, Greenplum, Vertica, HBase, InfiniDb,

InnoDB, MariaDB, neo4j, Netezza, TeraData, RedShift,

Riak, RavenDB, Solr, Spark, VoltDB,

With over 200 dbs, it’s

difficult to list all: Checkout

the below link

https://prashanthpanduran

ga.wordpress.com/2013/12

/23/why-nosql-ok-but-why-

so-many/

Some of them are

dataware housing

Solutions, While some are

data processing engines

Hana In-Memory database GemFire, Hekaton, Aerospike, BigMemory, DataBlitz,

EhCache, eXtremeDB, FuelDB, HazelCast, MonetDB,

Coherence, VoltDB

Any Key value store can be

used for the same, some of

the enterprises have

experimented using NoSQL

store used as a cache and

unstructured db solution

Linux Operating System SUSE, FreeBSD, Solaris, Debian/Ubuntu, Windows Server,

Mac OS X, RHEL

Only Server OS included in

the list

SockJS Web Socket like

object

Web socket, socket.io, atmosphere, SignalR,

Alchemy, Fleck

Couple of them listed

are open source

Libev Event Loop LibEvent, Asio, Nginx, epoll

Fabrik Visual programming

IDE

Also known for

Content construction

Kits (CCK)

IDE: Visual Studio, Eclipse, NetBeans, Aptana

CCKS: Seblod, K2, chronoform, Zoo,

Breezingforms, Cobalt, FlexiContent

Java Programming

language

C, C++, Python, C#, PHP, Javascript, Ruby, R,

Matlab, Objective-C, Visual Basic, Perl, Swift,

Scala, Shell, GO, LISP, SAS, F#, Groovy, Lua

Some of them listed

are web programming

languages adiitionals:

HTML, SQL, Haskell

Twisted Event driven network

programming

framework

Tornado, Django, Asyncio,

AWS Cloud provider Azure, Rackspace, CenturyLink, Salesforce,

Engineyard, Google, OpenStack, SAP,

CloudBees, CumuLogic, Eucalyptus, Gigaspaces,

Mulesoft, Parallels, Pivotal, puppet Labs, Ravello,

Rightscale, SoftwareAG, Xively, AT & T, Cisco,

Comcast, EMC, GoGrid, CSC, HP, IBM

smartcloud, Joyent,

The list includes

infrastructure, platform,

storage and security

cloud providers

Lucene Test search Engine

Library

Azure Search, Autonomy, Solr, GSA, Attivio,

DTSearch, elasticSearch, endeca, FAST,

MarkLogic, Nutch, Sphinx, Sketchy, Scumblr

A few NOSQL

databases have been

used for the same, This

list does not include all

the NOSQL databases

that could be used

Adobe Air Cross-platform

runtime

Cordova(Phonegap), Appcelerator, Qt, Sencha,

cocos2d-x, Xamarin, ionic, Kony, mono, xcode

The ones listed here are

cross platform as well as

mobile development

platforms.

Sensu Monitoring

Framework

Zabbix, Nagios, icinga, monit, Riemann, statsd,

graphite, zenoss, collectd, munin, cacti, new

Relic, ganglia, splunk, sentry, dynatrace,

datadog, skylight, zenoss, observium, spiceworks,

solarwinds, fiddler, wireshark, httpwatch, firebug,

soapUI, OpManager

The list includes some of

the: Infrastructure

monitoring

Searching, monitoring

and analysing, Network

monitoring

Scalable distributed

monitoring system

PagerDuty

NeoLoad

Incident

management system

and performance

testing and

monitoring

OpsGenie, VictorOps, xmatters, pingdom,

Gomez, webpagetest, monitis, uptrends,

keynote, OpsView, Apache JMeter, LoadRunner,

WebLOAD, Appvance, NeoLoad, LoadUI, WAPT,

Loadster, LoadImpact, Soasta, Rational

Performance Tester, Testing Anywhere,OpenSTA,

QEngine (ManageEngine), Loadstorm,

CloudTest, Httperf, SilkPerformer, BlazeMeter,

Visual Studio Test Suite,

Also includes web site

monitoring

Cloud based quality

testing

Performance

monitoring

Chef IT Automation Puppet, ansible, salt, docker, Jenkins,

Capistrano, saltstack

Configuration

management, SCCM

memCache Distributed memory

object caching

Apc, memcached, dynacache, ehcache,

xcache,

key value based

NOSQL databases are

also used

Razor Physical and virtual

hardware provisioning

solution

Axemblr, Cobbler, JuJU, SaltCloud, Dell Crowbar,

Ansible, CFEngine, Chef

Perforce Version Management

and Content

collaboration

Git, SVN, TFS, bitbucket, ClearCase, Subversion

Pytheas ITIL assets management

software

Remedy (BMC), Assyst (Axios), FrontRange, EasyVista,

Hornbill, HP Service Manager, SmartCloud Control Desk

(IBM), ServiceNow

IT incident management, IT

problem management, IT

change management, IT

release governance, IT user

self-service, IT request

management, IT

knowledge management,

IT service support analytics

and reporting, IT SLA

management

Ref: Gartner

ZUUL Service that provides

dynamic routing,

monitoring, resiliency

and security

Nginx, lightpd, Netscaler, HAProxy, Radware,

CoyotePoint, Barracuda, Kemp, Varnish, Avast, Norton,

Kaspersky, Mcafee, AVG, Avast, Bitdefender, F5,

PaloAlto, Cisco ASA, Cisco ACE, Foundary, Juniper SSG,

MS TMG

Can be firewall, router, web

load balancing server,

proxy Server etc.

Feign Java http client binder Retrofit, JAX-RS, web socket,

Jersey, CXF, Apache HC

Includes transport libraries

Hive Querying and managing

large datasets residing in

distributed storage

Impala, BigSQL, HAWQ

AWS ELB Elastic Load balancing Nginx, HAProxy, Route53,

Azure Traffic Manager, F5

Port-bound servers, sticky sessions, TCP session

reassignment, automatic unfail, slow start, SynGuard,

dynamic feedback protocol, NAT, maximum

connection, Round Robin, Least Connections,

Weighted Round Robin, Weighted Least Connections,

Fastest Response

Layer 4 and Layer 7 load balancing

Cloud Load balancing features: Dedicated (static) IP

address,SSL termination, Multiple protocols, Advanced

access control, Connection logging, Advanced

algorithmic routing, Session persistence, Connection

throttling, Node management, High availability

Content caching, Persistent connections, Gzip

compression, Regionalized load balancers

gZip Application used for

file compression and

decompression

httpZip,deflate, 7zip, bzip2, zlib

Akamai Content delivery

network

Azure CDN, Cloudfront, Torbit, Incapsula,

Cotendo, Fastly

HTML 5 frameworks

Javascript

Frameworks

https://www.facebook.com/notes/prashanth-panduranga/frameworks/10152107517972934

OpenStack Open Source Cloud

computing platform

OpenStack currently has the following features:

Compute(Nova), Object Storate (Swift), Block

Storage (Cinder), Networking (Neutron),

Dashboard (Horizon), Identity Service (Keystone),

Image Service (Glance), Telemetry (Ceilometer),

Orchestration (Heat), Database (Trove), Bare

Metal Provisioning (Ironic), Multiple tenant cloud

messaging (Zaqar), Elastic Map Reduce

(Sahara)

Hadoop Distributed storage and distributed processing of very large data sets on computer clusters

Aegisthus Bulk Data Pipeline out of Cassandra

Eureka Eureka is a REST (Representational State Transfer) based service that is primarily used in the AWS

cloud for locating services for the purpose of load balancing and failover of middle-tier servers

Genie Federated Job Execution Engine

Clojure Dynamic programming language that targets the Java Virtual Machine

PigPen Map-Reduce for Clojure

Governator Governator is a library of extensions and utilities that enhance Google Guice to provide:

classpath scanning and automatic binding, lifecycle management, configuration to field

mapping, field validation and parallelized object warmup

Inviso Visualize Hadoop performance

Ribbon Ribbon is a Inter Process Communication (remote procedure calls) library with built in software

load balancers

Hystrix Hystrix is a latency and fault tolerance library designed to isolate points of access to remote

systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex

distributed systems where failure is inevitable

Suro Distributed data pipeline

Aminator A tool for creating EBS AMIs

Lipstick Pig Visualization framework

Zeno In-Memory Data Propagation Framework

Blesk Lightweight client for pushing notifications to web based applications/sites

Turbine Turbine is a tool for aggregating streams of Server-Sent Event (SSE) JSON data into a

single stream. The targeted use case is metrics streams from instances in an SOA

being aggregated for dashboards

Priam Co-Process for backup/recovery, Token Management, and Centralized

Configuration management for Cassandra

Workflowable Workflowable is a Ruby gem that allows adding flexible workflow functionality to

Ruby on Rails Applications

s3mper S3mper is a library that provides an additional layer of consistency checking on top

of Amazon's S3 index through use of a consistent, secondary index

Astyanax Java Client for Apache Cassandra

Denominator Denominator is a portable Java library for manipulating DNS clouds. Denominator

has pluggable back-ends, including AWS Route53, Neustar Ultra, DynECT, Rackspace

Cloud DNS, OpenStack Designate, and a mock for testing

GCViz Garbage Collector Visualization framework

Curator The Curator Framework is a high-level API that greatly simplifies using ZooKeeper. It

adds many features that build on ZooKeeper and handles the complexity of

managing connections to the ZooKeeper cluster and retrying operations

Staash A language-agnostic as well as storage-agnostic web interface for storing data into

persistent storage systems, the metadata layer abstracts a lot of storage details and

the pattern automation APIs take care of automating common data access patterns

Edda Edda is a Service to track changes in cloud deployments

Brutal An asyc centered chat bot framework for python programmers written using the twisted

framework

CassJMeter JMeter plugin to run cassandra tests

Glisten Groovy library for building JVM applications with Amazon Simple Workflow (SWF)

Pig Platform for analyzing large data sets

Spark Engine for big data processing, with built-in modules for streaming, SQL, machine learning and

graph processing

Karyon Framework and a library for a cloud ready web service. Blueprint for the services. It contains

Bootstrapping, Libraries and Lifecycle Management, Runtime Insights and Diagnostics,

Pluggable Web Resources, Cloud-Ready hooks

EBS Elastic Block store, persistent block level storage volume

Curler A Gearman worker which cURLs to do work

archaius, , Library for configuration management API

ZooKeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing

distributed synchronization, and providing group services

Parallel processing - Explicit and Implicit parallelism, batch parallelism, asynchronous programming, segregating

layers, distributing workloads, Load balancing, multi- tenancy, scaling out on all layers, sharding, partitioning,

CAP preference, reads, writes, statelessness, logging and telemetry, automating, SOA adoption, caching,

throttling, distributing requests across multiple zones, effective usage of CDNs, Auto provisioning, Auto scaling,

compression, queuing, workload distribution, batch processing, designing system with fault tolerance,

redundancy, Consistency, Availability, Partition Tolerance, event processing, web sockets, cloud computing, fog

computing, Grid Computing, Client side workload distribution, In-Memory processing, Proxies, No single points

of failure. Resilience to failure, Graceful degradation, Recoverability from failure, design for failure, Database

Transactions, Client side transactions, two-phase commit, Auto-commit, Partition Everything, DB operations

ordering, Considerations for Eventual consistency, Functional Segmentation, Application Pools, Prevention of

session state, Async Everywhere, Index, Structured Indexes, text indexes, entity indexes, Fuzzy match indexes,

pre-aggregated indexes, pre-calculated indexes, embedded value indexes, join indexes, link indexes, De-

Normalized Indexes (all kinds) are all important considerations for a highly successful and scalable website.

Rest assured if you have considered all the above factors in your architecture you are on your way to create a scalable one. Do let me know if you have questions regarding any particular subject and I will be glad to write up on the same.

Reference

www.HighScalability.com

has some awesome architecture blogs, I have derived the technology stack from multitude of articles

hosted by highscalability and a few from Netflix blogs

PRASHANTH B PANDURANGA

https://prashanthpanduranga.wordpress.com/

Telephone (India): 7259767006, 7760824680

[email protected]