Zero to Thousands TPS: Private Cloud Operations Workshop Maudrit Martinez Apigee Paul Mibus Apigee
Zero to Thousands TPS: !Private Cloud Operations Workshop !
Maudrit Martinez Apigee
Paul Mibus Apigee
Agenda
9:30 – 10:00 Apigee Overview
10:00 – 11:00 Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00 Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45 Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30 Platform Operations
©2015 Apigee. All Rights Reserved.
Intelligent API platform enables the digital value chain
INTELLIGENT API PLATFORM
Intelligent API Platform
User Connected Experience
Developer API API Team Backend
©2015 Apigee. All Rights Reserved.
Apigee Edge Design"
Design first. Document Smart. Full support for Swagger 2.0
Develop Configuration: Over 30 ready-to-use and "configurable policies Code: Built-in support for Node, JavaScript and Java extensibility BaaS
Secure End-to-end security Threat protection Access control Simple OAuth implementation for your APIs PCI and HIPAA compliance
Publish Turnkey developer portal
MonitorCentralized control, decentralized development
Multi-tenant architecture Billions of API calls, including large spikes
Analyze Complete visibility– from app end to backend
Automatically and continuously collect "all API-traffic data out of the box
Monetize Flexible rate plans
Internationalization support Usage tracking
Limits and notifications
Scale Self-service
State @ scale Flexible deployment
©2015 Apigee. All Rights Reserved.
Apigee Edge – API management
API Services
API Team
Developer Services
Monitor, manage, and measure success
Analytics Services
Increase adoption and value of APIs
Build, manage, scale, and secure APIs and Apps API"
Gateway OAuth & Security
Policies & Programmability
Versioning & Governance
API Services
Developer"Portal
Smart"Docs
API"BaaS Monetization
Developer Services
Ops & Business Metrics
Developer"Analytics
App"Performance
Custom"Reports
Analytics Services
Flexible Deployment Self-Service State @ Scale
Connected Experience
Business User
Developer
©2015 Apigee. All Rights Reserved.
Config - Over 30 ready-to-use and configurable policies Code - Built-in support for Node, JavaScript, Java and Python extensibility
Apigee Edge policies – Build APIs faster
Manage interactions with API consumers and optimize performance
Secure APIs and protect back-end systems from attack
Transform, translate and reformat data for easy consumption Extend with
programming when you need it
©2015 Apigee. All Rights Reserved.
Public Cloud = Private Cloud
Flexible deployment options
Apigee Cloud
Multi-Datacenter Deployment
1+ Billion "calls / month
7 regions
>99.9%"API availability
19 availability zones
3,000+"servers
~1/2 billion"transactions / day
~20 ms"average latency
Private Cloud"leading Telco
©2015 Apigee. All Rights Reserved.
Agenda 09:30 – 10:00 Apigee Overview
10:00 – 11:00 Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00 Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45 Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30 Platform Operations
©2015 Apigee. All Rights Reserved.
Apigee Edge is comprised of several stateless components that use infrastructure services to persist data • Gateway: Routing and processing API calls • Apigee UIs: Enterprise UI, Developer Portal • Infrastructure Services: Persistence and queuing of run time data • Management Server: Provider of REST APIs for all configuration tasks Note: Monetization is part of Developer Services and leverages Gateway, Analytics Services and Management Server
Apigee Edge architecture – High level
ANALYTICS SERVICES
API SERVICES
BaaS"Backend
DEVELOPER SERVICES
Analytics Backend
Management Server
Gateway
Apigee "UIs
Infrastructure Services
Developer Portal
Client Backend
Apigee Open Source
Legend:
©2015 Apigee. All Rights Reserved.
Apigee Edge architecture – High level
Load Balancer
Infrastructure Services BaaS
Backend
Developer Portal
Analytics Backend
Management Server
Gateway
Infrastructure Services
Apigee "UIs
Gateway
Infrastructure Services
Infrastructure Services BaaS
Backend
Developer Portal
Analytics Backend
Management Server
Gateway
Infrastructure Services
Apigee "UIs
Gateway
Infrastructure Services
DATA
REP
LICATIO
N
DC-1
DC-2
Load Balancer
Client
Backends
Apigee Open Source
Legend:
©2015 Apigee. All Rights Reserved.
Apigee Edge architecture – Component view
Apigee Open Source
Legend:
• Router handles all incoming API traffic and dispatches it. The Router terminates the HTTP request, handles the SSL traffic, and uses the virtual host name, port, and URI to steer requests to the appropriate node.
• Message Processor handles API traffic for a specific organization and environment and which executes all policies.
• Apache Cassandra stores application configurations, distributed quota counters, API keys, and OAuth tokens for applications running on the gateway.
• Apache ZooKeeper contains configuration data about all the services of the zone and which notifies the different servers of configuration changes.
• OpenLDAP contains organization user and roles. • Management Server offers an API that is used by the Central
Services server to communicate with the servers in each on-premises installation.
• QPID Server manages queuing system for analytics data. • Postgres Server manages analytics database.
API SERVICES
DEVELOPER SERVICES
Router
Enterprise "UI
Qpid / Ingest Server
postgreSQL
Postgres Server
Developer Portal mySQL
BaaS Stack BaaS UI
Management Server
cassandra openldap zookeeper qpidd
BAAS
ANALYTICS SERVICES
Message Processor
Client Backend
©2015 Apigee. All Rights Reserved.
Apigee Edge architecture – Technology Stack
Management Zookeeper Consistency
Availability
Partition Tolerance
Our customers value "the ability to have highly available API expanded across geographically
dispersed sites.
Our customers value accurate metrics to "
help drive data driven business decisions.
Our customers value the ability to centralized management of distributed components.
Apigee Edge CAP
Theorem
In theoretical computer science, the CAP theorem, also known as Brewer's theorem, states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:
• Consistency - all nodes see the same data at the same time. • Availability - a guarantee that every request receives a response about whether it succeeded or failed. • Partition tolerance - the system continues to operate despite arbitrary message loss or failure of part of the system.
Text Credits: Wikipedia ©2015 Apigee. All Rights Reserved.
Apache Cassandra
15 ©2015 Apigee. All Rights Reserved.
DC 1
CS CS CS
DC 2
CS CS CS
“Apache Cassandra is an open source distributed database management system. It is an Apache Software Foundation top-level project designed to handle very large amounts of data spread out across many commodity servers while providing a highly available service with no single point of failure.” -- Wikipedia
http://cassandra.apache.org/
Cassandra characteristics: • All nodes are equal. Not master/slave or primary/secondary. • An application can read/write data from any node. • Data replication. Apigee Edge uses replication factor 3. • Consistency managed by application. Apigee Edge uses one
and local quorum. Cassandra is used by Apigee for a variety of purposes, including: • Storage of developer, application and API Product data • Storage of access and refresh tokens • Storage of key-value map data • Audit logs • Custom analytics report models
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
Apache Zookeeper
16 ©2015 Apigee. All Rights Reserved.
DC 1
ZKV ZKV ZKV
DC 2
ZKV ZKV ZKO
“Apache ZooKeeper is a software project of the Apache Software Foundation, providing an open source distributed configuration service, synchronization service, and naming registry for large distributed systems.” "
– Wikipedia Zookeeper is used by Apigee as a distributed configuration registry, tracking component location, configuration and status data. With some exceptions, it is NOT required to process API requests.
• Leader: The node that controls coordination of writes across distributed Zookeeper nodes
• Voters: Nodes that can vote on change proposal made by the Leader
• Observers: Do not vote on change proposals and must forward all writes to the Leader
http://zookeeper.apache.org/
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
Apache Qpid
17 ©2015 Apigee. All Rights Reserved.
DC 1
QD QD
DC 2
QD QD
“Apache Qpid, an open-source (Apache 2.0 licensed) messaging system, implements the Advanced Message Queuing Protocol. It provides transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support.” "
– Wikipedia Qpid is used by Apigee Edge as messaging system for analytics and monetization data.
http://qpid.apache.org/
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
• Apigee Edge architecture offers great flexibility when it comes to deployment options. • Edge can be deployed from a single VM to a multi-datacenter active/active configuration. • Edge has been designed from the ground up to be a true cloud scale solution, capable of running on
both virtualized (including AWS) and physical hardware.
Apigee Edge architecture – Deployment options
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
Node 1
MS
UI
R
MP
ZK
CS
AD
QIS
PS
QD
PG
Demo Non-Production Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKO
Production Node 1 Node 2 Node 3 Node 4 Node 5
MS
UI R R
QIS MP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS
©2015 Apigee. All Rights Reserved.
1. Know your business
Topology selection – Key considerations
Business SLAs Infrastructure
2. Understand traffic patterns 3. Plan for growth (2X, 5X, 10X?)
4. Know your API proxies 5. Remember everything fails 6. API Traffic, Analytics and Management, Developer Service components can and should scale independently. R R
MP MP
CS CS CS PS QIS QD PG
MS UI OL
Analytics
Management
DP MY Developer
R MP API Traffic CS
ZK
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
• Horizontally scalable. Components can be installed and configured to run from a single node "(all in one setup) all the way to a multi DC centers, active/active, globally distributed setup.
• Ability to stack components. • Distributed data replicated using eventual consistency. • Asynchronous analytics data capture and processing. • Centralized configuration for distributed components. • Multitenant by design. • Management via APIs and UI console.
Apigee Edge architecture – Architecture characteristics
DC2 DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
ELB
OL
ELB Load balancer
Node 6
R
MP
Node 11 Node 12 Node 13 Node 14 Node 15 Node 17
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO ZKV
Node 18 Node 19 Node 20
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
• Planet footprint is driven by customer requirements, transaction volumes, availability and reliability among others drive component stacking and number of nodes.
• R, MP and CS are critical components to handle live API traffic. • Management and analytics components are not required on every DC. • For high availability, ZK and QD should be available in all DCs. • Analytics data replication via Master/Slave or Publish/Subscribe.
Apigee Edge architecture – Horizontal scalability
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
DC2 DC2
DC2 DC1 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 10 Node 11 Node 12 Node 13 Node 14 Node 15 Node 16 Node 17 Node 18
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
ZKV
QD QD
PGS
MS
UI
OL
DC<N> Node 11 Node 12 Node 13 Node 14 Node 15 Node 16 Node 17
R R
QIS MP MP QIS
ZKV
CS
ZKV
CS CS
ZKV
QD QD
ZKV
©2015 Apigee. All Rights Reserved.
• Given the responsibility and capabilities offered by each component, scalability requirements and how they are implemented may vary.
• In most scenarios, scaling to accommodate higher number of TPS or API calls may impact only components serving live API traffic.
• Analytics and management components may grow in number mostly driven by demanding high availability requirements for those capabilities provided by analytics and management components.
Apigee Edge architecture – Scaling by capability
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
QIS PS QIS
QD QD PGM
R R
MP MP
CS CS CS
MS UI OL
ZK ZK ZK
API Traffic
Analytics
Management
DP MY Developer
©2015 Apigee. All Rights Reserved.
Agenda 09:30 – 10:00 Apigee Overview
10:00 – 11:00 Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00 Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45 Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30 Platform Operations
©2015 Apigee. All Rights Reserved.
Apigee Edge architecture – Multitenant by design
DC2 DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
ELB
OL
ELB Load balancer
Node 6
R
MP
Node 11 Node 12 Node 13 Node 14 Node 15 Node 17
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO ZKV
Node 18 Node 19 Node 20
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
Multitenancy “Multitenancy is a reference to the mode of operation of software where multiple independent instances of one or multiple applications operate in a shared environment. The instances (tenants) are logically isolated, but physically integrated. The degree of logical isolation must be complete, but the degree of physical integration will vary. The more physical integration, the harder it is to preserve the logical isolation. The tenants (application instances) can be representations of organizations that obtained access to the multitenant application (this is the scenario of an ISV offering services of an application to multiple customer organizations). The tenants may also be multiple applications competing for shared underlying resources (this is the scenario of a private or public cloud where multiple applications are offered in a common cloud environment).” - Gartner
Multitenancy = Software capabilities to support shared infrastructure "while providing tenants with data and processing partitioning.
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Multitenancy
Region
Pod
Org
Virtual Host
Environment
Regional Services are shared by many pods across a single geographical region and maintain state and provide an API that works across the pods in a region
Pods are a collection of servers that share logical functions such as a Gateway Pod or an Analytics Pod.
Orgs provide logical grouping to secure access to API management services. Orgs are associated with pods for servers – they can either share hardware or be isolated
Environments are virtual routes that allow API bundles to be deployed and tested within and Org. Environments can be associated with pods for servers independently of the Org.
Virtual Hosts are similar to Apache Virtual hosts and route traffic to environments based on ports and domain names.
Planet Planets represent an entire physical environment and can encompass multiple regions and pods
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Multitenancy
Data Center 1 Data Center 2 Data Center N
Planet
Org 1
Environment 1
Environment N
Org N
Environment 1
Environment N
• A Planet can expand multiple DCs. Organization and Environment can expand across the planet. • Data partitioning by Organization and Environment. Processing partitioning by Org+Env can be configured.
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Organizational structure Organiza3on
Environment Cache Resource
KVM Resource
Target Server
User
Role
Permission
Permission
Extension Resource
Key / Trust Store
Vault
API Product
Company
Developer
Applica3on
Key / Token
KVM Resource
Virtual Host
API Proxy
Extension Resource
Vault
Deployment
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Physical partitioning
Tenant A
Tenant B
Tenant C
Tenant A
Tenant B
Tenant C
Tenant D
Tenant E
Tenant D
Tenant E
Tenant A
Tenant B
Tenant C
Tenant D
Tenant E
Inbound Routing API Call Processing Data Storage
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Organizational structure
DC2 DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
Node 11 Node 12 Node 13 Node 14 Node 15 Node 17
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO ZKV
Node 18 Node 19 Node 20
Organization Environment Central POD
Gateway POD
Analytics POD
Planet
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – API traffic data flow
32 ©2015 Apigee. All Rights Reserved.
Legend:
• Routers send requests to Message Processors in their Gateway pod.
• If there are two or more gateway pods in a region, then routers will ignore message processors in the other gateway pods.
• Message Processors respect the region as their scope.
• For two data center the same rules apply as for one datacenter.
• All Apigee components are configured to only use the Cassandra nodes in their region / data center.
Region / DC 1
R
MP
R
MP
Load balancer
Client
CS CS CS
Backend
Gat
eway
Pod
1
R
MP
R
MP Gat
eway
Pod
2
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend: API call flow Analytics flow
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – API traffic data flow
33 ©2015 Apigee. All Rights Reserved.
R+MP Configuration options
R
MP
R
MP
Load balancer
Tenant Aware Routing
• Rs direct traffic to appropriate MPs, load balancing between them.
• Default behavior. • Health check heartbeat allows R to automatically take
MP out of/into rotation if unresponsiveness is detected.
R
MP
R
MP
Load balancer
Server Affinity
• R can also be configured to connect to a dedicated MP. Using server affinity, all traffic handled by a R is exclusively sent to its corresponding MP.
• This configuration option offers customer the ability to isolate R/MP for dedicated use cases without impacting API traffic flowing across other R/MP within the same pod.
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend: API call flow Analytics flow
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Analytics data flow
34 ©2015 Apigee. All Rights Reserved.
• Ingest services in Qpid Ingest Server will collect Analytics data from all queues and store in PostgreSQL.
• Postgres Server aggregates analytics data asynchronously.
• Message Processors respect the region as their scope and will offload analytics data to all Apache Qpid queues in their region / data center.
R
MP
R
MP
Load balancer G
atew
ay P
od
QD QD
Cent
ral P
od
PGM
QIS QIS
PS
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend: API call flow Analytics flow
©2015 Apigee. All Rights Reserved.
DC 2 DC 2 DC 1 DC 1
Apigee Edge Architecture – Analytics data flow
35 ©2015 Apigee. All Rights Reserved.
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend: API call flow Analytics flow
QIS
PS PS
QD
PGM PGS
MP Writes
Writes
Reads
Writes
QIS QD
MP Writes
Writes
Reads
QIS
PS
QD
PGM
MP Writes
Reads
Writes
QD
MP Writes
QIS
PS
PGM
Writes
Reads
Reads Reads
Reads/Writes
Reads/Writes Reads/Writes
• Analytics data is partitioned per Environment. • Analytics data size generated by MPs per transaction is about 1kb. • PostgreSQL contains Raw Data Tables and Aggregated Data Tables. Raw Data Tables grow as
analytics data is collected. Appropriate data retention policy and purge processes are required. • Custom reports allow customers to define user-defined queries which run against raw data.
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Network Zoning
36 ©2015 Apigee. All Rights Reserved.
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend: API call flow Analytics flow
Data
Data
DMZ
App
DMZ Core
R R
QIS
MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
R
MP
ZKV
DP
MY
QIS
MP MP
QIS
QD QD
MS
UI
OL
MP
R R R
PS
ZKV
CS
ZKV
CS CS
PGM
ZKV
DP
MY
QIS
MP MP
QIS
QD QD MS
UI
OL
MP
R R R
PS
ZKV
CS
ZKV
CS CS
PGM
ZKV
DP
MY
Single Zone Two Zones Three Zones • Edge architecture does not impose
network zone requirements. Network zoning will be driven by customer operation and security requirements.
• Firewalls and security appliances between zones should accommodate the connectivity "and traffic characteristics of Apigee components without adding "latency overhead.
• Keep API traffic as direct as possible. Keep MP dependencies close.
• Pay attention to cross-zone connections.
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Components Connectivity
37 ©2015 Apigee. All Rights Reserved.
QD QIS
CS ZK
CS ZK
CS ZK
AD / OL
PG-standby PS
Target Backend
PG: 5432
QD: 5672
customer specific
10389
ZK: 2181 CS: 9160, 7199
PG: 5432
PG-master PS
ZK: 2181 CS: 9160, 7199
UI MS
QIS: 4529
PS: 4530 PG: 5432
PS
PS
MP
MS MS
MS
MS
QIS
MP
QIS
MS
ZK: 2181 2888, 3888
CS: 7000 7199, 9160
PG: 5432* PG
PG
QIS: 8083 1102
PS: 8084 1103
8081 1100
MS: 8080
UI: 9000
4528
M
M
M
MP 8082 1101
M
8998, 4528 R
customer specific
MP
MS
4527
MY DP 3306
MS: 8080
DP
DP
client / load balancer
R
R Router
MP Message Processor
CS Cassandra
MS Management Server
OL OpenLDAP
ZK Zookeeper
QD Apache Qpid
QS Qpid Server
PG PostgreSQL
PS Postgres Server
DP Developer Portal
MY MySQL
AP
I tra
ffic
Con
figur
atio
n A
naly
tics
Dev
UI Management UI
Legend: from
svc:port
1-way
2-way
to
management calls HTTP or JMX)
M
Notes: • For CS all nodes
talk to all other CS nodes
• For ZK all nodes talk to all other ZK nodes
• * both directions to enable PostgreSQL failover
QD: 5672
R
Single DC View
©2015 Apigee. All Rights Reserved.
Apigee Edge Architecture – Components Connectivity
38 ©2015 Apigee. All Rights Reserved.
R Router
MP Message Processor
CS Cassandra
MS Management Server
OL OpenLDAP
ZK Zookeeper
QD Apache Qpid
QS Qpid Server
PG PostgreSQL
PS Postgres Server
DP Developer Portal
MY MySQL
AP
I tra
ffic
Con
figur
atio
n A
naly
tics
Dev
UI Management UI
Legend: from
svc:port
1-way
2-way
to
management calls HTTP or JMX)
M
Notes: • For CS all nodes
talk to all other CS nodes
• For ZK all nodes talk to all other ZK nodes
• * both directions to enable PostgreSQL failover
Multi DC View
^
Data Center 1 Data Center 2
MP
Target Backend
R
QD QIS
CS ZK
CS ZK
CS ZK
OL
PG-master PS
UI MS
CS ZK
CS ZK
CS ZK
OL
QD QIS
PG-standby PS
R
MP
UI MS
PG: 5432
10389
PG: 5432
4527 1100
4528 1101
4527 1100
4528 1101
ZK: 2181, 2888, 3888 CS: 7000, 7199, 9160
PG
QIS
MS MS
^ Target Backend
©2015 Apigee. All Rights Reserved.
Agenda 09:30 – 10:00 Apigee Overview
10:00 – 11:00 Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00 Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45 Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30 Platform Operations
©2015 Apigee. All Rights Reserved.
System Requirements – Hardware Specification
• Cassandra – Local storage with SSD or fast HDD supporting 2000 IOPS
• PostgreSQL – Local storage with SSD or fast HDD supporting 1000 – 8000 IOPS – CPU, Memory and Size the storage according to the data retention
requirements. Implement archive / purge process and keep less data for best performance.
– Operations Guide provides formula to estimate storage needs.
Type C
ZKV
CS
Type G
QIS
QD
Type H
PS
PGM
Type A
MS
UI
Type B
OL
Type Components CPU Memory Disk A UI, MS 2 core 4 GB 60 GB B OL 2 core 4 GB 60 GB C ZK, CS 8 core 16 GB 250 GB local storage with SSD or fast HDD supporOng 2000 IOPS D R, MP 4/8 core 8/16 GB 100 GB E R 4 core 8 GB 60 GB F MP 8 core 16 GB 100 GB G QIS, QD 8 core 16 GB 500 GB local storage with SSD or fast HDD supporOng 1000 IOPS H PS, PG 8 core 16 GB 500 GB to1 TB local storage with SSD or fast HDD supporOng 4000-‐8000
IOPS
Type F
MP
Type D
R
MP
Type E
R
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
©2015 Apigee. All Rights Reserved.
System Requirements – Software Dependencies
https://apigee.com/docs/api-services/reference/supported-software
Operating System • Red Hat Enterprise Linux (64-bit):
o 6.3, 6.4, 6.5, 6.6, 7.0 • CentOS (64-bit):
o 6.3, 6.4, 6.5, 6.6, 7.0 • Oracle Linux (64-bit):
o 6.5 JDK • Oracle JDK 1.7 • OpenJDK 7
SSL/TLS • 1.0 • 1.2 Other software • Cassandra 2.0.15 • Zookeeper 3.4.5 • QPID 0.14 • PostgreSQL 9.3 • Play (UI) 2.3.4 • OpenLDAP 2.4
©2015 Apigee. All Rights Reserved.
Private Cloud Installer – Software Dependencies • awk • basename • bash • chkconfig • curl • date • dirname • echo • expr • grep
• hostname • id • ls • perl • pgrep (from procps) • ps • pwd • python • rpm • rpm2cpio
• sed • sudo • tar • tr • uname • unzip • useradd • wc • yum
Complete list of prerequisites can be found in Apigee Edge Install and Configuration Guide, Page 19. In addition to the tools above, some nodes require the installation of additional software components such as: • ntp (all nodes) • rsync (PostgreSQL nodes) • openldap-clients openldap-servers (Management Server node)
©2015 Apigee. All Rights Reserved.
Apigee Edge Installation Process
DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
1. Install data store hosts 2. Install management hosts
• Install LDAP first if using standalone LDAP hosts
3. Install router, message processor 4. Qpid hosts 5. PostgreSQL hosts 6. Configure PostgreSQL replication, if needed 7. Create organization(s) and environment(s) If applicable, install: 7. Developer portal 8. Install monetization 9. App services (BaaS)
1 2 3 4 5
©2015 Apigee. All Rights Reserved.
Private Cloud – Software Install Apigee Private Cloud (apigee-edge-4.15.07.00.zip) has everything required to install and configure Apigee Edge and BaaS components except for the Developer Portal. The Developer Portal is distributed using a different package (DeveloperServices-4.15.07.00.tar).
Apigee Edge installation steps: 1. Acquire a license key and copy it to all nodes. 2. Download the Apigee Private Cloud from ftp.apigee.com and copy it to all nodes. 3. Installation downloads and installs required system software via Yum. It requires Internet connection or local
repository. 4. Unzip apigee-edge-4.<YY>.<MM>.<V>.zip 5. Run the primary installation script:
/<unzip-‐location>/apigee-‐edge-‐4.<YY>.<MM>.<V>/apigee-‐install.sh -‐j /usr/java/default -‐r <inst-‐root> -‐d <data-‐root>
©2015 Apigee. All Rights Reserved.
Private Cloud – Software Setup 1. Once Apigee binaries are installed, run:
/<inst-‐root>/apigee4/share/installer/apigee-‐setup.sh
2. Setup script requires a profile type and information about the system, provided interactively or via response file (silent install).
3. Create an organization, environment(s) and organization administrator, run:
/<inst-‐root>/apigee4/bin/setup-‐org.sh
©2015 Apigee. All Rights Reserved.
Private Cloud – Silent Install • Silent install provides a way for specifying, in advance, all necessary values used by apigee-setup.sh
• Silent install provides a way for specifying, in advance, all necessary values used by apigee-setup.sh
/<inst-‐root>/apigee4/share/installer/apigee-‐setup.sh -‐p ds -‐f <response-‐file-‐name>
• The response files contain a number of variables definitions to be used by apigee-setup.sh. A response file per DC/Region will be needed since some values are unique per DC/Region.
Single machine setups: • aio = All In One (Gateway and Analytics Standalone)
Cluster node setup for ZooKeeper and Cassandra (min 3 nodes): • ds = Datastore Cluster Node LDAP setup for OpenLDAP: • ld = LDAP Node Separate components setup: • ld = LDAP Node • ms = Gateway Management Server • r = Gateway Router • mp = Gateway Message Processor • rmp = Gateway Router and Message Processor • qs = Analytics Qpid Server • ps = Analytics Postgres Server • mo = Monetization Server
Note: The list above doesn’t show all the available options. Refers to Apigee Edge Install and Configuration Guide, section Basic Host Installation, page 35 for details.
©2015 Apigee. All Rights Reserved.
Private Cloud – Silent Response File • IP1= • IP2= • IP3= • IP4= • IP5=
• HOSTIP=$(hostname -i) • MSIP=$IP1 • ADMIN_EMAIL= • APIGEE_ADMINPW= • LICENSE_FILE=/root/opdk/license.txt • USE_LDAP_REMOTE_HOST=n • LDAP_TYPE=1 • APIGEE_LDAPPW= • ENABLE_AX=y • MP_POD=gateway • REGION=dc-1
• USE_ZK_CLUSTER=y • ZK_HOSTS="$IP1 $IP2 $IP3" • ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3" • USE_CASS_CLUSTER=y • CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" • CASS_USERNAME= • CASS_PASSWORD= • SKIP_SMTP=y • SMTPHOST= • SMTPPORT=25 • SMTPUSER= • SMTPPASSWORD= • SMTPSSL=n • BIND_ON_ALL_INTERFACES=y
©2015 Apigee. All Rights Reserved.
Apigee Edge Install Lab Scope • 5 nodes Apigee Edge Install • VMs instances running in Amazon AWS • Installation and setup using silent install process. • Setup order DS, MS, RMP, SAX
AWS Instances IP • http://iloveapis.com/workshops/ops/ Documentation • ApigeeEdgePrivateCloud-Install-Config-Guide.pdf
o Page 35 - Basic Host Installation o Page 47 – Onboarding o Page 120 - Appendix A: Silent Installation o Page 121 - 5-host Clustered Installation
Node 1 Node 2 Node 3 Node 4 Node 5
MS
UI R R
QIS MP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS
2: MS
1: DS
3: RMP 4: SAX
©2015 Apigee. All Rights Reserved.
Apigee Edge Install Lab - Prerequisites Preparation
A. Software, sample install file and JDK are located in /root/opdk/
B. Install prerequisites o In all nodes:
§ Install JDK (rmp -ivh <rmp>) § yum -‐y install ntp
o In node 1: § yum -‐y install openldap-‐clients openldap-‐servers
o In node 4 and 5: § yum -‐y install rsync
C. In all nodes, unzip apigee-‐edge-‐4.15.07.00.zip
D. Update silent install file with private IP addresses corresponding to your AWS instances. Copy silent install files to all nodes.
Additional considerations: • For the purpose of the lab, we took care of some prerequisites. When planning for real world installation, prerequisites described in Apigee
Edge Install and Configuration Guide, Page 19-35 should be completed.
©2015 Apigee. All Rights Reserved.
Apigee Edge Install Lab – Installation and Setup Install Steps A. Unzip apigee-edge-4.15.07.00.zip
B. ./apigee-‐install.sh -‐j /usr/java/default -‐r /opt -‐d /opt Setup Steps 1. /opt/apigee4/share/installer/apigee-‐setup.sh -‐p ds -‐f /root/opdk/response.txt
2. /opt/apigee4/share/installer/apigee-‐setup.sh -‐p ms -‐f /root/opdk/response.txt
3. /opt/apigee4/share/installer/apigee-‐setup.sh -‐p rmp -‐f /root/opdk/response.txt
4. /opt/apigee4/share/installer/apigee-‐setup.sh -‐p sax -‐f /root/opdk/response.txt
Node 1 Node 2 Node 3 Node 4 Node 5
MS
UI R R
QIS MP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS
2: MS
1: DS
3: RMP 4: SAX
©2015 Apigee. All Rights Reserved.
Apigee Edge Install Lab – Installation and Setup Setup Master-Standby Replication for PostgreSQL
• Apigee Edge Install and Configuration Guide, Page 59 Onboarding
• Apigee Edge Install and Configuration Guide, Page 47 • /opt/apigee4/bin/setup-org.sh
Access Apigee Edge
• http://<management-server-public-ip>:9000
©2015 Apigee. All Rights Reserved.
Agenda 09:30 – 10:00 Apigee Overview
10:00 – 11:00 Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00 Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45 Apigee Edge Install Lab
BREAK (15 MINS)
16:00 – 17:30 Platform Operations
©2015 Apigee. All Rights Reserved.
Platform Operations - Walkthrough • Directory Structure
o /<inst-‐root>/apigee4
• Start / Stop / Status o /<inst-‐root>/apigee4/bin/all-‐start.sh o /<inst-‐root>/apigee4/bin/all-‐status.sh o /<inst-‐root>/apigee4/bin/all-‐stop.sh
• Logs file o /<inst-‐root>/apigee4/var/log
• Management UI o http://<management-‐server-‐host>:9000
• Management API o http://apigee.com/docs/management/apis o http://<management-‐server-‐host>:8080
©2015 Apigee. All Rights Reserved.
Platform Operations – Backup and Restore • Backup/restore
o Edge includes backup scripts for: § LDAP § Cassandra § ZooKeeper § PostgreSQL § UI (custom reports) § Component UUIDs
o <inst-root>/apigee4/bin/backup.sh • High Availability and Disaster Recovery
o Multiple copies of the data o Eventual consistency o Horizontal scalability and resiliency
DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV
©2015 Apigee. All Rights Reserved.
Platform Operations – Recurring Maintenance • Cassandra nodes repair
/opt/apigee4/share/apache-‐cassandra/bin/nodetool repair -‐pr
• Periodically prune detailed analytics data based on retention requirements
/opt/apigee4/bin/pg-‐data-‐purge.sh
©2015 Apigee. All Rights Reserved.
Platform Operations – Component Level Monitoring
JVM Statistics – Threads,
heap, GC. Platform APIs.
System Statistics – CPU, Memory, Disk, Network.
DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 7 Node 8 Node 9 Node 10
R R
QIS MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV
Monitoring Tool
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
• System-Level Checks – CPU, Memory, Disk, Load, Network
• Process/Application Checks – Thread statistics, Memory utilization
• Components monitoring – JMX – Metrics – Management API (http://<host>:<port>/v1/
servers/self/up) – Router health check of the Message
Processer – Log monitoring
• HeartBeat • Mark down/mark up events • Logging Policy
©2015 Apigee. All Rights Reserved.
• API-Level Checks – API calls 20X responses – API call error within Apigee – API call error on backend – JMX – Metrics
• Message Flow Checks – Number of active clients – Number of responses (10X, 20X, 30X, 40X and 50X) – Connect failures
• API Health - https://pages.apigee.com/api-health
Platform Operations – APIs Monitoring
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal BaaS Server
Router Message Processor Enterprise UI
Management Server Postgres Server Qpid/Ingest Server
Openldap PostgreSQL Apache Qpid
MySQL Zookeeper Cassandra
Server/Virtual Machine POD
Legend:
LB
Backend System(s)
R+MP
R
MP
R
MP
Monitoring Tool
Monitoring Tool
Edge Analytics
©2015 Apigee. All Rights Reserved.