Private, Public, Hybrid Unique Challenges with OpenStack Gathering Info CMDB Ryszard Chojnacki, an OPS CMDB blueprint worker
Private, Public, Hybrid Unique Challenges with OpenStackGathering Info CMDB
Ryszard Chojnacki, an OPS CMDB blueprint worker27 October, 2015
Covering what?• Define a scenario for collecting data• Show actual payload headers used in one application• Set the stage for a CMDB blueprints direction:
ETL vs. Federation approaches
Extract, Transform & LoadSend data as appropriate
• Allows for– Complex and low cost
queries– Can be built to
accommodate loss– History, what changed last
week
FederationAccess the sources of information in “real time”
• Allows for– What is the situation NOW!– Works well for OpenStack
APIs depending on use-case
Set the scene
Imagine this ScenarioYou have hardware, data and applications spread over multiple locations – how can you aggregate meta data into 1 place
Local source:provisioning as example
{ “fqdn": “compute-0001.env1.adomain.com", “serial": “USE1234567", “os_vendor": “Ubuntu", “os_release": “12.04", “role": “compute-hypervisor",}
Suppose provision systems are created such that there is 1 for each environment
• Each system has a limited scope
• Each system must be uniquely identifiable to permit data aggregation
1
2
3
Global source:asset management
Payload type rack_info:{ “rack_id”: “r0099”, “datacenter”: “Frankfurt”, “tile”: “0404”}
Payload type rack_contents:{ “in_rack”: “r00099”, “serial”: “USE1234567”, “u”: 22, “pid”: “qy799a”}
Suppose that there is a single asset management tool that covers al environments
• Scope is global
• Unique ID still employed
• The example has more than one type of data for:
• Each rack – rack_info• Each asset – rack_contents
1
2
3
Snapshot Header
Message formatspayload
{"payload": { . . . }}
Separate logically, by encapsulating data into a payload document
For example, put here:• Provisioning data• Rack data• Asset data
Message formatsversion
{ "version": { "major": 1 // provider extension possible here; minor, tiny, sha1, … }, "payload": { . . . }}
“Schema” version for the payload
• Major versions same indicates no incompatible changes made to schema
• Where compatible snapshot live process will occur
Note: Documents don’t have schemas, but there must be some required, plus optional key/value pairs, so that consumers of the data can rely programmatically on it
Message formatswhen was that?
{ "batch_ts": <epoch_time>, "record_ts": <epoch_time>, “batch_isodate”: “2015-01-16 16:07:21.503680”, . . . "version": { "major": 1 }, "payload": { . . . }}
Useful for understanding how old is the data I’m seeing
• Batch timestamp must be constant for all records in the same batch
• Record is when the record was exported/message created – maybe the same as batch
• Updated, if available, is when the data was last changed in the source system
Note human readable _isodate forms, not used in processing
Message formats
{ "source": { "system": "rackreader.adomain.net" "type": "rack_info", "location": "env" }, "batch_ts": <epoch_time>, "record_ts": <epoch_time>, "import_ts": <epoch_time>, "version": { "major": 1 }, "payload": { . . . }}
Provides where the data came from and the type of data
• System: usually fqdn of source system
• Location: the scope of the system
• Type: describes the payload content, and is tied to the schema
Message formats
{ "record_id": “r00099-rackreader.adomain.net", "msg_type": "snapshot", "source": { "system": "rackreader.adomain.net" "type": "rack_info", "location": "env" }, "batch_ts": <epoch_time>, "record_ts": <epoch_time>, "import_ts": <epoch_time>, "version": { "major": 1 }, "payload": { . . . }}
Mark the content with a unique ID for that record, and how to process
• A combination of an identifier in the source system plus an fqdn makes for a very globally unique value
• This value is the primary key for all data operations on the record
• This is a “snapshot” how that is processed described shortly
Implementation
Philosophy employed• Operating at large scale expect to have issues
– Small % error X a big number = some degree of loss• Tolerant of loss• Considerate of resources• Wanted history
– Need easy access to the very latest• Need a flexible [document] schema – this is JSON
– Provider/Agent is the owner of the schema for its data– Need a way to converge; communities of practice
Snapshot versus event based updates
Example• Snapshot updates every 8h
– Larger data set but not very frequent
• Live updates as they occur– Tiny data as they occur
• Result– Minimal network utilization– Small overhead on source
• Use the combination that best suits the need
We run 2 collections
• Snapshot• Has history
• Live• Has only the latest
Snapshots update Live
Message type overview
Snapshot• Snapshot
– Defines a Snapshot record• Batch record count
– Defines how many items in a batch
– Only If sizes match update Live
• Required to know what to delete
Live• Overwrite
– Live overwrite a complete doc for a single record
• Delete– Delete from live a single
record– Never affects Snapshot
Message formatssnapshot_size
{ "msg_type": "snapshot_size", "source": { "system": "rackreader.adomain.net" "type": "rack_info", "location": "env" }, "size": 3, "batch_ts": <epoch_time>}
If the consumer receives the size of messages indicated then the update of live is possible
Any records received are always placed in the snapshot collection
Message formatsoverwrite
{ "msg_type": "overwrite", "record_id": “r00099-rackreader.adomain.net", "source": { "system": "rackreader.adomain.net" "type": "rack_contents", "location": "env" }, "version": { "major": 1 }, "record_ts": <epoch_time>, "payload": { . . . }}
• Separate the header info from the payload data
Message formatsdelete
{ "msg_type": “delete", "record_id": “r00099-rackreader.adomain.net", "source": { "system": "rackreader.adomain.net" "type": "rack_contents", "location": "env" }, "version": { "major": 1 }, "record_ts": <epoch_time>,}
• Separate the header info from the payload data
Direction
Noteworthy• If we lose an event we catch up in the batch update• If we lose a batch, data is just 1 batch cycle stale
• Several companies have arrived at this position
• Records are fairly small– rabbitMQ friendly– Easy to search in your data store
CMDB blueprint• Set the stage for a CMDB blueprints direction:
– Collect– Store– Query
• Focus on the Collection framework• Community of Practice
– Share common stuff; hopefully every expanding domain– Permit ad-hoc sources, for what you have now
Thank-you!
Message processing
A live update to “B” goes straight in here
And is later updated again by the snapshot
B
There are 2 collections of data; snapshot and live
Snapshot always keeps growing
Live only has 1 entry per record