Clojure From the Ground Up BuildingOur First Clojurebased Microservices
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Clojure From the Ground UpBuilding Our First Clojure-‐based Microservices
About the Speaker
John Carnell is a Senior Cloud Engineer at Interactive Intelligence in Raleigh, North Carolina. John is a prolific speaker and writer. He has spoken at such national conferences as the No Fluff Just Stuff Software Symposium, Internet Expo and the Data Warehousing Institute.
John has authored, co-‐authored and been a technical reviewer for a number of technology books and industry publications.
John can be reached at [email protected].
Disclaimer: The views and opinions expressed in this presentation are solely the opinion of the presenter and do not reflect or represent the position or opinion of Interactive Intelligence or any of its subsidiaries.
About Interactive Intelligence
• Interactive Intelligence is a leading provider of call center technologies and communication
• We offer a wide variety of software products and have two core product lines:• CIC – Traditional on-‐premise call center solution• PureCloud – Cloud-‐based telephony and communication solution
• We have been in business for approximately 20 years and have development and sales offices throughout the world, with the core software development being done in:• Indianapolis, Indiana -‐ CIC• Raleigh, North Carlolina – PureCloud
• Our Raleigh has been around for about 3 years and has approximately 100+ developers located there
About PureCloud• Cloud-‐based call center and communication software solution
• Runs primarily in AWS • EC2• Dynamo• SQS• S3
• Microservices based architecture• Every team is responsible for their own services from beginning to end and choose the technology stack they are going to work ands support• Approximately 100+ REST-‐based Microservices written in Java, Node, Go, Python (and now Clojure)• Heavy use of an event communication using Kafka
Why We Started Looking at Clojure
Sometimes Java is Just So Heavy…..
ChugChug
Chug
I wonder why my hair is going grey.
What we were trying to build: Edge Provisioning
Our Experiences Lets do a little walkthroughof our experiences.
This is a no-‐navel gazing zone.
Clojure and REST-‐Based Microservice Development
Core Framework for Microservices
Jetty Servlet Container
Compojure
Ring Middleware
Request Handler
• Strong Servlet Container• SSL Support/ High Performance
• Routing API • Swagger-‐based Interface Definition
• HTTP Request/Response Abstraction• Middleware Pipelines
• Pure Clojure Functions• Could be composed and tested independently of the Framework
Why did we choose Compojure and Ring framework?• I have worked in a lot of development frameworks.
• Frameworks can save time, but when you first start using them its Pure Frickin Magic (PFM)
• Then you often discover there is a price for magic……
Wanted to started with a light-‐weight abstraction layer that our team could reason about.
Defining an Endpoint using Ring and Compojure
Ring Handlers are Just Functions
Ring Middleware: All Requests Get Put Through a Pipe
Custom Middleware is Easy
Ring middleware has let to some very elegant solutions to problems. We solved an interesting problem in the Swagger UI with minimal effort with a custom middleware
Making and Parsing REST Calls
What Do You Do With This?{"organizationId" organization-‐id, "siteId" site-‐id, "asgId" asg-‐id, "instanceInfo" [ {"instanceId" (:instance-‐id instance1-‐info), "privateIp" (:instance-‐privateip instance1-‐info), "publicIp" (:instance-‐publicip instance1-‐info), "imageId" "ami-‐2ca6b546", "state" "running", "organizationId" organization-‐id, "edgeId" nil}, {"instanceId" (:instance-‐id instance2-‐info), "privateIp" (:instance-‐privateip instance2-‐info), "publicIp" (:instance-‐publicip instance2-‐info), "imageId" "ami-‐2ca6b546", "state" "running", "organizationId" organization-‐id, "edgeId" nil}]})(defn build-‐get-‐aws-‐asg-‐response "Builds a mock aws-‐asg-‐response based on the ASG" [asg-‐name organization-‐id site-‐id instance1-‐info instance2-‐info] {:auto-‐scaling-‐groups [{:tags [{:value organization-‐id, :key "cluster", :resource-‐type "auto-‐scaling-‐group", :resource-‐id asg-‐name, :propagate-‐at-‐launch true} {:value site-‐id, :key "layer", :resource-‐type "auto-‐scaling-‐group", :resource-‐id asg-‐name, :propagate-‐at-‐launch true} {:value organization-‐id, :key "organizationId", :resource-‐type "auto-‐scaling-‐group", :resource-‐id asg-‐name, :propagate-‐at-‐launch true} {:value site-‐id, :key "siteId", :resource-‐type "auto-‐scaling-‐group", :resource-‐id asg-‐name, :propagate-‐at-‐launch true} {:value "23", :key "version", :resource-‐type "auto-‐scaling-‐group", :resource-‐id asg-‐name, :propagate-‐at-‐launch true}], :health-‐check-‐type "EC2", :default-‐cooldown 600, :auto-‐scaling-‐group-‐arn"arn:aws:autoscaling:us-‐east-‐1:490606849374:autoScalingGroup:d9d42949-‐8227-‐4cb6-‐870a-‐884c081cf0f3:autoScalingGroupName/ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70-‐49b4-‐9b7c-‐fd9d9565cccc-‐ASG-‐v23", :availability-‐zones ["us-‐east-‐1a" "us-‐east-‐1d"], :launch-‐configuration-‐name "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70-‐49b4-‐9b7c-‐fd9d9565cccc-‐LC-‐v23", :health-‐check-‐grace-‐period 600, :max-‐size 2, :desired-‐capacity 2, :min-‐size 2, :instances [{:launch-‐configuration-‐name "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70-‐49b4-‐9b7c-‐fd9d9565cccc-‐LC-‐v23", :instance-‐id (:instance-‐id instance1-‐info), :health-‐status "Healthy", :availability-‐zone "us-‐east-‐1a", :lifecycle-‐state "Pending:Wait"} {:launch-‐configuration-‐name "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70-‐49b4-‐9b7c-‐fd9d9565cccc-‐LC-‐v23", :instance-‐id (:instance-‐id instance2-‐info), :health-‐status "Healthy", :availability-‐zone "us-‐east-‐1d", :lifecycle-‐state "Pending:Wait"}], :vpczone-‐identifier "subnet-‐7bfbc70f,subnet-‐80a9d0a8", :load-‐balancer-‐names [], :enabled-‐metrics [], :created-‐time "2016-‐06-‐10T08:29:29.288-‐04:00", :termination-‐policies ["OldestInstance"], :auto-‐scaling-‐group-‐name asg-‐name, :suspended-‐processes []}]})(defn get-‐ec2-‐instance-‐data "Mocks out EC2 instance data" [instance-‐info] {:reservations [{:instances [{:monitoring {:state "enabled"}, :tags [{:value "opsTestService-‐jim", :key "Name"} {:value "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70-‐49b4-‐9b7c-‐fd9d9565cccc-‐ASG-‐v23", :key "aws:autoscaling:groupName"} {:value "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70", :key "cluster"} {:value "23", :key "opsTestService-‐jim_version"} {:value "49b4-‐9b7c-‐fd9d9565cccc", :key "layer"} {:value "opsTestService-‐jim", :key "role"} {:value "49b4-‐9b7c-‐fd9d9565cccc", :key "siteId"} {:value "23", :key "version"} {:value "ad1ea37c-‐98c6-‐49b4-‐9b7c-‐fd9d9565fd70", :key "organizationId"} {:value "dev", :key "environment"}], :root-‐device-‐type "ebs", :private-‐dns-‐name "ip-‐172-‐18-‐12-‐25.ec2.internal", :hypervisor "xen", :subnet-‐id "subnet-‐80a9d0a8", :key-‐name "dev.ops", :architecture "x86_64", :security-‐groups [{:group-‐id "sg-‐69880d12", :group-‐name "edge-‐asg-‐template-‐InstanceSecurityGroup-‐1VX898CQDNTXY"}], :source-‐dest-‐check true, :root-‐device-‐name "/dev/xvda", :virtualization-‐type "hvm", :product-‐codes [], :instance-‐type "c3.large", :ami-‐launch-‐index 0, :image-‐id "ami-‐2ca6b546", :state {:name "running", :code 16}, :state-‐transition-‐reason "", :network-‐interfaces [{:description "", :private-‐dns-‐name "ip-‐172-‐18-‐12-‐25.ec2.internal", :subnet-‐id "subnet-‐80a9d0a8", :source-‐dest-‐check true, :private-‐ip-‐addresses [{:association {:public-‐dns-‐name "ec2-‐174-‐129-‐137-‐192.compute-‐1.amazonaws.com", :ip-‐owner-‐id "amazon", :public-‐ip (:instance-‐publicip instance-‐info)}, :private-‐ip-‐address (:instance-‐privateip instance-‐info), :private-‐dns-‐name "ip-‐172-‐18-‐12-‐25.ec2.internal", :primary true}], :network-‐interface-‐id "eni-‐8e0b90a8", :vpc-‐id "vpc-‐ba514bd8", :mac-‐address "12:cc:e7:16:7c:1f", :association {:public-‐dns-‐name "ec2-‐174-‐129-‐137-‐192.compute-‐1.amazonaws.com", :ip-‐owner-‐id "amazon", :public-‐ip (:instance-‐publicip instance-‐info)}, :status "in-‐use", :private-‐ip-‐address (:instance-‐privateip instance-‐info), :owner-‐id "490606849374", :groups [{:group-‐id "sg-‐69880d12", :group-‐name "edge-‐asg-‐template-‐InstanceSecurityGroup-‐1VX898CQDNTXY"}], :attachment {:status "attached", :device-‐index 0, :attach-‐time "2016-‐06-‐10T09:33:23.000-‐04:00", :delete-‐on-‐termination true, :attachment-‐id "eni-‐attach-‐781de2a2"}}], :vpc-‐id "vpc-‐ba514bd8", :ebs-‐optimized false, :instance-‐id (:instance-‐id instance-‐info), :iam-‐instance-‐profile {:id "AIPAJLCFTO5UAQFED3FXS", :arn "arn:aws:iam::490606849374:instance-‐profile/edge-‐asg-‐template-‐InstanceProfile-‐12CV1P3Z0GEKX"}, :public-‐dns-‐name "ec2-‐174-‐129-‐137-‐192.compute-‐1.amazonaws.com", :private-‐ip-‐address (:instance-‐privateip instance-‐info), :placement {:group-‐name "", :tenancy "default", :availability-‐zone "us-‐east-‐1d"}, :client-‐token "cc425936-‐57ca-‐4963-‐ab72-‐bfbceb93ac0c_subnet-‐80a9d0a8_1", :public-‐ip-‐address (:instance-‐publicip instance-‐info), :launch-‐time "2016-‐06-‐10T09:33:23.000-‐04:00", :block-‐device-‐mappings [{:device-‐name "/dev/xvda", :ebs {:status "attached", :attach-‐time "2016-‐06-‐10T09:33:24.000-‐04:00", :delete-‐on-‐termination true, :volume-‐id "vol-‐c8db8318"}} {:device-‐name "/dev/sdo", :ebs {:status "attached", :attach-‐time "2016-‐06-‐10T09:33:24.000-‐04:00", :delete-‐on-‐termination true, :volume-‐id "vol-‐7bdb83ab"}} {:device-‐name "/dev/sdp", :ebs{:status "attached", :attach-‐time "2016-‐06-‐10T09:33:24.000-‐04:00", :delete-‐on-‐termination true, :volume-‐id "vol-‐7cdb83ac"}}]}], :group-‐names [], :groups [], :owner-‐id "490606849374", :reservation-‐id "r-‐973dc335", :requester-‐id "226008221399"}]}
(ec2/describe-‐instances :instance-‐ids [instance-‐id])
You need something as cool as Bond
Specter
Specter
Clojure: Infrastructure and Middleware
I’m a services developer, I have people skills!
Introducing Amazonica• Amazon’s Java Client uses repeatable idioms
throughout their API
• Amazonica uses reflection and the API to build a simple DSL for invoking Amazon APIs.
• The documentation on their website is complete, but once you understand how they apply reflection its pretty easy to work your way back.
Amazonica: Simplify Amazon API Calls
Amazonica: Starting an Instance
Kafka: High Speed Event BusOur architecture communicates state changes in our services through events
We rely heavily on Kafka as our communication bus.
Built a small framework to register functions to handle messages.
91 lines of code and used our Broker Consumer
We pretty much reused the Java clients
Actually reused the HTTP handler code I had written for the delete endpoint.
I could do this because everything maps to a few basic types
Testing
Clojure: The Language
Tell me again to write thisservice in Java!!!
Concise Code In Action
Conciseness by the Numbers• 23 production level files
• 5 of those files are swagger definitions• Does not include the testing code
• Largest file is 213 lines of code
• This file has a large amount of text in it describing the endpoints
• The next largest file is 168 lines of code
• This file is has a bunch of individual property and map definitions in it.
Functional Example 1: Building a Pairing Token
Functional Example 1: Testing Pairing Tokens
The Real Power of Clojure: Simplified Data Structures
• Simple=General data structures not specific types.
• Clojure focuses on a using a few basic types with lots of powerful functions.
• Java and other static languages model the world through specific types that decompose the world into self-‐encapsulated data structures.
• This type specificity adds a huge amount of complexity to how we write our code.
Whoa!
{“organizationId” : “”,“id”:””,“label”: “”,“version”: “”,“dateCreated”: “”,“dateModified”:””,“modifiedBy”:””,“modifiedByApp:””,“createdBy”:””,”state”:””,“didPoolId”:””,“text”:””,”ownerType”:””,“ownerId”:””}
{:organizationId : “”:id:””:label: “”:version: “”
……}
• 601 lines of generated code• 1 mapping file• Have to write custom functions to
search and manipulate the data
• 1 function (read-‐str) to read in the data• Returns a 14 element hashmap• Every Clojure Hashmap function will work
with it.• Is the type safety Java provides worth the
complexity?
”It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.”-‐ Alan Perliss
Strong JVM Interoperability
Dude, What We Learned
Lessons Learned• Don’t get frustrated, you are going to have to change the way you think about writing code.
• Use standard data structures and be aware of the shape of your data.
• Write small functions
• Drive your exceptions to standard data structure
• Use the repl to poke at things
• Turn on auto—testing. Its awesome and will help you debug.
• Clojure is a great programming language that is concise. The claims about less code are true.
Whats Next• Stuart Sierra’s Component API
• Move from Lein to Clojure Boot
• More core.async please
• Keep learning
Closing Thoughts
“Clojure is the only language that I have seen where the longer a developer is writing code, the smaller the code base gets.”
-‐ Chris Miller
Shameless Self Promotion
Thanks
Appendix A: Libraries Used in Our Microservices
LibrariesProject Urls
compojure https://github.com/weavejester/compojure
Ring https://github.com/ring-‐clojure
specter https://github.com/nathanmarz/specter
amazonica https://github.com/mcohen01/amazonica
clj-‐http https://github.com/dakrone/clj-‐http
midje https://github.com/marick/Midje
lein http://leiningen.org/
boot https://github.com/boot-‐clj/boot
slingshot https://github.com/scgilardi/slingshot
Appendix B: Good Resources
Good Clojure Resources• Anything Rich Hickey Presents
• Simple Made Easy• Deconstructing the database• ClojureMade Simple
Related Documents
