Clocker How to Train Your Docker Cloud QCon London 2015; Docker Track Andrew Kennedy @grkvlt Introduc/on •  Andrew Kennedy –  Clocker Project Founder and Lead Engineer –  Open Source and Distributed Systems –  github.com/grkvlt •  CloudsoG Corpora/on –  Open Source Applica/on Management Specialists –  We’re Hiring… @grkvlt Agenda •  Clocker Introduc/on •  What is a Docker Cloud? •  Training a Docker Cloud •  Demonstra/on •  Roadmap @grkvlt Clocker IntroducBon What does it do? 1.  Spins up and Manages Docker Clouds 2.  Serves up Containers on Demand 3.  Manages Composite Applica/on Deployments on Docker @grkvlt What does it provide? •  Mul/ Host and Mul/ Container Applica/ons •  Seamless Networking –  Communica/on Between Services •  Orchestra/on and Clustering –  Control of Containers –  Container Management @grkvlt Who is using it? •  Tes/ng and Proof of Concept Stage –  Financial Services –  Insurance •  Produc/on –  Mul/-‐tenant Applica/on Trial –  Container per service –  Via IBM BlueMix @grkvlt Where can I ﬁnd it? •  Open Source •  Apache 2.0 Licensed •  h[p://clocker.io •  Status •  0.7.0 Beta Release •  0.8.0-‐SNAPSHOT Development @grkvlt What is a Docker Cloud? Clocker and Brooklyn •  What is it? –  Brooklyn Applica/on –  Brooklyn Loca/on •  What does it provide? –  First Class Docker Support in Brooklyn –  Op/mized Brooklyn Blueprints for Docker @grkvlt Apache Brooklyn •  Apache Brooklyn –  Open Source –  Java –  Donated to the Apache SoGware Founda/on –  ASF Incubator Status Apache Brooklyn • 
• 
• 
• 
Applica/on Management Pla_orm Deploy, Manage and Monitor Blueprints Provisioning, Installa/on and Customiza/on Management –  AutoScaling, Resilience, Performance, Security @grkvlt Apache jclouds •  Java Cloud Library •  API Agnos/c –  CloudStack, OpenStack, AWS EC2, GCE… •  Create Virtual Machines –  Return SSH Endpoint –  Manage Proper/es Apache jclouds •  Drivers for REST APIs •  Docker Driver –  Wri[en by @turlinux •  Virtual Container –  Using SSH Daemon –  Same Endpoint Type as VM –  Composi/on on any Image or Dockerﬁle @grkvlt Docker •  Popular –  Large Ecosystem –  Growing –  Complex •  Containers –  Isola/on –  Performance –  Composable @grkvlt Docker @grkvlt Brooklyn Blueprints • 
• 
• 
• 
• 
Describe Applica/ons OASIS CAMP Standard List of Services Tree Structure Sensors, Eﬀectors and Policies @grkvlt Blueprint Example name: appserver-‐w-‐policy services: -‐ type: brooklyn.entity.webapp.ControlledDynamicWebAppCluster initialSize: 1 memberSpec: $brooklyn:entitySpec: type: brooklyn.entity.webapp.jboss.JBoss7Server brooklyn.config: wars.root: http://search.maven.org/remotecontent?filepath=io/brooklyn/example/brooklyn-‐example-‐hello-‐world-‐sql-‐webapp/0.6.0/brooklyn-‐example-‐hello-‐world-‐sql-‐
webapp-‐0.6.0.war http.port: 8080+ java.sysprops: brooklyn.example.db.url: $brooklyn:formatString("jdbc:%s%s?user=%s\\&password=%s", component("db").attributeWhenReady("datastore.url"), "visitors", "brooklyn", "br00k11n") brooklyn.policies: -‐ policyType: brooklyn.policy.autoscaling.AutoScalerPolicy brooklyn.config: metric: $brooklyn:sensor("brooklyn.entity.webapp.DynamicWebAppCluster", "webapp.reqs.perSec.windowed.perNode") metricLowerBound: 10 metricUpperBound: 100 minPoolSize: 1 maxPoolSize: 5 -‐ type: brooklyn.entity.database.mysql.MySqlNode id: db name: DB HelloWorld Visitors brooklyn.config: datastore.creation.script.url: https://github.com/apache/incubator-‐brooklyn/raw/master/usage/launcher/src/test/resources/visitors-‐creation-‐script.sql @grkvlt Applica/on Management •  Policies •  Sensor Data Driven •  Auto Scaling •  Cluster Management •  Resilience and Failover •  Follow the Sun @grkvlt Docker Cloud •  Brooklyn Blueprint for Docker Cluster •  Docker Engine on Cloud VM –  Conﬁgura/on for TLS Cer/ﬁcates –  Setup Volumes (Logging?) –  Install Agents (Security? SDN?) @grkvlt Docker Cloud 1. 
2. 
3. 
4. 
@grkvlt On-‐demand Mul/-‐Tenant Hardware Independent Applica/on Level Architecture @grkvlt Clocker Features •  Applica/on Deployment –  Oasis CAMP YAML Blueprint –  TOSCA in Development –  Same as Core Brooklyn •  Mixed Des/na/ons –  Some Virtual Machines –  Some Bare Metal –  Some Containers @grkvlt Clocker Features •  Docker Extensions to Brooklyn –  Dockerﬁle or Image Speciﬁca/on for Installa/on –  Placement Strategies for Containers –  Create Docker Images and Networks •  Manages Docker Engine –  Deployment and Management –  Installa/on and Conﬁgura/on –  SoGware-‐Deﬁned Networking @grkvlt Clocker Orchestra/on Clocker @grkvlt Cloud Virtual Machine Docker Engine Container SDN Network Segment Training a Docker Cloud Training…? •  Container Management •  Placement and Provisioning •  Autonomics –  Policies and Scaling •  Headroom •  Networks @grkvlt Container Management •  Sources –  Docker Image Deﬁni/on –  Docker Hub –  Dockerﬁle –  Brooklyn En/ty Deﬁni/on •  Create Image Automa/cally –  Commit or Push for Reuse @grkvlt Container Management •  Installa/on of Services –  Deﬁned by Brooklyn or Dockerﬁle –  Common to all En/ty Instances •  Commit Image –  Available for next En/ty •  Push Image –  Available for all Hosts @grkvlt Container Management id: dockerfile-‐mysql name: "Docker Hub MySQL Database" origin: "https://registry.hub.docker.com/_/mysql/" locations: -‐ my-‐docker-‐cloud services: -‐ type: brooklyn.entity.container.docker.application.DockerfileApplication id: mysql name: "MySQL" brooklyn.config: docker.dockerfile.url: "https://s3-‐eu-‐west-‐1.amazonaws.com/brooklyn-‐clocker/mysql-‐5.6.tgz" docker.container.environment: MYSQL_ROOT_PASSWORD: "s3cr3t” @grkvlt Container Management id: dockerfile-‐mysql name: "Docker Hub LAMP Stack" locations: -‐ my-‐docker-‐cloud services: -‐ type: docker:mysql:5.7.5 id: mysql env: MYSQL_ROOT_PASSWORD: "s3cr3t" -‐ type: docker:grkvlt/myapp:latest id: application env: MYSQL_HOST: component(”mysql").attributeWhenReady("host.hostname") @grkvlt Container Placement •  Demand –  Adding an Applica/on –  Scaling exis/ng Applica/on •  Requirements –  Host Loca/on –  Service Resources –  CPU, Memory @grkvlt Container Placement •  Supply –  Choose a Host from available –  Create new Host •  Start Container –  Set CPU and Memory –  A[ach to Network @grkvlt Container Placement •  Placement Strategies –  Filter Hosts –  Sort Hosts •  Informa/on from – 
– 
– 
– 
@grkvlt Docker Underlying Machine Cloud Environment En/ty or Service Container Placement •  Placement Strategies –  Random, Depth or Breadth First –  CPU or Memory Usage –  Memory, CPU or Container Limits –  Geographic Constraints •  User Deﬁned –  Java Predicate @grkvlt Placement Strategy •  Determinis/c •  Simple –  Predicate and Comparator docker.container.strategies: -‐ $brooklyn:object: type: "brooklyn.location.docker.strategy.MaxContainersPlacementStrategy” brooklyn.config: maxContainers: 16 -‐ $brooklyn:object: type: "brooklyn.location.docker.strategy.CpuUsagePlacementStrategy” brooklyn.config: maxCpu: 0.75 @grkvlt Container Provisioning •  Provisioning Strategy –  New Docker Host Loca/on •  Constraints –  Docker Infrastructure Constraints –  En/ty or Applica/on Constraints •  User Deﬁned Strategies •  Intelligent Container Orchestra/on @grkvlt Autonomics •  Brooklyn Policies •  A[ached to En//es in Applica/on –  Nothing Docker Speciﬁc •  Elas/c Scaling –  Cluster Resizing –  Sensor Driven @grkvlt Applica/on Resilience •  Service Resilience and Replacement –  Restart Service and Container –  Applica/on Level, Not Infrastructure –  Same as Cloud •  Could be Integrated Further –  Snapshot Running Container for Restart @grkvlt Headroom •  Ensure resources available •  Based on MaxContainers strategy limit –  Or Percentage U/lisa/on –  Or CPU and RAM alloca/on •  Scale Docker Host Cluster Automa/cally –  Add new Docker hosts –  Remove empty Docker hosts @grkvlt SoGware-‐Deﬁned Networking •  Needed for Seamless Provisioning •  Host to Host Communica/on –  Same LAN Segment –  No Port Forwarding –  Natural Applica/on Conﬁgura/on •  Ini/al Driver was EPMD Applica/ons @grkvlt Networking Providers •  Standardized Interfaces •  Pluggable Providers –  Weave –  Metaswitch Calico –  IBM OpenDOVE –  Others? @grkvlt Networking Providers •  Implementa/on Agnos/c –  L2 over L3 etc. –  Similar to Hypervisor in Clouds •  Generic Interfaces –  Host Component –  Service Component (or Endpoint) @grkvlt Networking Capabili/es •  Provide Mul/ple Networks –  Single Applica/on or Shared –  Private Addresses –  Segmented by CIDR •  Docker Port Forwarding Access –  Debug Mechanism @grkvlt Clocker Networking SDN Gateway Internet Host @grkvlt SDN Bridge Container Container Networking Capabili/es •  IP Pool Controlled by Clocker •  Clocker Controls Subnet Alloca/on –  Applica/ons Segmented by CIDR •  Delegate to SDN or Cloud –  Bring your own IP •  Docker Port Forwarding Access @grkvlt Future Capabili/es •  Adding More SDN Providers •  Name Resolu/on –  BIND and DNSmasq or Weave DNS –  Needed for JMX et al –  Enables Many More En//es •  Adding More Supported Blueprints @grkvlt Virtual Network En/ty •  New Clocker En/ty •  Provisions a Virtual Network •  Looks for NetworkProvisioningExtension –  Registered in DockerLoca/on by SDN Provider –  Could also add to JcloudsLoca/on for Clouds •  Part of Applica/on Blueprint @grkvlt Network Provisioning •  Create Named Virtual Networks •  Uses Available Provider –  Currently Weave, Calico and IBM Supported -‐ type: brooklyn.networking.VirtualNetwork networkId: my-‐application cidr: 192.168.12.0/24 gateway: 192.168.12.1 dnsServers: -‐ 8.8.8.8 brooklyn.config: sdn.example.securityGroup: "my-‐security-‐group" @grkvlt Network Provisioning •  Allow Minimal (Zero!) Conﬁgura/on •  Use Sensible Defaults –  Also SDN Speciﬁc Conﬁgura/on •  Allocate Address Space on Demand -‐ type: brooklyn.networking.VirtualNetwork networkId: database-‐net cidr: 192.168.34.0/24 -‐ type: brooklyn.networking.VirtualNetwork networkId: couchbase-‐net @grkvlt Network Provisioning •  A[ach Containers to Networks brooklyn.config: network.list: -‐ couchbase-‐net -‐ management-‐net •  Create Networks as Required •  Also A[ach to VMs and Metal –  Where Supported @grkvlt DemonstraBon Features •  Orchestrated Docker 1.5.0 deployment with Weave SDN integra/on •  Automated a[achment of containers to mul/ple dynamic networks •  Brooklyn applica/on blueprints with network topology @grkvlt Clocker Roadmap Roadmap •  Latest Docker 1.5.0 Feature Support –  Stats command, Read-‐only containers, Named Dockerﬁles and more •  Swarm –  Add Clocker as an extension to Swarm itself? –  Or add features to exis/ng jclouds driver? •  Compose –  Support as alterna/ve blueprin/ng language @grkvlt Roadmap •  Networking –  DNS and DNSmasq Integra/on –  Rancher Integra/on –  Open vSwitch Integra/on •  Gemng Started –  Clocker Dockerﬁle @grkvlt Roadmap •  Repositories –  Docker Enterprise Hub –  Bintray, Ar/factory, Quay.io –  Private Repositories •  Applica/on Deﬁni/on –  TOSCA –  Docker Compose @grkvlt Roadmap •  Plugin API will be helpful •  Also projects like Powerstrip •  Provide services to Docker this way •  Orchestra/on •  Policies •  Consume wider range of services •  Networking and Storage @grkvlt Summary •  Clocker –  Brooklyn + jclouds + Docker + Weave + … –  @clockercentral •  Solves –  Composite Applica/on Management –  Docker Cloud Networking –  Container Placement and Provisioning @grkvlt QuesBons? Web Resources http://clocker.io/ http://brooklyn.io/ http://docker.io/ http://github.com/zettio/weave/ http://abstractvisitorpattern.co.uk/ @clockercentral @brooklyncentral @cloudsoft @grkvlt