Writing Docker monitoring agent with Go

Writing Docker monitoring agent with Go

!ainoya

About me

• Naoki Ainoya(!ainoya)

• iOS(new!)/Server/Infra engineer

• Swift/Scala/Go/Docker

• Recruit Marketing Partners Co, Ltd.

A tiny deployment pipeline tool Written in Go

!walter-cd/walter

Open source project

Introduction of Fune• An event emitter triggered by Docker Event API

• The agent process (fune-agent) communicates with docker daemon via its socket

• https://github.com/ainoya/fune

Docker daemonFu

ne a

gent

Cont

tain

er A

Cont

tain

er B

Cont

tain

er C

Nginx/Redis

vulcand(etcd)

Slack notification etc..

Listens docker events API

notifies each containers are created/died/stopped/started

Emits pluggable actions

Your arbitrary docker ecosystem

ECS/k8s/fleet/etc..

Docker Host

Deploy containers

Overview of Fune

• Docker Events API provides a lifecycle of container as JSON format

• create/start/die/stop

Overview of Fune

fune-agent listens these events

Docker daemonFu

ne a

gent

Cont

tain

er A

Cont

tain

er B

Cont

tain

er C

Listens docker events API

notifies each containers are created/died/stopped/started

Your arbitrary docker ecosystem

ECS/k8s/fleet/etc..

Docker Host

Deploy containers

Overview of Fune

fune-agent emits Action when received Docker events from event

Docker daemonFu

ne a

gent

Cont

tain

er A

Cont

tain

er B

Nginx/Redis

vulcand(etcd)

Slack notification etc..

Listens docker events API

notifies each containers are created/died/stopped/started

Emits pluggable actions

Your arbitrary docker ecosystem

ECS/k8s/fleet/etc..

Docker Host

Deploy containers

Overview of Fune

Docker daemonFu

ne a

gent

Cont

tain

er A

Cont

tain

er B

Nginx/Redis

vulcand(etcd)

Slack notification etc..

Listens docker events API

notifies each containers are created/died/stopped/started

Emits pluggable actions

Your arbitrary docker ecosystem

ECS/k8s/fleet/etc..

Docker Host

Deploy containers

• Action, for example;

• Set/Del a container information to Redis as FQDN/IP:Port pair for proxy

• Slack notification

• More details are next time :)

Today I talk about:

• Some tips I learned with writing fune

• Studied a lot of stuffs from coreos/etcd

Setup project

• Standard structure for working with Godeps correctly

• /src/github.com/ainoya/fune

./${GOPATH}/src!"" 9fans.net#   %"" go!"" code.google.com#   %"" p!"" github.com#   !"" ainoya/fune#   !"" barakmich#   !"" coreos etc...

Godeps for the package dependencies

• Using Godeps once, all dependencies copied into directory and rewrite import statement

• Then you need Godeps command only if package dependency is added

import "github.com/fsouza/go-dockerclient"// after `Godep save -r ./…`import "github.com/ainoya/fune/Godeps/_workspace/

src/github.com/fsouza/go-dockerclient"

Godeps for the package dependencies

• All dependencies are included in git repos

• But it makes dirty commit log!!

• Keep off its dirtiness by split commits :)

How about vendor feature in go1.5?

• go1.5 vendor feature have some problems yet

• ex) go  tool  "./…" doesn't works well

• https://github.com/golang/go/issues/11659

• See also ) http://deeeet.com/writing/2015/06/26/golang-dependency-vendoring/

Write tests

• Rerun automatically if files are changed

• rerun is awesome! (but ruby tool gem)

rerun -x -c -d directory1, directory2,... \—pattern '**/*.{go}' -- go test ./...

Be careful about race condition

• Test with "-­‐race" option if you often use goroutine in your code

• "-­‐cpu  N" option may reveal race condition you didn't expected

• I use "go  test  ./…  -­‐race  -­‐cpu  1,2,4" inside test script

WARNING: DATA RACEWrite by goroutine 8: sync.raceWrite() /usr/local/Cellar/go/1.5/libexec/src/sync/race.go:41 +0x2e sync.(*WaitGroup).Wait() /usr/local/Cellar/go/1.5/libexec/src/sync/waitgroup.go:124 +0xf9 github.com/ainoya/fune/emitter.TestBroadCast() /Users/ainoya/.go/src/github.com/ainoya/fune/gopath/src/github.com/ainoya/fune/emitter/emitter_test.go:57 +0x444 testing.tRunner() /usr/local/Cellar/go/1.5/libexec/src/testing/testing.go:456 +0xdc

Is Go1.5 compilation time slower than 1.4?

• "go  test" obviously takes time much longer

• It's probably because of GoGo compiler performanceRef) Performance section of https://golang.org/doc/go1.5

• It should be improved with Go1.6

Channel tips

• coreos/etcd implements a shutdown feature with using close(channel)type  Server  struct  {     name        string     channel  chan  struct{}     done        chan  struct{}     stop        chan  struct{}  }  

func  main()  {     done  :=  make(chan  struct{})     stop  :=  make(chan  struct{})  

  s  :=  &Server{done:  done,  stop:  stop,  hoge:  hoge}  

  go  s.run()  

  s.channel  <-­‐  struct{}{}  

  osutil.RegisterInterrputHandler(s.Stop)     <-­‐s.done  }

func  (s  *Server)  run()  {     defer  func()  {  close(s.done)  }()     for  {       select  {       case  <-­‐s.channel:         fmt.Println("do  something")       case  <-­‐s.stop:         return       }     }  

}  

func  (s  *S)  Stop()  {     select  {     case  s.stop  <-­‐  struct{}{}:     case  <-­‐s.done:       return     }     <-­‐s.done  }  

Channel tips

• coreos/etcd implements a shutdown feature with using close(channel)type  Server  struct  {     name        string     channel  chan  struct{}     done        chan  struct{}     stop        chan  struct{}  }  

func  main()  {     done  :=  make(chan  struct{})     stop  :=  make(chan  struct{})  

  s  :=  &Server{done:  done,  stop:  stop,  hoge:  hoge}  

  go  s.run()  

  s.channel  <-­‐  struct{}{}  

  osutil.RegisterInterrputHandler(s.Stop)     <-­‐s.done  }

func  (s  *Server)  run()  {     defer  func()  {  close(s.done)  }()     for  {       select  {       case  <-­‐s.channel:         fmt.Println("do  something")       case  <-­‐s.stop:         return       }     }  

}  

func  (s  *S)  Stop()  {     select  {     case  s.stop  <-­‐  struct{}{}:     case  <-­‐s.done:       return     }     <-­‐s.done  }  

Channel tips

• coreos/etcd implements a shutdown feature with using close(channel)type  Server  struct  {     name        string     channel  chan  struct{}     done        chan  struct{}     stop        chan  struct{}  }  

func  main()  {     done  :=  make(chan  struct{})     stop  :=  make(chan  struct{})  

  s  :=  &Server{done:  done,  stop:  stop,  hoge:  hoge}  

  go  s.run()  

  s.channel  <-­‐  struct{}{}  

  osutil.RegisterInterrputHandler(s.stop)     <-­‐s.done  }

func  (s  *Server)  run()  {     defer  func()  {  close(s.done)  }()     for  {       select  {       case  <-­‐s.channel:         fmt.Println("do  something")       case  <-­‐s.stop:         return       }     }  

}  

func  (s  *S)  Stop()  {     select  {     case  s.stop  <-­‐  struct{}{}:     case  <-­‐s.done:       return     }     <-­‐s.done  }  

Channel tips

• coreos/etcd implements a shutdown feature with using close(channel)type  Server  struct  {     name        string     channel  chan  struct{}     done        chan  struct{}     stop        chan  struct{}  }  

func  main()  {     done  :=  make(chan  struct{})     stop  :=  make(chan  struct{})  

  s  :=  &Server{done:  done,  stop:  stop,  hoge:  hoge}  

  go  s.run()  

  s.channel  <-­‐  struct{}{}  

  osutil.RegisterInterrputHandler(s.Stop)     <-­‐s.done  }

func  (s  *Server)  run()  {     defer  func()  {  close(s.done)  }()     for  {       select  {       case  <-­‐s.channel:         fmt.Println("do  something")       case  <-­‐s.stop:         return       }     }  

}  

func  (s  *S)  Stop()  {     select  {     case  s.stop  <-­‐  struct{}{}:     case  <-­‐s.done:       return     }     <-­‐s.done  }  

Channel tips

• coreos/etcd implements a shutdown feature with using close(channel)type  Server  struct  {     name        string     channel  chan  struct{}     done        chan  struct{}     stop        chan  struct{}  }  

func  main()  {     done  :=  make(chan  struct{})     stop  :=  make(chan  struct{})  

  s  :=  &Server{done:  done,  stop:  stop,  hoge:  hoge}  

  go  s.run()  

  s.channel  <-­‐  struct{}{}  

  osutil.RegisterInterrputHandler(s.Stop)     <-­‐s.done  }

func  (s  *Server)  run()  {     defer  func()  {  close(s.done)  }()     for  {       select  {       case  <-­‐s.channel:         fmt.Println("do  something")       case  <-­‐s.stop:         return       }     }  

}  

func  (s  *S)  Stop()  {     select  {     case  s.stop  <-­‐  struct{}{}:     case  <-­‐s.done:       return     }     <-­‐s.done  }  

Test goroutine easily"s.done" channel makes testing goroutine easily

//  https://github.com/coreos/etcd/blob/master/etcdserver/server_test.go#L1031  //  TestPublishRetry  tests  that  publish  will  keep  retry  until  success.  func  TestPublishRetry(t  *testing.T)  {     n  :=  &nodeRecorder{}     srv  :=  &EtcdServer{       cfg:            &ServerConfig{TickMs:  1},       r:                raftNode{Node:  n},       w:                &waitRecorder{},       done:          make(chan  struct{}),       reqIDGen:  idutil.NewGenerator(0,  time.Time{}),     }     time.AfterFunc(500*time.Microsecond,  func()  {  close(srv.done)  })     srv.publish(10  *  time.Nanosecond)  

Run go as Docker container• Go single binary with "SCRATCH"

• You can run a go stuff inside extremely simple container!

FROM  scratch  EXPOSE  8080  COPY  your-­‐golang-­‐app  /app/your-­‐golang-­‐app  ENV  PATH=/app:$PATH  

ENTRYPOINT  ["/app/your-­‐golang-­‐app"]  CMD  ["-­‐some-­‐option=haha"]

Round up

• Learned a lot of stuffs from well-knowned products (coreos/etcd)

• project structure, test, golang-way

See you next kyobashi.*!

• 9/7: kyobashi.dex

• 9/16: potatotips

• ???: kyobashi.???