Top Banner
GoLang: learning by doing Case: Migrating nearly 2 billion documents @jonatasdp
40

Go Lang migrating billions of documents

Jan 15, 2017

Download

Technology

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Go Lang  migrating billions of documents

GoLang: learning by doing

Case: Migrating nearly 2 billion documents

@jonatasdp

Page 2: Go Lang  migrating billions of documents

Go is awesome!

Page 3: Go Lang  migrating billions of documents

2 weeks experience

Page 4: Go Lang  migrating billions of documents

How it starts?

Page 5: Go Lang  migrating billions of documents

the challenge: migrate ~1.8 billion from a mongo server to another

Page 6: Go Lang  migrating billions of documents

WITHOUT STOP!

Page 7: Go Lang  migrating billions of documents

downtime migrationø

Page 8: Go Lang  migrating billions of documents

Billion of documents1.8

Page 9: Go Lang  migrating billions of documents

200K / MINUTE

Page 10: Go Lang  migrating billions of documents

1Million / 5 MIN

Page 11: Go Lang  migrating billions of documents

20Millions / hour

Page 12: Go Lang  migrating billions of documents

400Millions / DAY

Page 13: Go Lang  migrating billions of documents

5 days MIGRATING

Page 14: Go Lang  migrating billions of documents

weeks experience2 ● syntax● Basics

Page 15: Go Lang  migrating billions of documents

●Mgo● TERMUI● CLI

Page 16: Go Lang  migrating billions of documents

Migration algorithm● Two mongo connections: <origin> and <destiny>● <origin> receive records● <destiny> insert records● When record is on <destiny>● Remove from <origin>

Page 17: Go Lang  migrating billions of documents

FIRST TASTE● START WITHOUT TESTS :(● LEARNING ABOUT REFERENCES VS VARS● Structs for everything● Interface Comprehension● GO Routines● Locks and Mutex● Channels

Page 18: Go Lang  migrating billions of documents

MGO EXAMPLESimple insert

Page 19: Go Lang  migrating billions of documents

MGO EXAMPLEIs Not generic

Page 20: Go Lang  migrating billions of documents

structs

Page 21: Go Lang  migrating billions of documents

Structs & Interfaces

Page 22: Go Lang  migrating billions of documents

Interface {}

Page 23: Go Lang  migrating billions of documents

"Typecasting" the Interface

Page 24: Go Lang  migrating billions of documents

GOROUTINES

Page 25: Go Lang  migrating billions of documents

no wait group = 1.2M rpm

Page 26: Go Lang  migrating billions of documents

GOROUTINES + wait group = 200k rpm

Page 27: Go Lang  migrating billions of documents

State less GOROUTINESWith channels

Page 28: Go Lang  migrating billions of documents

GOROUTINES + channels

Page 29: Go Lang  migrating billions of documents

Channels

Page 30: Go Lang  migrating billions of documents

Channels

Page 31: Go Lang  migrating billions of documents

STRUCT METHODS

Page 32: Go Lang  migrating billions of documents

TERM UI

Page 33: Go Lang  migrating billions of documents

TERM UI

Page 34: Go Lang  migrating billions of documents

TERM UI colors

Page 35: Go Lang  migrating billions of documents

CLIgithub.com/codegagsta/cli

Page 36: Go Lang  migrating billions of documents

CLIgithub.com/codegagsta/cli

Page 37: Go Lang  migrating billions of documents

CLI

Page 38: Go Lang  migrating billions of documents

What I learned

● simplicity● naturally fast● easy to write async code● easy to maintain (mongo-migration#1)

Page 39: Go Lang  migrating billions of documents

OPEN SOURCE

https://github.com/ResultadosDigitais/mongo-migration

Page 40: Go Lang  migrating billions of documents

Thank you!

http://shipit.resultadosdigitais.com.br

@jonatasdp

http://ideia.me