Modern Black Mages Fighting in the Real World

Modern Black Mages Fighting in the Real World

Sep 9, 2016 in RubyKaigi 2016

@tagomoris Satoshi "Moris" Tagomori

Satoshi "Moris" Tagomori (@tagomoris)

Fluentd, MessagePack-Ruby, Norikra, ...

Treasure Data, Inc.

https://github.com/tagomoris/msgpack-inspect

http://docs.fluentd.org/articles/logohttp://www.fluentd.org/

Fluentd

• What is Fluentd? • Open Source Log Collector • Pluggable, Reliable, Less resource usage, Ease to use

• Versions of Fluentd • v0.12: stable versions (2014/12 - Now) • v0.14: versions for next stable (2016/05 - Now)

http://docs.fluentd.org/articles/logo

What's about this session?

• Introduce some patterns of "Black Magic"s (a.k.a. meta programming) in Ruby

• Show you some PRAGMATIC use of Black Magics

Fluentd v0.14 Release

Fluentd v0.14 API Update

• Everything changed :)

• Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::*

• Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods

http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details

Classes hierarchy (v0.12)

Fluent::Input F::Filter

F::Output

BufferedOutput

ObjectBuffered

TimeSliced Multi

Output F::BufferF::Parser

F::Formatter

3rd party plugins

Classes hierarchy (v0.14)

F::P::Input F::P::Filter F::P::Output

Fluent::Plugin::Base

F::P::BufferF::P::Parser

F::P::FormatterF::P::Storage

both ofbuffered/non-buffered

F::P::BareOutput(not for 3rd party

plugins)

F::P::MultiOutput

copyroundrobin

diff v0.12 v0.14

F::P::Output




plugins)

F::P::MultiOutput

copyroundrobin

F::Output

BufferedOutput

ObjectBuffered

TimeSliced Multi

Output

Super classes byhow to buffer data

All output pluginsare just "Output"

Basic Black Magics: Class and Mixin in Ruby

Class and Subclass in Ruby

class A

#bar

class B

#bar

super

B.new.bar

class A

#bar

class B

#bar

super

B.new.bar

module M

#bar

Introducing Methods by Mixin

class A

#bar

class B

#bar

super

B.new.bar

module M

#bar

Singleton Class of Ruby

#bar

B.new.singleton_class

class A

#bar

class B

#bar

super

b=B.new b.singleton_class.include M2 b.bar

module M

#bar

Adding Methods on An Instance (1)


#bar

M2

#bar

class A

#bar

class B

#bar

super

b=B.new b.extend M2 b.bar

module M

#bar

Adding Methods on An Instance (2)


#bar

M2

#bar

Back to Fluentd code :)

diff v0.12 v0.14

F::P::Output




plugins)

F::P::MultiOutput

copyroundrobin

F::Output

BufferedOutput

ObjectBuffered

TimeSliced Multi

Output

Super classes byhow to buffer data

All output pluginsare just "Output"

Fluentd v0.12 Fluent::Output

class Fluent::Output

#emit(tag, es, chain)

MyOutput

Engine calls plugin.emit(tag, es, chain)

@buffer

Fluentd v0.12 Fluent::BufferedOutput (1)

class Fluent::Outputclass BufferedOutput

#emit(tag, es, chain, key)

MyOutput


super(tag, es, chain, any_key)


@buffer

#emit(key, data, chain)

#format(tag,time,record)

#format_stream(tag,es)




MyOutput




@buffer





Fluentd v0.12 Fluent::TimeSlicedOutput



MyOutput#emit(tag, es, chain)


@buffer


#emit(tag, es, chain) class TimeSlicedOutput


Fluentd v0.12 Fluent::ObjectBufferedOutput



MyOutput#emit(tag, es, chain)


@buffer


#emit(tag, es, chain) class ObjectBufferedOutput

Fluentd v0.12 Fluent::BufferedOutput

class Fluent::Outputclass BufferedOutputMyOutput

@buffer calls #write in OutputThread

@buffer

chunk#write(chunk)

OutputThread

#pop

Fluentd v0.12 Fluent::TimeSlicedOutput


@buffer

MyOutput

class TimeSlicedOutput

OutputThread

#write(chunk)


#write calls chunk.keychunk

#pop

Fluentd v0.12 Fluent::ObjectBufferedOutput


@buffer

MyOutput

class ObjectBufferedOutput

OutputThread

#write(chunk)

#write(chunk)

#write_object(chunk_key, chunk)


chunk#pop

Fluentd v0.12 API Problems

• Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes

• counting input events • hook arguments/return values to update API

• Fluentd core didn't show fixed API

• Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for

arguments/return values

How can we solve this problem?

Fluent::Plugin::Output (v0.14)

Fluentd v0.14 Fluent::Plugin::Output

class Outputclass MyOutput

#process(tag, es)

Engine calls plugin.emit_events(tag, es)

@buffer

#write

#emit_events(tag, es)

#format(tag, time, record)

#write(chunk)

#try_write(chunk)

#emit_sync(tag, es)

#emit_buffered(tag, es)



Output calls plugin.write (or try_write)

@buffer

chunk

#write(chunk)

#try_write(chunk)

flush thread

#process(tag, es)


Fluentd v0.14 Design Policy

• Separate entry points from implementations • Methods in superclass control everything

• Do NOT override these methods! • Methods in subclass do things only for themselves

• not for data flow, control flow nor others

• Plugins have simple/straightforward call stack • Easy to understand/maintain

❤

How about existing v0.12 plugins?

Requirement:

(Almost) All Existing Plugins SHOULD Work Well WITHOUT ANY MODIFICATION

• Fluent::Compat namespace for compatibility layer

v0.14 Plugins & Compat Layer

F::P::Output

F::P::Base

v0.14 PluginsFluent::

Compat::Output

F::C::BufferedOutput

F::C::TimeSliced

Output

F::C::ObjectBuffered

Output

Fluent::OutputF::

BufferedOutput

F::TimeSliced

Output

F::ObjectBuffered

Output

v0.12 Plugins

Double Decker Compat Layer?

• Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-(

• Separate code into Fluent::Compat • and import it into Fluent top level



#process(tag, es)

Engine calls plugin.emit_events(tag, es)

@buffer

#write



#write(chunk)

#try_write(chunk)

#emit_sync(tag, es)





MyOutput




@buffer





Fluent::Plugin::Outputclass MyOutput@buffer

#write


v0.12 Plugins via Compat Layer: Best case (virtual)

Compat::BufferedOutput



#format_stream(tag,es) #handle_stream_*

#handle_stream_simple

#emit(tag, es, chain, key) #emit(tag, es, chain, key)


#write(chunk) flush thread


#write


v0.12 Plugins via Compat Layer: Best case (real)










#write


When plugin overrides #format_stream










#write


When plugin overrides #format_stream









default implementation for calling "super"


#write


When plugin overrides #emit






#emit(tag, es, chain) #emit(tag, es, chain, key)




#write












#write










This call doesn't happen, in fact

#emit doesn't return values!



#write











#emit calls @buffer.emit → NoMethodError !


#write












#write











#emit

1. #emit calls @buffer.emit with data to be written in buffer

0. plugin calls @buffer.extend to add #emit

2. @buffer.emit stores arguments into plugin's attribute

3. get stored data

4. call @buffer.write with data


#write












#write


Thinking about "chunk" instance ...









#write may call "chunk.key", but v0.14 chunk doesn't have #key !


#write



"chunk" has #metadata, and values of #key can be created via #metadata

Let's "chunk.extend" !

Where to do so?

?


Fluent::Plugin::OutputMyOutput@buffer

#write

C::BufferedOutput



#write(chunk)

BufferedChunkMixin

plugin.extend BufferedChunkMixin in #configure

Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...

Controlling Plugin Lifecycle

Plugin Lifecycle Updated

Methods(v0.12) • #configure • #start

• #before_shutdown • #shutdown

v0.12 Plugins often doesn't call "super"!

Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate

In v0.14, these methods MUST call "super"

• #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?

For Example: shutdown compat plugins


#shutdown

F::P::Output

super

#shutdown?

#shutdown

F::C::Output

#shutdown

MyOutput

#shutdown

It doesn't call "super"! We want to call this...

What We Want To Do:


#shutdown

F::P::Output

super

#shutdown?

#shutdown

F::C::Output

#shutdown

MyOutput

#shutdown

1. call #shutdown anyway

0. Fluentd core calls #shutdown

2. call #shutdown? to check "super" is called or not

3. call #shutdown of superclass forcedly!

What We Want To Do:


#shutdown

F::P::Output

super

#shutdown?

#shutdown

F::C::Output

#shutdown

MyOutput

#shutdown

How to make this point?

One More Magic! Module#prepend

class A

#bar

class B

#bar

super

B.new.bar

Wrapping Methods on a Class (1)


#bar

class A

#bar

class B

#bar

super

B.new.bar

module M

Wrapping Methods on a Class (2)


#bar

#bar

Using extend is powerful, but it should be done for all instances

How about wrapping methods for all instances of the class?

class A

#bar

class B

#bar

super

module M;def bar;super;end;end B.prepend M B.new.bar

module M

Wrapping Methods on a Class (3): Module#prepend


#bar

#bar

module M wraps B, and M#bar is called at first

What We Want To Do:


#shutdown

F::P::Output

super

#shutdown?

#shutdown

F::C::Output

#shutdown

MyOutput

#shutdown

THIS ONE !!!

What We Got :-)


#shutdown

F::P::Output

super

#shutdown?

#shutdown

F::C::Output

#shutdown

MyOutput

#shutdown

1. call #shutdown anyway

0. prepend CallSuperMixin at first

2. call #shutdown? to check "super" is called or not

3. if not, get method of superclass, bind self with it, then call it

Thank you @unak -san!

Beating Test Code

Testing: Capturing return values by Test Driver

OutputMyOutput@buffer

#write

#emit_events

#format

#emit_buffered

Output Plugin Test Driver

Create plugin instances

Feed test data into plugin

Testing: Capturing return values by Test Driver


#write

#emit_events

#format

#emit_buffered

We want to assert this return value!

Output Plugin Test Driver Feed test data into plugin

Using #prepend to capture return values


#write

#emit_events

#format

#emit_buffered


moduleM

#format

Store return value of "super"

Using #prepend ... doesn't work :-(


#write

#emit_events

#format

#emit_buffered


#format

Test code sometimes overwrites methods for many reasons :P

singletonclass

#format

😱

moduleM

One Another Magic: Stronger Than Anything

class A

#bar

class B

#bar

super

b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar

Another Study: How To Wrap Singleton Method?


#bar

module M

#bar

class A

#bar

class B

#bar

supermodule P

Another Study: How To Wrap Singleton Method?


#bar

b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend P b.bar

#bar

module M

#bar

Singleton class is a class, so it can be prepended :)

It's actually done in Test Driver implementation...

Using #prepend on singleton_class: Yay!


#write

#emit_events

#format

#emit_buffered


#format

singletonclass

#format😃

moduleM

Prepending modules on singleton_class overrides everything!

IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P

Do Whatever You Can For Users!

It Makes Everyone Happier!

... Except for Maintainers :(

Modern Black Mages Fighting in the Real World

Software