Writing Pluggable Software Tatsuhiko Miyagawa miyagawa@gmail.com Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo.

Writing Pluggable Software

Tatsuhiko Miyagawa miyagawa@gmail.com

Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo

Tatsuhiko MiyagawaTatsuhiko Miyagawa

For non-JP attendees …

If you find \ in the code,Replace that with backslash.

(This is MS' fault)

PlaggableSoftware

PluggableSoftware

Agenda

#1How to make

your app pluggable

#2TMTOWTDP

There's More Than One Way To Deploy Plugins

Pros/Cons by examples

First-of-all:Why pluggable?

Benefits

#1Keep the app design

and code simple

#2Let the app users

customize the behavior

(without hacking the internals)

#3It's fun to write

pluginsfor most hackers

(see: Plagger and Kwiki)

"Can your app do XXX?"

"Yes, by plugins."

"Your app has a bug in YYY""No, it's the bug in plugin

YYY,Not my fault."

(Chain Of Responsibilities)

Good EnoughReasons, huh?

#1Make your app

pluggable

Example

ack (App::Ack)

grep –r for programmers

Ack is a "full-stack"software now.

By "full-stack" I mean:

Easy installNo configurationNo way to extend

Specifically:These are hardcoded

Ignored directoriesFilenames and types

Ignored Directories

@ignore_dirs = qw( blib CVS RCS SCCS .svn

_darcs .git );

Filenames and languages mapping

%mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )],… perl => [qw( pl pm pod tt ttml t )],

What if makingthese pluggable?

DISCLAIMER

Don't get me wrong Andy,I love ack the way it is…

Just thought it can bea very good example for the

tutorial.

Quickstart:Class::Trigger

Module::Pluggable

Class::Trigger SYNOPSIS

package Foo;use Class::Trigger;

sub foo { my $self = shift; $self->call_trigger('before_foo'); # some code ... $self->call_trigger('after_foo');}

package main;Foo->add_trigger(before_foo => \&sub1);Foo->add_trigger(after_foo => \&sub2);

Class::TriggerHelps you to

implementObserver Pattern.

(Rails calls this Observer)

Module::Pluggable SYNOPSIS

package MyClass;use Module::Pluggable;

use MyClass;my $mc = MyClass->new();# returns the names of all plugins installed

under MyClass::Plugin::*my @plugins = $mc->plugins();

package MyClass::Plugin::Foo;sub new { … }1;

Setup plugins in App::Ack

package App::Ack;

use Class::Trigger;use Module::Pluggable require => 1;

__PACKAGE__->plugins;

Setup plugins in App::Ack

package App::Ack;

use Class::Trigger;use Module::Pluggable require => 1;

__PACKAGE__->plugins; # "requires" modules

Ignored Directories (Before)

@ignore_dirs = qw( blib CVS RCS SCCS .svn

_darcs .git );

Ignored Directories (After)

# lib/App/Ack.pm__PACKAGE__->call_trigger('ignore_dirs.add', \@ignore_dirs);

Ignored Directories (plugins)

# lib/App/Ack/Plugin/IgnorePerlBuildDir.pmpackage App::Ack::Plugin::IgnorePerlBuildDir;

App::Ack->add_trigger( "ignore_dirs.add" => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( blib ); },);

Ignored Directories (plugins)

# lib/App/Ack/Plugin/IgnoreSourceControlDir.pmpackage

App::Ack::Plugin::IgnoreSourcdeControlDir;

App::Ack->add_trigger( "ignore_dirs.add" => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( CVS RCS .svn _darcs .git ); },);

Filenames and languages (before)

%mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )],… perl => [qw( pl pm pod tt ttml t )],

Filenames and languages (after)

# lib/App/Ack.pm__PACKAGE__->call_trigger('mappings.add', \%mappings);

Filenames and languages (plugins)

package App::Ack::Plugin::MappingCFamily;use strict;

App::Ack->add_trigger( "mappings.add" => sub { my($class, $mappings) = @_; $mappings->{asm} = [qw( s S )]; $mappings->{cc} = [qw( c h xs )]; $mappings->{cpp} = [qw( cpp m h C H )]; $mappings->{csharp} = [qw( cs )]; $mappings->{css} = [qw( css )]; },);

Works great with few lines of

Now it's time to add Some useful stuff.

Example Plugin:Content Filter

sub _search { my $fh = shift; my $is_binary = shift; my $filename = shift; my $regex = shift; my %opt = @_;

if ($is_binary) { my $new_fh; App::Ack->call_trigger('filter.binary', $filename, \

$new_fh); if ($new_fh) { return _search($new_fh, 0, $filename, $regex, @_); } }

Example:Search PDF content

with ack

PDF filter plugin

package App::Ack::Plugin::ExtractContentPDF;use strict;use CAM::PDF;use File::Temp;

App::Ack->add_trigger( 'mappings.add' => sub { my($class, $mappings) = @_; $mappings->{pdf} = [qw(pdf)]; },);

PDF filter plugin (cont.)

App::Ack->add_trigger( 'filter.binary' => sub { my($class, $filename, $fh_ref) = @_; if ($filename =~ /\.pdf$/) { my $fh = File::Temp::tempfile; my $doc = CAM::PDF->new($file); my $text; for my $page (1..$doc->numPages){ $text .= $doc->getPageText($page); } print $fh $text; seek $$fh, 0, 0; $$fh_ref = $fh; } },);

PDF search

> ack --type=pdf Audreyyapcasia2007-pugs.pdf:3:Audrey Tang

Homework

Use File::ExtractTo handle arbitrary media files

Homework 2:

Search non UTF-8 files(hint: use Encode::Guess)You'll need another hook.

Summary

Class::Trigger+ Module::Pluggable= Pluggable app easy

#2TMTOWTDP

There's More Than One Way To Deploy Plugins

Module::Pluggable+ Class::Trigger

= Simple and Nicebut has limitations

In Reality, we need more control

over how plugins behave

1)The order of

plugin executions

2)Per user

configurationsfor plugins

3)Temporarily

Disable pluginsShould be easy

4)How to install

& upgrade plugins

5)Let plugins

have storage area

Etc, etc.

Examples:Kwiki

Plaggerqpsmtpd

Movable Type

I won't talk aboutCatalyst plugins

(and other framework thingy)

Because they're NOT

"plug-ins"

Install plugins And now you write

MORE CODE

95% of Catalyst plugins

Are NOT "plugins"But "components"

95% of these statistics is made up by the speakers.

Kwiki 1.0

Kwiki Plugin code

package Kwiki::URLBL;use Kwiki::Plugin -Base;use Kwiki::Installer -base;

const class_id => 'urlbl';const class_title => 'URL Blacklist DNS';const config_file => 'urlbl.yaml';

sub register { require URI::Find; my $registry = shift; $registry->add(hook => 'edit:save', pre =>

'urlbl_hook'); $registry->add(action => 'blacklisted_url');}

Kwiki Plugin (cont.)

sub urlbl_hook { my $hook = pop; my $old_page = $self->hub->pages->new_page($self-

>pages->current->id); my $this = $self->hub->urlbl; my @old_urls = $this->get_urls($old_page->content); my @urls = $this->get_urls($self->cgi-

>page_content); my @new_urls = $this->get_new_urls(\@old_urls, \

@urls); if (@new_urls && $this->is_blocked(\@new_urls)) { $hook->cancel(); return $self->redirect("action=blacklisted_url"); }}

Magic implementedin Spoon(::Hooks)

"Install" Kwiki Plugins

# order doesn't matter here (according to Ingy)Kwiki::DisplayKwiki::EditKwiki::Theme::BasicKwiki::ToolbarKwiki::StatusKwiki::Widgets# Comment out (or entirely remove) to disable# Kwiki::UnnecessaryStuff

Kwiki plugin config

# in Kwiki::URLBL plugin__config/urlbl.yaml__urlbl_dns: sc.surbl.org, bsb.spamlookup.net,

rbl.bulkfeeds.jp

# config.yamlurlbl_dns: myowndns.example.org

Kwiki plugins are CPAN modules

Install and Upgrade plugins

cpan> install Kwiki::SomeStuff

Using CPAN as a repository

Pros #1:reuse most of current CPAN infrastructure.

Using CPAN as a repository

Pros #2:Increasing # of

modules= good motivation

for Perl hackers

Cons #1:Installing CPAN deps

could be a mess(especially for Win32)

Cons #2:Whenever Ingy

releasesnew Kwiki, lots of plugins just break.

Kwiki plugin storage

return if grep {$page->id} @{$self->config->cached_display_ignore};

my $html = io->catfile( $self->plugin_directory,$page->id)->utf8;

Kwiki 2.0

Same as Kwiki 1.0

Except:plugins are now

in SVN repository

Plagger plugin

package Plagger::Plugin::Publish::iCal;use strict;use base qw( Plagger::Plugin );

use Data::ICal;use Data::ICal::Entry::Event;use DateTime::Duration;use DateTime::Format::ICal;

sub register { my($self, $context) = @_; $context->register_hook( $self, 'publish.feed' => \&publish_feed, 'plugin.init ' => \&plugin_init, );}

Plagger plugin (cont)

sub plugin_init { my($self, $context) = @_;

my $dir = $self->conf->{dir}; unless (-e $dir && -d _) { mkdir $dir, 0755 or $context->error("Failed to mkdir $dir: $!"); }}

Plagger plugin storage

$self->conf->{invindex} ||= $self->cache->path_to('invindex');

Plagger plugin config

# The order matters in config.yaml# if they're in the same hooksplugins: - module: Subscription::Config config: feed: - http://www.example.com/ - module: Filter::DegradeYouTube config: dev_id: XYZXYZ - module: Publish::Gmail disable: 1

Plugins Install & Upgrade

> notest cpan Plagger# or …> svn co http://…/plagger/trunk plagger> svn update

Plagger impl.ripped off by

many apps now

qpsmtpd

mod_perl for SMTP

Runs with tcpserver, forkserver or Danga::Socket standalone

Plugins: Flat files

rock:/home/miyagawa/svn/qpsmtpd> ls -F pluginsasync/ greylistingauth/ hosts_allowcheck_badmailfrom http_configcheck_badmailfromto ident/check_badrcptto logging/check_badrcptto_patterns miltercheck_basicheaders parse_addr_withhelocheck_earlytalker queue/check_loop quit_fortunecheck_norelay rcpt_okcheck_relay relay_onlycheck_spamhelo require_resolvable_fromhostcontent_log rhsblcount_unrecognized_commands sender_permitted_fromdns_whitelist_soft spamassassindnsbl tlsdomainkeys tls_cert*dont_require_anglebrackets virus/

qpsmtpd plugin

sub hook_mail { my ($self, $transaction, $sender, %param) = @_;

my @badmailfrom = $self->qp->config("badmailfrom") or return (DECLINED);

for my $bad (@badmailfrom) { my $reason = $bad; $bad =~ s/^\s*(\S+).*/$1/; next unless $bad; $transaction->notes('badmailfrom', $reason) … } return (DECLINED);}

Actually qpsmtpdPlugins are "compiled"to modules

my $eval = join("\n", "package $package;", 'use Qpsmtpd::Constants;', "require Qpsmtpd::Plugin;", 'use vars qw(@ISA);', 'use strict;', '@ISA = qw(Qpsmtpd::Plugin);', ($test_mode ? 'use Test::More;' : ''), "sub plugin_name { qq[$plugin] }", $line, $sub, "\n", # last line comment without newline? );

$eval =~ m/(.*)/s; $eval = $1;

eval $eval; die "eval $@" if $@;

qpsmtpd plugin config

rock:/home/miyagawa/svn/qpsmtpd> ls config.sample/config.sample:IP logging

require_resolvable_fromhostbadhelo loglevel rhsbl_zonesbadrcptto_patterns plugins

size_thresholddnsbl_zones rcpthosts

tls_before_authinvalid_resolvable_fromhost relayclients tls_ciphers

config/plugins

# content filtersvirus/klez_filter

# rejects mails with a SA score higher than 2spamassassin reject_threshold 20

config/badhelo

# these domains never uses their domain when greeting us, so reject transactions

aol.comyahoo.com

Install & Upgrade plugins

Just use subversion

MT plugins are flat-files

(or scripts that call modules)

MT plugin code

package MT::Plugin::BanASCII; our $Method = "deny";use MT; use MT::Plugin;

my $plugin = MT::Plugin->new({ name => "BanASCII v$VERSION", description => "Deny or moderate ASCII or Latin-1

comment",}); MT->add_plugin($plugin);MT->add_callback('CommentFilter', 2, $plugin, \

&handler);

MT plugin code (cont)

sub init_app { my $plugin = shift; $plugin->SUPER::init_app(@_); my($app) = @_; return unless $app->isa('MT::App::CMS'); $app->add_itemset_action({ type => 'comment', key => 'spam_submission_comment', label => 'Report SPAM Comment(s)', code => sub { $plugin->submit_spams_action('MT::Comment',

@_) }, } );

MT plugin storage

require MT::PluginData;my $data = MT::PluginData->load({ plugin => 'sidebar-manager', key => $blog_id },);unless ($data) { $data = MT::PluginData->new; $data->plugin('sidebar-manager'); $data->key($blog_id);}$data->data( \$modulesets );$data->save or die $data->errstr;

Order control

MT->add_callback('CMSPostEntrySave', 9, $rightfields, \&CMSPostEntrySave);

MT->add_callback('CMSPreSave_entry', 9, $rightfields, \&CMSPreSave_entry);

MT::Entry->add_callback('pre_remove', 9, $rightfields, \&entry_pre_remove);

Defined in plugins. No Control on users end

Conclusion

Flat-filesvs.

Modules

Flat-files:☺ Easy to install (Just grab it)

☻ Hard to upgradeOK for simple plugins

Modules:☺ Full-access to Perl OO

goodness☺ Avoid duplicate efforts of CPAN ☻ Might be hard to resolve deps.

Subversion to the rescue(could be a barrier for newbies)

Nice-to-haves:Order control

Temporarily disable pluginsPer plugin config

Per plugin storage

Resources

Class::Triggerhttp://search.cpan.org/dist/Class-Trigger/Module::Pluggablehttp://search.cpan.org/dist/Module-Pluggable/Ask Bjorn Hansen: Build Easily Extensible Perl

Programshttp://conferences.oreillynet.com/cs/os2005/view/e_sess/6806qpsmtpdhttp://smtpd.develooper.com/MT pluginshttp://www.sixapart.com/pronet/plugins/Kwikihttp://www.kwiki.org/Plaggerhttp://plagger.org/

Writing Pluggable Software Tatsuhiko Miyagawa miyagawa@gmail.com Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo.

Documents

Y. Miyagawa (AIST)CCP2006 Gyengju Plasma analysis for PIII.....

Sphinx && Perl Houston Perl Mongers - May 8th, 2014

Tilting GoogleMaps -...

Brno Perl Mongers 28.5.2015 - Perl family by mj41

Discussant’s Remarks Tatsuhiko Kawashima, Gakushuin...

Shibuya Perl Mongers#12 No Sql Couch Db

SCT Endcap Module Initial Alignments Using Survey Data Paul....

SRITHAI MIYAGAWA CO ,LTD

MIRACLE MONGERS AND THEIR METHODS - Higher Intellect ·...

ASIAA Interferometry Summer School – 2006 Introduction –...

Leaders, managers and power mongers

strace for Perl Mongers

Tatsuhiko Matsushita PhD candidate, Victoria University of...

RabbitMQ for Perl mongers

Tatsuhiko Matsushita (University of Tokyo) Vocab@Vic 2013...

Offline Data Analysis Software Steve Snow, Paul S Miyagawa.....