Smoking docker

Smoke & Mirrors:Containerizing Perl Testing

Steven LembarkWorkhorse [email protected]

What is “Smoking”?● High-level sanity check, not detailed analysis.

Does it catch fire when we power it on?

● CPAN runs tests as part of installation.

We are unusual that way.

● CPAN::Reporter reports results of tests.● CPAN::Reporter::Smoker builds with reporting.

Why do you smoke?● Verify personal modules.● Validate project platform.● Addict (CPAN smoker).

What do you smoke?● Whatever is in your sandbox.● A project and all of its supporting code.● All of CPAN, or the RECENTS file.

How do you smoke?● “make -wk -C myproject all test” in your sandbox.● Nightly build, git hooks, Travis CI.● CPAN or CPANM smoke testers.

Risks of Smoking● Pull in arbitrary code.

Maybe broken.

Possibly malicious.

More likely just stupid

system( rm -rf / *.foobar );

Second-hand Smoke● Permanent side effects.● Left over test files.● Wasted disk space.● Bandwidth.● Still possible with personal tests:

Dependent modules.

Protect Yourself● Install into a dedicated sandbox.● Disposable environment.

One approach: Virtual Machines● Heavy footprint.● Fully configure a running machine to run a test.● Inefficient use of hardware.

Lightweigit: “chroot”● Filesystem isolation only.● Tests can interfere with network, processes.● Files pollute permenant storage.● No resource limits.

Another approach: Process isolation.● Keep processes from interfering with one another.● Lightweight: Share logger, daemons, filesytems...● More efficient use of CPU.● Less work to set up.

● Not always easy to do right.

Jails & Zones

Better way: lxc● Approach via IBM.

Mainframe LPAR ported to linux.● Uses process groups.● Excellent control.

● One problem: People used them like VM's.● Two problem: lxc has hundreds of switches.

lxc for the Masses

● Docker: 80/20 of lxc.

Medium isolation.● Generate an “image”.

Read-only stack of AUFS mounts.● Run a “container”.

Mountpoint + resources + isolation.● Decent manglement tools.

AUFS with lxc

● Images are layered.

Read-only.

● "Container layer"

updated.

discarded.

Good and bad:

No permenant updates.

Not so nice● Images look good for smoking:

Read-only underneath: minimal harm.

Re-cycle O/S layer with Perly build.

Allows mix'n match testing.

● Catch:

No fixes or upgrades to smokebox.

Changes require complete rebuild.

One fix: Volumes● Described as “data volumes”.● Can store anything.● Like Perl, build directory, CPAN mirror.

● Mountpoints and symlinks.

Dir's or files from O/S into container.

● Mount RO or RW.

Maintain Perl & Smoker.

Hybrid approach● O/S is an image.

● Perl, CPAN on a volume.

O/S tools build and maintain perl.

perl maintains /var/lib/CPAN

perly volumes● Tempting: perlbrew dir's as volumes.

Problems with storage in /home.

Hard to share across users.

Storage depends on user account.

Multiple accounts duplicate space.

● Alternate: Let Docker manage the volumes

Simpler sharing.

/var/lib/docker [hopefully] has more space.

Manage via docker.

Perl Testers Wiki● Good instructions for CPAN::Reporter::Smoker.● Start with an O/S image and get a working smoker.● CPANM has its own version.

"minismokebox"

Grab a distro● Docker Hub has hundreds of distro images.● I'll use Gentoo here: it includes build environment.

No need to install gcc, make...

“stage3” is the starting point.

Installing Stage3● docker import $tarball into gentoo-stage3:YYYYMMDD.● Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.

root's .exrc .bashrc● Used to bulid perl, run smoke.

Building perl, smoker● Volumes for /opt, /var/tmp & script files.

Standard paths simplify build scripts.● “docker run” with command of build script.

build perl.

install & run minicpan.

configure reporting & smoker.

CPAN update to test reporting.

Aside: --tmpfs● docker run –tmpfs has a problem:

The volume has execution turned off.

Fine for data.

Useless for build.● Avoid the issue with a tmpfs volume.

Volume must be private to build.

Creating a tmpfs

● Private tmp volume via PID.

● tmpfs usually cleaned up automatically on the way out.

● Use volume rm to be sure.

tmp=”var-tmp-$BASHPID”;$docker volume create

--driver=local--opt type=tmpfs--opt device=tmpfs--opt o=size=320M$tmp

&&docker run

...-v $tmp:/var/tmp...

;# may fail due to non-existant volume.docker volume rm $tmp;

Volumes of Volumes

● Temp volume -v $tmp:/var/tmp

● Perl tarball -v $PWD/perl 5.24.1.tar.gz:/var/tmp/perl.tar.gz‑

● Build script -v $PWD/build-perl:/var/tmp/build-perl

● Perl install -v opt-perl-5.24.1:/opt

● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

Volumes of Volumes


● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz




Volumes of Volumes






Volumes of Volumes






Start the ball rolling

● Assemble a container.● Dispatch the build script.

● “-t” avoids issues with core test that uses column/row counts.

● “-a” gets output from container into log.

$docker volume create \ --driver=local \ --opt type=tmpfs \ --opt device=tmpfs \ --opt o=size=320M \ $tmp &&$docker volume create \ --driver=local \ $perl &&$docker run --rm \ -t \

--name=”build-$perl”\ -a stdout \ -a stderr \ "${vols[@]}" \ $image \ /var/tmp/build-perl \2>&1 | tee build.out;

docker volume rm $tmp;

Building Perl● The fun starts in /var/tmp/build-perl● Mostly here-scripts.● Install & configure smoketest engine.● Includes common non-smoke modules.

Building PerlHOME='/var/lib/CPAN';

( cd /var/tmp; gzip -dc < perl.tar.gz | tar xf -; cd perl*;

jobs=$(grep '^processor' /proc/cpuinfo | wc -l);export TEST_JOBS=$jobs;

./Configure \ -de \ -Dprefix=/opt/perl5 \ -Dman1dir='none' \ -Dman3dir='none' \ -Doptimize='-O2 -pipe -march=native' \ && /usr/bin/make -j$jobs all test_harness install;

) ||exit -1;

rm -rf /var/tmp/perl*;

● Generic “perl.tar.gz” via docker volume mapping.

● /opt/perl5 is in external volume.

● Don't knead man pages.

Building PerlHOME='/var/lib/CPAN';

● /home is in the O/S image!

Changes in container layer are discarded!!

● Persistent files go in /var/lib/CPAN or /opt.

Building Perl

TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness install;

Files=2410, Tests=851044, 223 wallclock secs(172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)

Aside: Testing perl builds● Use tmpfs for /opt instead of disk volume.

Fast, disposable storage for validating your build scripts.

● git clone <whatever> /scratch;

cd /scratch/whatever;

● docker run … -v /scratch/whatever:/var/tmp/whatever … ;

● build script does cd /var/tmp/whatever; make all test install;

Testing bleeding edge perl● Replace tar xvf with volume:

-v /gitclone/perl-5.25.X:/var/tmp/perl

● Rest of it stays the same.

Configure and make run quickly.

Minimal overhead if core modules don't change.

Lots “up to date” messages from CPAN.

Minor setups

● “h2ph” validates $PATH.

● Pick your remote server.

export PATH=/opt/perl5/bin:$PATH;

h2ph -r -l /usr/include || exit -1;

# at this point install seems usable

lib=$(ls -d /opt/perl5/lib/5* | tail -1);site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);

cpan='/var/lib/CPAN';build='/var/tmp/CPAN';

remote='http://mirror.uic.edu/CPAN/';local="$cpan/sources/";

CPAN uses /var/lib, /var/tmp

● Push CPAN config onto stable storage.

● CPAN config is specific to perl version.

perl -MCPAN -e 'shell'<<CPAN;yeso conf cpan_home $cpano conf histfile $cpan/histfileo conf prefs_dir $cpan/prefso conf build_dir $buildo conf keep_source_where $localo conf auto_commit yeso conf build_dir_reuse yeso conf check_sigs yeso conf inhibit_startup_message yeso conf halt_on_failure noo conf build_cache 1o conf index_expire 1o conf commitCPAN

TLC

● Update CPAN.● Deal with modules

that require user input, special choices.

(CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;

perl -MCPAN -e 'install CPAN';

perl -MCPAN -e shell <<CPAN;install Module::Signature2CPAN

AUTOMATED_TESTING=1; # avoid interactive prompt

perl -MCPAN -e 'shell <<CPAN;install Term::Readline::Perl

CPAN)

● Common.● Smoke modules.

● Iterate them with ${modules[@]}.

modules=( YAML::XS JSON::XS CPAN::SQLite IO::Socket::SSL Net::SSL Net::SSLeay HTTP::Date Test::Most Metabase::Resource Module::Version Module::Version Bundle::CPAN Log::Log4perl Bundle::CPAN CPAN::Mini CPAN::Reporter CPAN::Testers Test::RequiresInternet WWW::RobotRules LWP::UserAgent WWW::Mechanize CPAN::Reporter::Smoker Bundle::CPAN::Reporter::Smoker::Tests Bundle::CPANReporter2 App::SmokeBox::Mini App::cpanminus::reporter);

Other modules

Generate ID file for Reporter

● ~ is /var/lib/CPAN

Not /home!

( mkdir ~/.cpantesters; cd ~/.cpantesters; metabase-profile <<END;Steven [email protected]/secretEND chmod 0400 metabase_id.json;)

Configure CPAN::Reporter

# Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes# Would you like me configure CPAN::Reporter now? [yes] <enter># email_from? [] [email protected]# edit_report? [default:ask/no pass/na:no] <enter># send_report? [default:ask/yes pass/na:yes] <enter># transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] <enter># Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] <enter>

perl -MCPAN -e shell <<CPAN;o conf init [email protected]

CPAN

Voyeurism● Fixing the build uses “docker exec”.● Attach to a running container.● Does not require ssh.● Allows fixing, re-running the build.

Voyeurism● Fixing the build uses “docker exec”.● Attach to a running container.● Does not require ssh.● Allows fixing, re-running the build.

● Or just watching it:

docker exec -it build-perl-5.24.1 'top';

Access a running container● Say you get something like:

t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12.BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.# Looks like your test exited with 2 before it could output anything.t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)Failed 11/11 subtests

Access a running container● Say you get something like:

● Q: How do you fix it?

t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12.BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.# Looks like your test exited with 2 before it could output anything.t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)Failed 11/11 subtests

Access a running container● Exec into the container.● Use CPAN to install the module.● Which leaves you with:

docker exec -it build-perl-5.24.1 bash –-login;

Aside: Container ID● docker ps shows what is running.

Includes the container id & name:

$ docker ps;CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..." 28 minutes ago Up 28 minutes build-perl-5.24.1

# same result:

$ docker exec -it 493d9f285faf 'bash login';‑‑root@493d9f285faf ~ #

$ docker exec -it build-perl-5.4.21 'bash login';‑‑root@493d9f285faf ~ #

Through the looking glass● Inside the container you are root.

Kernel's proc struct has unchanged EUID.

● Limited view of the system.

Only container proc's in “ps” or “top”.

Nice on a crowded server.

Runlevel restrictions are ineffective!

● Run “perl -MCPAN” or “make install”.

Helpful if a non-essential test fails.

Through the Looking Glass● Outside docker you have a specific PID & EUID.● Inside docker the initial process has EUID 0 and PID 1.● EUID 0 is manageable.● PID 1 requires that the smoketest proc reap children.

Otherwise you end up with defunct proc's.

Will eventually eat all proc slots with long-running container.

One fix is Docker::Reaper.

Remember:● Changes to the image go to a separate AUFS layer.● They are discarded.● Fixing files in /etc/ will rescue this build, not the next one.

● Changes to /opt & /var/lib/CPAN are persistent.

TMTOWTDI: minicpan● Outside of docker.

Maintain CPAN cache in /var/lib.

● Inside docker.

Use /var/lib.

Running repeated jobs● Docker can automatically restart running jobs.● docker run -d restart=always ...‑‑

Downside: Lots of proc's started up at once.

Issues with tmpfs on system restarts.● --restart=unless-stopped

Allows stopping volumes on the way down.

Start singly on way back up.

Smoketest lifecyle● Run minicpan once on /var/lib/CPAN.

Global cache of modules.

● Run smoker with O/S images & perly vol's.

● This will only test *one* kernel!● Use VM's running docker for multi-kernel smoke tests.

Zombie apocolypse● Container startup command is process #1.● When it dies the container exits.● It has to reap all children.

Not a big issue for small, single-purpose containers.

Big problem for smoketests.

Fork & Reap

● Parent propagates signals.● Exits with $? from child process.● Loop reaps zombies.

Without this process slots will be exhausted.

If that happens: log on as SU and stop, rm the container.

for(;;) { my $pid = wait;

$pid != -1 or die "Lost child pid ($child)";

$pid != $child and next;

return $? }

Smoking

● Volumes for

perl

CPAN

/var/tmp● Run smoker.

Never exits.

vols=( "-v $perl:/opt:ro" "-v $cpan:$cpan" "-v $tmp:/var/tmp");( $docker volume create \ ... $tmp \ && exec $run \ -t \ --name="$cname" \ ${vols[@]} \ $image \ "/var/lib/CPAN/.cpantesters/docker-smoker") 2>&1 |tee $log;

Propsed change● Smoker processes recents and exit.

Simplifies iterating multiple perl verisons.

● Goal:

Product of images, perl vol's.

Automate bleed test.

Summary● Smoking can be hazardous.● LXC provides lightweight solution.● Mix O/S image, perl volume.

O/S is fixed.

Volume updated to maintain smoker.

● [email protected]:lembark/docker-smoker.git

Mostly shell kwikhaks.

Working to make it cleaner, perly.

Smoking docker

Technology