Jul 04, 2015
What is DTrace?
• Dynamic tracing framework
• Works on Solaris/FreeBSD/OSX/QNX/Linux
!2
Who uses it?
• Apple’s Xcode Instruments
• Cloud Analytics in Joyent SmartDataCenter
• Oracle’s ZFS Storage Appliance analytics
• Brendan’s Flame Graphs
• Command line DTraceToolkit
!3
Xcode Instruments
!4
Cloud Analytics
!5
Oracle’s ZFS-SA analytics
!6
Flame Graphs
!7
DTraceToolkit
!8
Terminology
• probe
• provider
• module
!9
How it works?dtrace (1)
kernel space
user spacelibdtrace
DTrace
!10
/dev/dtrace
libproc
Providers
dtrace: kernel space
fbt:
io:
syscall:
tcp:
dtmalloc:
lockstat:
profile:
!11
and others:
dtrace(1)• Command line frontend
• Shows available probes/providers
• Enable probes and attach to processes
• D Language compiler
• Works via libdtrace and libproc
• ETC…!12
Available probes
!13
To see all probes on the system you need to run: # dtrace -l [ … many lines … ]
To see probes by dtrace provider: # dtrace -l -n 'dtrace:::' ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR
D program structure
provider:module:function:probe / predicates / { action; }
D language is like C and AWK:
!14
First D program
!15
# dtrace -q -n ' > syscall::open:entry { > printf("%s\t%s\n", execname, copyinstr(arg0)); > } > profile:::tick-1s { > exit(0); > } > ' !tmux /dev/null sh /etc/libmap.conf sh /var/run/ld-elf.so.hints sh /lib/libedit.so.7 sh /lib/libncurses.so.8 sh /lib/libc.so.7
Built-in Variables (not all)
args pid probeprov
errno probefunc timestamp
execname probemod vtimestamp
!16
Aggregating FunctionsFunction Name Arguments
count none
sum scalar expression
avg scalar expression
min scalar expression
max scalar expression
lquantize scalar expression, lower bound, upper bound, step value
quantize scalar expression
!17
Tracing applications
!18
• Trace with pid provider
• Trace with USDT (Userland Statically Defined Traces)
DTrace integration examples
Please clone repo (seeing code is required for next slides):
http://clck.ru/8wWyv
!19
int length_fast(char *); int length_slow(char *); int main(void) { int counter = 0; int c_loop = 0; char *Lines[] = { "x", "tt", "Test", NULL }; char *line; printf("%10s %4s\n", "STRING", "LEN"); while (15 > c_loop) { while (NULL != Lines[counter]) { line = Lines[counter]; printf("%10s: %02d\n", line, length_fast(line)); printf("%10s: %02d\n", line, length_slow(line)); ++counter; } ++c_loop; counter = 0; sleep(c_loop); } ! return 0; }
!20
#00 pid provider
!21
pid$target::length_*:entry { self->ts = timestamp; printf("String: %s\n", copyinstr(arg0)); }
00_script.sh
pid$target::length_*:return /self->ts/ { @num[probefunc] = count(); @sum[probefunc] = sum(timestamp - self->ts); @q[probefunc] = quantize(timestamp - self->ts); ! self->ts = 0; } !profile:::tick-10sec { exit(0); }
!22
END { printf("Count of call functions:\n"); printa(@num); printf("Time of call functions:\n"); printa(@sum); printf("Distribution of times by functions:\n"); printa(@q); }
!23
Count of call functions: ! length_fast 6 length_slow 6 !Time of call functions: ! length_fast 434394 length_slow 500954
Distribution of times by functions: ! length_fast value ------------- Distribution ------------- count 2048 | 0 4096 |@@@@@@@@@@@@@@@@@@@@ 3 8192 |@@@@@@@ 1 16384 | 0 32768 | 0 65536 | 0 131072 |@@@@@@@@@@@@@ 2 262144 | 0 ! length_slow value ------------- Distribution ------------- count 4096 | 0 8192 |@@@@@@@ 1 16384 |@@@@@@@ 1 32768 |@@@@@@@@@@@@@ 2 65536 | 0 131072 |@@@@@@@@@@@@@ 2 262144 | 0
result
!24
#01 First USDT providerdtrace_demo.d:
provider demo { probe start(); probe stop(); };
!25
Generate header file: # dtrace -h -s dtrace_demo.d
#01 Adding probes while (1) { if (DEMO_START_ENABLED()) DEMO_START(); while (NULL != Lines[counter]) { [ … lines … ] } counter = 0; sleep(1); if (DEMO_STOP_ENABLED()) DEMO_STOP(); }
!26
Compile applications
!27
Make object file: # clang -c main.c -DENABLE_DTRACE !Run dtrace(1) on object file: # dtrace -G -s dtrace_demo.d main.o !Link objects to application: # clang -lelf main.o dtrace_demo.o -o demo
#01 Trace application#!/usr/sbin/dtrace -Zs !#pragma D option quiet !demo*:::start { self->ts = timestamp; } !demo*:::stop { printf("time: %d\n", timestamp - self->ts); self->ts = 0; }
!28
01_script.sh# scripts/01_script.sh time: 1006057576 time: 1008003117 time: 1007984334 time: 1008002385 time: 1034989225 time: 1007995235 time: 1008000837 time: 1007991571 time: 1007992224 time: 1007998339
result
#02 Probes with arguments
provider demo { probe start(char *); probe stop(int); };
dtrace_demo.d:
!29
Generate header file: # dtrace -h -s dtrace_demo.d
#02 Adding probesint length_fast(char *string) { int i; ! if (DEMO_START_ENABLED()) DEMO_START(string); ! for (i = 0; '\0' != *string; ++i) { ++string; } if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i; }
!30
int length_slow(char *string) { int i; if (DEMO_START_ENABLED()) DEMO_START(string); ! for (i = 0; '\0' != *string; ++i) { ++string; usleep(i*i); } if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i; }
#!/usr/sbin/dtrace -Zs !#pragma D option quiet !BEGIN { self->start = timestamp; self->ts = 0; printf("Start tracing....\n\n"); } !demo*:::start { self->ts = timestamp; printf("String: %10s ", copyinstr(arg0)); }
!31
!demo*:::stop /self->ts/ { printf("len: %2d time: %6d \n", arg0, timestamp - self->ts); @[arg0] = quantize((timestamp - self->ts) / 1000); self->ts = 0; } !END { printf("Time %d.%d sec\n", (timestamp - self->start)/1000/1000/1000, (timestamp - self->start)/1000/1000/100); }
!32
# scripts/02_script.sh Start tracing.... !String: tt len: 2 time: 24425 String: Test len: 4 time: 5131 String: Test len: 4 time: 39085 String: x len: 1 time: 12314 String: x len: 1 time: 11881 String: tt len: 2 time: 4949 String: x len: 1 time: 11692 String: x len: 1 time: 8848 String: tt len: 2 time: 4430 String: tt len: 2 time: 24967 String: Test len: 4 time: 4561 String: Test len: 4 time: 37770 ^C Time 12.126 sec ![ … skipped distributions …]
result
#03 Evolution interface
!33
typedef struct { int dummy; } dline_t; typedef struct { int dummy; } demo_t; !provider demo { probe start(); probe stop(dline_t *dp) : (demo_t *dp); }; !#pragma D attributes Evolving/Evolving/ISA provider demo provider #pragma D attributes Private/Private/Unknown provider demo module #pragma D attributes Private/Private/Unknown provider demo function #pragma D attributes Private/Private/ISA provider demo name #pragma D attributes Evolving/Evolving/ISA provider demo args
dtrace_demo.d:
#03 Fill structureint length_fast(char *string) { dline_t data; int i; if (DEMO_START_ENABLED()) { data.mystr = string; DEMO_START(); } [ … some action … ] if (DEMO_STOP_ENABLED()) { data.len = i; DEMO_STOP(&data); } return i; }
typedef struct { const char *mystr; int len; } dline_t;
demo_provider.h:
!34
#03 First DTrace librarytypedef struct { char *mystr; int len; } dline_t; !typedef struct { string str; int len; } demo_t; !#pragma D binding "1.7" translator translator demo_t < dline_t *p > { str = copyinstr(*(uintptr_t *)copyin((uintptr_t)&p->mystr, sizeof (char*))); len = *(uint64_t *)copyin((uintptr_t)&p->len, sizeof (p->len)); };
!35
demo.d:
copy demo.d to /usr/lib/dtrace!or run:!# dtrace -L /path/to/dtrace/libs some args
#03 Access to structure
demo*:::stop /self->ts/ { printf("String: %10s len: %2d time: %6d\n", args[0]->str, args[0]->len, timestamp - self->ts); @[args[0]->str] = quantize((timestamp - self->ts) / 1000); self->ts = 0; }
03_script.sh:
!36
# scripts/03_script.sh Start tracing.... !String: x len: 1 time: 5178 String: x len: 1 time: 3485 String: tt len: 2 time: 1415 String: tt len: 2 time: 334482 String: Test len: 4 time: 1673 String: Test len: 4 time: 899032 ^C Time 2.20 sec [ … skipped distributions … ]
!37
result
libusdt
• Creates DTrace probes at runtime
• Bindings to Lua/Ruby/node.js/Perl
!38
libusdt and tarantool
Please clone repo recursively from dtrace branch:
http://clck.ru/8wX24See README.DTrace for more details
!39
lua usdt
provider = usdt.provider("init_script", "tarantool_box") probe = provider:probe("dt_hello", "entry", "char *", "int") provider:enable() !function dt_hello(mystr) probe:fire(mystr, string.len(mystr)) end
init.lua:
!40
# dtrace -l -n 'init_script*:::' ID PROVIDER MODULE FUNCTION NAME 249651 init_script18132 tarantool_box dt_hello entry
!41
# dtrace -q -n 'init_script*:::entry { printf("String %10s len: %02d\n", copyinstr(arg0), arg1) } tick-2s {exit(0);}' String Test len: 04 String tt len: 02 String xxx len: 03
localhost> for _, val in ipairs({"Test", "tt", "xxx"}) do dt_hello(val) end --- ...
Example
References• http://dtrace.org/
• http://wiki.FreeBSD.org/
• http://www.dtracebook.com
• http://docs.oracle.com/cd/E18752_01/html/819-5488/docinfo.html
• http://www.youtube.com/watch?v=i47XhPsAVTQ
• http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html
• http://www.joyent.com/developers/resources/cloud-analytics-in-joyent-smartdatacenter
• http://www.brendangregg.com/dtracetoolkit.html
• https://github.com/brendangregg/dtrace-cloud-tools
• https://github.com/brendangregg/FlameGraph
• https://github.com/davepacheco/mod_usdt
• https://github.com/chrisa/libusdt
• https://github.com/chrisa/lua-usdt