SherLog: Error Diagnosis by Connecting Clues from Run-time Logs Ding Yuan 1 Haohui Mai 1 Weiwei Xiong 1 Lin Tan 1 Yuanyuan Zhou 2 Shankar Pasupathy 3 1 University of Illinois at Urbana-Champaign 2 University of California, San Diego 3 NetApp, Inc. ASPLOS’10, March 13-17, 2010, Pittsburgh, Pennsylvania, USA. July 18, 2013 Lisong Guo (LIP6/REGAL) July 18, 2013 1 / 13
24
Embed
SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
SherLog: Error Diagnosis by Connecting Clues fromRun-time Logs
ASPLOS’10, March 13-17, 2010, Pittsburgh, Pennsylvania, USA.
July 18, 2013
Lisong Guo (LIP6/REGAL) July 18, 2013 1 / 13
Introduction
Scenario — Postmortem In-Production Debugging with Logspostmortem VS. prediction (model checking, static analysis etc.)in-production VS. in-house (runtime instrumentation etc.)log VS. others (bug reports, deployment configuration etc.)
Subtasks of Debuggingreproduce the bug (procedure non-related to source code)infer the failure-inducing execution pathinfer the conditions along the failure-inducing execution path
Research QuestionHow can we help a developer to debug in the scenario above ?
Lisong Guo (LIP6/REGAL) July 18, 2013 2 / 13
Introduction
Scenario — Postmortem In-Production Debugging with Logspostmortem VS. prediction (model checking, static analysis etc.)in-production VS. in-house (runtime instrumentation etc.)log VS. others (bug reports, deployment configuration etc.)
Subtasks of Debuggingreproduce the bug (procedure non-related to source code)infer the failure-inducing execution pathinfer the conditions along the failure-inducing execution path
Research QuestionHow can we help a developer to debug in the scenario above ?
Lisong Guo (LIP6/REGAL) July 18, 2013 2 / 13
Introduction
Scenario — Postmortem In-Production Debugging with Logspostmortem VS. prediction (model checking, static analysis etc.)in-production VS. in-house (runtime instrumentation etc.)log VS. others (bug reports, deployment configuration etc.)
Subtasks of Debuggingreproduce the bug (procedure non-related to source code)infer the failure-inducing execution pathinfer the conditions along the failure-inducing execution path
Research QuestionHow can we help a developer to debug in the scenario above ?
Lisong Guo (LIP6/REGAL) July 18, 2013 2 / 13
Approach
IdeaManual Inspection −→ Automatic Inference
A tool that takes the run-time logs and source code as inputs, and thenproduces some debugging hints for developers (i.e connecting the dots)
all possible and valid failure-inducing execution paththe evolution of value on certain variables along the inferred paths
Usage Scenariorun the tool to get a list of interesting pathsexamine the values of certain suspicious variables along some pathrepeat the previous step until the root cause of the bug is found
Lisong Guo (LIP6/REGAL) July 18, 2013 3 / 13
Approach
IdeaManual Inspection −→ Automatic Inference
A tool that takes the run-time logs and source code as inputs, and thenproduces some debugging hints for developers (i.e connecting the dots)
all possible and valid failure-inducing execution paththe evolution of value on certain variables along the inferred paths
Usage Scenariorun the tool to get a list of interesting pathsexamine the values of certain suspicious variables along some pathrepeat the previous step until the root cause of the bug is found
Lisong Guo (LIP6/REGAL) July 18, 2013 3 / 13
Design
ComponentsLog Parsing: locate the logging statements in the source codePath Inference: infer the failure execution paths and the constraintsValue Inference: infer the value evaluation along the given paths
Lisong Guo (LIP6/REGAL) July 18, 2013 4 / 13
Log Parsing
ObjectivesIdentifying the Logging Points and Variables in the source code.
Objectivesinfer the failure-inducing execution pathinfer the constraints of variables along the path
Constrained Sequence Matching Problem (NP-Complete?)based on Saturn, a static analysis framework for C programsmatch the control & data flow with the sequence of log messagesconvert the path searching problem into a set of declarative rules
Glance of Implementationcustomized control-flow: main → log@4 → b1@10 → c@16 → log@25
Objectivesinfer the failure-inducing execution pathinfer the constraints of variables along the path
Constrained Sequence Matching Problem (NP-Complete?)based on Saturn, a static analysis framework for C programsmatch the control & data flow with the sequence of log messagesconvert the path searching problem into a set of declarative rules
Glance of Implementationcustomized control-flow: main → log@4 → b1@10 → c@16 → log@25
Objectivesinfer the failure-inducing execution pathinfer the constraints of variables along the path
Constrained Sequence Matching Problem (NP-Complete?)based on Saturn, a static analysis framework for C programsmatch the control & data flow with the sequence of log messagesconvert the path searching problem into a set of declarative rules
Glance of Implementationcustomized control-flow: main → log@4 → b1@10 → c@16 → log@25
Limitationsskip the analysis on the functions of non-log-generating
I therefore it might return incorrect resultsno alias analysis for pointers
I but the underlining framework Saturn support alias analysisspecial treatments on some external routines/functions
I abort, exit, setjmp, longjmp etc.
Lisong Guo (LIP6/REGAL) July 18, 2013 8 / 13
Value Inference
Objectiveinfer the value-flow of certain variables, given the execution paths
Algorithmmodel the assignment relationship among memory locations asguarded points-to graph
I predicate(position, variable, value, constraint)symbolically execute the inferred failure path forwards
I refine the constraint according to the scope of variablesI incrementally update the graph at each step
generate the sequence of value evolution (value-flow)
Lisong Guo (LIP6/REGAL) July 18, 2013 9 / 13
Value Inference
Objectiveinfer the value-flow of certain variables, given the execution paths
Algorithmmodel the assignment relationship among memory locations asguarded points-to graph
I predicate(position, variable, value, constraint)symbolically execute the inferred failure path forwards
I refine the constraint according to the scope of variablesI incrementally update the graph at each step
generate the sequence of value evolution (value-flow)
Lisong Guo (LIP6/REGAL) July 18, 2013 9 / 13
Evaluation
Methodologymanually reproduce and diagnose the real-world bugscollect path summaries at runtimecompare the result of SherLog with the reproduction information
Metricsuseful: SherLog infers a subset of bug reproduction informationcomplete: SherLog infers all the bug reproduction information
Lisong Guo (LIP6/REGAL) July 18, 2013 10 / 13
Evaluation
Methodologymanually reproduce and diagnose the real-world bugscollect path summaries at runtimecompare the result of SherLog with the reproduction information
Metricsuseful: SherLog infers a subset of bug reproduction informationcomplete: SherLog infers all the bug reproduction information
Lisong Guo (LIP6/REGAL) July 18, 2013 10 / 13
Evaluation
Methodologymanually reproduce and diagnose the real-world bugscollect path summaries at runtimecompare the result of SherLog with the reproduction information
Metricsuseful: SherLog infers a subset of bug reproduction informationcomplete: SherLog infers all the bug reproduction information
Lisong Guo (LIP6/REGAL) July 18, 2013 10 / 13
Evaluation
Lisong Guo (LIP6/REGAL) July 18, 2013 11 / 13
Assumptions/Limitations
Assumptionssufficient logging messages
I reasonable density distribution of logging statementsI reasonable amount of logging statements being activated
well-match between the bug manifestation path and the log tracesequential and single-threaded log messages
I cannot handle multi-thread concurrent program
Technical Limitationsskip the functions that do not involve in log productiondo not parse the complex constructs of C programming language, suchas pointer arithmetics
Adam J. Oliner, Archana Ganapathi, Wei Xu: Advances and challenges inlog analysis. Commun. ACM 2012
Ding Yuan, Jing Zheng, Soyeon Park, Yuanyuan Zhou, Stefan Savage:Improving Software Diagnosability via Log Enhancement. ASPLSO 2011
Wei Xu, Ling Huang, Armando Fox, David A. Patterson, Michael I. Jordan:Detecting Large-Scale System Problems by Mining Console Logs. ICML 2010
Thomas Reidemeister, Mohammad Ahmad Munawar, Miao Jiang, Paul A. S.Ward: Diagnosis of recurrent faults using log files. CASCON 2009
Trishul M. Chilimbi, Ben Liblit, Krishna Mehra, Aditya V. Nori, and KapilVaswani: HOLMES: Effective statistical debugging via efficient pathprofiling. ICSE 2009