DARPA DARPA A Metrics System for Continuous A Metrics System for Continuous Improvement of Design Technology Improvement of Design Technology Andrew B. Kahng and Stefanus Andrew B. Kahng and Stefanus Mantik Mantik
Dec 19, 2015
DARPADARPA
A Metrics System for Continuous A Metrics System for Continuous Improvement of Design TechnologyImprovement of Design Technology
Andrew B. Kahng and Stefanus MantikAndrew B. Kahng and Stefanus Mantik
12/9/99
2
Motivation: Complexity of the Design ProcessMotivation: Complexity of the Design Process
Ability to make silicon has outpaced ability to design itAbility to make silicon has outpaced ability to design it Complex data, system interactionsComplex data, system interactions SOCSOC
more functionality and customization, in less timemore functionality and customization, in less timedesign at higher levels of abstraction, reuse existing design design at higher levels of abstraction, reuse existing design
componentscomponentscustomized circuitry must be developed predictably, with less riskcustomized circuitry must be developed predictably, with less risk
Key question: “Will the project succeed, i.e., finish on Key question: “Will the project succeed, i.e., finish on schedule and under budget while meeting performance schedule and under budget while meeting performance goals?”goals?”
SOC design requires an organized, optimized design SOC design requires an organized, optimized design processprocess
12/9/99
3
Value of CAD Tools Improvement Not ClearValue of CAD Tools Improvement Not Clear
What is $ value of a “better” scheduler, mapper, placer?What is $ value of a “better” scheduler, mapper, placer?
What is $ value of GUI, usability, …?What is $ value of GUI, usability, …?
What is the right objective?What is the right objective?min wirelength min wirelength routable routablemin literals min literals amenable to layout amenable to layout
Value well-defined only in context of overall design Value well-defined only in context of overall design processprocess
12/9/99
4
What is the Design Process?What is the Design Process?
Not like any “flow/methodology” bubble chartNot like any “flow/methodology” bubble chartbacks of envelopes, budgeting warsbacks of envelopes, budgeting warschanged specs, silent decisions, e-mails, lunch discussionschanged specs, silent decisions, e-mails, lunch discussionsad hoc assignments of people, tools to meet current needsad hoc assignments of people, tools to meet current needsproprietary databases, incompatible scripts/tools, platform-proprietary databases, incompatible scripts/tools, platform-
dependent GUIs, lack of usable standardsdependent GUIs, lack of usable standardsdesign managers operate on intuition, engineers focus on tool design managers operate on intuition, engineers focus on tool
shortcomingsshortcomings Why did it fail?Why did it fail?
““CAD tools”CAD tools”““inexperienced engineers”inexperienced engineers”
Must measure to diagnose, and diagnose to improveMust measure to diagnose, and diagnose to improve
12/9/99
5
What Should be Measured?What Should be Measured?
Many possibilitiesMany possibilitiesrunning a tool with wrong options, wrong subset of standardrunning a tool with wrong options, wrong subset of standardbug in a translator/readerbug in a translator/readerassignment of junior designer to project with multiple clocksassignment of junior designer to project with multiple clocksdifference between 300 MHz and 200 MHz in the specdifference between 300 MHz and 200 MHz in the specchanging an 18-bit adder into a 28-bit adder midstreamchanging an 18-bit adder into a 28-bit adder midstreamdecision to use domino in critical pathsdecision to use domino in critical pathsone group stops attending budget/floorplan meetingsone group stops attending budget/floorplan meetings
Solution: record everything, then mine the dataSolution: record everything, then mine the data
12/9/99
6
Design Process Data CollectionDesign Process Data Collection
What revision of what block was what tool called on?What revision of what block was what tool called on?by whom?by whom?when?when?how many times? With what keystrokes?how many times? With what keystrokes?
What happened within the tool as it ran?What happened within the tool as it ran?what was CPU/memory/solution quality?what was CPU/memory/solution quality?what were the key attributes of the instance?what were the key attributes of the instance?what iterations/branches were made, under what conditions?what iterations/branches were made, under what conditions?
What else was occurring in the project?What else was occurring in the project?e-mails, spec revisions, constraint and netlist changes, …e-mails, spec revisions, constraint and netlist changes, …
Everything is fair game; bound only by server bandwidthEverything is fair game; bound only by server bandwidth
12/9/99
7
Unlimited Range of Possible DiagnosesUnlimited Range of Possible Diagnoses
User performs same operation repeatedly with nearly User performs same operation repeatedly with nearly identical inputsidentical inputs tool is not acting as expectedtool is not acting as expectedsolution quality is poor, and knobs are being twiddledsolution quality is poor, and knobs are being twiddled
Email traffic in a project:Email traffic in a project:
missed deadline, missed revised deadline; people disengaged; missed deadline, missed revised deadline; people disengaged; project failedproject failed
On-line docs always open to particular pageOn-line docs always open to particular pagecommand/option is unclearcommand/option is unclear
12/9/99
8
METRICS System ArchitectureMETRICS System Architecture
Metrics Data Warehouse
Tool Tool Tool
xmitter xmitter xmitter
Data-Mining
Reporting
Inter/Intra-net
Server
JavaApplets
WebBrowsers
Wrapper orembedded
12/9/99
9
METRICS TransmitterMETRICS Transmitter
No functional change to the toolNo functional change to the tooluse API to send the available metricsuse API to send the available metrics
Low overheadLow overheadexample: standard-cell placer usingexample: standard-cell placer using
Metrics API Metrics API < 2% runtime overhead < 2% runtime overhead
even less overhead with bufferingeven less overhead with buffering
Won’t break the tool on transmittal failureWon’t break the tool on transmittal failurechild process handles transmission child process handles transmission
while parent process continues its jobwhile parent process continues its job
initToolRun()
sendMetrics()
sendMetrics()
sendMetrics()
sendMetrics()
12/9/99
10
METRICS TransmitterMETRICS Transmitter
EDATool
Tool wrapper
EDATool
API
JavaServlet Oracle8i
Inter/Intra-net
XML
XML
XML SQL
12/9/99
11
Transmitter ExampleTransmitter Example
/** API Example **//** API Example **/int main(int argc, char * argv[ ] )int main(int argc, char * argv[ ] ){{
......toolID = initToolRun( projectID, flowID );toolID = initToolRun( projectID, flowID );......printf( “Hello World\n” );printf( “Hello World\n” );sendMetric( projectID, flowID, toolID,sendMetric( projectID, flowID, toolID,
““TOOL_NAME”, “Sample” );TOOL_NAME”, “Sample” );sendMetric( projectID, flowID, toolID,sendMetric( projectID, flowID, toolID,
““TOOL_VERSION”, “1.0” );TOOL_VERSION”, “1.0” );......terminateToolRun( projectID, flowID,terminateToolRun( projectID, flowID,
toolID );toolID );return 0;return 0;
}}
## Wrapper example## Wrapper example( $File, $PID, $FID ) = @ARGV;( $File, $PID, $FID ) = @ARGV;$TID = `initToolRun $PID $FID`;$TID = `initToolRun $PID $FID`;open ( IN, “< $File” );open ( IN, “< $File” );while ( <IN> )while ( <IN> ){{
if ( /Begin\s+(\S+)\s+on\s+(\S+.*)/)if ( /Begin\s+(\S+)\s+on\s+(\S+.*)/){{
system “sendMetrics $PID $FID $TID \system “sendMetrics $PID $FID $TID \TOOL_NAME $1”;TOOL_NAME $1”;
system “sendMetrics $PID $FID $TID \system “sendMetrics $PID $FID $TID \START_TIME $2”;START_TIME $2”;
}}......
}}system “terminateToolRun $PID $FID \system “terminateToolRun $PID $FID \
$TID”;$TID”;
12/9/99
12
Example of METRICS XMLExample of METRICS XML
<? xml version=“1.0” ?><METRICSPACKET>
<REQUEST><TYPE> TOOL </TYPE><PROJECTID> 173 </PROJECTID><FLOWID> 9 </FLOWID><PARAMETER> 32 </PARAMETER>
</REQUEST><METRICS>
<PROJECTID> 173 </PROJECTID><FLOWID> 9 </FLOWID><TOOLID> P32 </TOOLID><DATETIME> 93762541300 </DATETIME><NAME> TOOL_NAME </NAME><VALUE> CongestionAnalysis </VALUE>
</METRICS></METRICSPACKET>
12/9/99
13
Current Testbed: A Metricized P&R FlowCurrent Testbed: A Metricized P&R Flow
LEF
DEF
Placed DEF
QP ECO
Legal DEF CongestionMap
WRoute
Capo Placer
Routed DEF
CongestionAnalysis
Incr. WRoute
Final DEF
METRICS
12/9/99
14
METRICS ReportingMETRICS Reporting
Web-based Web-based platform independentplatform independent
accessible from anywhereaccessible from anywhere
Example: correlation plots created on-the-flyExample: correlation plots created on-the-flyunderstand the relation between two metricsunderstand the relation between two metrics
find the importance of certain metrics to the flowfind the importance of certain metrics to the flow
always up-to-datealways up-to-date
12/9/99
15
METRICS ReportingMETRICS Reporting
JavaServlet Oracle8i
Inter/Intra-net
SQL
WEBBrowser
LocalGraphing
Tool(GNUPlot)
dataplot
Request
ReportRequest
Report
Request
Data
3rd PartyGraphing
Tool(Excel,Lotus)
Wrapper
Data
Future implementation
12/9/99
16
Example ReportsExample Reports
Congestion vs WL
# Via vs WL
12/9/99
17
METRICS StandardsMETRICS Standards
Standard metrics naming across toolsStandard metrics naming across toolssame name same name same meaning, independent of tool supplier same meaning, independent of tool supplier
generic metrics and tool-specific metricsgeneric metrics and tool-specific metrics
no more ad hoc, incomparable log filesno more ad hoc, incomparable log files
Standard schema for metrics databaseStandard schema for metrics database
12/9/99
18
Generic and Specific Tool MetricsGeneric and Specific Tool Metrics
tool_name chartool_version chartool_vendor charcompiled_date mm/dd/yyyystart_time hh:mm:ssend_time hh:mm:sstool_user charhost_name charhost_id charcpu_type charos_name charos_version charcpu_time hh:mm:ss
Generic Tool Metrics
num_cells integernum_nets integerlayout_size doublerow_utilization doublewirelength doubleweighted_wl double
num_layers integernum_violations integernum_vias integerwirelength doublewrong-way_wl doublemax_congestion double
Placement Tool Metrics
Routing Tool Metrics
12/9/99
19
Current StatusCurrent Status
Completion of METRICS server with Oracle8i, Completion of METRICS server with Oracle8i,
Servlet, and XML parserServlet, and XML parser
Initial transmittal API in C++Initial transmittal API in C++
METRICS wrapper for Cadence P&R toolsMETRICS wrapper for Cadence P&R tools
Simple reporting scheme for correlationsSimple reporting scheme for correlations
12/9/99
20
Additional InfrastructureAdditional Infrastructure
Industrial standard network discoveryIndustrial standard network discovery Jini, UPNP (Universal Plug & Play), SLP (Salutation)Jini, UPNP (Universal Plug & Play), SLP (Salutation)
SecuritySecurity encryption for XML dataencryption for XML data
SSL (Secure Socket Layer)SSL (Secure Socket Layer)
user id & password authentication (reporting)user id & password authentication (reporting)
registered users (transmitting)registered users (transmitting)
3rd party reporting tool3rd party reporting tool MS Office integration, Crystal report, …MS Office integration, Crystal report, …
Data miningData mining
12/9/99
21
METRICS DemoMETRICS Demo
Transmission of metricsTransmission of metricsAPI inside toolsAPI inside tools
Perl wrapper for log filesPerl wrapper for log files
ReportingReportingcorrelation reportscorrelation reports
progress on current tool run, flow, designprogress on current tool run, flow, design
12/9/99
22
Potential Benefits to Project ManagementPotential Benefits to Project Management
Accurate Resource Prediction At any point in Design CycleAccurate Resource Prediction At any point in Design Cycle up front estimates for People, Time, Technology, EDA Licenses, IP re-use...up front estimates for People, Time, Technology, EDA Licenses, IP re-use... go/no go at earliest pointgo/no go at earliest point
Accurate Project Post-mortemsAccurate Project Post-mortems Everything tracked - tools, flows, users, notesEverything tracked - tools, flows, users, notes Optimize for next Project based on past resultsOptimize for next Project based on past results No “loose”, random data or information left at Project end (log files!!!)No “loose”, random data or information left at Project end (log files!!!)
Management ConsoleManagement Console Web-based, status-at-a-glance of Tools, Designs, Systems at any point in Web-based, status-at-a-glance of Tools, Designs, Systems at any point in
projectproject
No wasted resourcesNo wasted resources prevent out of sync runsprevent out of sync runs no duplication of data or effortno duplication of data or effort
12/9/99
23
Potential Benefits to Tools R&DPotential Benefits to Tools R&D
Methodology for continuous tracking data over entire lifecycle of Methodology for continuous tracking data over entire lifecycle of
instrumented toolsinstrumented tools
More efficient analysis of realistic dataMore efficient analysis of realistic data no need to rely only on extrapolations of tiny artificial “benchmarks”no need to rely only on extrapolations of tiny artificial “benchmarks”
no need to collect source files for test cases, and re-run in houseno need to collect source files for test cases, and re-run in house
Facilitates identification of key design metrics, effects on toolsFacilitates identification of key design metrics, effects on tools standardized vocabulary, schema for design/instance attributesstandardized vocabulary, schema for design/instance attributes
Improves benchmarkingImproves benchmarking apples to apples, and what are the apples in the first placeapples to apples, and what are the apples in the first place
apples to oranges as well, given enough correlation researchapples to oranges as well, given enough correlation research
12/9/99
24
Potential Research Enabled by METRICSPotential Research Enabled by METRICS
Tools:Tools: scope of applicabilityscope of applicability predictabilitypredictability usabilityusability
Designs:Designs: difficulty of design or manufacturingdifficulty of design or manufacturing verifiability, debuggability/probe-abilityverifiability, debuggability/probe-ability likelihood of a bug escapelikelihood of a bug escape $ cost (function of design effort, integratability, migratability, ...)$ cost (function of design effort, integratability, migratability, ...)
Statistical metrics, time-varying metricsStatistical metrics, time-varying metrics What is the appropriate abstraction of manufacturing process for What is the appropriate abstraction of manufacturing process for
design?design? Impact of manufacturing on design productivityImpact of manufacturing on design productivity Inter- and intra-die variationInter- and intra-die variation Topography effectsTopography effects Impact, tradeoffs of newer lithography techniques and materialsImpact, tradeoffs of newer lithography techniques and materials
12/9/99
25
Ongoing WorkOngoing Work
Work with EDA, designer community to establish Work with EDA, designer community to establish standardsstandards tool users: list of metrics needed for design process optimizationtool users: list of metrics needed for design process optimization tool vendors: implementation of the metrics requested with the tool vendors: implementation of the metrics requested with the
standardized namingstandardized naming Improve the transmitterImprove the transmitter
add message bufferingadd message buffering““recovery” system for network / server failurerecovery” system for network / server failure
Extend METRICS system to include project management Extend METRICS system to include project management tools, email communications, etc.tools, email communications, etc.
Additional reports, data miningAdditional reports, data mining