Scaling xtext

Scaling XtextLieven Lemiengre

Sigasi

● IDE for Hardware Description Languages○ VHDL, (System)Verilog

● Using Xtext for 4 years

● Large user base○ (commercial, free, students)

Our company goal

● Assist hardware designer

● High quality interactive front-end compiler○ Instant error reporting

■ parsing, semantic, linting, style checking○ Visualisations, design exploration, ...○ Integrate with ecosystem

■ other compilers, simulators, synthesizers

The challenge

● Large projects○ > 250 KLOC is not uncommon○ design + external libraries○ big files

■ some libraries are distributed as 1 file■ generated hardware cores

●

Adopting Xtext

● Started with the early Xtext 2.0 snapshots

● Initial performance analysis○ Clean build performance of a big project (330k LOC)

■ > 20 minutes■ > 2 GB

○ Editing big files (> 1 MB)■ unusable

Adopting Xtext

● Started with the early Xtext 2.0 snapshots

● Initial performance analysis○ Clean build performance of a big project (330k LOC)

■ > 20 minutes → < 1 min■ > 2 GB → < 1 GB memory

○ Editing big files (> 1 MB)■ unusable → usable with reduced editor

● Xtext framework improvements

● Measure → analyze → improve or cheat○ faster build○ reduce memory usage○ UI responsiveness

Improving performance

Overview

● Analysing build performance○ Analyze the build

■ Macro build measurements■ Key performance points

○ Reduce workload○ Parallelize the build

● Analyzing UI issues○ Monitoring the UI thread○ Saveguards

Analyzing builds: builder overview

Global indexing

LinkingValidation

Custom Validation

Global index

Eclipse resources

warnings errorsresource

descriptions

Builder Participants

resource changes

?

Analyzing builds: metrics

● For each build○ # of files being build○ timing: Global index, Linking, Validation, Individual

builder participants● Instrument by overriding

ClusteringBuilderState & XtextBuilder● Example:

Building 134 resources, timing: {

global index=1806,

linking=378,

validation=823,

totalLinkingAndValidation=1364

}

Analyzing builds: resource loads

● Observation: ○ Most time spent in resource loads○ Certain files are loaded multiple times?!

Global indexing Linking Validation Builder

Participants

resources

LOAD

POTENTIAL RELOAD

POTENTIAL RELOAD

Analyzing builds: Memory footprint

Global index

ResourceSet

● Global index○ always loaded →limit size○ use hashing: name + hash of contents

● Reduce EMF sizederived model -> Xcore modelhttp://www.sigasi.com/content/view-complexity-your-xtext-ecore-model

Optimize loading

● What is resource load?○ Parse○ build EMF model○ build Node model

● Parallelise○ parse multiple files simultaneously○ ~3 time faster loads on 4 core machine○ EMF synchronization issues

● Cache○ serialize EMF and Node model in a cache○ 3-4 time faster loads

Linking

Global indexing Linking Validation Builder

Participants

● Language specific○ VHDL vs Verilog

● Avoiding linking○ expressions that we couldn’t handle (yet)○ library files, only when used in user-code

● Many iterations○ lazy linking vs eager linking○ From 40% of build time to 20%

Linking

Pseudo-algorithm● In your IScopePovider

○ Find a eContainer that can be linked eagerly○ Link everything

■ unresolved links become UnlinkedDeclaration

● In your DefaultLinkingService○ getLinkedObjects(...)

■ already resolved? → skip it

● Disable notifications

Validation

Global indexing Linking Validation Builder

Participants

● Combine validations to avoid model traversals

● Local analysis, do global validations moved into builder participant

● Avoid validation○ disabled validations○ libraries: errors & warnings are suppressed anyway

● Monitor

Track performance

● Nightly build● log build times

UI responsiveness

● Measuring: detect a blocked UI thread○ initially Svelto https://github.com/dragos/svelto○ now our own method & logging○ Eclipse Mars

● Improvements○ UI is for drawing only!○ Make sure everything is cancellable

● Safeguards○ certain services should never be executed on the UI

thread => check & log

Lightweight Editor (fallback)

● Syntax-highlighting + markers● For files > 1 MB● Based on ContentTypes extension point

Two ContentTypes (based on file size)

<extension point="org.eclipse.core.contenttype.contentTypes">

<content-type ...

describer="com.sigasi...FullVhdlContentDescriber"

name="VHDL editor"

<describer class="...FullVhdlContentDescriber" />

</content-type>

<content-type ...

describer="com.sigasi....LightweightVhdlContentDescriber"

name="Lightweight VHDL editor"

<describer class="...LightweightVhdlContentDescriber" />

</content-type>

</extension>

Future work

● Continuous process

● Cache global index info per resource?

● Linking without node model?

● Avoid project locking build (Xtext 2.8?)

Come talk to us about...

● Documentation generation● Fancy linking algorithms / type systems● Graphical views● Cross-language support● Testing Xtext-plugins● Lexical macros● Manage large amount of validations● ...

+1 0 -1

Sign in: www.eclipsecon.org

Evaluate the sessions