Build an IDE - Eclipse 2009

© 2007 by «Author»; made available under the EPL v1.0 | Date | Other Information, if necessary© 2009 by IBM; made available under the EPL v1.0 | Mar 23, 2009

PIMP Your Eclipse: Building an IDE using IMP

Robert M. Fuhrer, Philippe Charles,Stanley M. Sutton Jr.(IBM T. J. Watson Research Center)Jurgen Vinju(Centrum Wiskunde en Informatica)

2 Presentation Title | Presentation Subtitle | © 2007 by «Author»; made available under the EPL v1.0 PIMP Your Eclipse: Building an IDE using IMP | © 2009 by IBM; made available under the EPL v1.02

Outline

Introduction IDE Development Process Overview (20 mins) The Target Language (5 mins) Interactive Exercises: Developing an IDE (2 hours) Advanced Topics (30 mins) IMP Architectural Overview (10 mins) Where to go from here


IMP Goals

Facilitate the development of robust, high-functionality, language-specific IDEs in Eclipse

Enable language developers and users to get on with the interestingwork

Ultimately: make it easier to approximate the functionality of theEclipse JDT for other languages


IMP Target: Desired IDE Functionality

New Project/Type/… creation wizards refactoring launch & debug: launch configs,

breakpoints, backtraces, values,evaluation

Eclipse JDT, IntelliJ set a very high bar!

syntax highlighting, compiler annotations,hover help, source folding, formatting…

structural views

navigation (hyperlinks, “Open Type”, …)

content assist, quick fixes

compiler w/ incremental build,automatic dependency tracking


Nicely-formatted rich text hovers for documentation of given entity

Quick fixes: “Remove catch clause” when no one throws the corresponding exception

“Change method signature” – add a formal parameter of the appropriate type

“Add package to Export-Package directive in bundle manifest”

Refactorings: “Introduce Parameter” – make selected expression within a method a parameter at

the method’s call sites

“Infer Type Arguments” – migrate application to generics by determining theappropriate type parameters for each instance of an already generic type

“Introduce Type Parameter” – help make existing classes generic

Raise the level of abstraction at which programmer interacts with the code

void foo() { try { // doesn’t throw Foo } catch (Foo f) { // … }}

void foo() { // M not exported M m = new M();}

The “High Bar” of Language-specific Functionality

void foo() { //…}void bar() { foo(3);}

void foo() { // doesn’t throw Foo}

void foo(int i) { //…}void bar() { foo(3);}

<Manifest.mf>// …Export-Packages: org.foo.MPackage

void foo() { bar(3);}void bar(int N) { for(int i=0; i < N; i++){ //… }}

void foo() { bar();}void bar() { for(int i=0; i < 3; i++){ //… }}

void foo() { List<String> l= new ArrayList<String>(); l.add(“hi”); String s; s= /*(String)*/l.get(0);}

void foo() { List l= new ArrayList(); l.add(“hi”); String s; s= (String) l.get(0);}

class Cell<T> { T data; T getData() { return data; } void setData(T o) { data= o; }}

class Cell { Object data; Object getData() { return data; } void setData(Object o) { data= o; }}


IMP IDE for LPG

Source text isLPG grammarspecification

Navigableoutline ofgrammarsections

Hoverannotations

Syntaxcoloring

Hyperlinking(not shown)

Parsing anderror reporting

Sourcefolding

And there’s more


IMP Approach Keep IDE developer focused on language concepts

data structures (e.g. ASTs) and processing, not Eclipse UI componentry Leverage common themes, structures, concepts, semantics

Encapsulate common views/behavior within IMP runtime framework Define responsibilities for language-specific behavior as “IDE services” or

“language services” with corresponding extension points & APIs Encapsulate language and language processing concepts within IMP APIs

Remain technology-agnostic: parsing, analysis, … Design for customization Use meta-tooling to reify the IDE development flow

Templates/generators/wizards for IDE services Language-definition support (grammars, parsers)

Ease the IDE development process Generate language ASTs and parsers from grammar (e.g., LPG) Direct developer focus to relevant APIs and customization sites Provide DSLs to ease service implementations (preferences, presentation, etc.) Cheat sheets, documentation, default implementations, examples

Enable IDE developers to get on with the interesting work!


IMP & the Eclipse Plug-in Architecture Plug-in – set of contributions

Your IDE: one or more of these Smallest unit of Eclipse functionality Big example: HTML editor Small example: action that creates zip files

Extension point – named entity for collectingcontributions

Example: extension point for workbenchpreference UI

Generally, one per IMP language/IDE service

Extension – a contribution (an IMP service impl) Example: IMP outline tree model builder

IMP Runtimeplug-in

IDE plug-in A

extension (svc impl)extension point

IDE plug-in B


Installing IMP

As of 3/23/09, we recommend either 3.3.2 or 3.4.2

Many issues w/ 3.4.2 and p2 installs/updates, so please (for now): Use fresh 3.4.2 install

Install only IMP and IMP-related features in one shot

Vote for bugzilla bug on 3.4.2 p2:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=268619


Installing IMP - Eclipse 3.3


Installing IMP - Eclipse 3.3 (cont.)






Installing IMP - Eclipse 3.3 (cont.) - Select Features

Except: IMP Program AnalysisUI depends on Zest (a graphlayout engine) - which is notpart of 3.3 Classic, so disablethis unless you installed itpreviously…

Select everything on theIMP Update Site!


Installing IMP - Eclipse 3.3 (cont.) - Licenses


Installing IMP - Eclipse 3.3 (cont.) - Go for it!






Installing IMP - Eclipse 3.4 (cont.) - Add Site


Installing IMP - Eclipse 3.4 (cont.) - Select Features


Installing IMP - Eclipse 3.4 (cont.) - Confirm


Installing IMP - Eclipse 3.4 (cont.) - Licenses


Outline

Introduction IDE Development Process Overview The Target Language Interactive Exercises: Developing an IDE Advanced Topics IMP Architectural Overview Where to go from here


IMP Development Process Overview Define “language descriptor”

Identify base language (if any), file name extensions, unique ID to locatecorresponding service implementations …

Define (or reuse) lexical and grammar specifications Used as basis for many language/IDE services Use any parser generator you like! (LPG, SDF, Antlr, …) IMP includes LPG tooling: creates a grammar skeleton; you complete it; ASTs

auto-generated Scannerless parsing is an option (for tricky languages like JavaScript, AspectJ) or: be macho and do it all yourself (with a little help from IMP)

Define language service implementations Wizard/compiler creates plugin metadata to hook into framework Wizard inserts example code for a simple language (not derived from AST, yet) Complete service implementation as desired In any order, though a few constraints (e.g., ref resolver before content assist)

IDE is usable at each step!


IMP Development Process Overview (cont.)

Lather, rinse, repeat,…

Some IDE services are “factored” ⇒ multiple levels of customization more investment ⇒ more language-specific behavior

IDE developer chooses a point in the spectrum from generic to fully custom

E.g. for hover help, if the IDE developer: does nothing

user gets trivial syntactic feedback on the kind of construct (feh) supplies a reference resolver

user gets a presentation of the referenced entity (can be good enough) supplies a documentation provider

user gets the documentation on the referenced entity (better) etc.


IMP Development Process Overview: Cheat Sheets

“Click toPerform”

Descriptionof stepProcess

overview

Initiateeach step


IMP Development Process Overview: User-Visible “IDEServices” Source editor Content assistance Hover help Outline view Quick outline Token coloring Indexed search (via PDB)° Compiler/tool annotations and

markers Hyperlinking Auto-editing Source folding

Annotation hover Formatting View filtering° Refactoring contributions Editor action contributions Preference pages Call graph° Type hierarchy°

° Under development


* no coding required ° under development

IMP Development Process: IDE DeveloperResponsibilities Language description* Parser; message handling Content Proposer Token Colorer Folding Updater Model Listener Index Contributor ° Auto-edit Strategy Editor actions Formatter

Structured source differ° Reference resolvers Incremental builder Compiler Nature enabler* Label Provider Image Decorator Outline content provider Refactoring contributions Documentation provider Preference service and pages

(via simple DSL)

All services are optional!


Outline

Introduction IDE Development Process OverviewThe Target Language Interactive Exercises: Developing an IDE Advanced Topics IMP Architectural Overview Where to go from here


The Target Language - “LEG”

The “Little Expression Grammar” Imperative, procedural, block-structured language Minimal control structures: if-then-else, while Type variable decls, assignments Only simple scalar types: int, double, boolean Functions with formal parameters, return statements Entry point: main()

int factorial(int n) { if (n <= 1) return 1; return n * factorial(n-1);}int main() { return factorial(10);}


Outline



Tutorial Exercises Exercise 1: Prep’ing the Chassis

a) Registering your language w/ IMPb) Grammar & lexerc) Launch time!d) Grammar mods

Exercise 2: Wheels and Chromea) Syntax coloringb) Defining the outlinec) Making source foldable

Exercise 3: Engine Modsa) Resolving referencesb) Providing content assistancec) Providing an incremental builderd) Supporting ‘Mark Occurrences’

Exercise 4: Fuzzy Dicea) Enhancing hover helpb) Preference pages

Advanced Topics (time allowing): Source code formatting

Presentation specifications

Error recovery

Providing a custom editor name &icon

Using other kinds ofparser/compiler front ends

Simple breakpoint support

Deployment the Easy Way


Exercise 1: Prep’ing the Chassis


Exercise 0: Create a New Plugin Project


Exercise 0:Create a Plugin Project(cont.)


Exercise 0:Create a Plugin Project(cont.)


Exercise 0: Switch to the IMP IDE Perspective


Exercise 1a: Registering Your Language with IMP

New ->Programming

languagedescriptor

Languageproject and

IDFilename

extensions Parentlanguage

(leave blank)


Exercise 1a: Registering Your Language with IMP (cont.)

As a side-effect, this will also generate a new “bundle activator” class Overwrites the one generated by the “New Plugin Project” wizard

Adds functionality such as access to language ID, plugin log, etc.

Nothing interesting in the previously-generated one, so just accept the“nag box”


New extension oforg.eclipse.imp.runtime.languageDescription

Result of Creating New Language Descriptor


Exercise 1b: Parsing/Lexing: The Grammar and Lexer Specs

ParseController

class

Languageoptions

Auto-generate

AST classes


Exercise 1b: Anatomy of an LPG Grammar

%Options - package decl, enable AST gen, etc. - typically keep as is

%Globals - placed at beginning of generated Parser source file, e.g.additional imports

%Terminals - declare aliases for terminals that invalid Java™ symbols

%Start - identify the top-level grammar construct - usually somethinglike “sourceFile”

%Rules - the grammar rules - the good stuff

%Headers - additional Parser class members, nested classes

-- this is a comment in an LPG grammar

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.


Exercise 1b: Anatomy of an LPG Grammar (cont.) declaration ::= primitiveType identifier

compilationUnit$$functionDeclaration ::= %empty | compilationUnit functionDeclaration

identifier ::= IDENTIFIER /. IAst decl; public void setDeclaration(IAst decl) { this.decl = decl; } public IAst getDeclaration() { return decl; } ./

standard left-recursive rulefor a possibly-empty list

grammar annotation torequest ArrayList-likeAPI for generated AST

additional members forgenerated Java rule class

1 interface per non-terminal

1 class per rule


Exercise 1b: Anatomy of an LPG Lexer

%Options - package decl, etc. - typically keep as is

%Globals - additional imports

%Export - identify the terminal symbols to be exported to the parser

%Terminals - aliases for characters (optional)

%Start - ‘Token’ - typically keep as is

%Rules - the token syntax - same formalism/power as for grammars!(more powerful than regexps used by scanner generators like lex/flex)

-- this is a comment in an LPG grammar


Exercise 1b: Anatomy of an LPG Lexer (cont.)

Token ::= identifier /. checkForKeyWord(); ./ | number /. makeToken($_NUMBER); ./ | DoubleLiteral /. makeToken($_DoubleLiteral); ./ | white /. skipToken(); ./ | slc /. makeComment($_SINGLE_LINE_COMMENT); ./

number ::= digit | number digit

identifier ::= letter | identifier letter | identifier digit

%EXPORT Foo ➜ makeToken($_Foo)

syntax of a NUMBER token

valid token types

create “adjunct” token


Exercise 1b: Potential Issues Running LPG Builder

How to know if LPG builder ran correctly: LEGParser.java, LEGLexer.java, etc. generated in same package as grammar Package created containing generated AST classes

1. LPG Builder may not work when Eclipse is installed in a path that containsembedded spaces.• Remedy: Move Eclipse install folder to a path without embedded spaces

2. LPG executable doesn’t run properly• Cause: On Mac OS and Linux(?), the LPG executable file is not always marked as

executable on the file system.• Remedy: Run

chmod a+x /path/to/lpg-macosx_x86

on the file identified in the Error Log View


Result of Creating New Grammar & Parser

Sample grammartemplate instantiated

and opened —contains example

specificationsParser/AST classes

auto-generatedby LPG Builder

Developer pointedto starting point

for customization


Exercise 1c: Running Your IDE


Exercise 1c (cont.): Running Your IDE



Makes sure your plugindependencies are correctbefore launching (avoidsClassNotFoundExceptions)



Platform-specific LPG pluginfragments (whichever onesaren’t intended for theplatform you’re running)

Annotation-processingplugins (only valid forJava 6.0)

Trim your launch configto avoid irrelevantvalidation complaints






Running LEG Editor with Live Parsing

Project withsource

Source textwith

parse errorannotations

It’s alive!


Exercise 1d: Enhancing the Grammar

Add support for the usual comparison operators “<=“ and “>=“ to thelexer and grammar, which our definition of factorial() uses


Exercise 1d (cont.): Enhancing the Grammar (soln.)

Add to LegParser.g:

%Terminals LESS_EQUAL ::= '<=’ GREATER_EQUAL ::= ’>=’%End

-- …

%Rules | expression ’<=' term | expression ’>=' term%End

Add to LegLexer.gi:%Export -- … GREATER_EQUAL LESS_EQUAL%End

%Rules | ’<' '=’ /.makeToken($_LESS_EQUAL); ./ | ’>' '=’ /.makeToken($_GREATER_EQUAL); ./%End


Exercise 2: Wheels and Chrome


Exercise 2a: Token Coloring


Exercise 2a: Token Coloring (cont.)public class LEGColorer extends TokenColorerBase implements ITokenColorer, LEGParsersym {

protected final TextAttribute identifierAttr, numberAttr, keywordAttr; // …

public LEGColorer() { super(); Display d = Display.getDefault(); numberAttr = new TextAttribute(d.getSystemColor(SWT.COLOR_DARK_BLUE), null, SWT.BOLD); keywordAttr = new TextAttribute(d.getSystemColor(SWT.COLOR_DARK_RED), null, SWT.ITALIC);}

public TextAttribute getColoring(IParseController controller, Object o) { IToken token= (IToken) o;

switch (token.getKind()) {

case TK_IDENTIFIER: return identifierAttr; case TK_NUMBER: return numberAttr; // … default: if (((SimpleLPGParseController) controller).isKeyword(token.getKind())) return keywordAttr; return super.getColoring(controller, token); }}

public IRegion calculateDamageExtent(IRegion seed) { return seed;}

}

N.B.: Colors are “managed system resources” - very limited in number.IMPORTANT: Use either system Colors or Colors from a fixed palette.If too many Colors are instantiated, Eclipse may crash!

returns null for the default font

or, return an expanded region to re-color

or whatever things your getTokenIterator() returns

token kind symbols defined by lexer spec

good resource: http://colorbrewer.org


Exercise 2a (cont.): Enhancing the Token Colorer

Add appropriate code to LegTokenColorer to handle single-linecomments and doubles

Hints: Add a TextAttribute field decl for the comment attribute and initialize it

Use numberAttribute for doubles

Add cases to the switch statement in getColoring()

Use JDT hyperlinking (or “Find Declaration”) to locate TK_IDENTIFIER inorder to find the appropriate token symbols to use


Exercise 2a (cont.): Enhancing the Token Colorer (soln.)public class LEGTokenColorer implements ITokenColorer { TextAttribute commentAttribute; // … public TokenColorer() { // … commentAttribute = new TextAttribute( display.getSystemColor(SWT.COLOR_GREEN), null, SWT.ITALIC); // … } public TextAttribute getColoring(IParseController ctlr, Object o) { // … switch (token.getKind()) { // … case TK_DoubleLiteral: return numberAttribute; case TK_SINGLE_LINE_COMMENT: return commentAttribute; // … } }

}


Result of New Token Colorer

Implementationtemplate

instantiatedand opened

Contains exampleimplementation —returns appropriateTextAttribute based

on token type

Developer pointedto starting point

for customization


Exercise 2b:Defining the Outline


Exercise 2b: Defining an Outline (Model Builder)public class LEGModelBuilder extends TreeModelBuilderBase {

public void visitTree(Object root) { if (root == null) return; ASTNode rootNode= (ASTNode) root; LEGModelVisitor visitor= new LEGModelVisitor();

rootNode.accept(visitor); } private class LEGModelVisitor extends AbstractVisitor { // … public boolean visit(block n) { pushSubItem(n); return true; }

public void endVisit(block n) { popSubItem(); }

public boolean visit(declarationStmt0 n) { createSubItem(n); return true; } }

}

create item w/ children

“close” item w/ children

create item w/ no children

generated, but if you useanother kind of visitor…

LPG-generated visitor class

continue visiting sub-structure


Exercise 2b: Enhancing the Model Builder

Add functionDeclaration nodes to the model

Hints: Add visit() and endVisit() overrides to the visitor class for

functionDeclaration, and call pushSubItem() andpopSubItem()


Exercise 2b: Defining an Outline (Label Provider)

public class LEGLabelProvider implements ILabelProvider {

private static ImageRegistry sImageRegistry = Activator.getInstance().getImageRegistry();

private static Image DEFAULT_IMAGE = sImageRegistry.get(ImageResources.LEG_DEFAULT_IMAGE);

private static Image FILE_IMAGE = sImageRegistry.get(ImageResources.LEG_FILE); private static Image FILE_WITH_WARNING_IMAGE = sImageRegistry.get(ImageResources.LEG_FILE_WARNING); private static Image FILE_WITH_ERROR_IMAGE = sImageRegistry.get(ImageResources.LEG_FILE_ERROR);

// …

A: image resource management

allocate image resources with care -use an ImageRegistry (as provided byIMP-generated bundle activator class)


Exercise 2b: Defining an Outline (Label Provider) public Image getImage(Object element) {

if (element instanceof ISourceEntity) { ISourceEntity entity= (ISourceEntity) element; return getImageFor(entity.getResource()); } if (element instanceof IResource) { return getImageFor((IResource) element); } ASTNode n= (element instanceof ModelTreeNode) ? (ASTNode) ((ModelTreeNode) element).getASTNode() : (ASTNode) element; return getImageFor(n); }

public Image getImageFor(IResource res) { if (res instanceof IFile) { int sev = MarkerUtils.getMaxProblemMarkerSeverity((Ifile) res, IResource.DEPTH_ONE); switch(sev) { case IMarker.SEVERITY_ERROR: return FILE_WITH_ERROR_IMAGE; case IMarker.SEVERITY_WARNING: return FILE_WITH_WARNING_IMAGE; default: return FILE_IMAGE; } } return null; // use default image }

B: providing images

IMP Project Explorer

Package Explorer, Navigator

IMP outline page

decorate files w/ marker indicators


Exercise 2b: Defining an Outline (Label Provider)

public static Image getImageFor(ASTNode n) { return DEFAULT_IMAGE; }

B: providing images


Exercise 2b: Defining an Outline (Label Provider) public String getText(Object element) { ASTNode n= (element instanceof ModelTreeNode) ? (ASTNode) ((ModelTreeNode) element).getASTNode() : (ASTNode) element;

if (n instanceof IcompilationUnit) return "Compilation unit"; if (n instanceof block) return "Block"; // … if (n instanceof functionDeclaration) { functionHeader hdr = (functionHeader) ((functionDeclaration) n).getfunctionHeader(); return hdr.getidentifier().toString(); } return "<???>"; }

C: providing text labels


Exercise 2b: Enhancing the Label Provider

Add formal argument types to the label produced forfunctionDeclarations

Add custom icons for functionDeclarations and for the main()function.

Hints: Copy & modify icons from the “icons” folder of the LEG project to create

2 new icons (e.g. “funcDecl.gif” and “mainDecl.gif”)

Copy the image resource maintenance code in the bundle Activator classand ILEGResources interface for these 2 new icons.

Add case to LEGLabelProvider.getImageFor(ASTNode n) forfunctionDeclaration


Exercise 2b: Enhancing the Label Provider (soln.) public String getText(Object element) { ASTNode n= (element instanceof ModelTreeNode) ? (ASTNode) ((ModelTreeNode) element).getASTNode() : (ASTNode) element; // … if (n instanceof functionDeclaration) { functionHeader hdr = (functionHeader) ((functionDeclaration) n).getfunctionHeader(); StringBuilder sb= new StringBuilder(); sb.append(hdr.getType()); sb.append(" "); sb.append(hdr.getidentifier().toString()); sb.append("("); declarationList parms = hdr.getparameters(); for(int i=0; i < parms.size(); i++) { if (i > 0) sb.append(", "); sb.append(parms.getdeclarationAt(i).getprimitiveType()); } sb.append(")"); return sb.toString(); } // … }


Exercise 2b: Enhancing the Label Provider (soln.)public interface ILEGResources { // … public static final String FUNC_DECL= "funcDecl"; public static final String MAIN_DECL= "mainDecl"; // …

}

public class Activator extends PluginBase { protected void initializeImageRegistry(ImageRegistry reg) { // … path= ICONS_PATH.append(”funcDecl.gif"); imageDescriptor= createImageDescriptor(bundle, path); reg.put(ILEGResources.FUNC_DECL, imageDescriptor);

path= ICONS_PATH.append(”mainDecl.gif"); imageDescriptor= createImageDescriptor(bundle, path); reg.put(ILEGResources.MAIN_DECL, imageDescriptor); }

}


Exercise 2b: Enhancing the Label Provider (soln.)public class LEGLabelProvider implements ILabelProvider { // … private static Image FUNC_DECL_IMAGE = sImageRegistry.get(ILEGResources.FUNC_DECL); private static Image MAIN_DECL_IMAGE = sImageRegistry.get(ILEGResources.MAIN_DECL); // … public static Image getImageFor(ASTNode n) { if (n instanceof functionDeclaration) { functionDeclaration fd= (functionDeclaration) n; if(fd.getfunctionHeader().getidentifier().toString().equals("main")) return MAIN_DECL_IMAGE; else return FUNC_DECL_IMAGE; } return DEFAULT_IMAGE; }

}


Exercise 2c: Making Source Foldable


Exercise 2c: Making Source Foldablepublic class LEGFoldingUpdater extends LPGFolderBase {

private void makeFoldable(ASTNode n) { makeAnnotation(n.getLeftIToken(), n.getRightIToken()); }

private class FoldingVisitor extends AbstractVisitor { public boolean visit(block n) { makeFoldable(n); return true; } };

public void sendVisitorToAST(HashMap newAnnotations, List annotations, Object astRoot) { ASTNode theAST= (ASTNode) astRoot; prsStream = theAST.getLeftIToken().getPrsStream(); AbstractVisitor abstractVisitor = new FoldingVisitor(); theAST.accept(abstractVisitor); makeAdjunctAnnotations(); }

}

N.B.: Folding is line-based, so if your regiondoesn’t include an EOL, it won’t be foldable

override for each node to make foldablemake it so!

optional: makes comments (aka “adjuncts”) foldable

continue visiting sub-structure


Exercise 2c: Enhancing the Source Folder

Add functionDeclarations to the set of foldable entities Hint: add an override for functionDeclaration to the visitor class,

and call makeFoldable() Be sure to return true, since you want the visitor to visit the

functionDeclaration‘s body


Exercise 2c: Enhancing the Source Folder (soln.)

private class FoldingVisitor extends AbstractVisitor { // … public boolean visit(functionDeclaration n) { makeFoldable(n); return true; }

}


Exercise 3: Engine Mods


Prelude to Exercise 3a: Language ServicesUnderlying Hover Help and Hyperlinking Hover help:

Use ISourcePositionLocator to identify AST node under text cursor If there’s an IReferenceResolver:

If this AST node is a reference, use the IReferenceResolver to determine thecorresponding declaring AST node

If there’s an IDocumentationProvider: Pass resulting entity to IDocumentationProvider

Else: Obtain source text for target

Hyperlinking: Use ISourcePositionLocator to identify AST node under text cursor If this AST node is a reference, use the IReferenceResolver to determine the

corresponding declaring AST node If the resolver produces something other than the original node:

Use the ISourcePositionLocator to locate the declaration’s text (possibly inanother file)


Exercise 3a: Resolving References


Exercise 3a: Resolving Referencespublic class LEGReferenceResolver implements IReferenceResolver {

public LEGReferenceResolver () { }

public String getLinkText(Object node) { return node.toString(); }

public Object getLinkTarget(Object node, IParseController ctlr) { if (node instanceof Iidentifier && ctlr.getCurrentAst() != null) { identifier id = (identifier) node; LEGParser parser = (LEGParser) ((SimpleLPGParseController)controller).getParser(); SymbolTable<IAst> symtab = parser.getEnclosingSymbolTable(id);

return symtab.findDeclaration(id.toString()); }

return null; }

}

Get the text to be shown in the popup post-itview before hyperlinking to that node

If the given AST node is a reference to something,return the target of the reference (usually a decl).


Exercise 3a: Resolving References(Digression on LEG’s Use of Symbol Tables)

functionDeclaration ::= functionHeader block /. SymbolTable<IAst> symbolTable;

public void setSymbolTable(SymbolTable<IAst>symbolTable) { this.symbolTable = symbolTable; }

public SymbolTable<IAst> getSymbolTable() { return symbolTable; } ./

N.B.: In LEG, each function decl defines a local scope

add a local SymbolTable field

setter for parser to set up the scope

getter for name lookup to find the scope

How to look up references in LEG?

Other possible sources of binding info: index (using PDB), compiler front-end

<LegParser.g excerpt>


Exercise 3a: Resolving References(Digression on LEG’s Use of Symbol Tables)

private final class SymbolTableVisitor extends AbstractVisitor { public boolean visit(block n) { n.setSymbolTable(symbolTableStack.push( new SymbolTable<IAst>(symbolTableStack.peek()))); return true; } public void endVisit(block n) { symbolTableStack.pop(); } public boolean visit(declaration n) { IToken id = n.getidentifier().getIToken(); SymbolTable<IAst> symbol_table = symbolTableStack.peek();

if (symbol_table.get(id.toString()) == null) symbol_table.put(id.toString(), n); else emitError(id, "Illegal redeclaration of " + id.toString()); return true; } public boolean visit(identifier n) { IToken id = n.getIDENTIFIER(); IAst decl = symbolTableStack.peek().findDeclaration(id.toString());

if (decl == null) emitError(id, "Undeclared variable " + id.toString()); else n.setDeclaration(decl); return true; }

}

push symtab for new scope

pop symtab when leaving scope

install var decl in symtab

find decl in symtab, put direct ref in ident node

<LegParser.g excerpt>


Exercise 3a Digression: ISourcePositionLocator

public interface ISourcePositionLocator { Object findNode(Object astRoot, int offset);

Object findNode(Object astRoot, int startOffset, int endOffset);

int getStartOffset(Object entity);

int getEndOffset(Object entity);

int getLength(Object entity);

IPath getPath(Object entity);}

textual offsets are in chars

the innermost AST node whose textual extent contains the given text offset

the innermost AST node that spans the given text extent

“entity”: AST node, token, or type system object

the char offset of the beginning of the textual extent of the given entity

the char offset of the end of the textual extent of the given entity

the char length of the textual extent of the given AST node

the workspace-relative or file-system absolute path to the compilationunit (source or compiled, if no source) containing the given AST nodeor ISourceEntity. The path is in Eclipse "portable" format, using '/' for thepath separator. See org.eclipse.core.runtime.IPath#toPortableString().

How do we locate source entities?You can obtain an ISourcePositionLocator fromIParseController.getNodeLocator()


Exercise 3b: Providing Content Assistance


Exercise 3b: Providing Content Assistance

public class LEGContentProposer implements IContentProposer {

public ICompletionProposal[] getContentProposals(

IParseController ctlr,

int offset,

ITextViewer viewer) { // …}

}

offset at which contentassist was requestedviewer for the editorrequesting assistance

gives access to AST, etc.


Exercise 3b: Content Assistance: SourceProposals

public class SourceProposal implements ICompletionProposal { String proposal;

String newText;

String prefix;

int offset;

int cursorLoc;

String addlInfo;}

Shown to the user in the popup view

The text to be added to the source buffer

The prefix being completed (if any;typically for completing an identifier)

The location of the text being completed(buffer-relative)

Where to place the cursor once the textis inserted, if the proposal is accepted(buffer-relative)

Info to show in a popup pane alongside theproposal list when this proposal is selected


Exercise 3b: Content Assistance: SourceProposals

proposal

newText = “wait()”

addlInfo

offset prefix = “wa”

cursorLoc = just after “wait()”


Exercise 3b: Providing Content Assistancepublic class LEGContentProposer implements IContentProposer {

public ICompletionProposal[] getContentProposals(IParseController ctlr, int offset, ITextViewer viewer) { List<ICompletionProposal> result = new ArrayList();

if (ctlr.getCurrentAst() != null) { IToken token = getToken(ctlr, offset); String prefix = getPrefix(token, offset); LEGParser parser = (LEGParser) ((SimpleLPGParseController) ctlr).getParser(); ISourcePositionLocator locator = ctlr.getNodeLocator(); ASTNode node = (ASTNode) locator.findNode(ctlr.getCurrentAst(), token.getStartOffset(), token.getEndOffset()); if (node != null) { result= computeProposals(token, prefix, node, parser); } } else { result.add(new ErrorProposal("No proposals available - syntaxerrors", offset)); }

return result.toArray(new ICompletionProposal[result.size()]); } // …


Exercise 3b: Providing Content Assistance (cont.)private List<ICompletionProposal> computeProposals(String prefix,

ASTNode node, int offset, LEGParser parser) { List<ICompletionProposal> result = new ArrayList(); if (node.getParent() instanceof Iexpression || node.getParent() instanceof assignmentStmt || node.getParent() instanceof BadAssignment || node.getParent() instanceof returnStmt) { Map<String, IAst> decls = collectVisibleDecls( parser.getEnclosingSymbolTable(node)); List<IAst> matchingDecls = filterSymbols(decls, prefix);

for (IAst decl: matchingDecls) { result.add(createProposalForDecl(decl, prefix, offset)); } } else { result.add(new ErrorProposal("No completion exists for prefix: " + prefix, offset)); } return result;

}


Exercise 3b: Providing Content Assistance (cont.)

private Map<String,IAst> collectVisibleDecls( SymbolTable<IAst> innerScope) {

Map<String,IAst> result = new HashMap<String,IAst>();

// Move outward from innermost enclosing scope for (SymbolTable<IAst> s = innerScope; s != null; s = s.getParent()) { for (String key: s.keySet()) { if (! result.containsKey(key)) // omit shadowed declarations result.put(key, s.get(key)); } } } return result;

}


Exercise 3b: Providing Content Assistance (cont.)private SourceProposal createProposalForDecl(IAst decl, String prefix, int offset) { String propDescrip = "", newText = "";

if (decl instanceof declaration) { newText = ((declaration) decl).getidentifier().toString(); propDescrip = ((declaration) decl).getprimitiveType() + " " + newText; } else if (decl instanceof functionDeclaration) { functionDeclaration fdecl = (functionDeclaration) decl; functionHeader fhdr = fdecl.getfunctionHeader(); declarationList parms = fhdr.getparameters(); newText = fhdr.getidentifier() + "("; for (int i = 0; i < parms.size(); i++) { Ideclaration parm = parms.getdeclarationAt(i); newText += ((declaration0) parm).getprimitiveType() + (i < parms.size() - 1 ? ", " : ""); } newText += ")"; propDescrip = fhdr.getType() + " " + newText; } return new SourceProposal(propDescrip, newText, prefix, offset);

}

Here’s the Beef!

handle vardecls

handle funcdecls


Exercise 3c:Providing anIncremental Builder

…SMAP support is only useful ifyour compiler generates Java code…

Enabling this will cause a “line mappingtable” to be generated in each class file


Exercise 3c: Providing an Incremental Builder IDE developer has these responsibilities:

a) Pass options from preferences page to compile

b) Identify suitable source files

c) Route compiler messages to markers

d) Inform IMP of inter-compilation-unit dependencies

e) Provide “Nature Enabler” to configure builder on existing projects

f) [optional] Provide “New Project Wizard” that configures nature (JDT doesthis)

g) Oh, and build :-) Two basic options for the compiler:

Call out to an existing compiler

Write your own, starting from AST (IMP is not a compiler framework – useany one of the really good ones already out there! Polyglot, JAstAdd, …)


Exercise 3c: Providing an Incremental BuilderDigression on Natures & Builders Builders

A builder instance is associated with a given project (accessible fromyour builder class via getProject())

A builder scans the project for relevant input (source) files to process A builder creates markers on resources to indicate build problems

Natures A project has one or more natures Natures are often associated with languages (but could also be

associated with tools) A nature is typically used to enable various bits of functionality E.g., the JDT’s “Java nature” enables the JDT’s Java compiler The nature typically adds the associated builder(s) to the project’s list of

“build commands”


Exercise 3c: Providing an Incremental Buildera) Pass options from preferences to compiler

public class BuilderBase extends IncrementalProjectBuilder { protected IPreferencesService fPrefService = new PreferencesService(null, getPlugin().getLanguageID());

} protected IProject[] build(int kind, Map args, IProgressMonitor mon) { if (fPrefService.getProject() == null) { fPrefService.setProject(getProject()); // Use project-specific prefs } }

}

public class LegBuilder extends BuilderBase { protected void compile(IFile file, IProgressMonitor mon) { // … fEmitDiagnostics = fPrefService.getBooleanPreference( LegConstants.P_EMITDIAGNOSTICS); // … }

public String getIncludePath() { return fPrefService.getStringPreference(LegConstants.P_INCLUDEPATH); }

}


Exercise 3c: Providing an Incremental Builderb) Identifying suitable source files

public class MyBuilder extends BuilderBase { // … protected boolean isSourceFile(IFile file) { return !file.isDerived() && file.getFileExtension().equals(“g”);

}

protected boolean isNonRootSourceFile(IFile file) { return !file.isDerived() && file.getFileExtension().equals(“gi”); }

}“non-root” files are scanned fordependencies, but not passedto compile()

“root” files are both passed tocompile() and scanned fordependencies


Exercise 3c: Providing an Incremental Builderc) Route compiler messages to markersBuilderBase.createMarker(IResource errorResource, int startLine, int startChar, int endChar, String description, int severity)

where errorResource is typically a source file

where startChar, endChar are file- (not line-) relative offsets

where severity is one of: IMarker.SEVERITY_INFO IMarker.SEVERITY_WARNING IMarker.SEVERITY_ERROR


Exercise 3c: Providing an Incremental Builderd) Inform IMP of inter-compilation-unit dependenciespublic class BuilderBase { protected DependencyInfo fDependencyInfo;

protected abstract void collectDependencies(IFilefile);

}

Typically: parse file, traverse AST and look for references to entities inother files

Register dependencies via calls to DependencyInfo method

public void addDependency(String fromPath, String uponPath);


Exercise 3c: Providing an Incremental Buildere) Providing a “Nature Enabler”public class EnableNature implements IWorkbenchWindowActionDelegate { private IProject fProject;

public EnableBuilder() { } public void dispose() { } public void init(IWorkbenchWindow window) { }

public void run(IAction action) { new MyNature().addToProject(fProject); }

public void selectionChanged(IAction action, ISelection selection) { if (selection instanceof IStructuredSelection) { IStructuredSelection ss= (IStructuredSelection) selection; Object first= ss.getFirstElement();

if (first instanceof IProject) { fProject= (IProject) first; } else if (first instanceof IJavaProject) { fProject= ((IJavaProject) first).getProject(); } } }}

all generated code;no editing needed

Use the “New NatureEnabler” Wizard

track selection so action isready to run when invoked

action class - shows upin context menu


Exercise 3c: Providing an Incremental BuilderUsing a Console for Diagnosticspublic class MyBuilder extends BuilderBase { private static final String BUILDER_CONSOLE = ”My Builder";

protected MessageConsoleStream getConsoleStream() { findConsole(BUILDER_CONSOLE).newMessageStream(); }

protected void compile(IFile file, IProgressMonitor m) { MessageConsoleStream msgStream = getConsoleStream(); // … msgStream.println(“…”); }

}


Exercise 3c: Providing an Incremental BuilderThe Poor-Man’s Compilerpublic class LEGBuilder extends BuilderBase { // … protected void compile(final IFile file, IProgressMonitor m) { try { // … LEGCompiler compiler= new LEGCompiler(PROBLEM_MARKER_ID); compiler.compile(file, monitor); // … // runParserForCompiler(file, monitor);

doRefresh(file.getParent()); // N.B.: Assumes generated // files go into parent folder } catch (Exception e) { // catch all Exceptions: any exception could break the // builder infra-structure. getPlugin().logException(e.getMessage(), e); } }

}


Exercise 3c: Enhancing the Poor-Man’s Compiler

Add support to the compiler for the “<=“ and “>=“ operators you addedto the language earlier

Hints: Add endVisit() overrides to the TranslatorVisitor for AST node

types expression8 and expression9 and generate the appropriate Javacode

See how the compiler already handles “== “


Exercise 3c: Enhancing the Poor-Man’s Compiler (soln.)

public class LEGCompiler { // … private final class TranslatorVisitor extends AbstractVisitor { // … public void endVisit(expression8 n) { String right= (String) fTranslationStack.pop(); String left= (String) fTranslationStack.pop(); fTranslationStack.push(left + " <= " + right); } public void endVisit(expression9 n) { String right= (String) fTranslationStack.pop(); String left= (String) fTranslationStack.pop(); fTranslationStack.push(left + " >= " + right); } }

}


A Cautionary Tale: Adapting an Existing Compiler Front-endfor Use in an IDE

An existing compiler doesn’t necessarily make a good IDE back-end: “Synthetic” AST nodes (e.g. Java default constructor) not explicitly marked Syntactic desugaring done too early – difficult/impossible to recover original Source position information is often fake, missing, or inaccurate

E.g., parent extent doesn’t contain that of its children; bogus synthetic nodeextent

Can’t correlate source selections to AST nodes Can’t reproduce source text from AST (e.g., after rewriting)

In short: ASTs of even well-written compilers make transformation hard Analysis results (e.g., type inference, control- and data-flow) usually not exposed

in a reusable manner Error recovery – may be optimized for parsing speed vs. good diagnostics Error messages – may get “error cascade” Engineering trade-offs between speed of batch compilation vs. incremental

compilation

or: “Hey, your front-end is my back-end!”


Exercise 3d: Supporting ‘Mark Occurrences’


Exercise 3d: Supporting ‘Mark Occurrences’

• In IMP, there can be many kinds of “occurrences,” even per language:• definition/reference (“classic” JDT behavior)

• use/def (data flow)

• executes-in-parallel

• whatever definition suits your language

public interface IOccurrenceMarker extends ILanguageService {

public String getKindName();

public List getOccurrencesOf(IParseController ctlr, Object astNode);

}

used to identify this particularkind of occurrence in the UI

produces the list of AST nodes (from the samefile) that represent occurrences of astNode


Exercise 4: Fuzzy Dice


Exercise 4a: Enhancing Hover Help with aDocumentation Provider


Exercise 4a: Enhancing Hover Help with aDocumentation Provider

public class LEGDocumentationProvider implements IDocumentationProvider { public String getDocumentation(Object entity, IParseController ctlr) { if (entity == null) return null;

if (entity instanceof ASTNode) { // Possibilities: source text, associated docs (e.g. JavaDoc).

// Produce HTML for nicer hovers!

if (entity instanceof declaration) { declaration decl = (declaration) entity;

return decl.getprimitiveType().toString() + " " + decl.getidentifier().toString(); } }

return null; }

}


Exercise 4a: Enhancing the Documentation Provider

Enhance the IDocumentationProvider:

If the entity passed in is a functionDeclaration, find theimmediately-preceding comment (aka “adjunct” in LPG) that starts withthe “LEG Doc” prefix (“//**”), if any, and return that as thedocumentation.

Hints Use ASTNode.getPrecedingAdjuncts() to access the relevant

comments


Exercise 4a: Enhancing the Documentation Provider(soln.)public class LEGDocumentationProvider implements IDocumentationProvider{ public String getDocumentation(Object entity, IParseController ctlr){ // … if (entity instanceof functionDeclaration) { functionDeclaration fdecl= (functionDeclaration) entity; IToken[] adjuncts= fdecl.getPrecedingAdjuncts();

for(int i=0; i < adjuncts.length; i++) { String adjStr= adjuncts[i].toString(); if (adjStr.startsWith("//**")) { return adjStr.substring(4); } } } // … }

}


Exercise 4b: Adding a Preferences Page


package org.imp.leg.preferences;

page Leg {fields { boolean UseDefaultIncludePath { defvalue true; } dirlist IncludePathToUse { tooltip "A semicolon-separated folder list to search for include files"; defvalue ".;..;${pluginResource:org.eclipse.imp.leg/include}"; } unless UseDefaultIncludePath string SourceFileExtensions { tooltip "A comma-separated list of file name extensions identifying thesource files to process"; defvalue "leg"; } boolean GenerateLog { tooltip ”Place diagnostics from the build process in a log file"; defvalue true; } int MaxLogEntries { label "Maximum # of log entries"; range 0 .. 1000; defvalue 100; } if GenerateLog

}

Exercise 4b: Anatomy of a Preferences Specification

“Preference Page”UI components

generated directlyfrom this!

page name in UI - can be hierarchical,e.g., “Leg.Compiler”

package where generated code goes



page Leg { fields { boolean UseDefaultIncludePath }}

page Leg.Compiler { fields { boolean GenerateLog // …

}}

hierarchical page name


Exercise 4b: Anatomy of a Preferences SpecificationField specifications:

field-type field-key { label “Label on pref page”; tooltip “…”; defvalue …;}

field-types:boolean, color, combo,dirList, double, file,font, int, radio, string

If field-key == CamelCaseLikeThis, and no explicit label,label => “Camel Case Like This”

default if no defvalue:boolean => false,color => black,combo/radio => first listed value,file/dirList/string => “”,int/double => range min (0 if none),font => N/A

field-key => constant in LEGConstants.java, used incalls to IPreferencesService.getXXXPreference(String field-key)

int MaxLogEntries { label "Maximum # of Log Entries"; tooltip “Log is truncated to this size”; defvalue 100;}



radio/combo fields:combo field-key { values { valueKey1 “valueLabel1”, … }; defvalue valueKeyi;}

conditional fields:field-type field-key { // …} if boolean-field-key

constrained int/double fields:int TabSize { range 1 .. 20; }

substitution variables in defvalue(also allowed in user-specified values): ${os}, ${arch}, ${ws}, ${nl} ${workspaceLoc} ${projectLoc:projectName} ${pluginLoc:bundleID} ${pluginVersion:bundleID} ${pluginResource:bundleID/path}


Outline

Introduction IDE Development Process Overview The Target Language Interactive Exercises: Developing an IDEAdvanced Topics IMP Architectural Overview Where to go from here


Advanced Topics 1: Source Formatting

Generic rule-based rewrite engine Rules expressed using combination of

Pattern-matching (over concrete syntax) "Box" language layout operators (H - horizontal, V - vertical, A - array, I -

indent, etc.) with parameters (hs - horizontal sep, is = indentation) [CWI] Example: V [ H hs=0 [ “if” “(“ “<Expression>” “)” “{“ ]

I is=$blockIndent [ “<StatementList>” ] “}”]

Pattern example (extracted from above rule): if (<Expression>) { <StatementList> }

Meta-variables act as placeholders for syntactic constructs (e.g. “<Expression>”,“<StatementList>”)

User-settable formatting parameters (e.g. “$blockIndent”) automatically surfaced inIDE preferences (at least some day :-) )


Advanced Topics 1: Source Formatting (cont.)

1. Create service extension using “New Formatting Spec” wizard2. Create an extended parser/scanner for your language for the pattern

matcher Add tokens for “meta-variables” (e.g. <Expression> or

<StatementList> in prev. example)

Meta-variables serve as placeholders for syntactic constructs thatappear in the rules

Specify multiple “start symbols” Formatting rules are concrete syntax, but aren’t complete programs

in the target language Each rule is parsed independently (of course)

3. Create/edit formatting rules using IMP-supplied rule editor

4. Safe (no code loss): set of rules need not cover the entire language


text { font normal = "courier"; IDENTIFIER = { color = "BLUE"; font = normal; style = { bold }; } NUMBER = { color = "RED"; style = { regular }; } DoubleLiteral = { color = "YELLOW"; style = { regular }; } SINGLE_LINE_COMMENT = { color = "GREEN"; font = normal; style = { italic }; } } outline { node functionDeclaration; node block; node declarationStmt0; node declarationStmt1; } foldable { node functionDeclaration; }

}

package org.eclipse.imp.foo;import foo.imp.parser.Ast.*;

language foo { presentation { icon funcIcon = "func.gif"; icon blockIcon = "block.gif"; icon declIcon = "decl.gif";

functionDeclaration f = {

label = {:f.getfunctionHeader().getidentifier(). toString() :}; icon = funcIcon; } block b = { label = "Block"; icon = blockIcon; } declarationStmt0 d = { label = {:d.getdeclaration().getprimitiveType() + " " +d.getdeclaration().getidentifier().toString():}; icon = declIcon; } }

Advanced Topics 2: Presentation Specifications

AST node types

token kinds

{: Java code :}

icon file paths

New!


Advanced Topic 3: Error Handling Errors are the norm! ⇒ must not cripple the IDE! All analyses must produce something reasonable wherever possible

LPG: systematic, semi-automatic errorrecovery for parsing & creating “prosthetic”AST nodes (work in progress)

void A() { int x= 5; foo blah; for(int i=0; i < a.length; i++) { int y= a[i] * a[j]; x += y; }}

dangling ref

mangled statementA()

body

int x= 5; BadStmt for

int i=0;

body

i < a.length i++

header

declStmt

… a[j] …


Advanced Topic 4: Adding a custom editor name & icon

Add an extension like the following to your plugin.xml:

<extension point="org.eclipse.ui.editors"> <editor class="org.eclipse.imp.editor.UniversalEditor" extensions="g,gi" icon="icons/grammarfile.gif" id="lpg.editor" name="LPG Grammar Editor"> </editor> </extension>

Coming soon: Will use existing attributes (language name, icon) in thelanguage descriptor to do the above automatically


Advanced Topic 5: Using Other Kinds of Parserspublic interface IParseController extends ILanguageService { Language getLanguage();

void initialize(IPath filePath, ISourceProject project, IMessageHandler handler);

ISourceProject getProject();

IPath getPath();

Object parse(String input, boolean _, IProgressMonitor monitor);

Object getCurrentAst();

Iterator getTokenIterator(IRegion region);

ISourcePositionLocator getNodeLocator();

ILanguageSyntaxProperties getSyntaxProperties();

IAnnotationTypeInfo getAnnotationTypeInfo();}


Advanced Topic 5: Using Other Kinds of Parsers (cont.)class MyParseController extends ParseControllerBase { public Object parse(String input, boolean _, IProgressMonitor m) { // Parse the given input (editor buffer’s contents), // and return the resulting AST. See ParseControllerBase // for the path, the containing project, and where to store // the resulting AST. }

public Iterator getTokenIterator(IRegion region) { // Return an Iterator over tokens, or anything else you // want to syntax color. Your ITokenColorer must understand // what to do with whatever the Iterator produces. }

public ISourcePositionLocator getNodeLocator() { // Return the ISourcePositionLocator for this language. }

public ILanguageSyntaxProperties getSyntaxProperties() { // … (see next slide) }

}

your responsibilities


Advanced Topic 5: Using Other Kinds of Parsers (cont.)

interface ILanguageSyntaxProperties { public String getSingleLineCommentPrefix();

public String getBlockCommentStart();

public String getBlockCommentContinuation();

public String getBlockCommentEnd();

public String[][] getFences();

public String getIdentifierConstituentChars();

public int[] getIdentifierComponents(String ident);}

e.g. { { "[", "]" }, { "(", ")" }, { "{", "}" }, { "/.", "./" ]

Enables various editor commands(e.g. “Go to Matching Fence”)


Advanced Topic 6: Simple Breakpoint Support [Applies only if your compiler generates Java™ source]

Add line breakpoint support by adding SMAP (JSR-44) attributes to generatedJava class files

Arrange for your compiler to insert “//#line” comments in generated Javasource to indicate location of corresponding original source

• Have your nature’s configure() method add the SMAPINature to new projects,and identify Java as a down-stream builder:

public class MyNature extends ProjectNatureBase { // … public void addToProject(IProject project) { super.addToProject(project); new SmapiProjectNature("g").addToProject(project); } // … protected String getDownstreamBuilderID() { return "org.eclipse.jdt.core.javabuilder"; }}

file-name extension ofyour source files


Advanced Topic 6: Simple Breakpoint Support

IMP/SMAPI classfile post-processor: looks for your language’s source files corresponding to Java source files scans for “//#line” comments

builds a line mapping table

inserts the appropriate SMAP attributes into the classfile


Advanced Topic 7: Deployment the Easy Way

N.B.: deployment != ~employment


Outline



IMP Components

Runtime framework for core IDE component behavior Views: editor, outline page, markers, annotations, etc. Incremental build support Analysis …

Meta-tooling for language-specific IDEs Language-definition support for syntax, well-behaved ASTs, analyses DSLs to more easily implement language services: preferences,

presentation (beta), source formatting (beta) Quick fix/content assist, refactoring support – transformation & analysis

languages (coming) …


IMP Language Service Dispatching

IMP Source Editor& Eclipse Platform

AnalysisScheduler

Compiler Front-End

Analyzers

Source File“foo.leg”

…{ “*.leg” ⇒ “leg” language }…

Language Registry

ServiceControllers

Fact Basebindings, call graph,

type hierarchy,…

Tokens ASTetc.

Decorated ASTfor “foo.leg”

Bindings Types

Ref Resolver

Doc Provider

Indexer

Syntax Colorer

Outliner

Formatter

IDE Services for “leg”

Refactoring

src Δ’s

on demand

etc.

src Δ’s

language-specificlanguage-independent


IMP Language Service Scheduling

more expensive analysis

Quick Fix, Refactoring

Outline, Pkg ExplorerUpdates

Syntax color updates

Index Creation

Indexed Search

Lexical Analysis

Syntactic Analysis

Specialized Analyses

Model Update Broadcaster

Explicit User Actions

Editing Actions

Background Jobs

Presentation RepairerDamage region

idle time

src text


IMP Support for Language Service & Analysis Implementation

Lexical Analysis

SyntaxColoring Syntactic Analysis (Parsing)

Binding Analysis

ContentAssist

EagerParsing

Flow Analyses

Quick FixRefactoring

Refactoring-SpecificAnalyses

Search

IndexCreation

LexicalSpecification

GrammarSpecification,

Auto-generated AST’sGrammar Annotations:Scoping & Definitions

DeclarativePresentationSpecifications

Index EntrySpecifications

AST PatternMatching

WALA Analyses &IMP AnalysisFramework

Declarative ASTRewrite

SpecificationsError Recovery

Grammar Specs,Declarative Assist

Specifications

more expensive analyses

Outlining/Folding

Type Analysis

LanguageInheritance


IMP Analysis Architecture

Analysis & TransformationSpecifications

Specification Compiler(s)

Analysis-specificData Schemas

Analysis-specificAlgebraic Data Types

ConstraintSolver

StateSolver

SourceFact Extractors

Fact Browser& ViewersRefactorings

RelationalSolver

Fact Base{type,context,value}*

Pluggable AnalysisManager

RelationalEngine

ConstraintEngine

StateEngine

Core Data StructureImplementations

(relations, maps, sets,…)

GenericLayer

Analysis-SpecificLayer

SpecificationLanguage

Layer

AnalysisClients

analyzer registration

Search


IMP Analysis Languages

Base Language(Sets, Relations, Maps,Algebraic Data Types,

Operators, Fact access)

Constraint-basedAlgorithm Language

(constraint rep’n,update ops)

State Machine-basedAlgorithm Language

(FSM rep’n,state ops)

Relation-basedAlgorithm Language

(fixed-point ops)

core data structures,primitives, operators,strong type system

domain-specificrep’ns,

operators,semantics

LPG Family FactExtraction Language

Polyglot Family FactExtraction Language

SDF Family FactExtraction Language

JAstAdd Family FactExtraction Language

language extension

Analyzers

Extractors concrete syntax,pattern matching


IMP Static Analysis Support Using WALA

WALA extensible static analysis framework General framework encompassing many classic analyses

pointer analysis, escape analysis, effects analysis, call graph, type analysis multiple precisions (CHA, RTA, 0-CFA, 1-CFA, etc.)

General iterative solver framework for expressing new analyses

Robust, highly scalable (capable of analyzing millions of LOC)

Handles static and dynamic languages

Currently supports Java, JavaScript, PHP, X10

Open-source (http://wala.sourceforge.net) Adding support to WALA for a new language:

Implement translator from your source AST’s into WALA AST’s

Define new instruction types for WALA IR as needed (~12 for X10)

Implement constraint handlers for new IR instructions to enable existing analyses(e.g. pointer analysis, effects analysis, escape analysis)


WALA IR Generation

IBM JS AST

Polyglot Java AST

Polyglot X10 AST

JS CAst AST

Java CAst -> IRtranslator

IBM JS -> CAsttranslator

Rhino JS AST

Polyglot Java -> CAsttranslator

Generic OO -> CAsttranslator services

DOMO IR

X10 -> CAsttranslator

OO CAst AST

Java CAst AST

X10 CAst AST

Eclipse Java AST

Generic Java -> CAsttranslator

Eclipse Java -> CAsttranslator

Rhino JS -> CAsttranslator

Generic JS -> CAsttranslator

Generic OOCAst -> IRtranslator

X10 CAst -> IRtranslator

JS CAst -> IRtranslator

inheritancedata flow


Related Work

GUIDE (C. Laffra): Inspiration for early IMP prototype LDT Dynamic Languages Toolkit (Eclipse Technology Project)

DLTK focuses on dynamic languages; IMP doesn’t

DLTK uses generic language model for program representation; IMPdoesn’t rely on a single model

DLTK not based on meta-tooling; IMP is

DLTK aims for language interoperability, IMP for IDE and languageextensibility


Outline



Where to Go from Here

IMP http://www.eclipse.org/imp

LPG (formerly JikesPG) http://sourceforge.net/project/lpg

X10 http://x10.sourceforge.net/

WALA http://wala.sourceforge.net/

Polyglot http://www.cs.cornell.edu/projects/polyglot/

© 2007 by «Author»; made available under the EPL v1.0 | Date | Other Information, if necessary© 2009 by IBM; made available under the EPL v1.0 | Mar 23, 2009

The End

Questions?

Build an IDE - Eclipse 2009

Documents

maketoken

referenced entity

cast translator

lpg builder

source text

void foo

exercise 1c

iast decl