From DOT to Dotty

From DOT to Dotty

Martin Odersky

ScalaX, London, 8 December 2016

DOT: Foundations

dotty: Prototype

… for future Scala

Why Do Foundations Matter?

• They help ensure properties such as type soundness.

• They serve as a feedback loop for language design.

• They help detect hidden connections between language features.

Why Not Pick Existing Foundations?

• Because they would lead to variants of existing languages.

• Foundations are formative!

Our AimWe are looking for a minimal theory that can model

1. type parameterization,2. modules,3. objects and classes.

minimal: We do not deal with inheritance here.

Our AimWe are looking for a minimal theory that can model

1. type parameterization,2. modules,3. objects and classes.

There were several attempts before, including νObj,which was proposed as a basis for Scala (ECOOP 2003).

But none of them felt completely canonical or minimal.

Related: 1ML, which can model (1) and (2) by mapping to System F.

Dependent Types• We will model modules as objects with type members.

• This requires a notion of dependent type - the type referred to by a type member depends on the owning value.

• In Scala we restrict dependencies to paths.

• In the calculus presented here we restrict it further to variables.

Variable xPath p = x | p.a

Dependent Types in CodeWe can model heterogeneous maps like this:

Dependent Types in Code

Dependent Types in Code

Foundations: DOT

The DOT calculus is intended to be a minimal foundation of Scala.

Its type structure is a blueprint for the types used internally in the compiler.

DOT Terms• Translated to Scala notation, the language covered by

DOT is:Value v = (x: T) => t Function

new { x: T => d } Object

Definition d = def a = t Method definitiontype A = T Type

Term t = v Valuex Variablet1(t2) Applicationt.a Selection{ val x = t1; t2 } Local definition.

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } Recursion

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } Recursion

Should Scala have these?

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } Recursion

Will replace the

T1 with T2

syntax

DOT TypesThe Types covered by DOT are:

Type T = Any Top typeNothing Bottom typex.A Selection(x: T1) => T2 Function{ def a: T } Method declaration{ type T >: T1 <: T2 } Type declarationT1 & T2 Intersection{ x => T } RecursionScala has only refinements

T { d1 … dn}

with this as self reference.

DOT Syntax in Greek

Note: terms are in ANF form.This simplifies some things, but is not essential.

Type Assignment

Definition Type Assignment

Subtyping

ExpressivenessSimple as the model is, it is actually quite expressive.

Directly representable:

▶ type parameters▶ variance▶ nominal typing▶ generative modules ▶ self types ▶ ADTs and simple classes

Requires smallish extension:

▶ Classes with inheritance

Meta TheorySimple as the model is, the soundness proof of DOT was surprisingly hard.

▶ Attempts were made since about 2008.

▶ Previous publications (FOOL 12, OOPSLA 14) reportabout (some) advances and (lots of) difficulties.

▶ Essential challenge: Subtyping theories areprogrammer-definable.

Programmer-Definable TheoremsIn Scala and DOT, the subtyping relation is given in part by user-definable definitions:

type T >: S <: U { T: S .. U }

This makes T a supertype of S and a subtype of U. By transitivity, S <: U.

So the type definition above proves a subtype relationship which was potentially not provable before.

Bad BoundsWhat if the bounds are non-sensical?

Example:type T >: Any <: Nothing

By the same argument as before, this implies that Any <: Nothing

Once we have that, again by transitivity we get S <: T for arbitrary S and T.

That is, the subtyping relations collapses to a single point!

This means that most proof techniques for soundness fail.

Dealing with itObservation: To prove preservation, we need to reason at the top-level only about environments that arise from an actual computationSuch environments correspond to run-time stores which binds variables to values. And values have guaranteed good bounds because all type members in definitions are aliases.By an elaborate argument one can make use of this observation to show soundness.

For Details

Consequences for Language Design• So far: Some soundness issues were known, but it was

not clear how to fix them.

• Can one impose restrictions to guarantee good bounds?

• Has been tried for a while but was not complete.

• The meta theory taught us a principle to ensure soundness:

Every prefix p of a type selection p.A must be a computed value.

Things To Avoidtrait BadBounds { type A >: Any <: Nothing }

lazy val x: BadBounds = ???

BadBounds # A

val x: BadBounds = null

Need to drastically restrict types we can write in a lazy val: Only concrete types with good bounds are allowed.Need to drastically restrict types we can write in a projection: Only concrete types with good bounds are allowed.

Things To Avoidtrait BadBounds { type A >: Any <: Nothing }

lazy val x: BadBounds = ???

BadBounds # A

val x: BadBounds = nullNeed to track null in the type system (straightforward)

Need to track initialization status(hard)

dottydotty is the working name for our new Scala compiler.

• Builds on DOT in its internal data structures.

• Generics get expressed as type members.

• Supports the next iteration(s) of the Scala programming language.

A Whirlwind Tour Of Dotty Constructs

Dropped Features

DelayedInit

Macros

Existential Types

Procedure Syntax

Early Initializers

General Type Projection

def run() { ... }

Will be rewritten automatically to

def run(): Unit = { ... }

class Foo extends DelayedInit

class C extends {val x = e

} with D

Use trait parameters instead

T # X

- Was shown to be unsound for general types T.- Projection C#X from class types C still available.

(the reflection based kind)

def m(...) = macro impl(...)

C[U] forSome { type U }

Wildcards C[_]still supported.

Implemented New Features

Multiversal Equality

Intersection TypesUnion types

Trait parameters

Function arityadaptation

pairs.map((x, y) => x + y)

instead of

pairs.map {case (x, y) => x + y

}

T & U

- replaces T with U- is commutative

T | U

avoids huge lubs

@static methods and fieldsnon-blocking lazy vals

trait T(x: Int) { ... }object O {@static val x = ...@static def f() = ...

}

lazy val x = ... // thread-local

@volatilelazy val x - ... // thread-safe,

// avoids dead-locks

type-safe ==, !=

And Further Down the Road ?

Implicit Function Types

scala.metascrap all 22’s

effects

Implicit Function Types

What Are Implicit Function Types?

and what is “Contextual Abstraction”?

“Abstraction”

The ability to name a concept

and use just the name afterward

“Contextual”The context comprises all the inputs that let a program do its work, including:

• configuration data

• capabilities

• dependency injection

• type class instances

Implicit Parameters• Technique of choice to pass inputs to program parts that

need them.

• Advantage over normal parameters:

No boilerplate code to pass them along the edges of a call graph.

• But we still need to declare them as parameters everywhere they are passed!

Example: Transaction Handling

Example (2)

Example (3)

Example Run

Can We Do Better?• Problem: Boilerplate code for declaring implicit

parameters

• Repeating this 3x does not look so terrible.

• But in the dotty compiler there are 2641(!) occurrences of

• We’d like to get rid of them.

Towards A SolutionLet’s massage the definition of f1 a bit:

f1’s right hand side is now an implicit function value.What is its type?

So far: Transaction => Int

From now on: implicit Transaction => Int

or, desugared: ImplicitFunction1[Transaction, Int]

Inside ImplicitFunction1ImplicitFunction1 can be thought of being defined as follows:

Analogously for all other arities.

Two Rules for Typing1. Implicit functions get implicit arguments just like implicit

methods. Given:

f expands to f(a)

2. Implicit functions get created on demand. If the expected type of b is implicit A => B, then

b expands to implicit _: A => b

Revised Example:

But where does current come from?

Revised Example (2)

Summary• Implicit function types are a neat way to abstract over

contexts.

• It’s a very powerful feature, because it allows one to inject implicit values in a scope simply by defining type.

• This opens up a lot of possibilities.

• I expect it will fundamentally affect the kind of Scala code in the future.

When Can I Expect This?

Scala 2.12

Scala 2.13

Scala 3.0TASTY, middle end?

stdlibcollections

dotty MVP

dotty 0.x releases

2016backend, classpath handling

Scala 2.14

2017

2018

This is open source work, depends on community’s contributions.à Roadmap is tentative, no promises:

“MVP” = minimal viable prototype

ContributorsTheory (DOT): Nada Amin, Tiark Rompf, Sandro Stucki, Samuel Grütter. based on previous work by Adriaan Moors, Donna Malayeri, Geoffrey Washburn, and others.

Implementation (dotty): Many contributors, including

Dmitry Petrashko Nicolas StuckiGuillaume Martres Sebastien DouraeneFelix Mulder Ondrej LhotakLiu Fengyun Vera Salvisberg

Close collaboration with scalac team

Adriaan Moors Seth TisueJason Zaugg Stefan ZeigerLukas Rytz

Find out more on scala-lang.org/blog

Thank You