What linq is about

What LINQ is aboutor, Playing with LINQ, MongoDB, and some old new patterns for developers.or, How to inject functional paradigms in imperative languagesor, parallelism made easy.

WebWorkersCamp, 03/07/2010

Pierre Couzy – Microsoft France

Who are you ?

• Worked in the Microsoft space for 20 years– Speaker, trainer, developer, architect, …

• Primarily interested in Web topics– Identity, architecture, WS-*, interop, cloud, …

• Recent interop/OSS activity– Worked on drupal for Sql server project– Works on automated drupal testing on Windows– Speaker at major PHP events

• DrupalCon, PHP Forum, Symfony live

blog.couzy.com / [email protected]

Agenda

• LINQ• More LINQ• LINQ for MongoDB• Going Parallel• Going Distributed• Q&A

• Disclosure: This session is heavily Microsoft-Biased (which is to be expected, I work there)

LINQ : Why ?

• C# is an imperative language• Which is nice• Except for a few things– Splitting data manipulation between your code

and your data store– Coding data manipulation– Applying new manipulation patterns to existing

data

Clumsy c# example

What we’d like is

• Express more intent, less implementation• Easy composition• Strong typing– This is subject to hot debate

• Be as declarative as possible within the limits imposed by C#

From C# to LINQ

Things to notice

• Lazy evaluation– If we step into this code, we see LINQ triggers its magic on

demand• Easy composition

– If we add new clauses after the LINQ expression is built but before it’s consumed, we simply merge the additions to the existing expression

• So far, we’ve worked on objects– Which is how many naive LINQ implementations work :

synchronously pipelining methods calls on collections• LINQ has an interesting twist

– Expression trees

Expression Trees and IQueryable

• Let’s take the same example and introduce AsQueryable()

• Upon execution, we can notice the compiler didn’t eat our LINQ expression– It’s still in there, using a tree format (an

AbstractSymbolTree)• That tree will be resolved at runtime– In the Linq to Objects case, we get exactly the same

behaviour. – The iteration pattern may still be yield-based, but this is left

to IQueryable’s implementation details.

Playing with expression trees

• Let’s tackle Linq To SQL– My expression tree is now pushed to the database

layer– If I want to block the flow to SQL, I just have to

insert a new node in the tree.– Which allows me to balance things• Using C# patterns and methods when I want them• Pushing to the database what it does best

IQueryable goodies

• So, if I want my things to be LINQ aware, I can either– Build some methods and let the IEnumerable<T>

do its pure object magic (I still get lazy eval)– Or, play with IQueryable • And do my own interpretation of the tree

– Let’s see MongoDB in LINQ

• Strongly-typed interaction when querying and updating collections.

• Improved interface to send common Mongo commands (creating indices, getting all the existing dbs and collections, etc.).

• Ultra-fast de/serialization of BSON to .Net CLR types and back.

• LINQ-to-Mongo• Support for Mono

LINQ Patterns

• Having a LINQ Expression is nice: you basically delegate the execution to someone else’s code

• Plus, your LINQ Expressions have every good aspect you usually expect from functional programming (ie, they are closures).

• So, they are perfectly suited to parallelisation• All you have to do is add another node to the

expression tree• Demo : AsParallel()

Writing MapReduce in LINQ public static IQueryable<R> MapReduce<S,M,K,R>(this IQueryable<S> source, Expression<Func<S,IEnumerable<M>>> mapper, Expression<Func<M,K>> keySelector, Expression<Func<K,IEnumerable<M>,R>> reducer){ return source.SelectMany(mapper).GroupBy(keySelector, reducer);}

• Notice that the calling syntax : Source.MapReduce(lMapper, lKeySelector, lReducer)is simply a new node that can be inserted in a larger expression tree.

• This MapReduce method can run on N cores using PLINQ, or on X machines (and N cores per machine) using DryadLINQ

Collection<T> collection;bool IsLegal(Key k);string Hash(Key);

var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

DryadLINQ = LINQ + Dryad

C#

collection

results

C# C# C#

Vertexcode

Queryplan(Dryad job)Data

Parallel executionvar logentries = from line in logs where !line.StartsWith("#") select new LogEntry(line);var user = from access in logentries where access.user.EndsWith(@"\ulfar") select access;var accesses = from access in user group access by access.page into pages select new UserPageCount("ulfar", pages.Key, pages.Count());var htmAccesses = from access in accesses where access.page.EndsWith(".htm") orderby access.count descending select access;

Thanks.