Building a web application with ontinuation monads

1

Building a web application withcontinuation monads

Seitaro Yuki / @pab_tech / DWANGO Co., Ltd.

2

OutlineIntroduction of DWANGO and Niconico

The account system of Niconico and its tasks

Problems of component technologies of existing webframeworks

How to construct web applications using continuationmonads

Updating the component using indexed continuationmonads.

3

Introduction of DWANGO and Niconico

DWANGO is a company that operates a video sharing andlive broadcast platform Niconico.

Niconico has 50 million accounts and 2.5 million ofpremium members who pay 500 yen a month.

It has been known as one of most major streaming mediaservice in Japan.

4

Functional programming in DWANGO

DWANGO also has been known as a company that hasadopted functional programming languages in Japan.

DWANGO is using a variety of functional languages such asScala and Erlang.

DWANGO publishes Japanese Scala textbook for in-housetraining to Github. https://github.com/dwango/scala_text

5

Decomposing Niconico services

Previously Niconico was the one large PHP system that hasmillions of lines.

We are separating the common foundation system fromthe large PHP system using Scala

And separating the content delivery system using Erlang

As a result, it became easy to add functions to the system,and resistant to failure.

6

The account system of Niconico

DWANGO has a variety of services such as e-books andslide services other than video and live streaming.

The functionality related to users has been aggregatedinto the account system.

The tasks of the account system are the following.

User registration

User authentication

Operation user information

Decision whether premium or not

7

Various interfaces required for the account system

The account system is used for various services, and theyhave many devices.

Therefore the account system is required variousinterfaces.

8

User registrationRegistration on a registration page

Registration via an API

Registration with premium registration

Registration with connectivity verification of E-mail or SMS

9

User authenticationAuthentication from E-mail address and password

Authentication using OAuth such as Facebook and Twitter

2-step authentication with TOTP

2-step authentication with E-mail

10

Various responsesHTML、JSON、XML

Redirects

Japanese, English and Chinese

11

Technical requirementsCSRF check using a token

Session management using HTTP Cookies

Adding user tracking ID to the response

Adding CORS header to the response

12

Error handlingIn Web applications, it should return a correct formatresponse as much as possible, even if a critical erroroccurs.

13

Components of web applicationsThe various factors described so far will be used incombination.

So, we want a component technology to decompose a webapplication.

And, we want to assemble freely components to constructa web application.

14

Typical component technologies of web applications.

public interface Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain);}

doFilter in Java Servlet Filter

ServletRequest means an HTTP Request

ServletResponse means an HTTP Response

FilterChain means a next Filter

15

Filter Examplepublic class ExampleFilter implements Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain) { // preprocess for request chain.doFilter(request, response); // postprocess for response }}

You can put a preprocess before calling the next chain.

You can also put a postprocess a�er calling the next chain.

These nested structures can be seen in various languagessuch as WSGI of Python, Rack of Ruby.

16

Typical Web application sequence diagramWeb application

Web Browser Web Server Filter A Filter B Main Processing

request

request

request

request

response

response

response

response

17

Example: Authentication verificationIt assumes that a user logins elsewhere and there is asession in HTTP cookies before this operation.

This component stores the session information in serverstorage such as Redis.

The component compares the cookie information and theserver information to check whether authenticated.

If the session is correct, then this component passes thesession information to the next one.

Otherwise, it redirects to a login form.

18

Authentication verification sequence diagram

Web Browser Web Server Authentication Filter Main Processing

request

request

alt ["Authentication is successful"]

request

request

response

response

response

["Authentication is failed"]

redirect

response

19

Example: CompressionA compression component doesn't do anything to arequest.

This component will compress a response only if therequest has an Accept-Encoding.

20

Compression sequence diagramWeb Browser Web Server Compression Filter Main Processing

request

request

request

request

response

response

alt [There is "Accept‒Encoding: gzip"]

response compress response

[There is not "Accept‒Encoding: gzip"]

response do nothing to response

response

21

Problems of the component technologies ofexisting web applications

public interface Filter { public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain);}

This filter method doesn't have much information on thetype and parameters.

We don't know what kind of parameter is passed to thenext component, what combination of components canbe.

22

The component of Play framework in Scalatrait ActionFunction[-R[_], +P[_]] { def invokeBlock[A](request: R[A], block: P[A] => Future[Result]): Future}

The type parameter R represents a type of a Request.

The type parameter P represents a type of a transformedRequest by this function.

The type parameter A represents a type of a body of arequest.

ActionFunction represents a transformation from R to P.

23

Example: AuthenticatedFunctionclass AuthenticatedRequest[A](session: Session, request: Request[A])

object AuthenticatedFunction extends ActionFunction[Request, AuthenticatedRequest] {

def invokeBlock[A]( request: Request[A], block: Authenticated[A] => Future[Result]): Future[Result] = ???}

A component to authenticate and take a sessioninformation from the request.

AuthenticatedRequest is added an information of asession to the Request.

AuthenticatedFunction is a function that converts Requestto AuthenticatedRequest.

24

Problem of ActionFunctionclass LanguageRequest[A](language: Language, request: Request[A])

object LanguageFunction extends ActionFunction[Request, LanguageRequest

AuthenticatedFunction.andThen(LanguageFunction) // Can not!

LanguageFunction takes a Language parameter from theAccept-Language of Request.

This function can't be combined withAuthenticatedFunction for the mismatched types of therequest.

25

Introduction of components usingcontinuation monads

26

ContinuationA

Continuation

A R

The continuation represents the rest of computation at agiven point in execution.

A is some point of execution.

R is the result of the whole computation.

That means A to R is the continuation at A.

27

Continuation-passing style (CPS)Caller A B R

pass a continuation after A

altnot execute a continuation

pass a continuation after B

altnot execute a continuation

pass a continuation after R

CPS is a programming style that continuations are first-class.

28

Continuation monadcase class Cont[R, A](run: (A => R) => R) { def map[B](f: A => B): Cont[R, B] = Cont(k => run(a => k(f(a)))) def flatMap[B](f: A => Cont[R, B]): Cont[R, B] = Cont(k => run(a => f(a).run(k)))}

A type parameter A means a value in the given point ofexecution.

A type parameter R means a result of this monad.

A function of A to R means the continuation at A.

Thus a continuation monad converts a function thatreceives the continuation to a monad.

29

How to construct Web applications withcontinuation monads.

30

Authentication verificationWeb Browser Web Server Authentication Filter Main Processing

request

request

alt ["Authentication is successful"]

request

request

response

response

response

["Authentication is failed"]

redirect

response

31

Code of Authentication verification// takes a session id from a cookie from the request.def readSessionIdFromCookie(request: Request): Option[SessionId] = ???

// takes a session information from Redis using the session id.def readSessionFromRedis(sessionId: SessionId): Option[Session] = ???

// returns a response that redirects to a login form.def redirectLoginForm: Response = ???

// k is a continuation has a type of Session to Response.// verifyAuth is the authentication verification component.def verifyAuth(request: Request): Cont[Response, Session] = Cont((k: Session => Response) => readSessionIdFromCookie(request) .flatMap(readSessionFromRedis) .map(k) .getOrElse(redirectLoginForm))

32

CompressionWeb Browser Web Server Compression Filter Main Processing

request

request

request

request

response

response

alt [There is "Accept‒Encoding: gzip"]

response compress response

[There is not "Accept‒Encoding: gzip"]

response do nothing to response

response

33

Code of Compression// takes a value of the Accept-Encoding in the request.def readAcceptEncoding(request: Request): List[Encoding] = ???

// compresses the response.def gzip(response: Response): Response = ???

// compress is the Compression component.def compress(request: Request): Cont[Response, Unit] = Cont((k: Unit => Response) => { val response = k(()) // if the Accept-Encoding header contains "gzip" if (readAcceptEncoding(request).contains(Encoding.GZIP)) gzip(response) // compresses the response else response // doesn't do anything to the response })

34

Compose continuation monads using for-expression in Scala

// a core part of web application.def mainProcessing(request: Request, session: Session): Cont[Response,

def webApplication(request: Request): Cont[Response, Response] = for { session <- verifyAuth(request) _ <- compress(request) response <- mainProcessing(request, session) } yield response

The for-expression in Scala is used for composing monads.

We can compose verifyAuth, compress, andmainProcessing using for-expression.

35

Combine continuation monad transformerwith Scala Future

import scalaz._import scalaz.std.scalaFuture._

object ActionCont { type ActionCont[R, A] = ContT[Future, R, A]

def recover[R, A] (cont: ActionCont[A, A]) (pf: PartialFunction[Throwable, Future[A]]): ActionCont[R, A] = ContT(cont.run_.recoverWith(pf).flatMap)}

Scala Future is a monad has two functionalities that are

asynchronous computation and

variant type of error value.

36

Problem of continuation monad component

Components of continuation monads are very useful.

But there is an issue that a type of response is fixed.

37

Problem of a fixed type of response// takes a value of the Accept in the request.def readAccept(request: Request): List[MediaType] = ???

def toJson(entity: Entity): Response = ???

def toXml(entity: Entity): Response = ???

def toJsonOrXml(request: Request): Cont[Response, Unit] = Cont((k: Unit => Response) => { val response = k(()) // if the Accept header contains "application/json" if (readAcceptHeader(request).contains(MediaType.JSON)) toJson(response) // TypeError: Entity expected but actually Response else toXml(response) // TypeError: Entity expected but actually Response })

38

Updating components using indexed monad

So we would like to update our components with indexedcontinuation monad.

Indexed monad is a monad has an another type parametercan be changed.

We can deal with changes in a parameter of the monadusing the indexed monad.

For example, the indexed state monads can change a typeof state.

39

Indexed continuation monadO

Continuation

A R

Added type parameter O means a result type ofcontinuation.

40

Code of indexed continuation monadcase class IndexedCont[R, O, A](run: (A => O) => R) { def map[B](f: A => B): IndexedCont[R, O, B] = IndexedCont(k => run(a => k(f(a)))) def flatMap[E, B](f: A => IndexedCont[O, E, B]): IndexedCont[R, E, B IndexedCont(k => run(a => f(a).run(k))) def contramap[I](f: I => O): IndexedCont[R, I, A] = IndexedCont(k => run(a => f(k(a))))}

By distinguishing a result type of a continuation from aresult type of the whole,

We can deal with changes in a result type.

And we think that a contramap function to be a pair with amap function.

41

Composition of indexed continuation monad

Using flatMap of IndexedCont, we can composeIndexedCont[R, O, A] and IndexedCont[O, E, B] toIndexedCont[R, E, B]

E RB

OA R

OEBIndexedCont[R, O, A]

IndexedCont[O, E, B]

IndexedCont[R, E, B]

42

Example of a component of indexedcontinuation monad

def toJsonOrXml(request: Request): IndexedCont[Response, Entity, Unit] = IndexedCont((k: Unit => Entity) => { val entity = k(()) // intermediate result if (readAcceptHeader(request).contains(MediaType.JSON)) toJson(entity) // not a compile error else toXml(entity) // not a compile error })

Unlike the previous example, we can deal with entitieshave a type of an intermediate result.

43

Example of contramap of indexedcontinuation monad

def toJsonOrXml(request: Request): IndexedCont[Response, Entity, Unit] = IndexedCont((_: Unit => Response)(())).contramap { if (readAcceptHeader(request).contains(MediaType.JSON)) toJson else toXml }

We can also use contramap in the conversion process ofresponse.

44

Continuation monads in the account systemof Niconico

Actually, there are about 50 independent components inthe account system of Niconico

Let's see some of them that classified by purpose.

45

General purpose componentsComponent Purpose

FormRequestCont To get Form parameters of the request

JsonRequestCont To get Form parameters of the request

HandlerCont To wrap a domain service function

UserTrackIdCont To add user tracking id to the response

46

Components for APIComponent Purpose

AuthorityCont To verify that clients has theauthority to API

MaintenanceCont To return 503 if the service is undermaintenance

ExceptionRendererCont To convert errors to the responseof API

ResponseFormatCont To determine whether JSON orXML that the service should return

47

Components for Web pagesComponent Description

RedirectVerificationCont To verify that a given RedirectURL is contained in the whitelist

RedirectLoginFormCont To redirect the login form ifauthentication verification failed

PasswordVerificationCont To ask a password to user beforedisplaying a page

CsrfTokenVerificationCont To check a CSRF token

48

Components construct other componentsComponent Description

FlowCont To construct components such as pre-processing and error handling component

ApiCont To construct components required for API

WebPageCont To construct components required for Webpages

49

Conclusion

We have seen the construction Web application usingcontinuation monads.

Components of continuation monads is flexible tocompose and easy to understand.

We are updating our components to indexed continuationmonads to change the result type of continuation.

We can make a variety of interfaces by combining thecontinuation monad, then we can concentrate on moreimportant domain logic.