Transcript
Highload reactive server with Netty
Dmitriy DumanskiyBlynk, CTO
Java blog : https://habrahabr.ru/users/doom369/topicsDOU : https://dou.ua/users/DOOM/articles/
Makers problem
+ = ?
Makers problem
● Http/s● Mqtt● WebSockets● Own binary protocol
Blynk
10000 req/sec3 VM * 2 cores, 60$
25% load10k of local installations
Why netty?
CassandraApache Spark
Elasticsearch
Graylog
Neo4j
Vert.x
HornetQInfinispan
Finagle
Async-http-clientFirebase
Akka
CouchbasePlay frameworkRedisson
Why netty?
~700k servers
Why netty?● Less GC
Why netty?● Less GC● Optimized for Linux based OS
Why netty?● Less GC● Optimized for Linux based OS● High performance buffers
Why netty?● Less GC● Optimized for Linux based OS● High performance buffers● Well defined threading model
Why netty?● Less GC● Optimized for Linux based OS● High performance buffers● Well defined threading model● HTTP, HTTP/2, SPDY, SCTP, TCP,
UDP, UDT, MQTT, etc
When to use?● Performance is critical
When to use?● Performance is critical● Own protocol
When to use?● Performance is critical● Own protocol● Full control over network
(so_reuseport, tcp_cork, tcp_fastopen, tcp_nodelay, etc)
When to use?● Performance is critical● Own protocol● Full control over network● Game engines (agario, slither,
minecraft)
When to use?● Performance is critical● Own protocol● Full control over network● Game engines● <3 reactive
Non-Blocking
● Few threads● No context switching● No memory consumption
Non-Blockingnew Channel
read / write
Selector
Thread
new Channel
read / write
new Channel
read / write
java.nio.channels.SelectorSelector selector = Selector.open();channel.configureBlocking(false);SelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); if (key.isReadable()) { ... } }}
Selector selector = Selector.open(); // creating selectorchannel.configureBlocking(false);SelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); if (key.isReadable()) { ... } }}
Selector selector = Selector.open(); channel.configureBlocking(false);//registering channel with selector, listening for READ events onlySelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); if (key.isReadable()) { ... } }}
Selector selector = Selector.open();channel.configureBlocking(false);SelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); //blocking until we get some READ events Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); if (key.isReadable()) { ... } }}
Selector selector = Selector.open();channel.configureBlocking(false);SelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); //now we have channels with some data Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); if (key.isReadable()) { ... } }}
Selector selector = Selector.open();channel.configureBlocking(false);SelectionKey key = channel.register(selector, SelectionKey.OP_READ);while(true) { selector.select(); Set<SelectionKey> selectedKeys = selector.selectedKeys(); Iterator<SelectionKey> keyIterator = selectedKeys.iterator(); while(keyIterator.hasNext()) { key = keyIterator.next(); //do something with data if (key.isReadable()) { key.channel() } }}
FlowSelector
SelectionKey
Channel
ChannelPipeline
FlowChannelPipeline
fireEvent()
invokeChannelRead() executor.execute()
invokeChannelRead()
Minimal setupServerBootstrap b = new ServerBootstrap();b.group( new NioEventLoopGroup(1), new NioEventLoopGroup()) .channel(NioServerSocketChannel.class) .childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();f.channel().closeFuture().sync();
Minimal setupServerBootstrap b = new ServerBootstrap();b.group( new NioEventLoopGroup(1), //IO thread new NioEventLoopGroup()) .channel(NioServerSocketChannel.class) .childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();f.channel().closeFuture().sync();
Minimal setupServerBootstrap b = new ServerBootstrap();b.group( new NioEventLoopGroup(1), new NioEventLoopGroup() //worker threads) .channel(NioServerSocketChannel.class) .childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();f.channel().closeFuture().sync();
Minimal setupServerBootstrap b = new ServerBootstrap();b.group( new NioEventLoopGroup(1), new NioEventLoopGroup() //worker threads) .channel(NioServerSocketChannel.class) .childHandler(new ChannelInitializer() {...}); //pipeline init
ChannelFuture f = b.bind(8080).sync();f.channel().closeFuture().sync();
Minimal setup
new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel ch) { final ChannelPipeline pipeline = ch.pipeline(); pipeline.addLast(new MyLogicHere()); }};
ChannelPipeline
ChannelPipeline
● Inbound event -> ChannelInboundHandler (CIHA)
● Outbound event -> ChannelOutboundHandler (COHA)
ChannelInboundHandlerpublic interface ChannelInboundHandler extends ChannelHandler { ... void channelRegistered(ChannelHandlerContext ctx); void channelActive(ChannelHandlerContext ctx); void channelRead(ChannelHandlerContext ctx, Object msg); void userEventTriggered(ChannelHandlerContext ctx, Object evt); void channelWritabilityChanged(ChannelHandlerContext ctx); ...}
void initChannel(SocketChannel ch) { ch.pipeline() .addLast(new MyProtocolDecoder()) .addLast(new MyProtocolEncoder()) .addLast(new MyLogicHandler());}
Own tcp/ip server
Channel
MyProtocolDecoder
MyLogicHandler
Own tcp/ip server
Channel
MyProtocolEncoder
MyLogicHandler
HandlersHttpServerCodec
ChannelTrafficShapingHandler
IdleStateHandler
ReadTimeoutHandler
ChunkedWriteHandler
SslHandler
LoggingHandler
RuleBasedIpFilter
StringDecoderJsonObjectDecoder
Base64DecoderJZlibDecoder
JZlibDecoder
Lz4FrameDecoder
ProtobufDecoderObjectDecoder
XmlFrameDecoder
void initChannel(SocketChannel ch) { ch.pipeline() .addLast(new HttpRequestDecoder()) .addLast(new HttpResponseEncoder()) .addLast(new MyHttpHandler());}
Http Server
void initChannel(SocketChannel ch) { ch.pipeline() .addLast(new HttpServerCodec()) .addLast(new MyHttpHandler());}
OR
void initChannel(SocketChannel ch) { ch.pipeline() .addLast(sslCtx.newHandler(ch.alloc())) .addLast(new HttpServerCodec()) .addLast(new MyHttpHandler());}
Https Server
void initChannel(SocketChannel ch) { ch.pipeline() .addLast(sslCtx.newHandler(ch.alloc())) .addLast(new HttpServerCodec()) .addLast(new HttpContentCompressor()) .addLast(new MyHttpHandler());}
Https Server + content gzip
@Overridepublic void channelRead(Context ctx, Object msg) { //pass flow processing to next handler super.channelRead(ctx, msg);}
Pipeline flow
@Override public void channelRead(Context ctx, Object msg) { //stop request processing return;}
Pipeline flow
public void channelRead(Context ctx, Object msg) { If (msg instanceOf LoginMessage) { LoginMessage login = (LoginMessage) msg; if (isSuperAdmin(login)) { ctx.pipeline().remove(this); ctx.pipeline().addLast(new SuperAdminHandler()); } }}
Pipeline flow on the fly
public void channelRead(Context ctx, Object msg) { ChannelFuture cf = ctx.writeAndFlush(response); cf.addListener(new ChannelFutureListener() { @Override public void complete(ChannelFuture future) { future.channel().close(); } });}
Pipeline futures
@Overridepublic void channelRead(Context ctx, Object msg) { ChannelFuture cf = ctx.writeAndFlush(response); //close connection after message was delivered cf.addListener(ChannelFutureListener.CLOSE);}
Pipeline futures
@Overridepublic void channelRead(Context ctx, Object msg) { ... ChannelFuture cf = ctx.writeAndFlush(response); cf.addListener(future -> { ... });}
Pipeline futures
public void channelRead(Context ctx, Object msg) { ChannelFuture cf = session.sendMsgToFriend(msg); cf.addListener(new ChannelFutureListener() { @Override public void complete(ChannelFuture future) { future.channel().writeAndFlush(“Delivered!”); } });}
Pipeline futures
Pipeline blocking IONon blocking pools Blocking pools
IO Event LoopsDB
Worker Event Loops
Mailing
File system
public void channelRead(Context ctx, Object msg) { if (msg instanceof HttpRequest) { HttpRequest req = (HttpRequest) msg; if (req.method() == GET && req.uri().eq(“/users”)) { Users users = dbManager.userDao.getAllUsers(); ctx.writeAndFlush(new Response(users)); } }}
Pipeline blocking IO
public void channelRead(Context ctx, Object msg) { if (msg instanceof HttpRequest) { HttpRequest req = (HttpRequest) msg; if (req.method() == POST && req.uri().eq(“/email”)) { mailManager.sendEmail(); } }
Pipeline blocking IO
public void channelRead(Context ctx, Object msg) { if (msg instanceof HttpRequest) { HttpRequest req = (HttpRequest) msg; if (req.method() == GET && req.uri().eq(“/property”)) { String property = fileManager.readProperty(); ctx.writeAndFlush(new Response(property)); } }}
Pipeline blocking IO
public void channelRead(Context ctx, Object msg) { ...
blockingThreadPool.execute(() -> {Users users = dbManager.userDao.getAllUsers();ctx.writeAndFlush(new Response(users));
});}
Pipeline blocking IO
Pipeline blocking IO
● Thread.sleep()
Pipeline blocking IO
● Thread.sleep()● java.util.concurrent.*
Pipeline blocking IO
● Thread.sleep()● java.util.concurrent.*● Intensive operations
Pipeline blocking IO
● Thread.sleep()● java.util.concurrent.*● Intensive operations● Any blocking IO (files, db, smtp, etc)
Pipeline blocking IO
● Thread.sleep()● java.util.concurrent.*● Intensive operations● Any blocking IO (files, db, smtp, etc)
@Overridepublic void channelInactive(Context ctx) { HardwareState state = getState(ctx.channel()); if (state != null) { ctx.executor().schedule( new DelayedPush(state), state.period, SECONDS ); }}
EventLoop is Executor!
public void channelRead(Context ctx, Object msg) { if (msg instanceof FullHttpRequest) { FullHttpRequest request = (FullHttpRequest) msg; User user = sessionDao.checkCookie(request); ... } super.channelRead(ctx, msg); }
Request state
private static AttributeKey<User> USER_KEY = AttributeKey.valueOf("user");
ctx.channel().attr(USER_KEY).set(user);
Request state
public void channelRead(Context ctx, Object msg) { if (msg instanceof FullHttpRequest) { FullHttpRequest request = (FullHttpRequest) msg; User user = sessionDao.checkCookie(request); ctx.channel().attr(USER_KEY).set(user); } super.channelRead(ctx, msg); }
Request state
if (isSsl(in)) { enableSsl(ctx);} else { if (isGzip()) { enableGzip(ctx); } else if (isHttp(in)) { switchToHttp(ctx); }}
Port unification
Back pressure
if (channel.isWritable()) { channel.writeAndFlush(msg);}
Back pressure
BackPressureHandler
coming soon...
Performance
Performance
https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=plaintext
<dependency> <groupId>io.netty</groupId> <artifactId>netty-transport-native-epoll</artifactId> <version>${netty.version}</version> <classifier>${os}</classifier></dependency>
Native transport
Bootstrap b = new Bootstrap();b.group(new EpollEventLoopGroup());b.channel(EpollSocketChannel.class);
Native transport
SslContextBuilder.forServer().sslProvider(SslProvider.OpenSsl);
JNI OpenSslEngine
<dependency> <groupId>io.netty</groupId> <artifactId>netty-tcnative-boringssl-static</artifactId> <version>${netty.boring.ssl.version}</version> <classifier>${os}</classifier></dependency>
JNI OpenSslEngine
● Netty-tcnative● netty-tcnative-libressl● netty-tcnative-boringssl-static
JNI OpenSslEngine
Own ByteBuf
Own ByteBuf● Reference counted● Pooling by default● Direct memory by default● LeakDetector by default● Reduced branches, range-checks
Own ByteBuf
● ByteBufAllocator.buffer(size);● ctx.alloc().buffer(size);● channel.alloc().buffer(size);
Less system calls
for (Message msg : messages) { ctx.writeAndFlush(msg);}
Less system calls
for (Message msg : messages) { ctx.write(msg);}ctx.flush();
Thread Model
ChannelFuture inCf = ctx.deregister();
inCf.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture cf) {
targetLoop.register(cf.channel())
.addListener(completeHandler);
}
});
Reusing Event Loop
new ServerBootstrap().group(new EpollEventLoopGroup(1), new EpollEventLoopGroup()
).bind(80);
Reusing Event LoopEventLoopGroup boss = new EpollEventLoopGroup(1);
EventLoopGroup workers = new EpollEventLoopGroup();
new ServerBootstrap().group(
boss,
workers
).bind(80);
new ServerBootstrap().group(
boss,
workers
).bind(443);
Use direct buffers
ctx.writeAndFlush(
new ResponseMessage(messageId, OK)
);
Use direct buffers
ByteBuf buf = ctx.alloc().buffer(3);//pool
buf.writeByte(messageId);
buf.writeShort(OK);
ctx.writeAndFlush(buf);
Less allocations
ByteBuf msg = makeResponse(...);msg.retain(targets.size() - 1);
for (Channel ch : targets) { ch.writeAndFlush(msg);}
Void promise
ctx.writeAndFlush(
response
);
Void promise
ctx.writeAndFlush(
response, ctx.voidPromise()
);
Reuse handlers
@Sharable
public class StringDecoder extends MessageToMessageDecoder<ByteBuf> {
...
}
Prefer context
ctx.channel().writeAndFlush();
Prefer context
ctx.channel().writeAndFlush();
ctx.writeAndFlush();
Simpler - faster
ChannelInboundHandlerAdapter
does nothing, but fast
Simpler - faster
ByteToMessageDecoder
does some work, but slower
Simpler - faster
ReplayingDecoder
does job for you, but slowest
Turn off leak detection
ResourceLeakDetector.setLevel(
ResourceLeakDetector.Level.DISABLED);
What else?
● ASCIIString● FastThreadLocal● Unsafe● Optimized Encoders
● Really fast● Low GC load● Flexible● Rapidly evolve● Cool support
Summary
● Hard● Memory leaks● Still have issues
Summary
https://github.com/blynkkk/blynk-server
top related