Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Login with AWS Elasticsearch Services.NE T CO R E

Overview

• Released in 2006

• Previous version not supported for structured login

• Introduced structured login in 2017 which is support for .NET standards

Overview

• The newest login framework

• Released in 2013

• The big difference between Serilog and the other frameworks is that it is designed to do structured login out of the box

• Serilog, was.. NLog also supports structured login, but it is was only added recently and the benchmarks suggest that using it has some serious performance implications.

Serilog

• Structured login support in Serilog

• Ease of configuration

• API is more modern

• Ability to add enrichers give you the ability to intercept and modify the messages (append properties to the log messages)

• Can store logs in different destinations (Sinks)

• Ex: Amozon Cloudwatch, Azure Blob Storage, Azure Analytics, Rolling File, Azure Event Hub, Elasticsearch

Serilog

NLog Benchmark

Serilog Benchmark

Conclusion

• Serilog log provider and the NLog log provider are similarly configured

• The results for Serilog are much better than the ones for NLog in both throughput and latency.

• In most cases latency for Serilog is half as much as the latency for NLog, and throughput twice as much.

Serilog

• Advantages

• Widely adopted

• A lot of documentation

• Great community

• Disadvantages

• A bit harder to learn when coming from log4net

NLog

• Advantages

• A lot of documentation

• Having been around a long time, there are lots of blog posts

• Easy to get started when coming from other login frameworks

• Disadvantages

• Structured login is still a bit behind Serilog

• C#-based API is harder to use than Serilog's fluent API

Centralized login in .Net Core using AWS Elasticsearch Service

Serilog & Kibana

• Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

• It is No Sql, distributed, full text database

• built on Apache Lucene and was first released in 2010

• Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of open source tools for data ingestion, enrichment, storage, analysis, and visualization

• Referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana)

Elasticsearch Used For?• Application search

• Website search

• Enterprise search

• login and log analytics

• Infrastructure metrics and container monitoring

• Application performance monitoring

• Geospatial data analysis and visualization

• Security analytics

• Business analytics

How does Elasticsearch work?

• Raw data flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications

• Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch.

Elasticsearch Index

• Collection of documents that are related to each other

• Stores data as JSON documents

• Document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data).

Logstash

• Core products of the Elastic Stack

• Used to aggregate and process data and send it to Elasticsearch

• An open source, server-side data processing pipeline

• Enables you to ingest data from multiple sources simultaneously and enrich and transform it before it is indexed into Elasticsearch

Kibana

• Data visualization and management tool for Elasticsearch

• Provides real-time histograms, line graphs, pie charts, and maps

• Includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.

Why use Elasticsearch?

• Elasticsearch is fast

• Elasticsearch is distributed by nature

• Elasticsearch comes with a wide set of features

• The Elastic Stack simplifies data ingest, visualization, and reporting

Elasticsearch Logging Process

How to Setup AWS Elasticsearch Service

• Setup Elastic Search Domain

• Configure Cluster

• Setup Access

• Setup Access Policy

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": "*"

},

"Action": "es:*",

"Resource": "arn:aws:es:us-east-1:762493928111:domain/youcab/*"

}

]

}

Elasticsearch Service Summary

.Net Core Application Setup

• Required Packages

• Elasticsearch.Net

• Elasticsearch.Net.AWS

• Serilog.Sinks.Elasticsearch

Setup AWS Elasticsearch Credentials (appsetting.config)

"AWS": {

"AccessKey": "AKIAJOUREKVWNFMYLKOA",

"SecretKey": "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",

"Region": "us-east-1",

"ElasticUrl": "https://search-youcab-paf2n6jkgms3y5kxtzajjywspm.us-east-1.es.amazonaws.com"

}

Logger Service Project Setup (.Net Core)

• Setup AWS credentials in Startup class

• Configure AWS Elasticsearch Connection String

Demonstration

Elasticsearch Available Methods

• AWS Elasticsearch

• Azure Elasticsearch

• Elasticsearch Cluster

Any Questions???