Plan your Log Platform at Google Cloud Platform

Plan your Log Platform at

Google Cloud PlatformSimon Su @ GCP

Plan your Log Platform at

Google Cloud Platform

end to end log collectorsstore and query

presentation

Popular log solutions

Fluentd is an open source data collector for unified logging layer

Google solutionUnlimited data collect and store

●

●

●●●●●

Processing InputDirect Input

Common Protocol InputSecure Forward Bridge

tail input

syslog input

● Client site using log transfer agent like fluentd or logstash or others as data transfer tool to BigQuery.

http input

tcp output

http output

tcp input

● GCE implements a fluentd service to open common protocol input sockets

● Client site should has a secure tunnel for data transfer.● Client site using common protocol like http(s), tcp, udp,

syslog or others to transfer log data to GCE gateway.

secure forward output

secure forward

input

● GCE implements a fluentd service to open secure forward input socket.

● Client site has a fluentd service to implement secure forward output as other data transfer gateway

● Other log client sent logs to the gateway

Summarize

BigQuery● Customize schema● Co-relation● Higher quota and

performance

Cloud Logging● Web console● Export available for

BQ, GCS, Pub/Sub● Configure input

source

BigQuery Streaming Insert (Output Plugin)

syslog input

http(s) input

tcp/udp input

secure forward input

other inputs

Secure Forward(Output Plugin)

TCP(Output Plugin)

BigQuery Streaming Insert (Output Plugin)

Input Plugins

Log Store LifecycleBest cost price value concern

CloudLogging

Nearline

●