The Log Stash Book

8/10/2019 The Log Stash Book

1/220

The Book

Log management made easy

James Turnbull

logstash


2/220

The Logstash Book

James Turnbull

January 26, 2014

Version: v1.3.4 (5f439d1)

Website: The Logstash Book
http://www.logstashbook.com/


3/220

Contents

Foreword 1

Who is this book for? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Credits and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 1

Technical Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Jan-Piet Mens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Paul Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Technical Illustrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Conventions in the book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Code and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Colophon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Copyright . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Chapter 1 Introduction or Why Should I Bother? 6

Introducing Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Logstash design and architecture . . . . . . . . . . . . . . . . . . . . . . . . 8

What's in the book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Logstash resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Getting help with Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

A mild warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 2 Getting Started with Logstash 12Installing Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

On the Red Hat family . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

i


4/220

Contents

On Debian & Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Testing Java is installed . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Getting Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Starting Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Our sample conguration le . . . . . . . . . . . . . . . . . . . . . . . . 15

Running the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . . 16

Testing the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Chapter 3 Shipping Events 21

Our Event Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Installing Logstash on our central server . . . . . . . . . . . . . . . . . . . 23

Installing a broker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

ElasticSearch for Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Creating a basic central conguration . . . . . . . . . . . . . . . . . . . 34

Running Logstash as a service . . . . . . . . . . . . . . . . . . . . . . . . 36

Installing Logstash on our rst agent . . . . . . . . . . . . . . . . . . . . . 39

Our agent conguration . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Installing Logstash as a service . . . . . . . . . . . . . . . . . . . . . . . 43

Sending our rst events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Checking ElasticSearch has received our events . . . . . . . . . . . . . 47

The Logstash Kibana Console . . . . . . . . . . . . . . . . . . . . . . . . 49

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Chapter 4 Shipping Events without the Logstash agent 57

Using Syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

A quick introduction to Syslog . . . . . . . . . . . . . . . . . . . . . . . 58

Conguring Logstash for Syslog . . . . . . . . . . . . . . . . . . . . . . 59

Conguring Syslog on remote agents . . . . . . . . . . . . . . . . . . . 62

Using the Logstash Forwarder . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Congure the Logstash Forwarder on our central server . . . . . . . . 72

Installing the Logstash Forwarder on the remote host . . . . . . . . . 77

Other log shippers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Beaver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Woodchuck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Version: v1.3.4 (5f439d1) ii


5/220

Contents

Others . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Chapter 5 Filtering Events with Logstash 88Apache Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Conguring Apache for Custom Logging . . . . . . . . . . . . . . . . . 90

Sending Apache events to Logstash . . . . . . . . . . . . . . . . . . . . 97

Postx Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Our rst lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Adding our own lters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Extracting from dierent events . . . . . . . . . . . . . . . . . . . . . . 111

Setting the timestamp . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Filtering Java application logs . . . . . . . . . . . . . . . . . . . . . . . . . 118

Handling blank lines with drop . . . . . . . . . . . . . . . . . . . . . . . 119Handling multi-line log events . . . . . . . . . . . . . . . . . . . . . . . 121

Grokking our Java events . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Parsing an in-house custom log format . . . . . . . . . . . . . . . . . . . . 127

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Chapter 6 Outputting Events from Logstash 137

Send email alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Updating our multiline lter . . . . . . . . . . . . . . . . . . . . . . . . 138

Conguring the email output . . . . . . . . . . . . . . . . . . . . . . . . 138

Email output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140Send instant messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Identifying the event to send . . . . . . . . . . . . . . . . . . . . . . . . 141

Sending the instant message . . . . . . . . . . . . . . . . . . . . . . . . . 143

Send alerts to Nagios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Nagios check types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Identifying the trigger event . . . . . . . . . . . . . . . . . . . . . . . . . 145

The nagios output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

The Nagios external command . . . . . . . . . . . . . . . . . . . . . . . 148

The Nagios service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Outputting metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Collecting metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Version: v1.3.4 (5f439d1) iii


6/220

Contents

StatsD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Setting the date correctly . . . . . . . . . . . . . . . . . . . . . . . . . . 153

The StatsD output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Sending to a dierent StatsD server . . . . . . . . . . . . . . . . . . . . 159Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Chapter 7 Scaling Logstash 161

Scaling Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Installing new Redis instances . . . . . . . . . . . . . . . . . . . . . . . 164

Test Redis is running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

Conguring Redis output to send to multiple Redis servers . . . . . . 166

Conguring Logstash to receive from multiple Redis servers . . . . . 167

Testing our Redis failover . . . . . . . . . . . . . . . . . . . . . . . . . . 168

Shutting down our existing Redis instance . . . . . . . . . . . . . . . . 170Scaling ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Installing additional ElasticSearch hosts . . . . . . . . . . . . . . . . . 171

Monitoring our ElasticSearch cluster . . . . . . . . . . . . . . . . . . . 175

Managing ElasticSearch data retention . . . . . . . . . . . . . . . . . . 176

More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

Scaling Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

Creating a second indexer . . . . . . . . . . . . . . . . . . . . . . . . . . 180

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Chapter 8 Extending Logstash 183Anatomy of a plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

Creating our own input plugin . . . . . . . . . . . . . . . . . . . . . . . . . 187

Adding new plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Adding a plugin via directory . . . . . . . . . . . . . . . . . . . . . . . . 191

Adding a plugin to the Logstash JAR le . . . . . . . . . . . . . . . . . 193

Writing a lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Writing an output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Index 200

Version: v1.3.4 (5f439d1) iv


7/220

List of Figures

1 Copyright . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1 The Logstash Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.1 Our Event Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2 The Logstash web interface . . . . . . . . . . . . . . . . . . . . . . . . . 50

3.3 The Logstash web interface's light theme . . . . . . . . . . . . . . . . . 51

3.4 Query results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.5 Specic events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.6 Basic query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.7 Advanced query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

3.8 Customizing the dashboard . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.9 Adding a panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.10The Dashboard control panel . . . . . . . . . . . . . . . . . . . . . . . . 56

4.1 Syslog shipping to Logstash . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.1 Apache log event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

5.2 Querying for 404 status codes . . . . . . . . . . . . . . . . . . . . . . . . 99

5.3 Postx log ltering workow . . . . . . . . . . . . . . . . . . . . . . . . 118

5.4 Tomcat log event workow . . . . . . . . . . . . . . . . . . . . . . . . . 126

5.5 The Grok debugger at work . . . . . . . . . . . . . . . . . . . . . . . . . 130

6.1 Java exception email alert . . . . . . . . . . . . . . . . . . . . . . . . . . 140

6.2 Jabber/XMPP alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

6.3 Apache status and method graphs . . . . . . . . . . . . . . . . . . . . . 157

6.4 Apache bytes counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1586.5 Apache request duration timer . . . . . . . . . . . . . . . . . . . . . . . 159

v


8/220

List of Figures

7.1 Logstash Scaled Architecture . . . . . . . . . . . . . . . . . . . . . . . . 162

7.2 Logstash Redis failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

7.3 ElasticSearch scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

7.4 The Paramedic ElasticSearch plugin . . . . . . . . . . . . . . . . . . . . 1767.5 Logstash indexer scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

8.1 Cow said "testing" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Version: v1.3.4 (5f439d1) vi


9/220

Listings

2.1 Installing Java on Red Hat . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Installing Java on Debian and Ubuntu . . . . . . . . . . . . . . . . . . 13

2.3 Testing Java is installed . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.4 Downloading Logstash . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5 Sample Logstash conguration . . . . . . . . . . . . . . . . . . . . . . 15

2.6 Running the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . 16

2.7 Logstash startup message . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.8 Tuning the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.9 Running Logstash interactively . . . . . . . . . . . . . . . . . . . . . . 18

2.10 A Logstash JSON event . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.11 A Logstash plain event . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1 Installing Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.2 Creating the Logstash directory . . . . . . . . . . . . . . . . . . . . . . 24

3.3 Downloading the Logstash JAR le . . . . . . . . . . . . . . . . . . . 243.4 Creating Logstash conguration directory . . . . . . . . . . . . . . . 24

3.5 Creating Logstash log directory . . . . . . . . . . . . . . . . . . . . . . 24

3.6 Installing Redis on Debian . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.7 Installing EPEL on CentOS and RHEL . . . . . . . . . . . . . . . . . . 26

3.8 Installing Redis on Red Hat . . . . . . . . . . . . . . . . . . . . . . . . 26

3.9 Changing the Redis interface . . . . . . . . . . . . . . . . . . . . . . . 26

3.10 Commented out interface . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.11 Binding Redis to a single interface . . . . . . . . . . . . . . . . . . . . 27

3.12 Starting the Redis server . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.13 Testing Redis is running . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.14 Telneting to the Redis server . . . . . . . . . . . . . . . . . . . . . . . 28

3.15 A Logstash index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

vii


10/220

Listings

3.16 Downloading ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . 31

3.17 Installing ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.18 Starting ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.19 Initial cluster and node names . . . . . . . . . . . . . . . . . . . . . . 323.20 New cluster and node names . . . . . . . . . . . . . . . . . . . . . . . 32

3.21 Restarting ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.22 Checking ElasticSearch is running . . . . . . . . . . . . . . . . . . . . 33

3.23 ElasticSearch status information . . . . . . . . . . . . . . . . . . . . . 33

3.24 ElasticSearch status page . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.25 Creating the central.conf le . . . . . . . . . . . . . . . . . . . . . . . 34

3.26 Initial central conguration . . . . . . . . . . . . . . . . . . . . . . . . 35

3.27 Copying the central init script . . . . . . . . . . . . . . . . . . . . . . . 37

3.28 Starting the central Logstash server . . . . . . . . . . . . . . . . . . . 37

3.29 Checking the Logstash server is running . . . . . . . . . . . . . . . . 38

3.30 Checking the Logstash process . . . . . . . . . . . . . . . . . . . . . . 38

3.31 Logstash log output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.32 Create the Logstash directory . . . . . . . . . . . . . . . . . . . . . . . 39

3.33 Download Logstash JAR le . . . . . . . . . . . . . . . . . . . . . . . . 39

3.34 Creating the remaining Logstash directories . . . . . . . . . . . . . . 40

3.35 Creating the Logstash agent conguration . . . . . . . . . . . . . . . 40

3.36 Logstash event shipping conguration . . . . . . . . . . . . . . . . . . 41

3.37 File input globbing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.38 File recursive globbing . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.39 Conguring the Logstash agent init script . . . . . . . . . . . . . . . . 44

3.40 Starting the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . 44

3.41 Checking the Logstash agent is running . . . . . . . . . . . . . . . . . 44

3.42 Checking the Logstash process is running . . . . . . . . . . . . . . . . 45

3.43 Logstash agent startup log output . . . . . . . . . . . . . . . . . . . . 45

3.44 Watching the Logstash shipper.log le . . . . . . . . . . . . . . . . . 46

3.45 Watching the Logstash central.log le . . . . . . . . . . . . . . . . . . 46

3.46 Testing Redis is operational . . . . . . . . . . . . . . . . . . . . . . . . 46

3.47 Connecting to Maurice via SSH . . . . . . . . . . . . . . . . . . . . . . 46

3.48 A Logstash login event . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.49 Querying the ElasticSearch server . . . . . . . . . . . . . . . . . . . . 48

3.50 Launching the Logstash KIbana web interface . . . . . . . . . . . . . 49

Version: v1.3.4 (5f439d1) viii


11/220

Listings

3.51 Logstash web interface address . . . . . . . . . . . . . . . . . . . . . . 50

4.1 A Syslog message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.2 Adding the `syslog` input . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3 The `syslog` input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604.4 Restarting the Logstash server . . . . . . . . . . . . . . . . . . . . . . . 61

4.5 Syslog input startup output . . . . . . . . . . . . . . . . . . . . . . . . 61

4.6 Conguring RSyslog for Logstash . . . . . . . . . . . . . . . . . . . . . 64

4.7 Specifying RSyslog facilities or priorities . . . . . . . . . . . . . . . . 64

4.8 Restarting RSyslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4.9 Monitoring les with the imle module . . . . . . . . . . . . . . . . . 65

4.10 SyslogNG s_src source statement . . . . . . . . . . . . . . . . . . . . . 67

4.11 New SyslogNG destination . . . . . . . . . . . . . . . . . . . . . . . . 67

4.12 New SyslogNG log action . . . . . . . . . . . . . . . . . . . . . . . . . 67

4.13 Restarting SyslogNG . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.14 Conguring Syslogd for Logstash . . . . . . . . . . . . . . . . . . . . . 68

4.15 Restarting Syslogd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.16 Testing with logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.17 Logstash log event from Syslog . . . . . . . . . . . . . . . . . . . . . . 71

4.18 Checking for openssl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.19 Generating a private key . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.20 Generating a CSR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.21 Signing our CSR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.22 Copying the key and certicate . . . . . . . . . . . . . . . . . . . . . . 744.23 Cleaning up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.24 Adding the Lumberjack input . . . . . . . . . . . . . . . . . . . . . . . 75

4.25 The Lumberjack input . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.26 Restarting Logstash for Lumberjack . . . . . . . . . . . . . . . . . . . 76

4.27 Checking Lumberjack has loaded . . . . . . . . . . . . . . . . . . . . . 77

4.28 Downloading the Forwarder . . . . . . . . . . . . . . . . . . . . . . . . 77

4.29 Installing the developer tools . . . . . . . . . . . . . . . . . . . . . . . 78

4.30 Installing Go on Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.31 Installing prerequisite Forwarder packages . . . . . . . . . . . . . . . 78

4.32 Installing FPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784.33 Creating a Forwarder DEB package . . . . . . . . . . . . . . . . . . . 78

4.34 Forwarder make output . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Version: v1.3.4 (5f439d1) ix


12/220

Listings

4.35 Installing the Forwarder . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.36 Creating the Forwarder conguration directory . . . . . . . . . . . . 79

4.37 Copying the Forwarder's SSL certicate . . . . . . . . . . . . . . . . . 80

4.38 Creating logstash-forwarder.conf . . . . . . . . . . . . . . . . . . . . . 804.39 The logstash-forwarder.conf le . . . . . . . . . . . . . . . . . . . . . 81

4.40 Testing the Forwarder . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.41 Test the Forwarder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.42 The Forwarder connection output . . . . . . . . . . . . . . . . . . . . 83

4.43 Forwarder events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.44 Installing the Forwarder init script . . . . . . . . . . . . . . . . . . . . 84

4.45 The Forwarder defaults le . . . . . . . . . . . . . . . . . . . . . . . . 84

4.46 Starting the Forwarder . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.47 Checking the Forwarder process . . . . . . . . . . . . . . . . . . . . . 85

4.48 Installing Beaver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.1 An Apache log event . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.2 The Apache LogFormat and CustomLog directives . . . . . . . . . . 91

5.3 Apache VirtualHost logging conguration . . . . . . . . . . . . . . . 91

5.4 The Apache Common Log Format LogFormat directive . . . . . . . 92

5.5 Apache custom JSON LogFormat . . . . . . . . . . . . . . . . . . . . . 93

5.6 Adding the CustomLog directive . . . . . . . . . . . . . . . . . . . . . 94

5.7 Restarting Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.8 A JSON format event from Apache . . . . . . . . . . . . . . . . . . . . 96

5.9 Apache logs via the le input . . . . . . . . . . . . . . . . . . . . . . . 975.10 Apache events via the Logstash Forwarder . . . . . . . . . . . . . . . 97

5.11 A Postx log entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

5.12 Unltered Postx event . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.13 File input for Postx logs . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.14 Postx grok lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.15 The grok pattern for Postx logs . . . . . . . . . . . . . . . . . . . . . 103

5.16 The syntax and the semantic . . . . . . . . . . . . . . . . . . . . . . . 103

5.17 The SYSLOGBASE pattern . . . . . . . . . . . . . . . . . . . . . . . . . 103

5.18 The SYSLOGPROG pattern . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.19 The PROG pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.20 Postx date matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.21 Converting semantic data . . . . . . . . . . . . . . . . . . . . . . . . . 105

Version: v1.3.4 (5f439d1) x


13/220

Listings

5.22 The Postx event's elds . . . . . . . . . . . . . . . . . . . . . . . . . . 105

5.23 A fully grokked Postx event . . . . . . . . . . . . . . . . . . . . . . . 106

5.24 Partial Postx event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

5.25 Creating the patterns directory . . . . . . . . . . . . . . . . . . . . . . 1075.26 Creating new patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

5.27 Adding new patterns to grok lter . . . . . . . . . . . . . . . . . . . . 108

5.28 Postx event grokked with external patterns . . . . . . . . . . . . . . 109

5.29 A named capture for Postx's queue ID . . . . . . . . . . . . . . . . . 110

5.30 Adding new named captures to the grok lter . . . . . . . . . . . . . 110

5.31 Postx event ltered with named captures . . . . . . . . . . . . . . . 111

5.32 Postx event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5.33 Updated grok lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5.34 Postx component tagged events . . . . . . . . . . . . . . . . . . . . . 112

5.35 Nested eld syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5.36 A grok lter for qmgr events . . . . . . . . . . . . . . . . . . . . . . . 113

5.37 The /etc/logstash/patterns/postx le . . . . . . . . . . . . . . . . . 114

5.38 A partial ltered Postx event . . . . . . . . . . . . . . . . . . . . . . 115

5.39 The date lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

5.40 Postx event timestamps . . . . . . . . . . . . . . . . . . . . . . . . . . 117

5.41 File input for Tomcat logs . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.42 A Tomcat log entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.43 A drop lter for blank lines . . . . . . . . . . . . . . . . . . . . . . . . 120

5.44 Examples of the conditional syntax . . . . . . . . . . . . . . . . . . . . 1205.45 Conditional inclusion syntax . . . . . . . . . . . . . . . . . . . . . . . . 121

5.46 Using the multiline codec for Java exceptions . . . . . . . . . . . . . 122

5.47 A Java exception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5.48 Another Java exception . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5.49 A multiline merged event . . . . . . . . . . . . . . . . . . . . . . . . . 124

5.50 A grok lter for Java exception events . . . . . . . . . . . . . . . . . 124

5.51 Our Java exception message . . . . . . . . . . . . . . . . . . . . . . . . 125

5.52 Grokked Java exception . . . . . . . . . . . . . . . . . . . . . . . . . . 125

5.53 Alpha log entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

5.54 File input for our Alpha logs . . . . . . . . . . . . . . . . . . . . . . . . 1285.55 Single Alpha log entry . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

5.56 A Grok regular expression for Alpha . . . . . . . . . . . . . . . . . . . 129

Version: v1.3.4 (5f439d1) xi


14/220

Listings

5.57 Alpha grok lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

5.58 Alpha date lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

5.59 Alpha environment eld . . . . . . . . . . . . . . . . . . . . . . . . . . 133

5.60 Setting the line eld to an integer . . . . . . . . . . . . . . . . . . . . 1345.61 A ltered Alpha event . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

6.1 The Tomcat multiline le input and codec . . . . . . . . . . . . . . . 138

6.2 The email output plugin . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6.3 The content of our email . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6.4 The le input for /var/log/secure . . . . . . . . . . . . . . . . . . . . 141

6.5 Failed SSH authentication log entry . . . . . . . . . . . . . . . . . . . 141

6.6 Failed SSH authentication grok lter . . . . . . . . . . . . . . . . . . 142

6.7 Failed SSH authentication Logstash event . . . . . . . . . . . . . . . . 143

6.8 The xmpp output plugin . . . . . . . . . . . . . . . . . . . . . . . . . . 144

6.9 A STONITH cluster fencing log event . . . . . . . . . . . . . . . . . . 146

6.10 Identify Nagios passive check results . . . . . . . . . . . . . . . . . . 146

6.11 The grokked STONITH event . . . . . . . . . . . . . . . . . . . . . . . 147

6.12 The Nagios output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

6.13 The Nagios output with a custom command le . . . . . . . . . . . . 148

6.14 A Nagios external command . . . . . . . . . . . . . . . . . . . . . . . . 149

6.15 A Nagios service for cluster status . . . . . . . . . . . . . . . . . . . . 150

6.16 JSON format event from Apache . . . . . . . . . . . . . . . . . . . . . 152

6.17 The Apache event timestamp eld . . . . . . . . . . . . . . . . . . . . 153

6.18 Getting the date right for our metrics . . . . . . . . . . . . . . . . . . 1546.19 The statsd output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

6.20 Incremental counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

6.21 Apache status metrics in Graphite . . . . . . . . . . . . . . . . . . . . 156

6.22 Apache method metrics in Graphite . . . . . . . . . . . . . . . . . . . 156

6.23 The apache.bytes counter . . . . . . . . . . . . . . . . . . . . . . . . . 157

6.24 The apache.duration timer . . . . . . . . . . . . . . . . . . . . . . . . . 158

6.25 The StatsD output with a custom host and port . . . . . . . . . . . . 159

7.1 Installing Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

7.2 Binding Redis to the external interface . . . . . . . . . . . . . . . . . 165

7.3 Start the Redis instances . . . . . . . . . . . . . . . . . . . . . . . . . . 1657.4 Test Redis is running . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

7.5 Multi instance Redis output conguration . . . . . . . . . . . . . . . 166

Version: v1.3.4 (5f439d1) xii


15/220

Listings

7.6 Restarting the Logstash agent for Redis . . . . . . . . . . . . . . . . . 167

7.7 Multiple Redis instances . . . . . . . . . . . . . . . . . . . . . . . . . . 168

7.8 Restart the Logstash agent . . . . . . . . . . . . . . . . . . . . . . . . . 168

7.9 Stopping a Redis instance . . . . . . . . . . . . . . . . . . . . . . . . . 1697.10 Redis connection refused exception . . . . . . . . . . . . . . . . . . . 169

7.11 Stopping a second Redis instance . . . . . . . . . . . . . . . . . . . . . 169

7.12 Remote agent event sending failures . . . . . . . . . . . . . . . . . . . 170

7.13 Shut down Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.14 Stop Redis starting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.15 Installing Java for ElasticSearch . . . . . . . . . . . . . . . . . . . . . 172

7.16 Download ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . 172

7.17 Install ElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

7.18 ElasticSearch cluster and node names . . . . . . . . . . . . . . . . . . 173

7.19 Grinner cluster and node names . . . . . . . . . . . . . . . . . . . . . 173

7.20 Sinner cluster and node names . . . . . . . . . . . . . . . . . . . . . . 173

7.21 Restarting ElasticSearch to recongure . . . . . . . . . . . . . . . . . 173

7.22 Checking the cluster status. . . . . . . . . . . . . . . . . . . . . . . . . 174

7.23 Installing Paramedic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

7.24 The Paramedic URL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

7.25 Deleting indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

7.26 Optimizing indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

7.27 Optimizing all indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

7.28 Getting the size of an index . . . . . . . . . . . . . . . . . . . . . . . . 1787.29 Setting up a second indexer . . . . . . . . . . . . . . . . . . . . . . . . 181

7.30 Starting second Logstash instance . . . . . . . . . . . . . . . . . . . . 181

8.1 The stdin input plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

8.2 Requiring the Logstash module . . . . . . . . . . . . . . . . . . . . . . 186

8.3 Requiring the LogStash::Inputs::Base class . . . . . . . . . . . . . . . 186

8.4 The plugin class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

8.5 The namedpipe framework . . . . . . . . . . . . . . . . . . . . . . . . 187

8.6 The namedpipe framework plugin options . . . . . . . . . . . . . . . 188

8.7 The namedpipe input conguration . . . . . . . . . . . . . . . . . . . 189

8.8 The namedpipe input . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1908.9 Creating plugins directories . . . . . . . . . . . . . . . . . . . . . . . . 191

8.10 Adding the namedpipe input . . . . . . . . . . . . . . . . . . . . . . . 192

Version: v1.3.4 (5f439d1) xiii


16/220

Listings

8.11 Running Logstash with plugin support . . . . . . . . . . . . . . . . . . 192

8.12 Registering the namedpipe input . . . . . . . . . . . . . . . . . . . . . 192

8.13 Creating directory structure . . . . . . . . . . . . . . . . . . . . . . . . 193

8.14 Copying the plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1938.15 Adding a plugin to the JAR le . . . . . . . . . . . . . . . . . . . . . . 193

8.16 Checking the plugin has been added . . . . . . . . . . . . . . . . . . . 193

8.17 Our sux lter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

8.18 Conguring the addsux lter . . . . . . . . . . . . . . . . . . . . . . 195

8.19 An event with the ALERT sux . . . . . . . . . . . . . . . . . . . . . . 196

8.20 Installing CowSay on Debian and Ubuntu . . . . . . . . . . . . . . . . 196

8.21 Installing CowSay via a RubyGem . . . . . . . . . . . . . . . . . . . . 196

8.22 The CowSay output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

8.23 Conguring the cowsay output . . . . . . . . . . . . . . . . . . . . . . 198

Version: v1.3.4 (5f439d1) xiv


17/220

Foreword

Who is this book for?

This book is designed for SysAdmins, operations sta, developers and DevOpswho are interested in deploying a log management solution using the open source

toolLogstash.

There is an expectation that the reader has basic Unix/Linux skills, and is familiar

with the command line, editing les, installing packages, managing services, and

basic networking.

NOTE This book focuses on Logstash version 1.2.0 and later. It is not recom-mended for earlier versions of Logstash.

Credits and Acknowledgments

Jordan Sissel for writing Logstash and for all his assistance during the writing

process.

Rashid Khan for writing Kibana.

Dean Wilson for his feedback on the book.

Aaron Mildenstein for his Apache to JSON logging postshereandhere.

R.I. Pienaar for his excellent documentation on message queuing.

1
http://untergeek.com/2013/09/11/getting-apache-to-output-json-for-logstash-1-2-x/http://untergeek.com/2012/10/11/getting-apache-to-output-json-for-logstash/http://www.logstash.net/


18/220

Foreword

The ne folks in the Freenode #logstash channel for being so helpful as I

peppered them with questions, and

Ruth Brown for only saying "Another book? WTF?" once, proof reading the

book, making the cover page and for being awesome.

Technical Reviewers

Jan-Piet Mens

Jan-Piet Mens is an independent Unix/Linux consultant and sysadmin who's

worked with Unix-systems since 1985. JP does odd bits of coding, and has

architected infrastructure at major customers throughout Europe. One of hisspecialities is the Domain Name System and as such, he authored the book

Alternative DNS Serversas well as a variety ofother technical publications.

Paul Stack

Paul Stack is a London based developer. He has a passion for continuous inte-

gration and continuous delivery and why they should be part of what developers

do on a day to day basis. He believes that reliably delivering software is just as

important as its development. He talks at conferences all over the world on thissubject. Paul's passion for continuous delivery has led him to start working closer

with operations sta and has led him to technologies like Logstash, Puppet and

Chef.

Technical Illustrator

Royce Gilbert has over 30 years experience in CAD design, computer support,

network technologies, project management, business systems analysis for majorFortune 500 companies such as; Enron, Compaq, Koch Industries and Amoco Corp.

He is currently employed as a Systems/Business Analyst at Kansas State University

Version: v1.3.4 (5f439d1) 2
mailto:[email protected]://mens.de/http://twitter.com/jpmens


19/220

Foreword

in Manhattan, KS. In his spare time he does Freelance Art and Technical Illustra-

tion as sole proprietor of Royce Art. He and his wife of 38 years are living in and

restoring a 127 year old stone house nestled in the Flinthills of Kansas.

Author

James is an author and open source geek. James authored the two books about

Puppet (Pro Puppetand theearlier bookabout Puppet). He is also the author of

three other books includingPro Linux System Administration,Pro Nagios 2.0, and

Hardening Linux.

For a real job, James is VP of Engineering for Venmo. He was formerly VP of

Technical Operations for Puppet Labs. He likes food, wine, books, photographyand cats. He is not overly keen on long walks on the beach and holding hands.

Conventions in the book

This is an inline code statement.

This is a code block:

Listing 1: A sample code block

This is a code block

Long code strings are broken with .

Code and Examples

You can nd all the code and examples from the book on the websiteor you cancheck out theGit repo.

Version: v1.3.4 (5f439d1) 3
https://github.com/jamtur01/logstashbook-codehttp://www.logstashbook.com/code/index.htmlhttp://www.amazon.com/gp/product/1590594444?ie=UTF8&tag=puppet0e-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1590594444http://www.amazon.com/gp/product/1590596099?ie=UTF8&tag=puppet0e-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1590596099http://www.amazon.com/gp/product/1430219122?ie=UTF8&tag=puppet0e-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1430219122http://www.amazon.com/gp/product/1590599780?ie=UTF8&tag=puppet0e-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1590599780http://www.amazon.com/gp/product/1430230576/ref=as_li_ss_tl?ie=UTF8&tag=puppet0e-20&linkCode=as2&camp=217145&creative=399349&creativeASIN=1430230576


20/220

Foreword

Colophon

This book was written in Markdown with a large dollop of LaTeX. It was then

converted to PDF and other formats using PanDoc (with some help from scripts

written by the excellent folks who wroteBackbone.js on Rails).

Errata

Please email any Errata you ndhere.

Trademarks

Kibana and Logstash are trademarks of Elasticsearch BV. Elasticsearch is a regis-

tered trademark of Elasticsearch BV.

Version

This is version v1.3.4 (5f439d1) of The Logstash Book.

Copyright

Figure 1: Copyright

Some rights reserved. No part of this publication may be reproduced, stored in a

retrieval system or transmitted in any form or by any means, electronic, mechan-

ical or photocopying, recording, or otherwise for commercial purposes without

the prior permission of the publisher.

Version: v1.3.4 (5f439d1) 4
mailto:[email protected]://learn.thoughtbot.com/products/1-backbone-js-on-rails


21/220

Foreword

This work is licensed under the Creative Commons Attribution-NonCommercial-

NoDerivs 3.0 Unported License. To view a copy of this license, visithere.

Copyright 2014 - James Turnbull

ISBN: 978-0-9888202-1-0

Version: v1.3.4 (5f439d1)

Version: v1.3.4 (5f439d1) 5
mailto:[email protected]://creativecommons.org/licenses/by-nc-nd/3.0/


22/220

Chapter 1

Introduction or Why Should I

Bother?

Log management is often considered both a painful exercise and a dark art. In-

deed, understanding good log management tends to be a slow and evolutionary

process. In response to issues and problems, new SysAdmins are told: "Go look at

the logs." A combination ofcat, tail and grep (and often sed, awk or perl too)

become their tools of choice to diagnose and identify problems in log and event

data. They quickly become experts at command line and regular expression kung-

fu: searching, parsing, stripping, manipulating and extracting data from a humble

log event. It's a powerful and practical set of skills that strongly I recommend allSysAdmins learn.

Sadly, this solution does not scale. In most cases you have more than one host and

multiple sources of log les. You may have tens, hundreds or even thousands of

hosts. You run numerous, inter-connected applications and services across multi-

ple locations and fabrics, both physically, virtually and in the cloud. In this world

it quickly becomes apparent that logs from any one application, service or host

are not enough to diagnose complex multi-tier issues.

To address this gap your log environment must evolve to become centralized. The

tools of choice expand to include conguring applications to centrally log andservices like rsyslog and syslog-ng to centrally deliver Syslog output. Events

start owing in and log servers to hold this data are built, consuming larger and

6


23/220

Chapter 1: Introduction or Why Should I Bother?

larger amounts of storage.

But we're not done yet. The problem then turns from one of too little information

to one of too much information and too little context. You have millions or billions

of lines of logs to sift through. Those logs are produced in dierent timezones,

formats and sometimes even in dierent languages. It becomes increasingly hard

to sort through the growing streams of log data to nd the data you need and

harder again to correlate that data with other relevant events. Your growing

collection of log events then becomes more of a burden than a benet.

To solve this new issue you have to extend and expand your log management

solution to include better parsing of logs, more elegant storage of logs (as at les

just don't cut it) and the addition of searching and indexing technology. What

started as a simple grep through log les has become a major project in its own

right. A project that has seen multiple investment iterations in several solutions(or multiple solutions and their integration) with a commensurate cost in eort

and expense.

There is a better way.

Introducing Logstash

Instead of walking this path, with the high cost of investment and the potential of

evolutionary dead ends, you can start with Logstash. Logstash provides an inte-

grated framework for log collection, centralization, parsing, storage and search.

Logstash is free and open source (Apache 2.0licensed) and developed by Ameri-

can developer and Logging Czar at Dreamhost, Jordan Sissel. It's easy to set up,

performant, scalable and easy to extend.

Logstash has a wide variety of input mechanisms: it can take inputs from

TCP/UDP, les, Syslog, Microsoft Windows EventLogs, STDIN and a variety of

other sources. As a result there's likely very little in your environment that you

can't extract logs from and send them to Logstash.

When those logs hit the Logstash server, there is a large collection of lters that

allow you to modify, manipulate and transform those events. You can extract the

Version: v1.3.4 (5f439d1) 7
http://www.semicomplete.com/https://github.com/jordansisselhttp://www.apache.org/licenses/LICENSE-2.0.htmlhttp://logstash.net/


24/220


information you need from log events to give them context. Logstash makes it

simple to query those events. It makes it easier to draw conclusions and make

good decisions using your log data.

Finally, when outputting data, Logstash supports a huge range of destinations,

including TCP/UDP, email, les, HTTP, Nagios and a wide variety of network

and online services. You can integrate Logstash with metrics engines, alerting

tools, graphing suites, storage destinations or easily build your own integration

to destinations in your environment.

NOTE We'll look at how to develop practical examples of each of these input,lter and output plugins in Chapter 8.

Logstash design and architecture

Logstash is written in JRuby and runs in a Java Virtual Machine (JVM). Its archi-

tecture is message-based and very simple. Rather than separate agents or servers,

Logstash has a single agent that is congured to perform dierent functions in

combination with other open source components.

In the Logstash ecosystem there are four components:

Shipper: Sends events to Logstash. Your remote agents will generally only

run this component.

Broker and Indexer: Receives and indexes the events.

Search and Storage: Allows you to search and store events.

Web Interface: A Web-based interface to Logstash called Kibana.

Logstash servers run one or more of these components independently, which al-

lows us to separate components and scale Logstash.

In most cases there will be two broad classes of Logstash host you will probably

be running:

Version: v1.3.4 (5f439d1) 8


25/220


Hosts running the Logstash agent as an event "shipper" that send your appli-

cation, service and host logs to a central Logstash server. These hosts will

only need the Logstash agent.

Central Logstash hosts running some combination of the Broker, Indexer,Search and Storage and Web Interface which receive, process and store your

logs.

Figure 1.1: The Logstash Architecture

NOTE We'll look at scaling Logstash by running the Broker, Indexer, Search andStorage and Web Interface in a scalable architecture in Chapter 7 of this book.

Version: v1.3.4 (5f439d1) 9


26/220


What's in the book?

In this book I will walk you through installing, deploying, managing and extending

Logstash. We're going to do that by introducing you to Example.com, where you're

going to start a new job as one of its SysAdmins. The rst project you'll be in

charge of is developing its new log management solution.

We'll teach you how to:

Install and deploy Logstash.

Ship events from a Logstash Shipper to a central Logstash server.

Filter incoming events using a variety of techniques.

Output those events to a selection of useful destinations.

Use Logstash'sKibanaweb interface. Scale out your Logstash implementation as your environment grows.

Quickly and easily extend Logstash to deliver additional functionality you

might need.

By the end of the book you should have a functional and eective log management

solution that you can deploy into your own environment.

NOTE This book focusses on Logstash v1.2.0 and later. This was a major,somewhat backwards-incompatible release for Logstash. A number of options

and schema changes were made between v1.2.0 and earlier versions. If you are

running an earlier version of Logstash I strongly recommend you upgrade.

Logstash resources

The Logstash site(Logstash's home page).

The Logstash cookbook(a collection of useful Logstash recipes).

The Logstash source codeon GitHub.

Logstash's author Jordan Sissel'shome page,TwitterandGitHub account.

Version: v1.3.4 (5f439d1) 10
https://github.com/jordansisselhttps://twitter.com/jordansisselhttp://www.semicomplete.com/https://github.com/logstash/logstash/http://cookbook.logstash.net/http://www.logstash.net/http://kibana.org/


27/220


Getting help with Logstash

Logstash's developer, Jordan Sissel, has a maxim that makes getting help pretty

easy: "If a newbie has a bad time, it's a bug in Logstash." So if you're having

trouble reach out via the mailing list or IRC and ask for help! You'll nd the

Logstash community both helpful and friendly!

TheLogstash documentation.

TheLogstash cookbook.

TheLogstash users mailing list.

The Logstashbug tracker.

The #logstash IRC channel on Freenode.

A mild warning

Logstash is a young product and under regular development. Features are

changed, added, updated and deprecated regularly. I recommend you follow

development at theJira support site, on GitHuband review the change logs for

each release to get a good idea of what has changed. Logstash is usually solidly

backwards compatible but issues can emerge and being informed can often save

you unnecessary troubleshooting eort.

Version: v1.3.4 (5f439d1) 11
https://github.com/logstash/logstash/https://logstash.jira.com/secure/Dashboard.jspahttps://logstash.jira.com/secure/Dashboard.jspahttps://groups.google.com/forum/?fromgroups#!forum/logstash-usershttp://cookbook.logstash.net/http://logstash.net/docs/latest/


28/220

Chapter 2

Getting Started with Logstash

Logstash is easy to set up and deploy. We're going to go through the basic steps of

installing and conguring it. Then we'll try it out so we can see it at work. That'll

provide us with an overview of its basic set up, architecture, and importantly the

pluggable model that Logstash uses to input, process and output events.

Installing Java

Logstash's principal prerequisite is Java and Logstash itself runs in a Java Virtual

Machine or JVM. So let's start by installing Java. The fastest way to do this is via

our distribution's packaging system, for example Yum in the Red Hat family or

Debian and Ubuntu's Apt-Get.

TIP I recommend we install OpenJDK Java on your distribution. If you're run-ning OSX the natively installed Java will work ne (on Mountain Lion and later

you'll need to install Java from Apple).

12


29/220

Chapter 2: Getting Started with Logstash

On the Red Hat family

We install Java via the yum command:

Listing 2.1: Installing Java on Red Hat

$ sudo yum install java-1.7.0-openjdk

On Debian & Ubuntu

We install Java via the apt-getcommand:

Listing 2.2: Installing Java on Debian and Ubuntu

$ sudo apt-get -y install openjdk-7-jdk

Testing Java is installed

We can then test that Java is installed via the javabinary:

Listing 2.3: Testing Java is installed

$ java -version

java version "1.7.0_09"

OpenJDK Runtime Environment (IcedTea7 2.3.3)(7u9-2.3.3-0ubuntu1

~12.04.1)

OpenJDK Client VM (build 23.2-b09, mixed mode, sharing)

Version: v1.3.4 (5f439d1) 13


30/220


Getting Logstash

Once we have Java installed we can grab the Logstash package. Although Logstash

is written in JRuby, its developer releases a standalone jar le containing all of the

required dependencies. This means we don't need to install JRuby or any other

packages.

At this stage no distributions ship Logstash packages but you can easily build our

own using a tool likeFPMor from examples for RPM from hereorhereor DEB

fromhereor here.

TIP If we're distributing a lot of Logstash agents then it's probably a good ideato package Logstash.

We can download the jar le and rename it as logstash.jar:

Listing 2.4: Downloading Logstash

$ wget https://download.elasticsearch.org/logstash/logstash/

logstash-1.3.3-flatjar.jar -O logstash.jar

NOTE At the time of writing the latest version of Logstash is 1.3.3.

Starting Logstash

Once we have the jar le we can launch it with the java binary and a simple,

sample conguration le. We're going to do this to demonstrate Logstash workinginteractively and do a little bit of testing to see how Logstash works at its most

basic.

Version: v1.3.4 (5f439d1) 14
https://github.com/baseblack/logstash-debhttps://github.com/jbraeuer/logstash-debshttps://github.com/NumberFour/logstash-rpmshttps://github.com/mhorbul/logstash-rpm/https://github.com/jordansissel/fpm


31/220


Our sample conguration le

Firstly, let's create our sample conguration le. We're going to call ours sample

.confand you can see it here:

Listing 2.5: Sample Logstash conguration

input {

stdin { }

}

output {

stdout {debug => true

}

}

Oursample.confle contains two conguration blocks: one called inputand one

calledoutput. These are two of three types of plugin components in Logstash that

we can congure. The last type is filterthat we're going to see in later chapters.

Each type congures a dierent portion of the Logstash agent:

inputs - How events get into Logstash.

lters - How you can manipulate events in Logstash.

outputs - How you can output events from Logstash.

In the Logstash world events enter via inputs, they are manipulated, mutated or

changed in lters and then exit Logstash via outputs.

Inside each component's block you can specify and congure plugins. For ex-

ample, in the input block above we've dened the stdin plugin which controls

event input from STDIN. In the output block we've congured its opposite: the

stdout plugin, which outputs events to STDOUT. For this plugin we've added a

conguration option: debug. This outputs each event as a JSON hash.

Version: v1.3.4 (5f439d1) 15


32/220


NOTE STDIN and STDOUT are the standard streams of I/O in most applicationsand importantly in this case in your terminal.

Running the Logstash agent

Now we've got a conguration le let's run Logstash for ourselves:

Listing 2.6: Running the Logstash agent

$ java -jar logstash.jar agent -v -f sample.conf

NOTE Every time you change your Logstash conguration you will need torestart Logstash so it can pick up the new conguration.

We've used the javabinary and specied our downloaded jar le using the -jar

option. We've also specied three command line ags: agentwhich tell Logstash

to run as the basic agent,-v which turns on verbose logging and -f which species

the conguration le Logstash should start with.

TIP You can use the -vv ag for even more verbose output.

Logstash should now start to generate some startup messages telling you it is

enabling the plugins we've specied and nally emit:

Version: v1.3.4 (5f439d1) 16
http://en.wikipedia.org/wiki/Standard_streams


33/220


Listing 2.7: Logstash startup message

Pipeline started {:level=>:info}

This indicates Logstash is ready to start processing logs!

TIP You can see a full list of the other command line ags Logstash acceptshere.

Logstash can bea mite slow to start and will take a few moments to get started

before it is ready for input. If it's really slow you can tweak Java's minimum andmaximum heap size to throw more memory at the process. You can do this with

the-Xmsand -Xmxags. The-Xmsargument sets the initial heap memory size for

the JVM. This means that when you start your program the JVM will allocate this

amount of memory automatically. The-Xmx argument denes the max memory

size that the heap can reach for the JVM. You can set them like so:

Listing 2.8: Tuning the Logstash agent

$ java -Xms384m -Xmx384m -jar logstash.jar agent -v -f sample.

conf

This sets the minimum and maximum heap size to 384M of memory. Giving

Logstash more heap memory should speed it up but can also be unpredictable.

Indeed, Java and JVM tuning can sometimes have a steep learning curve. You

should do some benchmarking of Logstash in your environment. Keep in mind

that requirements for agents versus indexers versus other components will also

dier.

TIP There are some resources online that can help with JVM tuning here,here,hereandhere.

Version: v1.3.4 (5f439d1) 17
http://www.quora.com/JVM/What-are-some-useful-tips-for-tuning-programs-running-on-the-JVMhttp://www.semicomplete.com/blog/geekery/debugging-java-performance.htmlhttp://stackoverflow.com/questions/564039/jvm-performance-tuning-for-large-applicationshttp://www.caucho.com/resin-3.0/performance/jvm-tuning.xtphttp://cookbook.logstash.net/recipes/faster-startup-time/http://logstash.net/docs/latest/flags


34/220


Testing the Logstash agent

Now Logstash is running, remember that we enabled the stdin plugin? Logstash

is now waiting for us to input something on STDIN. So I am going to type "testing"

and hit Enter to see what happens.

Listing 2.9: Running Logstash interactively

$ java -jar logstash.jar agent -v -f sample.conf

output received {:event=>#"testing", "@timestamp"=>"2013-08-25

T17:27:50.027Z", "@version"=>"1", "host"=>"maurice.example.com

"}>, :level=>:info}

{

"message" => "testing",

"@timestamp" => "2013-08-25T17:27:50.027Z",

"@version" => "1",

"host" => "maurice.example.com"

}

You can see that our input has resulted in some output: a infolevel log message

from Logstash itself and an event in JSON format (remember we specied the

debugoption for the stdoutplugin). Let's examine the event in more detail.

Version: v1.3.4 (5f439d1) 18


35/220


Listing 2.10: A Logstash JSON event

{

"message" => "testing",

"@timestamp" => "2013-08-25T17:27:50.027Z",

"@version" => "1",

"host" => "maurice.example.com"

}

We can see our event is made up of a timestamp, the host that generated the

event maurice.example.com and the message, in our case testing. You might

notice that all these components are also contained in the log output in the @data

hash.

We can see our event has been printed as a hash. Indeed it's represented internally

in Logstash as a JSON hash.

If we'd had omitted the debugoption from the stdoutplugin we'd have gotten a

plain event like so:

Listing 2.11: A Logstash plain event

2013-08-25T17:27:50.027Z maurice.example.com testing

Logstash calls these formats codecs. There are a variety of codecs that Logstash

supports. We're going to mostly see theplainand jsoncodecs in the book.

plain - Events are recorded as plain text and any parsing is done using

filterplugins.

json - Events are assumed to be JSON and Logstash tries to parse the event's

contents into elds itself with that assumption.

We're going to focus on thejsonformat in the book as it's the easiest way to work

with Logstash events and show how they can be used. The format is made up of

a number of elements. A basic event has only the following elements:

Version: v1.3.4 (5f439d1) 19


36/220


@timestamp: AnISO8601 timestamp.

message: The event's message. Here testing as that's what we put into

STDIN.

@version: The version of the event format. This current version is 1.

Additionally many of the plugins we'll use add additional elds, for example the

stdin plugin we've just used adds a eld called host which species the host

which generated the event. Other plugins, for example the file input plugin

which collects events from les, add elds like pathwhich reports the le of the

le being collected from. In the next chapters we'll also see some other elements

like custom elds, tags and other context that we can add to events.

TIP Running interactively we can stop Logstash using the Ctrl-C key combina-tion.

Summary

That concludes our simple introduction to Logstash. In the next chapter we're

going to introduce you to your new role at Example.com and see how you can useLogstash to make your log management project a success.

Version: v1.3.4 (5f439d1) 20
http://en.wikipedia.org/wiki/ISO_8601


37/220

Chapter 3

Shipping Events

It's your rst day at Example.com and your new boss swings by your desk to tell

you about the rst project you're going to tackle: log management. Your job is

to consolidate log output to a central location from a variety of sources. You've

got a wide variety of log sources you need to consolidate but you've been asked

to start with consolidating and managing some Syslog events.

Later in the project we'll look at other log sources and by the end of the project all

required events should be consolidated to a central server, indexed, stored, and

then be searchable. In some cases you'll also need to congure some events to be

sent on to new destinations, for example to alerting and metrics systems.

To do the required work you've made the wise choice to select Logstash as your

log management tool and you've built a basic plan to deploy it:

1. Build a single central Logstash server (we'll cover scaling in Chapter 7).

2. Congure your central server to receive events, index them and make them

available to search.

3. Install Logstash on a remote agent.

4. Congure Logstash to send some selected log events from our remote agent

to our central server.

5. Install Logstash Kibana to act as a web console and front end for our logging

infrastructure.

21


38/220

Chapter 3: Shipping Events

We'll take you through each of these steps in this chapter and then in later chapters

we'll expand on this implementation to add new capabilities and scale the solution.

Our Event Lifecycle

For our initial Logstash build we're going to have the following lifecycle:

The Logstash agent on our remote agents collects and sends a log event to

our central server.

ARedisinstance receives the log event on the central server and acts as a

buer.

The Logstash agent draws the log event from our Redis instance and indexes

it.

The Logstash agent sends the indexed event toElasticSearch.

ElasticSearch stores and renders the event searchable.

The Logstash web interface queries the event from ElasticSearch.

Figure 3.1: Our Event Lifecycle

Version: v1.3.4 (5f439d1) 22
http://www.elasticsearch.org/http://redis.io/


39/220


Now let's set up Logstash to implement this lifecycle.

Installing Logstash on our central server

First we're going to install Logstash on our central server. We're going to build

an Ubuntu box calledsmoker.example.comwith an IP address of10.0.0.1as our

central server.

Central server

Hostname: smoker.example.com

IP Address: 10.0.0.1

As this is our production infrastructure we're going to be a bit more systematic

about setting up Logstash than we were in Chapter 1. To do this we're going to

create a directory for our Logstash environment and proper service management

to start and stop it.

TIP There are other, more elegant, ways to install Logstash using tools likePuppetorChef. Setting up either is beyond the scope of this book but there are

several Puppet modules for Logstash on the Puppet Forgeand aChef cookbook. I

strongly recommend you use this chapter as exposition and introduction on howLogstash is deployed and use some kind of conguration management to deploy

in production.

Let's install Java rst.

Listing 3.1: Installing Java

$ sudo apt-get install openjdk-7-jdk

Now let's create a directory to hold Logstash itself. We're going to use /opt/

logstash:

Version: v1.3.4 (5f439d1) 23
http://community.opscode.com/cookbooks/logstashhttp://forge.puppetlabs.com/modules?q=logstashhttp://www.opscode.com/chef/http://www.puppetlabs.com/


40/220


Listing 3.2: Creating the Logstash directory

$ sudo mkdir /opt/logstash

We'll now download the Logstash jar le to this directory and rename it to

logstash.jar.

Listing 3.3: Downloading the Logstash JAR le

$ cd /opt/logstash

$ sudo wget https://download.elasticsearch.org/logstash/logstash

/logstash-1.3.3-flatjar.jar -O logstash.jar

Now let's create a directory to hold our Logstash conguration:

Listing 3.4: Creating Logstash conguration directory

$ sudo mkdir /etc/logstash

Finally, a directory to store Logstash's log output:

Listing 3.5: Creating Logstash log directory

$ sudo mkdir /var/log/logstash

Now let's install some of the other required components for our new deployment

and then come back to conguring Logstash.

Installing a broker

As this is our central server we're going to install a broker for Logstash. Thebroker receives events from our shippers and holds them briey prior to Logstash

Version: v1.3.4 (5f439d1) 24


41/220


indexing them. It essentially acts as a "buer" between your Logstash agents and

your central server. It is useful for two reasons:

It is a way to enhance the performance of your Logstash environment byproviding a caching buer for log events.

It provides some resiliency in our Logstash environment. If our Logstash

indexing fails then our events will be queued in Redis rather than potentially

lost.

We are going to useRedisas our broker. We could choose a variety of possible

brokers, indeed other options includeAMQPand0MQ, but we're going with Redis

because:

It's very simple and very fast to set up.

It's performant.

It's well tested and widely used in the Logstash community.

Redis is a neat open source, key-value store. Importantly for us the keys can

contain strings, hashes, lists, sets and sorted sets making it a powerful store for a

variety of data structures.

Installing Redis

We can either install Redis via our packager manager or from source. I recommend

installing it from a package as it's easier to manage and you'll get everything you

need to manage it. However, you will need Redis version 2.0 or later. On our

Debian and Ubuntu hosts we'd install it like so:

Listing 3.6: Installing Redis on Debian

$ sudo apt-get install redis-server

On Red Hat-based platforms you will need to install the EPEL package repositories

to get a recent version of Redis. For example on CentOS and RHEL 6 to install

EPEL:

Version: v1.3.4 (5f439d1) 25
http://www.zeromq.org/http://www.amqp.org/http://redis.io/


42/220


Listing 3.7: Installing EPEL on CentOS and RHEL

$ sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/

i386/epel-release-6-8.noarch.rpm

And now we can install Redis.

Listing 3.8: Installing Redis on Red Hat

$ sudo yum install redis

NOTE If you want the source or the bleeding edge edition you can downloadRedis directly fromits site, congure and install it.

Changing the Redis interface

Once Redis is installed we need to update its conguration so it listens on all

interfaces. By default, Redis only listens on the 127.0.0.1 loopback interface. We

need it to listen on an external interface so that it can receive events from ourremote agents.

To do this we need to edit the /etc/redis/redis.conf (it's /etc/redis.conf on

Red Hat-based platforms) conguration le and comment out this line:

Listing 3.9: Changing the Redis interface

bind 127.0.0.1

So it becomes:

Version: v1.3.4 (5f439d1) 26
http://redis.io/download


43/220


Listing 3.10: Commented out interface

#bind 127.0.0.1

We could also just bind it to a single interface, for example our host's external IP

address10.0.0.1like so:

Listing 3.11: Binding Redis to a single interface

bind 10.0.0.1

Now it's congured, we can start the Redis server:

Listing 3.12: Starting the Redis server

$ sudo /etc/init.d/redis-server start

Test Redis is running

We can test if the Redis server is running by using the redis-clicommand.

Listing 3.13: Testing Redis is running

$ redis-cli -h 10.0.0.1

redis 10.0.0.1:6379> PING

PONG

When theredisprompt appears, then type PINGand if the server is running then

it should return a PONG.

You should also be able to see the Redis server listening on port 6379. You will

need to ensure any rewalls on the host or between the host and any agents allows

trac on port 6379. To test this is working you can telnet to that port and issue

the same PINGcommand.

Version: v1.3.4 (5f439d1) 27


44/220


Listing 3.14: Telneting to the Redis server

$ telnet 10.0.0.1 6379

Trying 10.0.0.1...

Connected to smoker.

Escape character is '^]'.

PING

+PONG

ElasticSearch for Search

Next we're going to installElasticSearchto provide our search capabilities. Elas-

ticSearch is a powerful indexing and search tool. As the ElasticSearch team puts

it: "ElasticSearch is a response to the claim: 'Search is hard.'". ElasticSearch is

easy to set up, has search and index data available RESTfully as JSON over HTTP

and is easy to scale and extend. It's released under the Apache 2.0 license and is

built on top of Apache's Lucene project.

When installing the Elasticsearch server you need to ensure you install a suitable

version. The ElasticSearch server version needs to match the version of the Elas-

ticSearch client that is bundled with Logstash. If the client version is 0.90.3 you

should install version 0.90.3 of the ElasticSearch server. Thecurrent documenta-tionwill indicate which version of ElasticSearch to install to match the client.

TIP Logstash also has a bundled ElasticSearch server inside it that we could use.To enable it see the embedded option of the elasticsearch plugin. For most purposes

though I consider it more exible and scalable to use an external ElasticSearch

server.

Version: v1.3.4 (5f439d1) 28
http://logstash.net/docs/latest/outputs/elasticsearchhttp://logstash.net/docs/latest/outputs/elasticsearchhttp://logstash.net/docs/latest/outputs/elasticsearchhttp://www.elasticsearch.org/


45/220


Introduction to ElasticSearch

So before we install it we should learn a little about ElasticSearch and how it

works. A decent understanding is going to be useful later as we use and scaleElasticSearch. ElasticSearch is a text indexing search engine. The best metaphor

is the index of a book. You ip to the back of the book1, look up a word and

then nd the reference to a page. That means, rather than searching text strings

directly, it creates an index from incoming text and performs searches on the index

rather than the content. As a result it is very fast.

NOTE This is a simplied explanation. See thesitefor more information andexposition.

Under the covers ElasticSearch uses Apache Lucene to create this index. Each

index is a logical namespace, in Logstash's case the default indexes are named for

the day the events are received, for example:

Listing 3.15: A Logstash index

logstash-2012.12.31

Each Logstash event is made up of elds and these elds become a document

inside that index. If we were comparing ElasticSearch to a relational database: an

index is a table, a document is a table row and a eld is a table column. Like a

relational database you can dene a schema too. ElasticSearch calls these schemas

"mappings".

NOTE It's important to note that you don't have to specify any mappings for op-erations, indeed many of searches you'll use with Logstash don't need mappings,

but they often makes life much easier. You can see an example of an Elastic-

1Not the rst Puppet book.

Version: v1.3.4 (5f439d1) 29
http://lucene.apache.org/core/http://www.elasticsearch.org/guide/


46/220


Search mappinghere. Since Logstash 1.3.2 a default mapping is applied to your

ElasticSearch and you generally no longer need to worry about setting your own

mapping.

Like a schema, mapping declares what data and data types elds documents con-

tain, any constraints present, unique and primary keys and how to index and

search each eld. Unlike a schema you can also specify ElasticSearch settings.

Indexes are stored in Lucene instances called "shards". There are two types of

shards: primary and replica. Primary shards are where your documents are stored.

Each new index automatically creates ve primary shards. This is a default setting

and you can increase or decrease the number of primary shards when the index

is created but not AFTER it is created. Once you've created the index the numberof primary shards cannot be changed.

Replica shards are copies of the primary shards that exist for two purposes:

To protect your data.

To make your searches faster.

Each primary shard will have one replica by default but also have more if required.

Unlike primary shards, this can be changed dynamically to scale out or make an

index more resilient. ElasticSearch will cleverly distribute these shards across theavailable nodes and ensure primary and replica shards for an index are not present

on the same node.

Shards are stored on ElasticSearch "nodes". Each node is automatically part of an

ElasticSearch cluster, even if it's a cluster of one. When new nodes are created

they can use unicast or multicast to discover other nodes that share their cluster

name and will try to join that cluster. ElasticSearch distributes shards amongst all

nodes in the cluster. It can move shards automatically from one node to another

in the case of node failure or when new nodes are added.

Version: v1.3.4 (5f439d1) 30
http://untergeek.com/2012/11/05/my-current-templatemapping/


47/220


Installing ElasticSearch

ElasticSearch's only prerequisite is Java. As we installed a JDK earlier in this chap-

ter we don't need to install anything additional for it. Unfortunately ElasticSearchis currently not well packaged in distributions but it is easy to download and cre-

ate your own packages. Additionally the ElasticSearch team does provide some

DEB packages for Ubuntu and Debian-based hosts. You can nd theElasticSearch

download page here.

As we're installing onto Ubuntu we can use the DEB packages provided:

Listing 3.16: Downloading ElasticSearch

$ wget https://download.elasticsearch.org/elasticsearch/

elasticsearch/elasticsearch-0.90.3.deb

Now we install ElasticSearch. We need to tell ElasticSearch where to nd our Java

JDK installation by setting theJAVA_HOMEenvironment variable. We can then run

thedpkgcommand to install the DEB package.

Listing 3.17: Installing ElasticSearch

$ export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386/$ sudo dpkg -i elasticsearch-0.90.3.deb

TIP You can also nd tar balls for ElasticSearch from which you can install orcreate RPM packages. There is an example RPM SPEC lehere.

Installing the package should also automatically start the ElasticSearch server but

if it does not then you can manage it via its init script:

Version: v1.3.4 (5f439d1) 31
https://github.com/tavisto/elasticsearch-rpmshttp://www.elasticsearch.org/download/http://www.elasticsearch.org/download/


48/220


Listing 3.18: Starting ElasticSearch

$ sudo /etc/init.d/elasticsearch start

Conguring our ElasticSearch cluster and node

Next we need to congure our ElasticSearch cluster and node name. ElasticSearch

is started with a default cluster name and a random, allegedly amusing, node

name, for example "Frank Kafka" or "Spider-Ham". A new random node name is

selected each time ElasticSearch is restarted. Remember that new ElasticSearch

nodes join any cluster with the same cluster name they have dened. So we want

to customize our cluster and node names to ensure we have unique names. To

do this we need to edit the /etc/elasticsearch/elasticsearch.ymlle. This is

ElasticSearch'sYAML-basedconguration le. Look for the following entries in

the le:

Listing 3.19: Initial cluster and node names

# cluster.name: elasticsearch

# node.name: "Franz Kafka"

We're going to uncomment and change both the cluster and node name. We're

going to choose a cluster name oflogstashand a node name matching our central

serve

The Log Stash Book

Documents