Troubleshooting Cassandra nodetool & system.log deep dive J.B. Langston, Senior Support Engineer
Jul 15, 2015
Troubleshooting Cassandranodetool & system.log deep dive
J.B. Langston, Senior Support Engineer
Company Confidential© 2014 DataStax, All Rights Reserved.
Agenda
2
1 Things to keep in mind
2 Useful Linux & Java tools
3 Useful nodetool commands
4 Things to look for in system.log
Company Confidential© 2014 DataStax, All Rights Reserved.
Troubleshooting Process
3
1 Ask what changed
2 Determine which nodes have problems
3 Find and understand errors
4 Examine bottlenecks
5 Determine root cause
6 Take corrective action
Company Confidential© 2014 DataStax, All Rights Reserved.
Troubleshooting Process
4
1 Ask what changed
2 Determine which nodes have problems
3 Find and understand errors
4 Examine bottlenecks
5 Share relevant information
6 Ask for help
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings• Application Code
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware
Company Confidential© 2014 DataStax, All Rights Reserved. 5
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?• Cassandra
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?• Cassandra• Kernel
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?• Cassandra• Kernel• JVM
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?• Cassandra• Kernel• JVM• Driver
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 5
• Does it work in another environment?
• Did you upgrade?• Cassandra• Kernel• JVM• Driver
• Change one thing at a time!
What changed?
• Did it work before?
• What did you change?• Settings• Application Code• Read/Write Load• Data Volume• Hardware• Network
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
• Startup
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
• Startup
• Foreground
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
• Startup
• Foreground• Coordinating requests
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads
Company Confidential© 2014 DataStax, All Rights Reserved. 6
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes• Compactions
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes• Compactions• Garbage collections
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes• Compactions• Garbage collections• Gossip
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 6
• Background• Flushes• Compactions• Garbage collections• Gossip• Hinted Handoff• Read Repair
Cassandra Processes
• Startup
• Foreground• Coordinating requests• Local reads• Local writes
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
Company Confidential© 2014 DataStax, All Rights Reserved. 7
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
• Network
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
• Network
• OS Resources/Limits
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
• Network
• OS Resources/Limits• File Handles
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
• Network
• OS Resources/Limits• File Handles• Processes
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 7
• Disk• Disk space• I/O bandwidth
• Network
• OS Resources/Limits• File Handles• Processes• Mapped Memory
System Resources
• CPU• Single Core utilization• Multi Core utilization
• Memory• Heap space• Off-heap space• OS page cache
• Garbage Collection
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per process
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per process
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk space
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilization
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usage
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections established
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilization
Company Confidential© 2014 DataStax, All Rights Reserved. 8
Linux monitoring commands
Command What it tells you…top CPU utilization and memory use per processtop -H CPU utilization per thread, memory use is still per processdf Free disk spaceiostat -x I/O bandwidth utilizationfree -m Memory and cache usagenetstat -an Network connections establishediftop Network bandwidth utilizationsar All (most) of the above, with history!
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each thread
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generation
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activity
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysis
Company Confidential© 2014 DataStax, All Rights Reserved. 9
Java monitoring commands
Command What it tells you…jstack -l Status and stack trace of each threadjmap -histo Types of objects on the heap (optionally only live objects)jmap -heap Size and usage of each java heap generationjstat -gccause Causes of gc activityjmap -dump Take a heap dump for further analysisMemoryAnalyzer Post-mortem heap-dump analysis
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster status
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single node
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single node
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single node
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local node
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local node
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progress
Company Confidential© 2014 DataStax, All Rights Reserved. 10
nodetool commands
Command What it tells you…status / ring Overall cluster statusinfo Status, memory usage, and caches for a single nodetpstats Statistics about each thread pool on a single nodecfstats Summary statistics for all tables and keyspaces on a single nodecfhistograms Detailed statistics for a specific table on the local nodeproxyhistograms Latency statistics for requests coordinated the local nodecompactionstats Compactions pending and in progressnetstats Network activity: streams, read repair, and in-flight commands
Company Confidential© 2014 DataStax, All Rights Reserved. 11
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 12
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 13
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 14
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 15
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 16
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 17
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 18
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 19
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 20
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 21
nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.173.171.164 22.78 MB 1 16.7% 67b3823f-6663-47d0-a04f-5914081e275c 1b DN 54.174.19.98 22.82 MB 1 16.7% 48d6f717-017b-4868-a525-b396d3f899aa 1b UN 54.174.245.247 22.68 MB 1 16.7% 6817e9ca-e79d-4fed-946e-7318bcfd5343 1b
Datacenter: us-west =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 54.153.107.100 22.72 MB 1 16.7% 4abf0a7a-00ef-441a-9f70-046cd9fe1c0c 1a UN 54.153.108.157 22.79 MB 1 16.7% 303f08dd-2a19-4175-98e7-97920232855b 1a UN 54.153.39.203 22.67 MB 1 16.7% d1a57a91-7aef-4878-a056-88949920724c 1a
Company Confidential© 2014 DataStax, All Rights Reserved. 22
nodetool ringNote: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: us-east ========== Address Rack Status State Load Owns Token -3074457345618258603 54.174.19.98 1b Down Normal 22.82 MB 16.67% -9223372036854775808 54.174.245.247 1b Up Normal 22.68 MB 16.67% -6148914691236517206 54.173.171.164 1b Up Normal 22.78 MB 16.67% -3074457345618258603
Datacenter: us-west ========== Address Rack Status State Load Owns Token 6148914691236517205 54.153.39.203 1a Up Normal 22.67 MB 16.67% 0 54.153.107.100 1a Up Normal 22.72 MB 16.67% 3074457345618258602 54.153.108.157 1a Up Normal 22.79 MB 16.67% 6148914691236517205
Company Confidential© 2014 DataStax, All Rights Reserved. 23
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 24
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 25
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 26
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 27
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 28
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 29
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 30
nodetool infoToken : -6148914691236517206 ID : 6817e9ca-e79d-4fed-946e-7318bcfd5343 Gossip active : true Thrift active : true Native Transport active: true Load : 22.68 MB Generation No : 1426523950 Uptime (seconds) : 1557 Heap Memory (MB) : 270.85 / 1842.00 Off Heap Memory (MB) : 0.11 Data Center : us-east Rack : 1b Exceptions : 0 Key Cache : size 2368 (bytes), capacity 96468992 (bytes), 104 hits, 128 requests, 0.813 recent hit rate, 14400 save period in seconds Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Company Confidential© 2014 DataStax, All Rights Reserved. 31
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 32
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 33
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 34
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 35
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 36
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 37
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 38
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 39
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 40
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 41
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 42
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 43
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 44
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 45
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 2 20487 0 0 RequestResponseStage 0 0 2787111 0 0 MutationStage 32 8713 1735129 0 0 ReadRepairStage 0 0 2641 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 34421 0 0 CacheCleanupExecutor 0 0 1 0 0 MigrationStage 0 0 1 0 0 MemoryMeter 1 1 71 0 0 FlushWriter 1 4 73 0 5 ValidationExecutor 0 0 42 0 0 InternalResponseStage 0 0 188 0 0 AntiEntropyStage 0 0 610 0 0 MemtablePostFlusher 1 4 358 0 0 MiscStage 0 0 43 0 0 PendingRangeCalculator 0 0 6 0 0 commitlog_archiver 0 0 0 0 0 CompactionExecutor 2 5 125 0 0 AntiEntropySessions 1 1 13 0 0 HintedHandoff 2 4 34 0 0
Message type Dropped RANGE_SLICE 0 READ_REPAIR 7 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 42993 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0
Company Confidential© 2014 DataStax, All Rights Reserved. 46
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 47
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 48
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 49
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 50
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 51
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 52
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 53
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 54
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 55
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 56
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 57
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 58
nodetool cfstatsKeyspace: Keyspace1 Read Count: 282781 Read Latency: 1.0155043372786716 ms. Write Count: 2700184 Write Latency: 0.07415620861393149 ms. Pending Tasks: 0
Table: Standard1 SSTable count: 5 Space used (live), bytes: 656644536 Space used (total), bytes: 656644536 Off heap memory used (total), bytes: 4221251 SSTable Compression Ratio: 0.0 Number of keys (estimate): 2182272 Memtable cell count: 215155 Memtable data size, bytes: 67222188 Memtable switch count: 72 Local read count: 282781 Local read latency: 1.016 ms Local write count: 2700943 Local write latency: 0.074 ms Pending tasks: 0 Bloom filter false positives: 379 Bloom filter false ratio: 0.00097 Bloom filter space used, bytes: 3897360 Bloom filter off heap memory used, bytes: 3897320 Index summary off heap memory used, bytes: 323931 Compression metadata off heap memory used, bytes: 0 Compacted partition minimum bytes: 259 Compacted partition maximum bytes: 310 Compacted partition mean bytes: 310 Average live cells per slice (last five minutes): 5.0 Average tombstones per slice (last five minutes): 0.0
Company Confidential© 2014 DataStax, All Rights Reserved. 59
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 60
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 61
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 62
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 63
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 64
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 65
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 66
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 67
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 68
nodetool cfhistogramsKeyspace1/Standard1 histograms
SSTables per Read 1 sstables: 14250 2 sstables: 42908 3 sstables: 97878 4 sstables: 126087 5 sstables: 745 6 sstables: 295 7 sstables: 122 8 sstables: 448 10 sstables: 37 12 sstables: 10 14 sstables: 1
Partition Size (bytes) 1131752 bytes: 1093 1358102 bytes: 479 1629722 bytes: 364 1955666 bytes: 343 2346799 bytes: 73 2816159 bytes: 134 3379391 bytes: 7 4055269 bytes: 0 4866323 bytes: 2
Cell Count per Partition 103 cells: 112 124 cells: 5130 149 cells: 2458 179 cells: 1704 215 cells: 1460 258 cells: 823 310 cells: 1327 372 cells: 1331 446 cells: 49295 535 cells: 9091 642 cells: 11892 770 cells: 3908 924 cells: 14532 1109 cells: 20898 1331 cells: 8491 1597 cells: 8140 1916 cells: 4010 2299 cells: 19476 2759 cells: 15112 3311 cells: 10812 3973 cells: 12763 4768 cells: 18597 5722 cells: 5765 6866 cells: 11340
Company Confidential© 2014 DataStax, All Rights Reserved. 69
nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839
Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 70
nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839
Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 71
nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839
Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 72
nodetool cfhistogramsWrite Latency (microseconds) 4 us: 83 5 us: 9004 6 us: 73288 7 us: 211239 8 us: 321305 10 us: 607511 12 us: 438101 14 us: 440503 17 us: 459036 20 us: 211763 24 us: 97240 29 us: 43564 35 us: 29579 42 us: 22575 50 us: 10442 60 us: 4842 72 us: 3566 86 us: 4323 103 us: 2038 124 us: 6192 149 us: 6850 179 us: 2657 215 us: 4536 258 us: 1839
Read Latency (microseconds) 42 us: 29 50 us: 132 60 us: 1225 72 us: 4791 86 us: 13031 103 us: 18526 124 us: 21151 149 us: 21085 179 us: 16528 215 us: 19259 258 us: 39965 310 us: 55015 372 us: 29344 446 us: 11347 535 us: 5692 642 us: 5395 770 us: 3550 924 us: 2649 1109 us: 1379 1331 us: 1084 1597 us: 1266 1916 us: 1492 2299 us: 2032 2759 us: 1494
Company Confidential© 2014 DataStax, All Rights Reserved. 73
nodetool proxyhistogramsRead Latency (microseconds) 124 us: 18 149 us: 445 179 us: 5866 215 us: 14541 258 us: 19123 310 us: 20941 372 us: 32847 446 us: 37464 535 us: 26451 642 us: 25291 770 us: 37870 924 us: 35865 1109 us: 30432 1331 us: 22929 1597 us: 17592 1916 us: 12895 2299 us: 9760 2759 us: 7887 3311 us: 6024 3973 us: 4751 4768 us: 3691 5722 us: 3038 6866 us: 2453 8239 us: 1991
Write Latency (microseconds) 29 us: 4 35 us: 60 42 us: 331 50 us: 471 60 us: 1334 72 us: 2856 86 us: 4661 103 us: 65741 124 us: 37999 149 us: 42697 179 us: 45598 215 us: 30515 258 us: 23481 310 us: 21831 372 us: 22741 446 us: 26045 535 us: 33045 642 us: 43533 770 us: 51021 924 us: 57496 1109 us: 61548 1331 us: 60741 1597 us: 57538 1916 us: 52255
Company Confidential© 2014 DataStax, All Rights Reserved. 74
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 75
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 76
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 77
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 78
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 79
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 80
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 81
nodetool compactionstatspending tasks: 7 compaction type keyspace table completed total unit progress Compaction Keyspace1 Standard1 65967769 154639724 bytes 42.66% Compaction Keyspace1 Standard1 53880334 227493794 bytes 23.68% Active compaction remaining time : 0h00m15s
pending tasks: 1 compaction type keyspace table completed total unit progress Validation Keyspace1 Standard1 74684443 95178582 bytes 78.47% Active compaction remaining time : n/a
Company Confidential© 2014 DataStax, All Rights Reserved. 82
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 83
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 84
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 85
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 86
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 87
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 88
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 89
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 90
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 91
nodetool netstatsMode: NORMAL Repair 028763b0-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.19.98 Receiving 6 files, 117949006 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-162-Data.db
851792/17950738 bytes(4%) received from /54.174.19.98 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
3786324/46561942 bytes(8%) sent to /54.174.19.98 Repair 020ed850-cc1e-11e4-a20c-a1d01a3fbf30 /54.174.245.247 Receiving 4 files, 93304584 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-161-Data.db
6094594/46561942 bytes(13%) received from /54.174.245.247 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
34195028/46561942 bytes(73%) sent to /54.174.245.247 Repair 018c88f0-cc1e-11e4-a20c-a1d01a3fbf30 /54.153.39.203 (using /172.31.10.65) Receiving 3 files, 49959102 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-160-Data.db
9371380/46561942 bytes(20%) received from /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-159-Data.db
2533414/2533414 bytes(100%) received from /54.153.39.203 Sending 2 files, 47709526 bytes total /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-158-Data.db
1147584/1147584 bytes(100%) sent to /54.153.39.203 /var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-157-Data.db
46561942/46561942 bytes(100%) sent to /54.153.39.203
Read Repair Statistics: Attempted: 39576 Mismatch (Blocking): 0 Mismatch (Background): 746
Pool Name Active Pending Completed Commands n/a 58 2545817 Responses n/a 0 2833081
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Default Location /var/log/cassandra/system.log
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.log
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Basic FormatLevelBasic Format
INFO
Basic Format
WARN
Basic Format
ERROR
Basic Format
DEBUG
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Basic FormatLevelBasic Format
INFO
Basic Format
WARN
Basic Format
ERROR
Basic Format
DEBUG
Basic FormatThread Type & ID
Basic Format
[CompactionExecutor:155]
Basic Format
[GossipTasks:1]
Basic Format
[AntiEntropySessions:1]
Basic Format
[OptionalTasks:1]
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Basic FormatLevelBasic Format
INFO
Basic Format
WARN
Basic Format
ERROR
Basic Format
DEBUG
Basic FormatThread Type & ID
Basic Format
[CompactionExecutor:155]
Basic Format
[GossipTasks:1]
Basic Format
[AntiEntropySessions:1]
Basic Format
[OptionalTasks:1]
Basic FormatDate & Time
Basic Format
2015-02-13 02:18:40,986
Basic Format
2015-02-17 19:47:37,331
Basic Format
2015-02-17 20:32:11,959
Basic Format
2015-02-20 11:29:14,056
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Basic FormatLevelBasic Format
INFO
Basic Format
WARN
Basic Format
ERROR
Basic Format
DEBUG
Basic FormatThread Type & ID
Basic Format
[CompactionExecutor:155]
Basic Format
[GossipTasks:1]
Basic Format
[AntiEntropySessions:1]
Basic Format
[OptionalTasks:1]
Basic FormatDate & Time
Basic Format
2015-02-13 02:18:40,986
Basic Format
2015-02-17 19:47:37,331
Basic Format
2015-02-17 20:32:11,959
Basic Format
2015-02-20 11:29:14,056
Basic FormatSource File
Basic Format
CompactionTask.java
Basic Format
Gossiper.java
Basic Format
CassandraDaemon.java
Basic Format
ColumnFamilyStore.java
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 92
system.log
Log SettingsLocationLog Settings
log4j.appender.R.File=/opt/dse-4.6.0/logs/system.logLog Settings
Logging Level
Log Settings
log4j.rootLogger=INFO,stdout,R # INFO, WARN, ERROR, or DEBUG
Log Settings
Class Override
Log Settings
log4j.logger.org.apache.cassandra.package.Class=DEBUG
Basic FormatLevelBasic Format
INFO
Basic Format
WARN
Basic Format
ERROR
Basic Format
DEBUG
Basic FormatThread Type & ID
Basic Format
[CompactionExecutor:155]
Basic Format
[GossipTasks:1]
Basic Format
[AntiEntropySessions:1]
Basic Format
[OptionalTasks:1]
Basic FormatDate & Time
Basic Format
2015-02-13 02:18:40,986
Basic Format
2015-02-17 19:47:37,331
Basic Format
2015-02-17 20:32:11,959
Basic Format
2015-02-20 11:29:14,056
Basic FormatSource File
Basic Format
CompactionTask.java
Basic Format
Gossiper.java
Basic Format
CassandraDaemon.java
Basic Format
ColumnFamilyStore.java
Basic FormatLine No.
Basic Format
(line 287)
Basic Format
(line 648)
Basic Format
(line 199)
Basic Format
(line 298)
Default Location /var/log/cassandra/system.log
Configuration File /etc/dse/cassandra/log4j-server.properties
Company Confidential© 2014 DataStax, All Rights Reserved. 93
Exceptionsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 94
Exceptions – Stack Tracejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 95
Exceptions – Organizationjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 96
Exceptions – Subsystemjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 97
Exceptions – Classjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 98
Exceptions – Nested Classesjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 99
Exceptions – Methodsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 100
Exceptions – Source Filejava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 101
Exceptions – Line Numberjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 102
Exceptions – Nested Exceptionsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more
Company Confidential© 2014 DataStax, All Rights Reserved. 103
Exceptions – Error Messageorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more
Company Confidential© 2014 DataStax, All Rights Reserved. 104
Exceptions - Organizationsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more
Company Confidential© 2014 DataStax, All Rights Reserved. 105
Exceptions - Subsystemsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more
Company Confidential© 2014 DataStax, All Rights Reserved.
Google Tips
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra
• Narrow or broaden as necessary
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra
• Narrow or broaden as necessary
DON’T:
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra
• Narrow or broaden as necessary
DON’T:
• Include source file + line number
106
Company Confidential© 2014 DataStax, All Rights Reserved.
Google TipsDO:
• Use exception and several package+class+method names• Use quotation marks around individual elements
• Use “site:” to limit search to relevant web sites• Mailing list: site:www.mail-archive.com/[email protected]• JIRA: site:issues.apache.org/jira/browse/CASSANDRA• StackOverflow: site:stackoverflow.com cassandra
• Narrow or broaden as necessary
DON’T:
• Include source file + line number• Include specific numbers and string
106
Company Confidential© 2014 DataStax, All Rights Reserved. 107
Exceptions – Search Termsjava.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:395) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize (CacheService.java:356) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:261) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:415) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore (ColumnFamilyStore.java:386) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:309) at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:266) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:376) at org.apache.cassandra.service.CassandraDaemon.activate (CassandraDaemon.java:480)
Company Confidential© 2014 DataStax, All Rights Reserved. 108
Exceptions – Search Termsorg.apache.thrift.transport.TTransportException: … at org.apache.thrift.transport.TIOStreamTransport.read at com.datastax.bdp.transport.server.TPreviewableTransport.readUntilEof at com.datastax.bdp.transport.server.TPreviewableTransport.preview at com.datastax.bdp.transport.server.TNegotiatingServerTransport.open at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at com.datastax.bdp.transport.server.TNegotiatingServerTransport$... at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run at java.util.concurrent.ThreadPoolExecutor.runWorker at java.util.concurrent.ThreadPoolExecutor$Worker.run at java.lang.Thread.run Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read at java.net.SocketInputStream.read at java.io.BufferedInputStream.fill at java.io.BufferedInputStream.read1 at java.io.BufferedInputStream.read at org.apache.thrift.transport.TIOStreamTransport.read ... 9 more
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
JVM vendor/version: … JVM vendor and version
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…
Heap settings for each generation
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…
Heap settings for each generation
Classpath: … Classpath (jar files and directories)
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…
Heap settings for each generation
Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)
Company Confidential© 2014 DataStax, All Rights Reserved. 109
Startup
What to look for… What it tells you…Logging initialized Node just restarted• DSE version: … • Cassandra version: …
Versions of major components
Loading settings from file: … Settings file locations• Data files directories: […] • Commit log directory: …
Data file and commit log location
JVM vendor/version: … JVM vendor and version• Heap size: … • Par Eden Space Heap memory: … • Par Survivor Space Heap memory: … • CMS Old Gen Heap memory: … • CMS Perm Gen Non-heap memory:…
Heap settings for each generation
Classpath: … Classpath (jar files and directories)JNA mlockall successful JNA is installed (make sure it is!)• Starting listening for CQL clients on ... • Listening for thrift clients…
Node is serving requests
Company Confidential© 2014 DataStax, All Rights Reserved. 110
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 111
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 112
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 113
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 114
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 115
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 116
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 117
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 118
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 119
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 120
Flushes
INFO [OptionalTasks:1] 2015-02-17 19:18:54,835 ColumnFamilyStore.java (line 794) Enqueuing flush of Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,836 Memtable.java (line 355) Writing Memtable-pdps@127960523(91418/471789 serialized/live bytes, 9370 ops)
INFO [FlushWriter:30] 2015-02-17 19:18:54,852 Memtable.java (line 395) Completed flushing /data/OpsCenter/pdps/OpsCenter-pdps-jb-3101-Data.db (26109 bytes) for commitlog position ReplayPosition(segmentId=1424226832202, position=38722)
Company Confidential© 2014 DataStax, All Rights Reserved. 121
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 122
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 123
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 124
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 125
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 126
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 127
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 128
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 129
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 130
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 131
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 132
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 133
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 134
Compactions INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,632 CompactionTask.java (line 105)
Compacting [SSTableReader(path='/data/system/hints/system-hints-ic-2498-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2493-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2499-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2497-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2495-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2500-Data.db'), SSTableReader(path='/data/system/hints/system-hints-ic-2494-Data.db'), SSTableReader(path=‘/data/system/hints/system-hints-ic-2496-Data.db’)]
INFO [CompactionExecutor:54106] 2015-01-24 12:57:04,635 CompactionController.java (line 156) Compacting large row system/hints:fa937b87-de5a-45e0-9b9d-379cc04b81b2 (487226212 bytes) incrementally
INFO [CompactionExecutor:54106] 2015-01-24 12:57:21,851 CompactionTask.java (line 262) Compacted 8 sstables to [/data/system/hints/system-hints-ic-2501,]. 359,979,223 bytes to 358,979,736 (~99% of original) in 17,219ms = 19.882093MB/s. 17 total rows, 6 unique. Row merge counts were {1:2, 2:2, 3:1, 4:0, 5:0, 6:0, 7:0, 8:1, }
Company Confidential© 2014 DataStax, All Rights Reserved. 135
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 136
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 137
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 138
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 139
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 140
Garbage Collections
INFO [ScheduledTasks:1] 2015-02-17 19:33:41,493 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 91197 ms for 3 collections, 15035796304 used; max is 16965173248
INFO [ScheduledTasks:1] 2015-02-17 19:31:03,013 GCInspector.java (line 116) GC for ParNew: 908 ms for 2 collections, 4428318872 used; max is 16965173248
WARN [ScheduledTasks:1] 2013-02-20 11:49:12,484 GCInspector.java (line 142) Heap is 0.8275743567425272 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Company Confidential© 2014 DataStax, All Rights Reserved. 141
Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)
InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [GossipTasks:1] 2015-02-17 14:39:48,841 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [RequestResponseStage:19] 2015-02-17 14:40:23,602 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [RequestResponseStage:18] 2015-02-17 14:40:58,971 Gossiper.java (line 887) InetAddress /10.100.12.130 is now UP
INFO [GossipTasks:1] 2015-02-17 14:40:58,971 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:41:33,667 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:29] 2015-02-17 14:42:42,668 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
Company Confidential© 2014 DataStax, All Rights Reserved. 142
Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)
InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [GossipTasks:1] 2015-02-17 14:39:48,841 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [RequestResponseStage:19] 2015-02-17 14:40:23,602 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [RequestResponseStage:18] 2015-02-17 14:40:58,971 Gossiper.java (line 887) InetAddress /10.100.12.130 is now UP
INFO [GossipTasks:1] 2015-02-17 14:40:58,971 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:41:33,667 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:29] 2015-02-17 14:42:42,668 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
Company Confidential© 2014 DataStax, All Rights Reserved. 143
Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)
InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [GossipTasks:1] 2015-02-17 14:39:48,841 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [RequestResponseStage:19] 2015-02-17 14:40:23,602 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [RequestResponseStage:18] 2015-02-17 14:40:58,971 Gossiper.java (line 887) InetAddress /10.100.12.130 is now UP
INFO [GossipTasks:1] 2015-02-17 14:40:58,971 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:41:33,667 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:29] 2015-02-17 14:42:42,668 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
Company Confidential© 2014 DataStax, All Rights Reserved. 144
Flapping INFO [GossipTasks:1] 2015-02-17 14:38:05,129 Gossiper.java (line 901)
InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:38:39,768 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:18] 2015-02-17 14:39:14,169 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [GossipTasks:1] 2015-02-17 14:39:48,841 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [RequestResponseStage:19] 2015-02-17 14:40:23,602 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
INFO [RequestResponseStage:18] 2015-02-17 14:40:58,971 Gossiper.java (line 887) InetAddress /10.100.12.130 is now UP
INFO [GossipTasks:1] 2015-02-17 14:40:58,971 Gossiper.java (line 901) InetAddress /10.98.12.216 is now DOWN
INFO [GossipTasks:1] 2015-02-17 14:41:33,667 Gossiper.java (line 901) InetAddress /10.100.12.130 is now DOWN
INFO [RequestResponseStage:29] 2015-02-17 14:42:42,668 Gossiper.java (line 887) InetAddress /10.98.12.216 is now UP
Company Confidential© 2014 DataStax, All Rights Reserved. 145
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 146
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 147
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 148
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 149
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 150
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 151
Hinted HandoffINFO [HintedHandoff:1] 2015-03-16 20:18:28,627 HintedHandOffManager.java (line 349)
Started hinted handoff for host: d1a57a91-7aef-4878-a056-88949920724c with IP: /54.153.39.203
INFO [HintedHandoff:1] 2015-03-16 20:18:37,292 HintedHandOffManager.java (line 381) Finished hinted handoff of 2492 rows to endpoint /54.153.39.203
Company Confidential© 2014 DataStax, All Rights Reserved. 152
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 153
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 154
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 155
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 156
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 157
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 158
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 159
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 160
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 161
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 162
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 163
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 164
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved. 165
RepairsINFO [AntiEntropySessions:1] 2015-03-16 17:24:44,097 RepairSession.java (line 246)
[repair #562f46d0-cc01-11e4-a84d-098a653a7013] new session: will sync /54.174.19.98, /54.153.108.157, /54.153.39.203, /54.174.245.247 on range (3074457345618258602,6148914691236517205] for Keyspace1.[Counter3, Standard1, SuperCounter1, Super1, Counter1]
INFO [RepairJobTask:2] 2015-03-16 17:24:48,102 RepairJob.java (line 161) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] requesting merkle trees for Standard1 (to [/54.153.108.157, /54.153.39.203, /54.174.245.247, /54.174.19.98])
INFO [AntiEntropyStage:1] 2015-03-16 17:24:52,749 RepairSession.java (line 166) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Received merkle tree for Standard1 from /54.174.19.98
INFO [RepairJobTask:4] 2015-03-16 17:24:52,757 Differencer.java (line 67) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.39.203 and /54.174.245.247 are consistent for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,438 Differencer.java (line 74) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Endpoints /54.153.108.157 and /54.174.19.98 have 6222 range(s) out of sync for Standard1
INFO [RepairJobTask:6] 2015-03-16 17:24:55,441 StreamingRepairTask.java (line 64) [streaming task #562f46d0-cc01-11e4-a84d-098a653a7013] Performing streaming repair of 6222 ranges with /54.153.108.157
INFO [StreamReceiveTask:1] 2015-03-16 17:25:01,524 StreamingRepairTask.java (line 92) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] streaming task succeed, returning response to /54.174.19.98
INFO [AntiEntropyStage:1] 2015-03-16 17:25:01,805 RepairSession.java (line 223) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] Standard1 is fully synced
INFO [AntiEntropySessions:1] 2015-03-16 17:25:01,806 RepairSession.java (line 284) [repair #562f46d0-cc01-11e4-a84d-098a653a7013] session completed successfully
Company Confidential© 2014 DataStax, All Rights Reserved.
Bringing it full circle…
166
1 Ask what changed
2 Determine which nodes have problems
3 Find and understand errors
4 Examine bottlenecks
5 Share relevant information
6 Ask for help
Thanks!Q&A