IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

BP110: Mastering Your LogsEverything You Should Know about Logging in IBM Domino

Ben Menesi, Ytria

Head of Product at Ytria

IBM Notes Administration & Development for

various enterprise IBM customers

Certified Advanced Domino Developer & Admin

and Security Professional v7+

Speaker at ICS / ESS technology conferences around the globe

IBM Champion for Collaboration Solutions 2014 & 2015

ca.linkedin.com/in/benedekmenesi

Speaker: Ben Menesi

@BenMenesi


http://twitter.com/BenMenesi

Founder, CEO & CTO at Ytria

15+ years IBM Notes & API development

Speaker at ICS conferences & published author

IBM Design Partner

ca.linkedin.com/pub/eric-houvenaghel/0/37/941

Co-AuthorEric Houvenaghel

ca.linkedin.com/pub/eric-houvenaghel/0/37/941

Domino

Server

The

Admin

1.• Events take place

2. • Events are logged

3.• Some events captured

4• Few events addressed

5

• Lots of events missed, angry users on the phone

Everything about LOG.NSF

– Understanding different log events

– Extracting additional data

– Log Retention, Searching the log and more

User Activity Logging

– Understanding what user activities are logged

– Utilizing Recorded user activity

Replication Logs

– Replication History tips & tricks

– Understanding what / where / why

– Replication Triangulation

Agenda

LOG.NSF

What is (or isn’t) logged in log.nsf?

How can we extract more data?

Best Practices for retention, searching and more

Almost everything from your Domino server (HTTP Web Server is another topic!)

– Generally 90%+ of what’s in your Domino console will be in here

• Except for Traveler and some DAOS command logs

Log.nsfWhat is being logged?

Log.nsf

replica

amgrrouter

MAIN Log.nsf setting in your server’s Notes.ini file

Logfilename: typically log.nsf

Log_option: 1 = Log to the console, 2 = Force DB fixup when opening log file, 4 = Full document scan

Days: # of days logs will be kept*

Size: Size of log text in event documents*

Days2: Optional # of days to keep activity trends data documents

Log.nsfLog file structure

LOG=logfilename, log_option, cant_touch_this, days, size, days2

Understanding your log views


SELECT Form=“Events”

SELECT Form=“Mail Routing”

SELECT Form=“Replication”

SELECT Form=“Security”

SELECT Form=“NNTPEvent”

SELECT Form=“PassthruConnection”

Server tasks simultaneously log into various log.nsf documents

Miscellaneous Events (form=Events)

– Size & content controlled by the Log= server notes.ini parameter (later)

Mail Routing Events (form=Mail Routing)

– Mainly updated by the SMTP Server and router tasks

Security Events (form=Security)

– E.g. ID Vault messages & errors

Replication Events (form=Replication)

– Replica & CLREPL tasks


Understanding your log views


Select Form=“Session”

Select Form=“Session”

SELECT Form=“Activity”

SELECT Form=“Activity” | Form=“Size”

User Session Activity (form=Session)

– Sessions this server had with users or other servers

Activity Data (form=Activity)

– Populated by the nightly Statistics Log task


Server tasks can log events to multiple types of documents!

– But no duplicate logs: different information!

– Example: Server 1 replicator opens session to Server 2 as per Connection Document settings


Misc.

Replication

Miscellaneous events: what do we see?

– Some high level info (Server, Start & Finish Time) and

– UI only shows ONE text type item with limited information

Log.nsf1.) Miscellaneous events

What the EventsR6 item tells us us

– Used to be a Rich Text item Prior to R6 (whew!)

– @If(@IsAvailable(Eventlist);EventList;@IsAvailable(Eventslist);EventsList;Events)

– EventsList item: only contains the Event time & description for each event

What is an event?


Time Status Code Severity Type

Addin Name Target Server Target DB Target User

Miscellaneous events – event attributes logged in multi-value Textitems


EventListTime &

Description

EventTime

EventSeverity

EventStatus

EventType

Event 1

Time

Severity

HEX Status

Type

Time &

Description

Event 2

Time

Severity

HEX Status

Type

Time &

Description

Event 3

Time

Severity

HEX Status

Type

Time &

Description

Event x

Time

Severity

HEX Status

Type

Miscellaneous events – EventSeverity

– Severities are recorded via numbers 0 – 5 where

• 0: Unknown

• 1: Fatal

• 2: Failure

• 3: Warning High

• 4: Warning Low

• 5: Normal


Miscellaneous events – EventType

– Multiple type identifiers in use

• Best not to mess with this stuff


Can we get more data?

– We can use formulas and the list of Severities, Events & Status Codes to get a better idea of each Misc. log document

Total # of events / document [Eventlist item]

– @Elements(Eventlist)

Total # of unique events [EventStatus item]

– @Elements(@Unique(EventStatus))

Highest severity / document [EventSeverity item]

– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]);


Customization options (Cont’d)

Once we have the _sev variable, use icons to display (from DDM resources)

– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+".gif“

# of Fatal severities / document [EventSeverity item]

– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"0";"")))

# of Failure severities [EventSeverity item]

– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":“1":"0";"")))


# of High Warning severities / document


# of Low Warning severities / document


# of Normal severities / document


# of Unknown severities / document



Customization options: this is what you’ll get

– Tip: we can make the Highest and severity columns sortable but careful with your indexes!


Mail Routing Events

– Same Structure as Miscellaneous events

– Incrementally fills documents (considering 40KB / Log= limit)

Log.nsf2.) Mail Routing Events

Time EventList EventStatus EventSeverity

Is there any additional data we can mine?

– Use same custom views (Severities, unique events, etc…)

• Tip: Messages can be traced based on the last 8 characters of their Universal ID

• Extract list of Messages transferred per Mail Routing Log, or their (unique) numbers.


Unique # of Messages transferred [EventList item]

– Could be used as a sortable column formula

• @Elements(@Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring(EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", ")))


List of unique message codes [EventList item]

– Could be used as a categorized column formula – help finding emails

• @Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring(EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", "))


Security Event documents

– Same Structure as Miscellaneous events

– Incrementally fills documents (considering 40KB / Log= limit)

– Typically you won’t even need a special view to understand unless you have a very high volume.

Log.nsf3.) Security Events

Replication Events <> Miscellaneous events

– Important: Logs documents PER SESSION

– Only Replication Sessions where the current server did the work!

• Search all participating server log files to get the big picture!

– What is stored in those replication events?

Log.nsf4.) Replication Events

InitiatedBy Server SourceServer

Body BytesIn BytesOut Pathname

EventList EventSeverity EventStatus EventType

What additional data can we extract and use?

– EventList & EventSeverity items ONLY created on error

– # of entries in the Pathname item tells us how many events

– Body item contains: Access, Additions, Updates, Deletes, Kbytes sent & received for each DB

• Body item is NOT multi-value (can’t really work with formulas)

• Body item is NOT part of the summary if more than 1 event is logged = can NOT be displayed in views

• Cluster Replication events are NOT properly logged unless you use the RTR_Logging parameter

• http://www-01.ibm.com/support/docview.wss?uid=swg21214739


http://www-01.ibm.com/support/docview.wss?uid=swg21214739

Interesting stuff: even when DB1 only has to SEND data to DB2, we can see data being received by DB1

– BytesIN & BytesOUT: NOT reliable:


Creating a custom replication view

Replicator / Cluster Replicator? [Initiatedby item]

Number of events [Pathname item]

– @Elements(Pathname)

Any Errors? [EventSeverity item]

– @If(@Elements(EventSeverity)>0;150;"")

Direction [Body item]

– _bl1:=@Contains(Body;"PULL");

– _bl2:=@Contains(Body;"PUSH");

– @If(_bl1=1&_bl2=1;"PULL-PUSH";_bl1=1&_bl2=0;"PULL";_bl2=1&_bl1=0;"PUSH";"-")


Creating a custom replication view (Cont’d)


Usage Session logs – what do we (not) see?

– IMPORTANT: Logging on a /USER/SESSION base!

– Contains a TON of raw data

Log.nsf5.) Usage Session Documents

What is an event?


Database Reads Writes Transactions

Bytes Read Bytes Written DB Open time

UserName Pathname Reads Writes

Transactions SentFromServer SentToServer Body

What additional data can we extract and use?

– PathName item (multi-value) contains list of databases accessed

• @Elements(PathName) gives us the # of DBs accessed for each session

– Rearrange columns in the Usage \ By User view


Searching your log file is painful

– Reason: you’ll be presented with any matching documents but you won’t be able to read between the lines

– Looks familiar?

– … and the sad part: 6 results would actually make us happy (RIGHT?)

Log.nsfHow to search?

Option #1: search using the Admin client

– TONS of parameters to specify – slow setup. Better for periodical search requirements


Option #1: search using the Admin client (Cont’d)

– Tip: do NOT use the Event Type tab!

• You can miss results because events are logged across types


Option #1: search using the Admin client (Cont’d)

– Tip: You can save queries in domadmin.nsf to reuse later

– BEWARE: Search results populate new documents!

• Avg. 180 events / document: single search with 1800 matches creates 10 documents

• Results can not be analyzed nor copied and are stored across multiple documents

– Conclusion: Useful for very specific queries that you may need periodically


Option #2: A quicker DIY solution (less customizable)

– Advantages: reads matches from multiple log documents and saves & displays them in one text file

– http://searchdomino.techtarget.com/tip/Easily-find-a-string-in-a-Lotus-Domino-server-log


http://searchdomino.techtarget.com/tip/Easily-find-a-string-in-a-Lotus-Domino-server-log

Option #3: Pro tip for data-range search (Kudos to Kim Greene)

– Select log documents and Actions \ Forward

• Aggregates EventList item contents in one document

• Use CTRL+F to search


Option #4 Great open source log parser via OpenNTF from JakobMajkilde

– Installed on server, access to Console Logs, Trace Logs, Log.nsf and more with a nice Xpages interface

– http://openntf.org/main.nsf/project.xsp?r=project/XPages%20Log%20File%20Reader


http://openntf.org/main.nsf/project.xsp?r=project/XPages Log File Reader

Option #5+ 3rd Party tools

– Use a tool like Ytria consoleEZ


Log deletions managed by parameter defined in log= server notes.ini parameter

– Deletions made at once, all entries older than 7 days removed

– Add to that: Deletion Stub Purge interval

• Default 90 days could result in =< 120 days old deletion stubs

• Deletions happen every 1/3rd of the time defined here

Log.nsfLog file retention

Known problem in previous Domino versions: log.nsf continues to grow and ignores log= settings

– Workaround: use “Remove documents not modified in the last (days)

• Beware: this field also defines Deletion Stub Purging time

• Tip: Deleted documents are removed WITHOUT creation of deletion stubs


Recommendation

– Use the “Remove documents note modified in the last (days)” setting

– Use console logs to keep old log data


Log_AgentManager

– 0 for no logging, 1 for partial & successful agent execution events, 2 for successful events only

Log_Replication

– 1 for logging when DB replicates, 2 for summary info about each DB, 3 for detailed info about each replicated note

• Great for debugging!

RTR_Logging

– 1 (Default), 4 for Logging replications (attempted & performed)

• Cluster replication will be logged

Log.nsfA few notes.ini parameters in the back pocket

Mail_Log_To_MiscEvents

– 0/1 whether to log mail events into Misc. event documents

• Tip: MailLogToEventsOnly=1 to NOT show router messages in the Domino Console

SMTPClientDebug=1

– Use temporarily to debug outbound mail problems

• Tip: don’t get confused by client, this one’s for your server!

HTTPLogUnauthorized=1

– Logs 401 HTTP errors in server console AND Misc. log events

• HTTP Users attempting to access resources that aren’t available and failed user authentication requests

• This is GREAT: brings a bit of web server logs into your log.nsf


Log_Console=2

– Logs all console commands even if prefixed with ! (By default !sh task won’t be logged!)

LOG_DisableTXNLogging=1

– 0/1 Take your Log.nsf out of DBs to be transaction logged

• Also clubusy.nsf and mail.box!

No_Force_Activity_Logging

– 0/1 Controls whether the statlog task automatically enables activity recording for all DBs (Default = 0)

• Tip even if disabled activity is being recorded in the Log.nsf Usage views


USAGE ACTIVITY

What are our most used databases?

How do we know what a given user has been up to?

How do we get user activity for multiple databases?

User Activity can be a *VERY* rich source of data

– Enable in the database properties window

• Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL dbs

• Tip: Set to confidential to prevent users with lower than Designer access from accessing it

Database User ActivityRecorded User Activity

Did you upgrade your database On Disk Structures?

– I’m not proud of it

– Tip: use Compact –REPLICA for upgrading system database ODS’s without downtime!

Database User ActivityBefore we go any further

Prior to ODS 48

– Entry size allocated 44 bytes, Object size 61600 bytes (max 1400 entries)

ODS 48+ (undocumented improvement)

– Entry size allocated 92 bytes, Object size 128800 bytes (max 1400 entries)

Curious: only two additional counters added, yet ODS 48+ entry is over 2x the size of those prior to ODS48: Why?

Database User ActivityUser Activity – What is really captured?

Date &

TimeReads Adds Updates Deletes User

Date &

TimeReads Writes User

Does Notes only capture as much as it displays?

– Nope

Database User ActivityUser Activity – What is really captured?

Reads Adds Updates Deletes

Data

Non

Data

Σ

Data

Non

Data

Σ

Data

Non

Data

Σ

Data

Non

Data

Σ

Differentiating between Data (as in: Document class notes) and Non-Data (as in: everything else such as Designs, ACLs, etc…) can help us look for more specific scenarios:

– Databases where no DOCUMENT class notes were read for some time

– Databases where no DOCUMENT class notes were added or updated since awhile

– Users who’ve deleted Designs or ACLs

– And more.

Database User ActivityUser Activity – Data & Non-Data

Makes no difference between users and servers

– Workaround: use a tool to parse the data and input exceptions

Only maintains 1400 entries

– Wait, this doesn’t need to be a problem!

Still answers questions like “What databases aren’t used on a regular basis?”

– Databases that contain 1400 entries are properly used apps

• @Date(FirstEntry) - @Date(LastEntry) tells us how frequently used

– Databases with lower than 1400 entries indicate they aren’t used that much

Database User ActivityUser Activity - Downside

How do we get the user activity for multiple databases

– Third party tools like http://www.agecom.com.au/useractivity

– …or Ytria’s databaseEZ

– …or we can implement it ourselves using the NotesUserActivity class:

• Uses the W32_NSFDbGetUserActivity API call (restrictive: no Data vs. Non-Data)

http://www.bananahome.com/ldd/sandbox.nsf/ByDate/c12a2fd2142758b68525688d00708397?OpenDocument

Database User ActivityUser Activity – How to get it?

http://www.agecom.com.au/useractivity

http://www.bananahome.com/ldd/sandbox.nsf/ByDate/c12a2fd2142758b68525688d00708397?OpenDocument

REPLICATION LOGSHow is replication logged?

What can we read out?

What to watch out for?

Set via Log_Replication=value

– 0 Do not log replication events

– 1 Log that DB is replicating (Default value)

– 2 Log summary info about each DB

– 3 Log info about each replicated document (both design and doc class)

– 4 Log info about each replicated field

Note: only impacts logging of replication events performed by the current server!

Replication Logs Replication verbosity logs

Keeps track of what the current database replicated with

– Tip1: Local <> Server replication leaves NO TRACE in server replica’s replication history!

– Tip2: When there’s nothing to replicate, no replication history entry created

Replication Logs Replication History

Feature aiming to optimize performance introduced in Domino 7

– Result: Replication History Entries indicating replication between A & C which never happened

Replication Logs Replication Triangulation

A

B

C

A – B

A – B

B – C

B – C

A – C

Idea: to prevent complete recalculation when replicating with an unknown server

– Results show 400% CPU increase in some cases (1700 servers)

– Disable (server side):

• NSF_REPLHIST_NO_TRI=1

• REPL_NO_WS_TRI_HIST=1

• REPL_NO_REMOTE_TRI_HIST=1

– Gotcha: need to clear replication history entries for all impacted DBs!

– Disable (client side):

• NSF_REPLHIST_NO_TRI=1 [No triangulated entries read]

• REPL_NO_WS_TRI_HIST=1 [No triangulated entries written]

Replication Logs Replication Triangulation

Purge Interval Replication Control

– New(ish) awesome option for preventing old documents from coming back (Introduced in Domino 8.5.3)

• Set via Replication Options > Space Savers

– DEBUG_REPL_PIRC=1 gives you date and summary info about documents not allowed to replicate

– DEBUG_REPL_PIRC=2 or greater provides more details about documents blocked by PIRC

Replication Logs PIRC logging

Please fill out your evaluation forms

– Your feedback is very important!

Feel free to get in touch!

– Contact - Ben Menesi

ThanksThank you for attending!


@BenMenesi


http://twitter.com/BenMenesi

Engage Online

SocialBiz User Group socialbizug.org

– Join the epicenter of Notes and Collaboration user groups

Social Business Insights blog ibm.com/blogs/socialbusiness

– Read and engage with our bloggers

Follow us on Twitter

– @IBMConnect and @IBMSocialBiz

LinkedIn http://bit.ly/SBComm

– Participate in the IBM Social Business group on LinkedIn

Facebook https://www.facebook.com/IBMConnected

– Like IBM Social Business on Facebook

https://reg.socialbizug.org/wispubs/socbizwidgets.nsf/homepage.xsp

http://www.ibm.com/blogs/socialbusiness

https://twitter.com/IBMConnect

https://twitter.com/IBMSocialBiz

http://bit.ly/SBComm

https://www.facebook.com/IBMConnected

Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from

IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial

publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS"

WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION,

INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted

according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have

used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which

IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and

discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific

situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any

relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide

legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested

those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the

capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-

party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE

IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual

property right.

IBM, the IBM logo, ibm.com, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®,

PartnerWorld®, Prove It!®, PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in

many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright

and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Notices and Disclaimers

IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

Software

log views log

different log events

nsflog file structure

size of log text

servers activity data

lots of events

additional data log

daos command logs log