Code performance workshop - PUG Challenge

Code performance

ABL Profiler & Indexes

2

compile-listing ain’t enough

▪ Intro

▪ ABL Profiler: your new best friend

▪ Indexes: it's easier to find things quickly when you know where you left them

▪ Conclusion

who are we?

▪ One named Peter

▪ One named Paul

4

one named Peter (so-called Judge)

[email protected]

Software Architect*

@ Progress since 2003

Integration-y stuff – Authentication Gateway, HTTP-Out,

Corticon et al

OE Best Practices / OERA / AutoEdge / CCS

4GL since 1996

* Aka programmer who knows PowerPoint

mailto:[email protected]

5

one named Paul (Koufalis)

[email protected]

Progress DBA and UNIX admin since 1994

Expert consulting related to technical aspects of

Progress and OpenEdge

Wide range of experience

Small 10 person offices to 3500+ concurrent users

AIX, HPUX, Linux, Windows…if Progress runs on it,

I’ve worked on it

Single malts and American bourbons enthusiast

mailto:[email protected]

6

▪ The oldest and most respected independent DBA consulting firm in the world

▪ Five of the world’s top OpenEdge DBAs

▪ Author of ProTop, the #1 FREE OpenEdge Database Monitoring Tool

http://protop.wss.com


7

compile-listing ain't enough

Everybody knows that there's a problem: "It's slow". Now what?

1. Find the source of the problem

2. Identify the errant code – HOW ?

3. Fix it. That's the easy part

8

why are we here?

▪ The purpose of this workshop is to help you identify and fix application performance

problems by

• Familiarising you with the ABL profiler

• (Re) Teaching you the index rules

▪ You can use these in dev, QA and prod environments

▪ We're not talking about

• How do you measure (relative) performance

• Database design

• Network traffic

• All the other crappy slow parts of your application

▪ Point of the labs is to IDENTIFY problems not necessarily fix 'em

▪ The code & db we have is intended to be 'realistic' aka crappy

▪ You can expect

• Some useful facts and info

• A lot of opinions

You know what you're doing

We expect your opinions

and input

expectations

The ABL Profiler

11

your next best friend: the abl profiler

▪ What is it?

• "Unsupported" tool that provides line-by-line execution timings

• "Officially" available since v9 via PROFILER handle

• Ignores UI blocking

• 11.6.0 added tooling support to PDSOE

• Needs more love, especially wrt OOABL

▪ How do I use it?

• In DEV all-in-one tooling in PDSOE (via launch configs)

• In PROD

– PROFILER system handle

– Roll-your-own tools

12

PDSOE launch config

PDSOE profiler views

PDSOE profiler views

15

profiling (production) code

▪ Set PROFILER attributes

ORDER MATTERS

• ENABLED: turns the feature on and off

• PROFILING: starts measuring timings

• LISTING: automatically generate

DEBUG-LIST for executed procedures

into DIRECTORY directory

• STATISTICS: dumps out various stats

• COVERAGE: If true, identify ALL

statements and IPs, whether run or not

▪ Add "magic" config

• Eight-finger CTRL-ALT-SHIFT-SPACE-

F7 in UI or a config file

• Turn it on/off, define output file etc

// Mandatory attributes. ORDER MATTERSprofiler:enabled = trueprofiler:file-name = session:temp-dir

+ '/profiler.out'profiler:description = 'working the shop'

// optional attributes. Listing require SOURCE// code, which shouldn't be on your PROD// systemprofiler:listings = falseprofiler:directory = session:temp-dirprofiler:trace-filter = '*':uprofiler:tracing = '':u

// from 10.2Bprofiler:statistics = true// coverage wins profiler:coverage = true

profiler:profiling = true

16

profiling (production) code

▪ When done, stop profiling and write out

the data

▪ Optionally add listings in later

if profiler:enabled then

do:

assign profiler:profiling = false

// optionally turn the feature off

profiler:enabled = false.

// profiling's not much use unless it's

// dumped to disk

profiler:write-data().

end.

17

profiling via command-line

▪ Specify values via a file on the command

–profile <profile-options-file>

• where profile-options-file contains

The profiler is enabled for the whole session

-COVERAGE-STATISTICS-DESCRIPTION "your description"-FILENAME "/path/to/profile.out"-LISTINGS "path/to/debug/listings/directory"-PROFILING-RAWDATA-TRACE-FILTER "*"-TRACING "procedure-name | line-number , ..."

18

profiling production faq1. What should I profile?

For an AppServer request, turn on in the activate event procedure to profile the entire ABL request. Don't forget to turn profiling off and/or dump the data

In UI, after data comes back from the AppServer, measure how long it takes to query and browse all 14 squillion records in the "thoroughly-endorse" table

See exactly which lines of code are run (code coverage). Works better with automated tests

You're in this workshop for a reason; I'm sure you have places in your apps that need profiling

2. Can I profile AppServer calls?

Yes, but it requires two profiler sessions (ie the profile data is not automatically measured on the server

3. I have no source on my production system, can I still profile my app?

Yes, absolutely!

4. What's the cost in using profiling at runtime?

Minimal / you shouldn't care

19

profiler output file format

The file contains 11 sections, which are in EXPORT format

• Separated by single-line . (dot / period)

1. Description of profiling session

2. Description of modules (e.g., procedures, functions, triggers)

3. Procedure call-tree information

4. Timing summaries for each executable line that was a part of the profiling session

5. Detailed tracing information

6. Coverage analysis

7. Operators info

8. Module detail

9. Session high-water marks

10. Session db info

11. Data written using the USER-DATA() method

new in v2

empty in v2

20

2 03/21/2017 "slow http call NO-SRC NO-LISTING" 12:43:46 "pjudge".254 "OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter" "DefaultRequestFilter.dbg" 30800235 "~HttpHeader OpenEdge.Net.HTTP.HttpHeader" "" 091 "OpenEdge.Net.HTTP.BuilderRegistry" "" 12769266 "KeySet OpenEdge.Core.Collections.KeySet" "" 0146 "EncodeString OpenEdge.Net.URI" "" 0189 "propGet_Registry OpenEdge.Net.HTTP.Filter.Writer.SetHeaderMessageWriterBuilder" "" 0126 "Example/dummy.r" "" 29514321 "GetByte OpenEdge.Core.Memptr" "" 0382 "~HttpClient OpenEdge.Net.HTTP.HttpClient" "" 0224 "AddUserAgent OpenEdge.Net.HTTP.HttpClient" "" 0213 "propGet_Registry OpenEdge.Net.HTTP.ResponseBuilder" "" 052 "OpenEdge.Logging.Filter.LoggerFilterNode" "" 52279385 "~ClientSocket OpenEdge.Net.ServerConnection.ClientSocket" "" 0

profiler output example (1)

1

2

3

21

coverage

119 "ReturnDataset" 3444647.119 "OutputDataset" 6525455566364.119 "OutputDataset" 56970717879.

22

119 "ReturnDataset" 3444647.119 "OutputDataset" 6525455566364.119 "OutputDataset" 56970717879.

coverage

23

profiler links

▪ https://community.progress.com/community_groups/openedge_general/w/openedgegeneral/

1980.profiler-control-tool

▪ Kbase: PROFILER Object attributes and methods

http://knowledgebase.progress.com/articles/Article/19495

• Lots of other kbases on the Profiler and related tooling

https://community.progress.com/community_groups/openedge_general/w/openedgegeneral/1980.profiler-control-tool

http://knowledgebase.progress.com/articles/Article/19495

Indexes

25

compile-listing ain’t enough

▪ Everybody knows that there's a problem: "It's slow".

• Now what?

▪ Last year we covered some of the tools available to help you find the errant code

• ProTop, LOG-MANAGER, VST data, -zqil

▪ Peter just covered profiler

▪ Now that we know where the problem is, let’s fix it

26

stupid index tricks

▪ We are going to cover the rules without going into all the dirty details

▪ There are a number of excellent presentations on index rules

• Mike Lonski’s numerous “Pick an Index” PPTs are my favourites

27

but before we get to the rules…

▪ This only applies to ABL and not SQL

▪ Rules are applied in hierarchical order to filter indexes

• This is important: Each rule is applied and the result is one or more remaining indexes

• Use a worksheet approach to make the rules "easy" to apply

▪ The first 5 rules only apply to a subset of indexes

• Compiler scans all fields in the query and selects all indexes that have leading components with those fields

▪ Field match rules must be contiguous

• “Equality” on fields 1 and 3 of the index counts as 1, not 2

Rules continue to be applied until there is only one index left

28

the rules

For the first-pass set of indexes, filter using the following rules:

1. Pre-select only indexes with leading components in the where clause

2. If CONTAINS use word-index

3. Unique index with all components involved in the equality matches

4. Most active equality matches

• Sorta kinda…full matches trump partial matches

• But only if more than 1 field (sometimes)

5. Most active range matches

6. Most active sort matches

If no "good" index is found, select one from

7. The primary index

8. First index alphabetically by name

30

order table indexes

Table Records Size Size Count LOBs Size Mean

PUB.Order 727285 63.9M 92 727285 -- -- --

Indexes

Flags Index Name Cnt Field Name

u CustOrder 2 + CustNum + Ordernum

OrderDate 1 + OrderDate

pu OrderNum 1 + Ordernum

OrderStatus 1 + OrderStatus

SalesRep 1 + SalesRep

w SRepW 1 + SalesRep

SRepDate 2 + SalesRep + OrderDate

DateSRep 2 + OrderDate + SalesRep

SDateOstat 2 + ShipDate + OrderStatus

for each order

Index Selection Rule

U

CustOrder OrderDate

PU

OrderNum OrderStatus SalesRep

W

sRepW SRepDate DateSRep SDateOstatCarrier

If “CONTAINS”, use word-index X X X X X X X X X

Unique index with all components

involved in the equality matchesX X X X X X X X X

Most active equality matches (Full

matches trump partial matches)X X X X X X X X X

Most active range matches X X X X X X X X X

Most active sort matches X X X X X X X X X

The primary index *

First index alphabetically by

name

where salesrep = "BBB"


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index X X X


involved in the equality matches X X X


matches trump partial matches) 1 / 1 X 1 / 2

Most active range matches

Most active sort matches

The primary index


name

1

3

where salesrep = "DKP" and orderdate = 09/05/2011


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index X X X X X


involved in the equality matches X X X X X


matches trump partial matches) X X X 2 / 2 2 / 2

Most active range matches X X

Most active sort matches X X

The primary index X X


name X

2

5

1

34

multiple index use

▪ Where clause includes “AND”

• ALL components of each index are involved in equality matches

• No unique indexes are involved

▪ Where clause includes “OR”

• Both sides of OR contain at least the lead component of an index

• Either equality or range match

▪ CAREFUL: return order not guaranteed

35

where salesrep = "BBB" and orderStatus = "Shipped"


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index X X X X


involved in the equality matches X X X X


matches trump partial matches) 1 / 1 1/ 1 X 1 / 2



The primary index


name

2

4

2

36

gotchas

▪ Expressions break bracketing

for each order no-lock where month(orderDate) = 1 ...

▪ BEGINS does NOT break bracketing

• Considered a range bracket

for each order no-lock where salesRep begins "D"

• Uses the order.salesRep index

▪ MATCHES breaks bracketing

▪ Temp-table rules are subtly different

▪ When in doubt, test and verify

• LOG-MANAGER is your friend

where month(orderDate) > 6


U

CustOrder OrderDate

PU


W









The primary index *


name

9

38

test and verify? how !?!

▪ Using the “salesrep begins …” example

▪ ProTop of course. It’s free.

▪ Query object handle q:index-information(i)

▪ Log-manager - see next slides

39

log-manager

▪ A terribly underused but awesomely amazing tool

▪ Allows you to leave debug messages in your code

• No more /* Message here vValue. */

▪ Create some secret hotkey sequence to activate

• I.e. you can turn it on in production for one user

▪ Writes detailed info to a log file

assign log-manager:logfile-name = "c:\temp\wshop.log"

log-manager:logging-level = 3 // There are more levels than you think

log-manager:log-entry-types = "4GLTrace,4GLTrans,QryInfo".

41

test and verify with log-manager

▪ Still with our “for each order no-lock where salesrep begins ‘d’ ” example:

Type: Dynamically Opened Query

PREPARE-STRING: for each order no-lock where salesrep begins 'd'

Prepared at Runtime

Client Sort: N

Scrolling: Y

Table: sports2000.Order

Indexes: SalesRep

Times prepared: 1

Time to prepare (ms): 0

DB Blocks accessed:

sports2000 : 366074

DB Reads:

Table: sports2000.Order : 182913

Index: Order.SalesRep : UNAVAILABLE

42

test and verify using protop

43

About ProTop

▪ Download free version at


▪ Local ChUI for real-time data

▪ Web portal for trending data

• Free and $$

▪ Monitoring and Alerts ($$)

▪ Free version is open source (GNU

GPL). Take a look under the hood

if you want to write your own

monitoring code

▪ Hit “h” to get the help screen

Options are CaSE-sENSiTiVe!!

http://protop.wss.com/

44

ProTop Web Dashboard

Graphical view of _TableStat/_Indexstat VST

These graphs are from the $$ part of the ProTop web

dashboard

45

conclusion

▪ The aim today was to show you areas you can look at for improving the performance of your

application

▪ From a code perspective, by

• coding appropriately to the indexes you have

• retrieving only data that's appropriate

• not leaving data "lying around"

▪ From a design/architecture perspective, by

• appropriately balancing the size and quantity of network requests

▪ From an infrastructure perspective, by

• making sure that you optimise the data transfers from the DB server

fin

for each order where


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index


involved in the equality matches


matches trump partial matches)



The primary index


name

where month(orderDate) > 6


U

CustOrder OrderDate

PU


W









The primary index *


name

6

where salesrep = "BBB"


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index X X X


involved in the equality matches X X X


matches trump partial matches) 1 / 1 X 1 / 2



The primary index


name

1

3

where salesrep = "BBB" and orderStatus = "Shipped"


U

CustOrder OrderDate

PU


W


If “CONTAINS”, use word-index X X X X


involved in the equality matches X X X X


matches trump partial matches) 1 / 1 1/ 1 X 1 / 2



The primary index


name

2

4

where salesrep = "DKP" and orderdate = 09/05/2011


U

CustOrder OrderDate

PU


W











name X

2

5

1

where salesrep = "DKP" and orderdate = 09/05/2011 by salesrep


U

CustOrder OrderDate

PU


W











name X

2

5

1

where salesrep >= "DKP" and orderdate >= 09/05/2011 by salesrep


U

CustOrder OrderDate

PU


W






matches trump partial matches) X X X X X

Most active range matches 2 / 2 2 /2

Most active sort matches 1 / 2 X

The primary index


name

2

5

1

Code performance workshop - PUG Challenge

Documents