B7: Caring for an Ailing AppServer Diego Canziani Senior Technical Support Engineer.

B7: Caring for an Ailing AppServer™

Diego CanzianiSenior Technical Support Engineer

© 2008 Progress Software Corporation2

Agenda

Anatomy of the AppServer Diagnosing Problems Help Treating your AppServer

AppServer Broker

Cl ients

Agents

Agent

Agent

Agent

Agent

client

Listening Port


Anatomy of the AppServer

What is the OpenEdge® AppServer?



Broker• JavaTM process

• Handles communication from clients to agents

• Manages client and agent pools

• Writes to <brokername>.broker.log

Or What is the Progress® AppServer?

<brokername>.broker.log



Agent• “C” process

• Handles requests and replies to the clients

• Communicates “STATUS” to broker

• Writes to <brokername>.server.log

Or What is the Progress AppServer?

<brokername>.server.log



State-aware State-reset Stateless State-free

4 Operating Modes

One to one

Many to few

- Default



Client to Broker gets redirected to Agent• State-aware

• State-reset

Client to Agent through the Broker• Stateless

• State-free

2 Connection Models



2 Connection Models

State-aware

State-reset

client

AppServer

Broker

Agent

Agent

Agent

Agent

Agent

2 Connection Models

Stateless

State-free



AppServer Broker Thread Pools

AppServer

Broker

Listening Thread L-3090

Client Threads

C-0001

C-0002

C-0003

C-0004

C-0005

C-0006

…

Server Threads

S-0001

S-0002

S-0003

S-0004

S-0005



Broker communication with Agents

Broker starts agent and waits for agent to respond, status set to:

STARTING

Once started the agent reports to broker and broker sets status to:

AVAILABLE

Client request agent, is redirected to agent port, broker sets status to:

CONNECTING

Client connects to agent, agent reports to broker, broker sets status to:

CONNECTED

Agent completes request, notifies broker, broker sets status to:

AVAILABLE



[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358)

[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p]

[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p]

[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p]

[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p]

[08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p]

[08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360)

Reading Log Files

[08/04/04@11:41:31.334-0400]P-001268 T-009968 2 AS AS

Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358)

AS 4GLTRACE Run connectAuthSvc “-H localhost -S 6789”

[performCardAuth - authcard.p]



AdminServer - Required• JAVA process

• Manages all OpenEdge Brokers and Servers

• Writes to admserv.log

NameServer - Optional• JAVA process

• Traffic manager from clients to AppServers

• Writes to <nameservername>.ns.log

Supporting Processes


Agenda



Diagnosing Problems

$DLC/bin/asbman -name asbroker1 -start

OpenEdge Release 10.1C01 as of Fri May 9 21:58:30 EDT 2008

Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)

Searching for asbroker1 (8288)

Connecting to asbroker1 (8276)

Unable to start asbroker1 (8297)

Startup Problems


Diagnosing Problems

=====================================================================[08/05/18@20:05:25.933-0400] P-000283 T-Main 1 --- ---

/usr1/stat/progress/101c/wrk/asbroker1.broker.log opened.[08/05/18@20:05:25.939-0400] P-000283 T-Main 1 --- --- Logging level set to =2[08/05/18@20:05:25.940-0400] P-000283 T-Main 1 --- --- Log entry types activated:

UBroker.Basic,[08/05/18@20:05:26.020-0400] P-000283 T-Main 2 UB Basic ubroker version: v101c (07-May-08) (8038)

[08/05/18@20:05:26.314-0400] P-000283 T-L-9090 2 UB Basic Started listenerthread: L-9090 (8043)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- Could not listen on port: 9090 (8044)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- java.net.BindException: Address already in use (errno:226)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.socketBind(Native Method)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket.bind(ServerSocket.java:325)

[08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket.<init>(ServerSocket.java:186)

Startup Problems - <brokername>.broker.log


Diagnosing Problems

$DLC/bin/asbman -name asbroker1 -start


Searching for asbroker1 (8288)Connecting to asbroker1 (8276)Starting asbroker1. Check status. (8296)

$DLC/bin/asbman -name asbroker1 -query


Searching for asbroker1 (8288)Connecting to asbroker1 (8276)Broker: asbroker1 not running (8313)

Startup Problems


Diagnosing Problems

[08/05/18@20:15:21.807-0400] P-000435 T-S-0004 2 UB Basic Started server: /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logfile /usr1/stat/progress/101c/wrk/asbroker1.server.log -ubpid 435 -Ms 1 -logname asbroker1 -logentrytypes ASPlumbing,DB.Connects -logthreshold 0 -numlogfiles 3 -ASID 4 –ubpropfile /usr1/stat/progress/101c/dlc/properties/ubroker.properties -svrefresh -ipver IPv4 -db mydatabase (8108)

[08/05/18@20:15:22.210-0400] P-000435 T-S-0004 1 UB ---------- Message from server : Startup Procedure Failed (8113)

[08/05/18@20:15:22.241-0400] P-000435 T-S-0004 1 UB ---------- IOException reading message from server : java.io.EOFException: pipe to server broken (8117)

[08/05/18@20:15:22.241-0400] P-000435 T-S-0004 2 UB Basic ERROR: cannot start server. (8100)

Startup Problems - <brokername>.broker.log


Diagnosing Problems

[08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704)

[08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- Error initializing the application server. (5479)

[08/05/18@20:29:13.992-0400] P-000667 T-000000 2 AS AS Application Server Shutdown. (5476)

[08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704)

[08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- Error initializing the application server. (5479)


Startup Problems - <brokername>.server.log


Diagnosing Problems

[08/04/03@13:03:33.470-0400] P-008824 T-004332 2 AS AS Starting application server for asbroker1. (5560)

[08/04/03@13:03:33.501-0400] P-008824 T-004332 2 AS AS Application Server Startup. (5473)

[08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r Database sports2000 not connected. (1006)

[08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r startup procedure ended with an ERROR condition. (8025)


Startup Problems - <brokername>.server.log


Diagnosing Problems

All “INITIAL” agents must start or the AppServer will fail to start• Set initial agents to start to zero

–“initialSrvrInstance=0” in ubroker.properties• (IPv6 enabled on a machine can cause agents not to

communicate with broker)

Add “–Djvmstart.debug=1” to the “jvmargs=“ in AdminServerPlugins.properties

• View $WRKDIR/jvmStart.log file

Startup Problems


Diagnosing Problems

Broker Name : asbroker1Operating Mode : State-resetBroker Status : ACTIVE Broker Port : 3090Broker PID : 25319Active Servers : 5Busy Servers : 0Locked Servers : 0Available Servers : 5Active Clients (now, peak) : (0, 6)Client Queue Depth (cur, max) : (0, 11)Total Requests : 124Rq Wait (max, avg) : (259 ms, 11 ms)Rq Duration (max, avg) : (5543 ms, 24 ms)

PID State Port nRq nRcvd nSent Started Last Change 25320 CONNECTED 02003 000020 000020 000020 May 19, 2008 06:28 May 19, 2008 07:28 25321 AVAILABLE 02018 000031 000031 000031 May 19, 2008 06:28 May 19, 2008 07:28 25323 AVAILABLE 02019 000014 000014 000014 May 19, 2008 06:28 May 19, 2008 07:2825322 CONNECTED 02006 000026 000026 000026 May 19, 2008 06:28 May 19, 2008 07:28 25324 AVAILABLE 02021 000033 000033 000033 May 19, 2008 06:28 May 19, 2008 07:28

Phantom Agents


Diagnosing Problems

ps -ef | grep _proapsv

statqa 25324 25319 0 06:28:47 ? 0:24 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf

statqa 25323 25319 0 06:28:46 ? 0:56 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf

statqa 25321 25319 0 06:28:45 ? 0:55 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 –log

Phantom Agents

protrace.25320

protrace.25322


Diagnosing Problems

Look at log files• <brokername>.broker.log

• <brokername>.server.log

• Database log files where agent is connected

Increase logging levels• srvrLoggingLevel=4 (extended)

• srvrLogEntryTypes=4GLTrace– UBNet - New for 10.1C

Diagnosing Application problems


Diagnosing Problems

Broker Name : asbroker1Operating Mode : StatelessBroker Status : ACTIVE Broker Port : 3090Broker PID : 25319Active Servers : 1Busy Servers : 0Locked Servers : 0Available Servers : 1Active Clients (now, peak) : (0, 1)Client Queue Depth (cur, max) : (0, 1)Total Requests : 11Rq Wait (max, avg) : (11 ms, 11 ms)Rq Duration (max, avg) : (24 ms, 11 ms)

PID State Port nRq nRcvd nSent Started Last Change 25320 SENDING 02003 000001 000001 00001 Apr 4, 2008 11:40 Apr 4, 2008 11:41

Hanging Agents


Diagnosing Problems

[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358)

[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p]

[08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p]

[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p]

[08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p]

[08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p]

[08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360)

Diagnosing Agent problems


Agenda



Help Treating your AppServer

-query• Get status and history of AppServer broker

and agents

-kill• Shutdown an AppServer with busy or hanging

agents

• May need to run twice in a row

asbman



Available on PSDN ABL source code available Read and Sort multiple log files Unsupported

LogRead



Don’t connect using the startup parameters• Any database down to a single database will

cause your AppServer agents to be unresponsive

Connect using the “CONNECT” call in the • srvrStartupProc=

• srvrConnectProc=

• srvrActivateProc=

Use “IF CONNECTED” ABL

Database Connections



New in 10.1B Described in ubroker.properties.README Read the CAUTION!

• Use only when directed by Tech Support

Periodically force the shutdown of all agents and allow new agents to start when requested

Lifespan



OpenEdge Management


D I S C L A I M E R

Under Development

The next slide includes information about potential future products and/or product enhancements.

What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here.

D I S C L A I M E R



New in 10.2A Described in ubroker.properties.README List of all clients known to the AppServer,

includes:• Host/IP

• Port

• Connected Agent

• More!

List AppServer Connections



Technical Support Knowledge Base (kbase) PSDN

Help from Progress


AppServer Broker

In Summary

Understand your AppServer Diagnose your Problem Care for your Ailing AppServer

Agent

Agent

Agent

Agent

client

Listening Port

Cl ients

Agents


Questions?


Thank You


B7: Caring for an Ailing AppServer Diego Canziani Senior Technical Support Engineer.

Documents

progress appserver

p slide

appserver slide

progress software corporation

clien t appserver broker

appserver client

appserver broker communication

openedge appserver