11i related sections of note 455194.11

1. Gather background information about the environmenta) Run the following SQL to provide information about the nodes

REMREM START OF SQLREM set echo onset timing onset feedback onset long 10000set pagesize 132set linesize 80col PLATFORM_CODE form a5col HOST form a20col DOMAIN form a30col WEBHOST form a30col VIRTUAL_IP form a20col status form a20col ConcMgr form a8col Forms form a8col WebServer form a8col Admin form a8col Database form a8col last_monitored form a40--select NODE_NAME, to_char(CREATION_DATE, 'DD-MON-RR HH24:MI') creation_date, PLATFORM_CODE, decode(STATUS,'Y','ACTIVE','INACTIVE') Status, decode(SUPPORT_CP,'Y', 'ConcMgr','No') ConcMgr, decode(SUPPORT_FORMS,'Y','Forms', 'No') Forms, decode(SUPPORT_WEB,'Y','Web', 'No') WebServer, decode(SUPPORT_ADMIN, 'Y','Admin', 'No') Admin, decode(SUPPORT_DB, 'Y','Rdbms', 'No') Database, to_char(LAST_MONITORED_TIME, 'DD-MON-RR HH24:MI:SS') last_monitored, NODE_MODE, SERVER_ADDRESS, HOST, DOMAIN, WEBHOST, VIRTUAL_IP, SERVER_IDfrom fnd_nodeswhere node_name != 'AUTHENTICATION'/REM REM END OF SQLREM

b) Provide the following Apache configuration files

Release 11i

All files located in the $IAS_ORACLE_HOME/Apache/Apache/conf directory All files located in the $IAS_ORACLE_HOME/Apache/Jserv/etc directory $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh $FND_TOP/secure/<SID>_<Hostname>/<SID>.dbc $APPL_TOP/admin/<SID>_<Hostname>.xml

You could use a script like that below to quickly gather this information:-

tar cvf mzConfigFiles_`hostname`_`date '+%m%d%y_%H%M'`.tar \ $IAS_ORACLE_HOME/Apache/Apache/conf/* \ $IAS_ORACLE_HOME/Apache/Jserv/etc/* \ $IAS_ORACLE_HOME/Apache/modplsql/cfg/* \ $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh \ $FND_TOP/secure/*/*.dbc \ $APPL_TOP/admin/*.xml

Collate the latest versions of all configuration files for Apache and OC4J configuration files, which are mainly located at $ORACLE_CONFIG_HOME

You can use the following command to package up these files:-

tar cvf configFiles_`hostname`.tar $ORA_CONFIG_HOME/10.1.2/* \ $CONTEXT_FILE \ $ADMIN_SCRIPTS_HOME \ $APPL_TOP/*.env \ $ORA_CONFIG_HOME/10.1.3/Apache \ $ORA_CONFIG_HOME/10.1.3/config \ $ORA_CONFIG_HOME/10.1.3/j2ee/oacore \ $ORA_CONFIG_HOME/10.1.3/j2ee/forms \ $ORA_CONFIG_HOME/10.1.3/j2ee/oafm \ $ORA_CONFIG_HOME/10.1.3/network \ $ORA_CONFIG_HOME/10.1.3/opmn \ $INST_TOP/ora/10.1.3/*.env

c) Confirm your network architecture and whether you are using a DMZ: A network topology diagram would be useful, otherwise please describe the eBusiness environment network architecture

2. Check latest Java Cache patches have been applied

Release 11i

Consider applying these patches :

Bug 7527878 "FIX FOR BUG 7484199 FOR ORACLE EBS 11i (JOC 10.1.3.3 for iAS 1.0.2.2)" Bug 7437603 "BACKPORT:6746642:RUP6:11I.ATG_PF.H:11.5.10.2:AITR12:INTERMITTENT OACORE APPLICA" ORBug 7389573 "STANDALONE RELEASE OF BUG 7286131 ON RUP5"

The above patches are all included in ATG RUP7

3. Confirm all eBiz nodes have been configured correctlya) Use the following SQL to list the relevant profile options for Java Caching

REMREM START OF SQLREM

set echo onset timing onset feedback onset long 10000set linesize 120set pagesize 132column SHORT_NAME format A30column NAME format A40column LEVEL_SET format a15column CONTEXT format a30column VALUE format A60 wrap select p.profile_option_name SHORT_NAME, n.user_profile_option_name NAME, decode(v.level_id, 10001, 'Site', 10002, 'Application', 10003, 'Responsibility', 10004, 'User', 10005, 'Server', 10007, 'SERVRESP', 'UnDef') LEVEL_SET, decode(to_char(v.level_id), '10001', '', '10002', app.application_short_name, '10003', rsp.responsibility_key, '10005', svr.node_name, '10006', org.name, '10004', usr.user_name, '10007', 'Serv/resp', 'UnDef') "CONTEXT", v.profile_option_value VALUE from fnd_profile_options p, fnd_profile_option_values v, fnd_profile_options_tl n, fnd_user usr, fnd_application app, fnd_responsibility rsp, fnd_nodes svr, hr_operating_units org where p.profile_option_id = v.profile_option_id (+) and p.profile_option_name = n.profile_option_name and p.profile_option_name in ('FND_CACHE_PORT_RANGE','JTF_DIST_CACHE_PORT','FLEXFIELDS:USE_JAVA_CACHE') and usr.user_id (+) = v.level_value and rsp.application_id (+) = v.level_value_application_id and rsp.responsibility_id (+) = v.level_value and app.application_id (+) = v.level_value and svr.node_id (+) = v.level_value and org.organization_id (+) = v.level_value order by short_name, level_set;REMREM END OF SQLREM

The JTF_DIST_CACHE_PORT profile option must be consistent for all nodes and needs to match the entry in jserv.properties. FND_CACHE_PORT_RANGE profile option should only be set when using DMZ configurations and is explained in Note 287176.1 "DMZ Configuration with Oracle E-Business Suite 11i" in section "5.9: Enable Distributed Oracle Java Object Cache Functionality"

In a DMZ environment, the port numbers used by both these two profile options need to be opened through the firewall. Also bear in mind that a common problem is to miscalculate the number of ports required so pay particular attention to the FND_CACHE_PORT_RANGE calculation

b) Use the following SQL to check the AQ setup and Workflow Agent Listener/WF Java Deferred Agent Listeners are running

REMREM START OF SQLREMset echo onset timing onset feedback onset pagesize 132set linesize 100col name form a30col queue_table form a30col enqueue_enabled form a7col dequeue_enabled form a7col retention form a20--select name, queue_table, enqueue_enabled, dequeue_enabled, retentionfrom dba_queueswhere owner = 'APPLSYS'and queue_type = 'NORMAL_QUEUE'order by 1/--set pagesize 132set linesize 90col COMPONENT_NAME form a50col COMPONENT_STATUS form a15col COMPONENT_TYPE form a25col STARTUP_MODE form a10col CONTAINER_TYPE form a5col INBOUND_AGENT_NAME form a20col OUTBOUND_AGENT_NAME form a20col CORRELATION_ID form a30col MAX_IDLE_TIME form 9999999remselect COMPONENT_NAME, COMPONENT_STATUS, COMPONENT_TYPE, STARTUP_MODE, CONTAINER_TYPE, INBOUND_AGENT_NAME, OUTBOUND_AGENT_NAME, CORRELATION_ID, MAX_IDLE_TIMEfrom FND_SVC_COMPONENTSorder by COMPONENT_NAME/REMREM END OF SQLREM

c) Check Event Group 'oracle.apps.fnd.bes.control.group' is registered and enabled in the Workflow System

REMREM START OF SQLREMREM List events related to Apps 11i Java CachingREMset echo onset feedback onset pagesize 132set linesize 100col name form a60col stat form a8col evt.licensed_flag form a1col generate_function form a65col ecl form a6col sub_stat form a8col sub_lic form a1col sub_cl form a6col rule_function form a45col source_type form a8--select evt.name, evt.type, evt.status stat, evt.licensed_flag, evt.generate_function, evt.java_generate_func, sub.STATUS sub_stat, sub.LICENSED_FLAG sub_lic, sub.SOURCE_TYPE source_type, sub.phase, sub.description, sub.rule_function rule_function, sub.java_rule_funcfrom wf_events evt, wf_event_subscriptions subwhere evt.guid = sub.event_filter_guidand evt.name = 'oracle.apps.fnd.bes.control.group'order by evt.name,evt.type, sub.source_type desc/---- check which events registered to bes.control.group--select name, statusfrom wf_eventswhere guid in (select member_guid from wf_event_groups where group_guid = (select guid from wf_events where name = 'oracle.apps.fnd.bes.control.group'))order by 2,1/--- --- Check 'oracle.apps.fnd.wf.ds.userRole.updated' event---select event_name, subscription_source_type,

subscription_phase, subscription_rule_data, subscription_rule_functionfrom WF_ACTIVE_SUBSCRIPTIONS_Vwhere event_name = 'oracle.apps.fnd.wf.ds.userRole.updated'/REM REM END OF SQLREM

d) List the nodes currently registered for Java Caching and the cache configuration details

REMREM START OF SQLREMset echo onset timing onset feedback onset long 10000set pagesize 132set linesize 90col APPLICATION_ID form 9999999col COMP_NAME form a40col COMPONENT_KEY form a40col LOADER_CLASS_NAME form a60col BUSINESS_EVENT_NAME form a80col HOSTNAME form a50REMREM Collect information about the registered nodesSELECT B.HOSTNAME, decode(B.LOAD_PICK_UP_FLAG,'t','TRUE','FALSE') load_pickup_flag, decode(B.PREFAB_ENABLED_FLAG,'t','TRUE','FALSE') prefab_enabled_flag, B.CACHE_SIZE, B.WSH_TYPE, T.DESCRIPTION, B.WEIGHT, to_char(B.LAST_UPDATE_DATE,'DD-MON-RR HH24:MI') last_update_datefrom JTF_PREFAB_WSH_POES_TL T, JTF_PREFAB_WSH_POES_B BWHERE B.WSH_PO_ID = T.WSH_PO_ID (+)order by B.HOSTNAME/REMREM Information about the cache component itemsREMselect COMP_NAME, COMPONENT_KEY, LOADER_CLASS_NAME, decode(TIMEOUT_TYPE,'it','IdleTime','ttl','TimeToLive','unknown') timeout_type, TIMEOUT, TIMEOUT_UNIT, APPLICATION_ID, decode(SGID_ENABLED_FLAG,'t','TRUE','FALSE') SGID_ENABLED, decode(DISTRIBUTED_FLAG,'t','TRUE','FALSE') Distributed, BUSINESS_EVENT_NAMEfrom JTF_PREFAB_CA_COMPS_Border by APPLICATION_ID,COMP_NAME/REM

REM END OF SQLREM

e) Run the following SQL to identify how frequently the listeners run and their service parameters

REMREM START OF SQLREMset echo onset timing onset feedback onset long 10000set pagesize 132set linesize 100col CONCURRENT_QUEUE_NAME form a20col USER_CONCURRENT_QUEUE_NAME form a35col MIN_PROCESSES form 9999col MAX_PROCESSES form 9999col SLEEP_SECONDS form 9999col SERVICE_PARAMETERS form a95--select FCQ.CONCURRENT_QUEUE_NAME, FCQ.USER_CONCURRENT_QUEUE_NAME, FCQS.MIN_PROCESSES, FCQS.MAX_PROCESSES, FCQS.SLEEP_SECONDS, FCQS.SERVICE_PARAMETERSfrom FND_CONCURRENT_QUEUE_SIZE FCQS, FND_CONCURRENT_QUEUES_TL FCQwhere FCQS.CONCURRENT_QUEUE_ID = FCQ.CONCURRENT_QUEUE_IDand FCQ.CONCURRENT_QUEUE_NAME like 'WF%'order by 1/REMset linesize 110col COMPONENT_NAME form a40col COMPONENT_STATUS form a8col parameter_name form a30col parameter_value form a25--select c.COMPONENT_NAME, c.COMPONENT_STATUS, p.parameter_name, v.parameter_valuefrom FND_SVC_COMP_PARAM_VALS v, fnd_svc_comp_params_b p, fnd_svc_components cwhere v.component_id = c.component_idand v.parameter_id = p.parameter_idand upper(c.COMPONENT_NAME) like upper('Workflow Java Deferred Agent Listener')order by 1,3/REMREM END OF SQLREM

f) Check the file system (Release 11i only)

For each Apps Middle Tier node, run the following Unix script. This script will pull out the most relevant configuration file settings and most relevant Java code version information

######################################################################## START OF SCRIPT ########################################################################LOG_FILE="mzJavaCacheChecks_`hostname`"(echo "Run started "dateecho " "##echo "Checking jserv.conf"echo "-------------------"grep "^ApJServGroup " $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.confecho " "echo "Checking jserv.properties"echo "-------------------------"grep -e "IASCacheProvidercacheProvider" -e "LONG_RUNNING_JVM" -e "CACHEMODE" -e "JTFDBCFILE" $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties echo " "echo "Checking zone.properties"echo "-------------------------"grep "^servlets.startup" $IAS_ORACLE_HOME/Apache/Jserv/etc/zone.properties ##echo " "echo "Checking Java class versions"echo "----------------------------"adident Header $OA_JAVA/oracle/apps/jtf/cache/*.class ##echo " "echo "Run completed "date) 2>&1 | tee ${LOG_FILE}.txt######################################################################## END OF SCRIPT ########################################################################

NOTE - if the line below is returned from this script, then it would be worth re-testing after commenting it out, as this has been known to contribute to Java Caching issues servlets.startup=oracle.apps.fnd.framework.OAStartupServlet

g) Run the Java Diagnostic tests and upload the results

These can be accessed by either :-

a) CRM HTML Administration responsibility Diagnostics--> CRM Foundation--> Caching Framework then select the "Run without Prerequisite" button to launch the tests this will produce two test reports which you can then view and "Upload to Support" or "Save" as a file

b) Direct navigation to the diagnostic tests http://<hostname>:<port>/OA_HTML/jtfqalgn.htm

4. Identify nature and scope of the issue This will confirm the extent of the issue by gathering the results of the following different tests. This will confirm the extent of the issue by gathering the results of the following different tests. Use the same browser session for all these tests to ensure the same JServ session is used for each test

THE FOLLOWING HIGHTLIGHTED ITEMS HAVE BEEN CONFIRMED – These are the issues happening in our environment:4.1. New responsibilities for existing users not showing

a. Login as an existing user, (for example TESTUSER1) then logout. This is to ensure the Java Cache has been initialized and to set the JServ session cookie b. Login as SYSADMIN user to add new self service framework responsibility to an existing user (TESTUSER1) b. Log out d. Log on as the TESTUSER1 user e. Confirm if the new responsibility is available or not f. If not, bounce Apache or clear the Java cache g. Login as the TESTUSER1 again, and confirm the new responsibility is now available

4.2 Changes to responsibilities not showing immediately a. Disable or re-enable self service framework responsibility. (blank out end date or introduce an end date of yesterday) b. Log off c. Log on d. Confirm if the change to the responsibility has taken effect or not e. Bounce Apache f. Re-check to see if change to the responsibility has taken effect or not

4.6. Time based testIf any of the above tests failed to display a change without bouncing Apache, re-do the same test but this time wait for 60 minutes after making the change before you login again to test if the change has taken effect.

This test is to confirm whether the change is not happening at all, or is just taking a long time to process

THE FOLLOWING HIGHTLIGHTED ITEMS HAVE BEEN TESTED IN OUR ENVIRONMENTS – Results were inconsistent, in that sometimes executing this process helped, in others there was no resolution of the problem:

Check which of the following potential workaround have a positive impact

a) After making a change that has not been propogated, run the following three concurrent requests, in order and confirm if this provides a workaround to the problem

a. "Sync responsibility role data into the WF table."b. "Synchronize WF LOCAL tables"

c. "Workflow Directory Services User/Role Validation" with the 'Fix dangling user/roles' and 'Add missing user/role assignments' set to "yes"

THESE two options WERE NOT TESTED as the access is not available to us:

b) In the "Functional Administrator" responsibility->core Services->Caching Framework->Global Configuration

Use the "Clear All" cache option to manually invalidate the Java Cache c) Does bouncing Apache solve the problem

BELIEVE THIS COULD ONLY BE DONE IN CONJUNCTION WITH PLANNED OUTAGE:5. Gather additional data to help identify the failure pointThese steps will ensure the appropriate data is gathered to track the error whilst it is occurring.

a) Release 11i only Enable Apache debug level logging by following note 249669.1 "How To Collect Apache and Jserv Debugging Details For Applications 11i" In particular the jserv.log logging should be enabled.

b) Release11i only Shutdown Apache and archive the following log files The following files from the $IAS_ORACLE_HOME/Apache/Apache/logs directory i) access_log.<ID Number> ii) error_log The following files from the $IAS_ORACLE_HOME/Apache/Jserv/logs directory iii) mod_jserv.log iv) jserv.log v) All files from the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm directory These will have filenames of the form <JVM GroupName>.<JVM Instance>.stdout|stderr For example OACoreGroup.0.stderr vi) $COMMON_TOP/rgf/<Instance>_<Hostname>/javacache.log

c) Run the following Unix script to gather data from the operating system with the environment still shutdown.

This will allow us to confirm that there are no rogue processes still running and all the tcp ports are free

######################################################################## START OF SCRIPT ########################################################################LOG_FILE="OS_`hostname`_`date '+%m%d%y_%H%M'`.txt"echo "**********************************************" >>$LOG_FILEecho "Run started " >> $LOG_FILEdate >>$LOG_FILEecho "Running as `id`" >> $LOG_FILEecho "**********************************************" >>$LOG_FILEecho "uname -a" >>$LOG_FILEuname -a >>$LOG_FILEecho "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE

echo "ulimit -a" >>$LOG_FILEulimit -a >>$LOG_FILEecho "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILEecho "netstat -a">> $LOG_FILEnetstat -a >> $LOG_FILEecho "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILEecho "ps -e -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args" >> $LOG_FILEps -e -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args >>$LOG_FILEecho "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILEecho "Run completed " >> $LOG_FILEdate >> $LOG_FILE######################################################################## END OF SCRIPT ########################################################################

d) Release 11i only Startup Apache

e) In OAM navigate to Workflow Manager Service Components--> select "Workflow Java Deferred Agent Listener" then click "Edit" button Click "Next" button --> Change "Log Level" to STATEMENT Click "Finish" button Restart the "Workflow Java Deferred Agent Listener" for the change to take effect

f) In OAM navigate to Workflow Manager Service Components--> select "Workflow Agent Listener Service" on the right hand side of the screen In the next screen it will list three Service Instances Select "Workflow Agent Listener Service" and stop the service Select "Workflow Agent Listener Service" then click "Edit" button Select the "Edit Service Parameters" button Change the value SVC_CONTAINER_LOG_LEVEL=4 to be SVC_CONTAINER_LOG_LEVEL=1 Click OK to save this change Click Save button Select "Workflow Agent Listener Service" and start the service The resulting log files are located in the $COMMON_TOP/admin/log/<instance>_<hostname> directory. Look for files of the filename format FNDCPGSC<ID Number>.txt after reproducing the issue and upload the updated files

g) Reproduce the problem then provide the following information:-

h) Rerun the Unix script from step (c) above

ii) Output from the following SQL

REMREM START OF SQLREMREM Check entries for WF_JAVA_DEFERRED, WF_JAVA_ERROR and WF_CONTROL queues

REMset echo onset timing onset feedback onset long 10000set pagesize 132set linesize 105col queue_name form a17col state form a15col how_many form 9999999999--select'wf_java_deferred' queue_name,decode(state, 0,'0 (Ready)', 1,'1 (Delayed)', 2, '2 (Retained)', 3, '3 (Exception)', to_char(state|| ' (Other)')) state,count(*) how_Manyfrom wf_java_deferredgroup by stateUNION ALLselect'wf_java_error' queue_name,decode(state, 0,'0 (Ready)', 1,'1 (Delayed)', 2, '2 (Retained)', 3, '3 (Exception)', to_char(state|| ' (Other)')) state,count(*) how_Manyfrom wf_java_errorgroup by stateUNION ALLselect'wf_control' queue_name,decode(state, 0,'0 (Ready)', 1,'1 (Delayed)', 2, '2 (Retained)', 3, '3 (Exception)', to_char(state|| ' (Other)')) state,count(*) how_Manyfrom wf_controlgroup by state/REMREM Check the difference between enqueue and dequeue time.REM REM This script queries the last 24 hours. REM col corr_id form a60col retry_count form 999col msg_state form a12col enq_time form a18col deq_time form a18col Elapsed_Sec form 999999999select 'wf_java_deferred' queue_name, retry_count retrys, msg_state, to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time, to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time,

to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99') Elapsed_Sec, corr_idfrom applsys.aq$wf_java_deferred awhere a.enq_time >= sysdate - (24/24) -- arbitrary 24 hoursUNION ALLselect 'wf_java_error' queue_name, retry_count retrys, msg_state, to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time, to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time, to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99') Elapsed_Sec, corr_idfrom applsys.aq$wf_java_error awhere a.enq_time >= sysdate - (24/24) -- arbitrary 24 hoursUNION ALLselect 'wf_control' queue_name, retry_count retrys, msg_state, to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time, to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time, to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99') Elapsed_Sec, corr_id from applsys.aq$wf_control awhere a.enq_time >= sysdate - (24/24) -- arbitrary 24 hoursorder by 5,6,7/REMREM END OF SQLREM

iii) The output from the following SQL to gather the STATEMENT level debug information

REMREM START OF SQLREMset echo onset timing onset feedback onset long 10000set pagesize 132set linesize 100col time_stamp form a17col user_name form a12col log_level form a10col module form a57 wrapcol message_text form a98 wrapcol session_id form 999999999999REMREM Show all messages for the last 1 hour REM select to_char(m.timestamp, 'DD-MM-RR HH24:MI:SS') time_stamp, u.user_name, l.meaning log_level, m.module, m.message_textfrom fnd_log_messages m, fnd_user u, fnd_lookup_values lwhere m.user_id = u.user_idand l.lookup_type = 'AFLOG_LEVELS'and l.lookup_code = m.log_level

and m.timestamp >= sysdate - (1/24) -- arbitary show all messages for last 1 hourorder by m.timestamp, m.log_sequence/REM REM END OF SQLREM

iv) Collect Java Thread Dump for the OACoreGroup JVMs

For each of the OACoreGroup JVMs, run the following operating system command

kill -3 <PID>

If you need help to identify the correct JVM process to run the kill command for, you can use the following procedure

Release 11i

cd $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm /sbin/fuser OACoreGroup.*.stdoutRelease 12

ps -ef | grep CLIENT | grep oacoreThis should return the process id (PID) for the OACoreGroup JVM(s)

v) Output from the following SQL which collects data from the Queue tables

REMREM START OF SQLREMset echo onset timing onset feedback onset long 10000set pagesize 1000set linesize 120col CORR_ID form a45col MSG_STATE form a13col ENQ_TIME form a18col DEQ_TIME form a18col RETRY_COUNT form 99999col EXPIRATION_REASON form a19col USER_DATA form a10000REMselect CORR_ID, MSG_STATE, to_char(ENQ_TIME,'DD-MON-RR HH24:MI:SS') enq_time, to_char(DEQ_TIME,'DD-MON-RR HH24:MI:SS') deq_time, RETRY_COUNT, EXPIRATION_REASON,

USER_DATAfrom applsys.aq$wf_java_deferred order by ENQ_TIME, DEQ_TIME/REM REM END OF SQLREM

vi) Apache log files as listed below

Release 11iThe following files from the $IAS_ORACLE_HOME/Apache/Apache/logs directory i) access_log.<ID Number> ii) error_log The following files from the $IAS_ORACLE_HOME/Apache/Jserv/logs directory iii) mod_jserv.log iv) jserv.log v) All files from the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm directory These will have filenames of the form <JVM GroupName>.<JVM Instance>.stdout|stderr For example OACoreGroup.0.stderr vi) $COMMON_TOP/rgf/<Instance>_<Hostname>/javacache.log