Middleware upgrade to Oracle FMW12c. DBA darbu organizēšana
Post on 11-Dec-2021
1 Views
Preview:
Transcript
DBA darbu organizēšana
Middleware upgradeto Oracle FMW12c.
Real Case stories.
Andrey Chervonets (Андрей Червонец)● working with Oracle products since 2001● Senior technical expert, DBA (Oracle, DB2, MySQL, ...)● Certified as:
● Oracle Database Administrator (8i-11g)● Oracle Application Server Administrator (9i/10g)● Oracle WebLogic Administrator● IBM DB2 Database Administrator
LinkedIn: http://www.linkedin.com/in/andreychervonets
MeetUp #1521.01.2016Riga, Latvia
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
● has founded SIA CoMinder un 2011:● to provide better DBA expert services● for customers interested in improvements
web: http://www.cominder.eu/
Agenda
• Project 1: to Oracle FMW 12.1.3
• Project 2: to Oracle Reports Server 12.2.1
• Summary.
• Q + A
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
DBA darbu organizēšanaIntroduction
to Upgrade...
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
Why this topic?!
• Software Continuous improvements (including Installation and Setup)
• no more challenging work for DBA/Admin?
• let's show
• Old Upgrade/Migration best practices – is it actual?:
• understand why do You (client, business, anyone) need this!
• read the documentation, release notes, etc.
• understand the architecture of ALL
• understand the risks
• prepare:
› in advance
› test what is possible in testing environment
• production: use software Release 2... ?
› or at least PSU/Bundle 1... ?
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
“Friends” and “Enemies”
• What/Who can be the “friend” or the “enemy”:
• Oracle Documentation ?!
• Oracle Support (experts and MOS notes) ?!
• Project Documentation ?!
• Default settings
• Your own experience
• “Everything You do not know - is not Your favour!”
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
DBA darbu organizēšanaProject 1
Java apps, SOA upgrade...
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
TA
• OLD:
• OS: VM: Linux RHEL/OL 5.8/5.9 - (3 test + 3 prod)
• FWM 11g
● java: jrockit R28.2.2
● WebLogic Server: 10.3.5
● other: 11.1.n.n
• DB: 11.2.0.3 (on HP-UX) – 2 PROD + 1 TEST (business data + FMW metada in 1)
• NEW:
• OS: VM: Oracle Linux 7.x (3 test + 3 prod)
• FMW 12.1.3:
● java: jdk-7u80
● WebLogic Server 12.1.3.0.0
• DB: 11.2.0.4 (on HP-UX):
● 2 PROD + 1 TEST (business data only)
● 2 PROF + 1 TEST (FMW metadata)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Installation
• OS
• Oracle Linux
• missing packages
• local firewall/ports
• FMW
– JDK + slow startup fix (securerandom.source=file:/dev/./urandom)
– WL + Infrastructure services (java -jar fmw_12.1.3.0.0_infrastructure.jar)
● do not run Domain Configuration Wizard yet!
– SOA Suite (java -jar fmw_12.1.3.0.0_soa.jar)
● ! into the same Oracle Base and Oracle Home!
– Patch 20163149 for performance improvements (with latest OPatch)
– install SOA Suite patch p20423408_121300_Generic
! note: review requirements after each PSU available
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Setup
• Setup FMW domain
– create FMR metadata repository database ( ./rcu -interactive ...)
– create WL domain (config.sh)
• Deploy custom applications
• EAR: tratata.ear,....
• SOA: tratata-composits,..
•
• Certificates in java keystores
• find what really needed
• Identity store
• Trusted store
• Configure start/stop of ALL
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation and recommended “best something...”
• versions references may not match
• copy+paste from old
• recommended directory structure
• Project Documentation
• incomplete description
• missing configuration steps
• copy+paste from old
• undocumented components
› keystores/certificates
› firewall requirements
• Oracle Bugs
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation mismatch example:
• directory structure model in 12.1.3. docs:../oracle/product/ORACLE_HOME../oracle/config/domains../oracle/config/applications
•
• default directory in installer still use 12.1.1 model:../oracle/product/ORACLE_HOME/user_projects/domains../oracle/product/ORACLE_HOME/user_projects/applications
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Using non-default, but “recommended” directories structure or due to other reasonmay face the following:
• NodeManager startup problems:
err n1:<Aug 17, 2015 6:11:56 PM EEST> <WARNING> <Domains file not found: /u01/app/oracle/product/FMW12.1.3/oracle_common/common/nodemanager/nodemanager.domains>
err n2:<Aug 17, 2015 6:11:57 PM EEST> <SEVERE> <Fatal error in NodeManager server: Identity key store file not found: /u01/app/oracle/product/FMW12.1.3/oracle_common/common/nodemanager/security/DemoIdentity.jks>
• Workaround (unpublished, found similar case published by community): find + copy:
find /u01/app/oracle/ -name nodemanager.domains...cp -p /u01/app/oracle/config/user_projects/domains/base_domain/nodemanager/nodemanager.domains /u01/app/oracle/product/FMW12.1.3/oracle_common/common/nodemanager/nodemanager.domains
find /u01/app/oracle/ -name DemoIdentity.jks...cp -p /u01/app/oracle/config/user_projects/domains/base_domain/security/DemoIdentity.jks /u01/app/oracle/product/FMW12.1.3/oracle_common/common/nodemanager/security/DemoIdentity.jks
• Make sure later which file(s) when used during NM, ManagedServer start/stop!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Project Documentation example
• missing configuration steps - “User Messaging service” during setup
• caused missing SOA “plugin” in EM and required functionality
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Project Documentation bad example:
• undocumented keystores/certificates
• used default keystores under old (11.1) ORACLE_HOME to store production credentials together with default set of certificates.
• outdated documentation
• and many old EXPIRED certificates, the same certificates in many keystores, certificates from testing environment migrated to production ages ago.
• How to resolve
• search for keystores on old server (+ default ones)
• check all VALID/EXPIRED certificates in every keystore
• ask application developers
• and application properties (most keystores path defined in cfg + default keystores)
• make client/developers accept for found JSK+certificates.
• copy or exp/imp/migrate to new (12c) server
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• How to find
– default ones:/u01/Middleware/wlserver_10.3/server/lib/.../u01/jrockit/jre/lib/security/cacerts...
– + search:find / -name "*.jks" -exec ls -l {} \;find / -name "*cacert*" -exec ls -l {} \;
– + find with WLST
● command: listKeyStores
● or by name:wls:/base_domain/serverConfig> find(name='KeyStoreFileName');Finding 'KeyStoreFileName' in all registered MBean instances …/Servers/AdminServer CustomIdentityKeyStoreFileName
/u01/Middleware/wlserver_10.3/server/lib/TATATA.jks/Servers/AdminServer CustomTrustKeyStoreFileName null/Servers/ManagedTT CustomIdentityKeyStoreFileName
/u01/Middleware/wlserver_10.3/server/lib/TATATA.jks/Servers/ManagedTT CustomTrustKeyStoreFileName null...
– + ask application developers (some may be even hard-coded!)
– + and application properties (most keystores path defined in cfg + default keystores)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• How to identify required certificates
– dump JKS keystore with keytool
${KEYTOOL_PATH} -list -v -keystore ${JKS_PATH}
– make report (searching, dumping and making summary may be automated with Shell)
– REMEMBER to document findings and required one (this will save time in next upgrades)!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Findings reports example – simple keystore dump
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Findings reports examples
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
DBA darbu organizēšanaProject 1
STATUS = MIGRATED!
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
DBA darbu organizēšanaProject 2
Reports Server upgrade
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
TA
• OLD:
– OS: VM: Windows Server 2003 R (64-bit)
– Oracle Forms (client-server) and Reports 6i (client-server -RBE, reports-server)
– DB: 9.2.0.8
• NEW:
• OS: VM: Windows Server 2012 (64-bit)
• Oracle Forms (untouched yet)
• Oracle Reports 12c (12.2.1)
• DB: 9.2.0.8 (untouched yet)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Installation
• OS
• create extra user: oracle
• Metadata DB:
• install or use other host. (! Note: 12c DBAC default settings make “eat” too much RAM!)
• FWM:
– Install JDK (jdk1.8.0_65+) – 32/64-bit
– Install Oracle WLS ( on 64-Bit Platforms Using a 64-Bit JDK)
● java -jar fmw_12.2.1.0.0_infrastructure.jar (w/o examples!!)
● ! NOT java -jar fmw_12.2.1.0.0_wls.jar
● do not run Domain Configuration Wizard!
– Install Forms and Reports 12.2.1
● fmw_12.2.1.0.0_fr_win64_Disk1_1of1.zip\setup_fmw_12.2.1.0.0_fr_win64.exe
● Option: Forms and Reports Deployment (other Forms Builder) 12.2.1.0.0
● do not run Domain Configuration Wizard yet!
– Apply PSU/Bundled/required one-off patches!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Setup
• Setup FMW domain
– create FMR metadata repository database ( ./rcu )
– create WL domain (config.sh)
• Create Reports specific components
• start NM, Admin Server first
• start WLS_REPORTS managed server (JVM)
• createReportsToolsInstance (with WLST) – MANDATORY!
• createReportsServerInstance (with WLST) – if Stand-Alone RS required
• Test it is working + reconfigure (12.2 RS is more secure out-of-the-box, manual intervention required to make it more admin-friendly ;)
• Automate start/stop of ALL
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation
• versions
• copy+paste from old
• missing information
• incorrect configuration
• Oracle Bugs
• documented
• undocumented (SR created, output: new MOS notes)
• old bugs reflection
• Default settings
• DB memory (DBCA recommends too much (70% of RAM or so) for just metadata repository DB,Be Careful!)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation issues example: Task: post-installation tasks
Documentation ( https://docs.oracle.com/middleware/1221/formsandreports/install-fnr/install.htm#FRINS394 )2.4.8 Configuring Reports Using the Configuration Wizard2.4.8.1 Provisioning a Machine“Follow these steps to provision a machine for Oracle Reports....Start Node manager… Start Adminserver… Use WLST to create Reports ComponentscreateReportsToolsInstance(instanceName='<reportsToolsName>', machine='AdminServerMachine')
createReportsServerInstance(instanceName='<reportsServerName>', machine='AdminServerMachine')
...Start WLS_REPORTSCreate a reports tools targeted to machine 'AdminServerMachine'- reptools1Create a reports server targeted to machine 'AdminServerMachine' – repsvr_<hostname> … “
Really:Remark 1: Once more?! Really – not! Do only once in correct place!Remark 2: createReportsServerInstance is only required if You need extra stand-alone reports server instance. Default in-process RS will be created with WLS_REPORTS by defaultwith name “rep_wls_reports_YOURSEREVR_HOSTNAME”Remark 3: createReportsTools – is mandatory!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation issues example:
Task: Make Unicode in PDF
• src1:
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation issues example: Task: Make Unicode in PDF
• src2:
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced
• Oracle Documentation issues example - Task: Make Unicode in PDF
• What is wrong with documentation:
• We should merge 9.7 and 11.3 to have full picture
• Path in MBean Browser is a bit different
• Really MBean Browser is optional – we can make changes in rwserver.conf only (+restart RS)
• Path to uifont.ali may not match the documentation depending on tool used (RS or Builder)
Really 2 files installed under the following path:
%DOMAIN_HOME%\config\fmwconfig\components\ReportsToolsComponent\ReportsTools01\guicommon\tk\admin%DOMAIN_HOME%\config\fmwconfig\components\ReportsToolsComponent\ReportsTools01\tools\COMMON
• Instructions does not make Unicode in PDF working!!
– Because of MISSING (not conmfugured by any install/setup tool) + UNDOCUMENTED registry setting REPORTS_INSTANCE (see MOS note 2091306.1 for configuration details)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – Oracle bugs
• Oracle Bugs – 1 – uifont.ali location
• something similar was in 11g
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – Oracle bugs
• Oracle Bugs – 1 – uifont.ali location
• similar to Bug 9090478, but not the same
• MOS Note 2981306.1 about REPORTS_INSTANCE created after our SR
• in 12c it is not bug, but just undocumented feature (no Bug created for the problem)
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – Oracle bugs
• Oracle Bugs – 2 – RS Engine crash if Fonts Subsetting used
• happened after REPORTS_INSTANCE is set to correct uifont.ali location
• anyone need to employ fonts subsetting to have unicode in PDF (even if RS on Windows)
• Bug 22224336 : REP-56048 WITH PDF SUBSETTING 12C, registered 3 weeks after 12.2.1
• One-off patch generated in 2 months: (Linux: 12.01.2016, Windows: 16.01.2016 )
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – Oracle bugs
• Oracle Bugs – 2 – RS Engine crash if Fonts Subsetting used
• Bugs happens, what is wrong with this one?!
• Unicode in PDF – is basic functionality - once more: anyone need to employ fonts subsetting to have unicode in PDF
• It was working in 11g. Why should it fail in 12c? (really it is still 11g RS engine)
• 2 months – is too much for critical bugs fix (IMHO)
• Q: Has is (RS 12.2) been ever tested before software was published?!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – Oracle bugs
• Other Bugs/Issues faced
• Fnchk.exe FMW 12c Fails to Start : The Program Can't Start Because MSVCR90.dll Is Missing From Your Computer. (Doc ID 2078252.1)
• Bug 22334822 : ISSUE WITH FOLDER ACCESS NOT SUPPORTING '*' TO ALLOW READ/WRITE TO ANY LOCATION" (still not fixed, use W/A with multiple locations delimited by “;”)
• arialb.ttf and arialbd.ttf : Differences / Explanations about these 2 Files for Arial Bold TTF (Doc ID 1407114.1) – check which is in your system and do not mind to fix wrong example in uifont.ali (some last lines)
• Size of PDF generated in 12c is much bigger then in 6i (for the same content).
• Other issues may happen depending on configuration required – check Doc ID 2073820.1
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues – how to avoid
• Oracle Reports 12.2 – has some new features and configuration changes
• More secured out-of-the-fox – we need manual “unlock” to make RS more admin-friendly
• New tool to change settings with web user interface – MBeans browser (link is present in EM).
• Config file location may be changed (compared to 11g) or does not match documentation:
› depending on platform (Win/Linux)
› used tools (RWServlet or just Reports Builder)
• Reports Builder has NO stand-alone installation. It should be installed together with Forms and Reports full inmstallation + WL domain configuration + metadata repository database.
• 'Oracle Reports Services - WLS_REPORTS' Not Showing Up In The Start Menu (Note:2077328.1)
• How to avoid:
• Read product (exact version) documentation, release notes in-advance
• Review fresh version of “Reports 12c Relevant Notes and Links Index” (Doc ID 2073820.1)
• Check correct config files used (Note:2071812.1)
• Enable trace/diagnostic ( Note:2070743.1 )
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – MOS
• Strange filters in MOS search – where is Oracle Reports Server?
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Issues faced – MOS
• Strange filters in MOS search – where is 12.2 version?
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
DBA darbu organizēšanaProject 2
STATUS = MIGRATION TESTED!but still resolve minor issues to make system more stable.
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
DBA darbu organizēšanaSummary
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
Summary
• 1. Upgrade/Migration rule #1:
• DO NOT PLAN to use Product Release/Version 1 in production – still actual
› at least until 1st PSU/Bundle patch collection
• 2. Version information (docs, release notes, MOS notes) should be reviewed carefully!
• 3. Looks like Oracle actively use popular development/support approaches:
• “Web2.0” (even in documentation)
• “early client involvement”
• 4. Oracle Support expert skills level (and time zone) ?!
• if You are lucky!
• escalation still useful
• 5. Sorry to say, but too much bugs for such $$$ (product price).
• 6. Knowledge is the power – DBA experts skills still required for challenging issues!
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
“Friends” and “Enemies”
• What/Who can be the “friend” and/or the “enemy”:
• Oracle Documentation ?!
• Oracle Support (experts and MOS notes) ?!
• Project Documentation ?!
• Default settings
• Your own experience
• “Everything You do not know - is not Your favour!”
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Best practices
• Old good Upgrade/Migration best practices:
• understand why do You (client, business, anyone) need this!
• read the documentation, release notes, etc.
• understand the architecture of ALL
• understand the risks
• prepare
› in advance
› test what is possible in testing environment
• production: do not plan Release 1 for production
› or at least PSU/Bundle 1... ?
• Review documentation/Release/MOS notes/Patches once more
› time to time
› before production
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
Best practices
• Keep in order (correct and up to date):
• Project DBA documentation (it is not just sheet of paper, but useful tool )
– Project Sys-admins documentation (it is not just sheet of paper, but useful tool )
– Project Developers documentation (it is not just sheet of paper, but useful tool )
• Your skills and knowledge - update periodically!
This will make migration/upgrade project more comfortable.
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
off-topic:
• Recommended Book to read:
“Time Management for System Administrators” ( By Thomas A. Limoncelli )
Copyright © 2016 CoMinder Ltd., http://www.cominder.eu/
DBA darbu organizēšanaQ + A
Vai tas ir svārīgi?!
Copyright © 2014, SIA CoMinder, http://www.cominder.eu
DBA darbu organizēšanaСпасибо за внимание!Paldies par uzmanību!Thanks for attention!
Andrey Chervonets
e-mail: a.chervonets@cominder.eu LinkedIn: http://www.linkedin.com/in/andreychervonetsweb: http://www.cominder.eu
MeetUp #1521.01.2016Riga, Latvia
Copyright © 2016, SIA CoMinder, http://www.cominder.eu
top related