Oracle error 20001: java.sql.SQLException: ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512

January 24, 2013, 6:31 pm

≫ Next: java.sql.SQLException: ORA-01017: invalid username/password; logon denied for glogdba after cloning OTM

≪ Previous: Workflow Mailer Notification not sending out mails

$

0

0

Anand  pinged me today.  After cloning an 11i instance, the following error was coming when AppsLocalLogin.jsp was accessed:

PRE>Unable to create anonymous session. Your session is no longer valid. Oracle error 20001: java.sql.SQLException: ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512: at "APPS.MO_UTILS", line 27 ORA-06512: at "APPS.MO_UTILS", line 232 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 2109 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 369 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 538 ORA-06512: at "APPS.FND_AOLJ_UTIL", line 217 ORA-06512: at line 1 has been detected in createSession(int, String,Hashtable)(userid=6','D3F9A80801171DD4E043447920036C9924997405341204718936747621840314').

META name="fwk-error-detail" content="oracle.apps.fnd.framework.OAException%3A+Application%3A+FND%2C+Message+Name%3A+SQL_PLSQL_ERROR.+Tokens%3A+ROUTINE+%3D+createSession(int%2C+String%2CHashtable)(userid%3D6'%2C'D3F9A80801171DD4E043447920036C9924997405341204718936747621840314')%3B+REASON+%3D+java.sql.SQLException%3A+ORA-20001%3A+Oracle+error+-6508%3A+ORA-06508%3A+PL%2FSQL%3A+could+not+find+program+unit+being+called+has+been+detected+in+fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG.%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+27%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+232%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+2109%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+369%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+538%0AORA-06512%3A+at+%22APPS.FND_AOLJ_UTIL%22%2C+line+217%0AORA-06512%3A+at+line+1%0A%3B+ERRNO+%3D+20001%3B+%0A%09at+oracle.apps.fnd.framework.server.OAExceptionUtils.processAOLJErrorStack(OAExceptionUtils.java%3A974)%0A%09at+oracle.apps.fnd.framework.OACommonUtils.processAOLJErrorStack(OACommonUtils.java%3A867)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.validateUser(OAPageBean.java%3A4762)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A713)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A515)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A436)%0A%09at+_oa__html._OA._jspService(_OA.java%3A84)%0A%09at+oracle.jsp.runtime.HttpJsp.service(HttpJsp.java%3A119)%0A%09at+oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java%3A417)%0A%09at+oracle.jsp.JspServlet.doDispatch(JspServlet.java%3A267)%0A%09at+oracle.jsp.JspServlet.internalService(JspServlet.java%3A186)%0A%09at+oracle.jsp.JspServlet.service(JspServlet.java%3A156)%0A%09at+javax.servlet.http.HttpServlet.service(HttpServlet.java%3A588)%0A%09at+org.apache.jserv.JServConnection.processRequest(JServConnection.java%3A456)%0A%09at+org.apache.jserv.JServConnection.run(JServConnection.java%3A294)%0A%09at+java.lang.Thread.run(Thread.java%3A662)%0A%23%23+Detail+0+%23%23%0Aoracle.apps.fnd.framework.OAException%3A+Application%3A+FND%2C+Message+Name%3A+FND_GENERIC_MESSAGE.+Tokens%3A+MESSAGE+%3D+java.sql.SQLException%3A+ORA-20001%3A+Oracle+error+-6508%3A+ORA-06508%3A+PL%2FSQL%3A+could+not+find+program+unit+being+called+has+been+detected+in+fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG.%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+27%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+232%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+2109%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+369%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+538%0AORA-06512%3A+at+%22APPS.FND_AOLJ_UTIL%22%2C+line+217%0AORA-06512%3A+at+line+1%0A%3B+%0A%09at+oracle.apps.fnd.framework.OAException.wrapperException(OAException.java%3A891)%0A%09at+oracle.apps.fnd.framework.OAException.wrapperException(OAException.java%3A865)%0A%09at+oracle.apps.fnd.framework.server.OAExceptionUtils.processAOLJErrorStack(OAExceptionUtils.java%3A980)%0A%09at+oracle.apps.fnd.framework.OACommonUtils.processAOLJErrorStack(OACommonUtils.java%3A867)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.validateUser(OAPageBean.java%3A4762)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A713)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A515)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A436)%0A%09at+_oa__html._OA._jspService(_OA.java%3A84)%0A%09at+oracle.jsp.runtime.HttpJsp.service(HttpJsp.java%3A119)%0A%09at+oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java%3A417)%0A%09at+oracle.jsp.JspServlet.doDispatch(JspServlet.java%3A267)%0A%09at+oracle.jsp.JspServlet.internalService(JspServlet.java%3A186)%0A%09at+oracle.jsp.JspServlet.service(JspServlet.java%3A156)%0A%09at+javax.servlet.http.HttpServlet.service(HttpServlet.java%3A588)%0A%09at+org.apache.jserv.JServConnection.processRequest(JServConnection.java%3A456)%0A%09at+org.apache.jserv.JServConnection.run(JServConnection.java%3A294)%0A%09at+java.lang.Thread.run(Thread.java%3A662)%0A%23%23+Detail+0+%23%23%0Ajava.sql.SQLException%3A+ORA-20001%3A+Oracle+error+-6508%3A+ORA-06508%3A+PL%2FSQL%3A+could+not+find+program+unit+being+called+has+been+detected+in+fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG.%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+27%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+232%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+2109%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+369%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+538%0AORA-06512%3A+at+%22APPS.FND_AOLJ_UTIL%22%2C+line+217%0AORA-06512%3A+at+line+1%0A%0A%09at+oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java%3A134)%0A%09at+oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java%3A289)%0A%09at+oracle.jdbc.ttc7.Oall7.receive(Oall7.java%3A590)%0A%09at+oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java%3A1973)%0A%09at+oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java%3A1119)%0A%09at+oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java%3A2191)%0A%09at+oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java%3A2064)%0A%09at+oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java%3A2989)%0A%09at+oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java%3A658)%0A%09at+oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java%3A736)%0A%09at+oracle.apps.fnd.common.WebAppsContext.createSession(WebAppsContext.java%3A2938)%0A%09at+oracle.apps.fnd.common.WebAppsContext.createSession(WebAppsContext.java%3A2825)%0A%09at+oracle.apps.fnd.common.WebAppsContext.createSession(WebAppsContext.java%3A2754)%0A%09at+oracle.apps.fnd.common.WebAppsContext.createAnonymousSession(WebAppsContext.java%3A5111)%0A%09at+oracle.apps.fnd.common.WebRequestUtil.setGuestSession(WebRequestUtil.java%3A1510)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.validateUser(OAPageBean.java%3A4760)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A713)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A515)%0A%09at+oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java%3A436)%0A%09at+_oa__html._OA._jspService(_OA.java%3A84)%0A%09at+oracle.jsp.runtime.HttpJsp.service(HttpJsp.java%3A119)%0A%09at+oracle.jsp.app.JspApplication.dispatchRequest(JspApplication.java%3A417)%0A%09at+oracle.jsp.JspServlet.doDispatch(JspServlet.java%3A267)%0A%09at+oracle.jsp.JspServlet.internalService(JspServlet.java%3A186)%0A%09at+oracle.jsp.JspServlet.service(JspServlet.java%3A156)%0A%09at+javax.servlet.http.HttpServlet.service(HttpServlet.java%3A588)%0A%09at+org.apache.jserv.JServConnection.processRequest(JServConnection.java%3A456)%0A%09at+org.apache.jserv.JServConnection.run(JServConnection.java%3A294)%0A%09at+java.lang.Thread.run(Thread.java%3A662)%0Aoracle.apps.fnd.framework.OAException%3A+Application%3A+FND%2C+Message+Name%3A+FND_GENERIC_MESSAGE.+Tokens%3A+MESSAGE+%3D+java.sql.SQLException%3A+ORA-20001%3A+Oracle+error+-6508%3A+ORA-06508%3A+PL%2FSQL%3A+could+not+find+program+unit+being+called+has+been+detected+in+fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG.%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+27%0AORA-06512%3A+at+%22APPS.MO_UTILS%22%2C+line+232%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+2109%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+369%0AORA-06512%3A+at+%22APPS.FND_SESSION_MANAGEMENT%22%2C+line+538%0AORA-06512%3A+at+%22APPS.FND_AOLJ_UTIL%22%2C+line+217%0AORA-06512%3A+at+line+1%0A%3B+%0A%09at+oracle.apps.fnd.framework.OAException.wrapperException(OAExcept

JSP Error

--------------------------------------------------------------------------------
Exception:
java.lang.NullPointerException

I concentrated on the first part of the error message:
ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512: at "APPS.MO_UTILS", line 27 ORA-06512: at "APPS.MO_UTILS", line 232

I tried calling this package and function by executing this query from sqlplus:  
SQL> conn apps/apps
SQL> select mo_utils.get_multi_org_flag() from dual;

select mo_utils.get_multi_org_flag() from dual
*

ERROR at line 1:

ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit
being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG.
ORA-06512: at "APPS.MO_UTILS", line 27
ORA-06512: at "APPS.MO_UTILS", line 232

I checked the code in the function mo_utils.get_multi_org_flag by:

select line,text
from dba_source
where name='MO_UTILS'
and line between 220 and 240;

LINE   TEXT
--------------------------------------------------------------------------------
FUNCTION Get_Multi_Org_Flag
RETURN VARCHAR2
IS

BEGIN
227
228
RETURN mo_global.is_multi_org_enabled;
229
230
EXCEPTION
231
WHEN OTHERS THEN
232
Generic_Error( 'fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG'
233
, sqlcode
234
, sqlerrm);
235
236
END Get_Multi_Org_Flag;

So the only line of code in this function is on line 237:

RETURN mo_global.is_multi_org_enabled;


I tried to execute this from sqlplus:

SQL> select mo_global.is_multi_org_enabled() from dual;

select mo_global.is_multi_org_enabled() from dual
*
ERROR at line 1:

ORA-04063: package body "APPS.MO_GLOBAL" has errors

Then I tried compiling the package body:

SQL> alter package APPS.MO_GLOBAL compile body;
Warning: Package Body altered with compilation errors.


SQL> show errors
Errors for PACKAGE BODY APPS.MO_GLOBAL:


LINE/COL ERROR
-------- -----------------------------------------------------------------
167/27 PLS-00201: identifier 'V$SQLAREA' must be declared
167/27 PL/SQL: Item ignored
302/4 PL/SQL: Statement ignored
302/53 PLS-00320: the declaration of the type of this expression is incomplete or malformed

I checked line 167:

LINE
----------
TEXT
--------------------------------------------------------------------------------
167
l_action_name v$sqlarea.action%TYPE;

That means, it is unable to query v$sqlarea.

So I logged in as sys and

SQL> grant select on v$sqlarea to apps;

grant select on v$sqlarea to apps
*
ERROR at line 1:
ORA-02030: can only select from fixed tables/views

Ok, that was a mistake, the correct grant is:

SQL> grant select on v_$sqlarea to apps;


Grant succeeded.

Compiled the package MO_GLOBAL:
SQL> alter package APPS.MO_GLOBAL compile body;

Package body altered.

Tested both the sqls which were previously erroring out:

SQL> conn apps/apps

Connected.

SQL> select mo_global.is_multi_org_enabled() from dual;

MO_GLOBAL.IS_MULTI_ORG_ENABLED()
--------------------------------------------------------------------------------
Y

SQL> select mo_utils.get_multi_org_flag() from dual;

MO_UTILS.GET_MULTI_ORG_FLAG()
--------------------------------------------------------------------------------
Y

We tried with IE and the same error still came.

So we tried with a new Firefox session, and we were able to login without issues.  IE also worked after clearing Temporary Internet Files.

That was an interesting troubleshoot.  I wonder, how the grant was missing in the first place.

---

java.sql.SQLException: ORA-01017: invalid username/password; logon denied for glogdba after cloning OTM

January 24, 2013, 6:48 pm

≫ Next: ORA-06502: PL/SQL: numeric or value error: hex to raw conversion error ORA-06512: at "APPS.WF_EVENT", line 3600

≪ Previous: Oracle error 20001: java.sql.SQLException: ORA-20001: Oracle error -6508: ORA-06508: PL/SQL: could not find program unit being called has been detected in fnd.plsql.MO_UTILS.GET_MULTI_ORG_FLAG. ORA-06512

$

0

0

After cloning OTM database and instance, when we started the services and tried accessing

Configuration and Administration > Cluster Management > Scalability Overview, we got this error in glog.exception.log

java.sql.SQLException: ORA-01017: invalid username/password; logon denied

I was able to diagnose that this was coming for glogdba user by switching on auditing on the database.

The encrypted password was correct for glogdba in glog.properties.

After a lot of R&D we found out that this was occurring because the encrypted password of glogdba user in data_source table was incorrect.


It had the encrypted password of the glogdba user in production. We reset the glogdba password to default glogdba after clone.

Doing this update statement and bouncing apache and weblogic did the trick:

update data_source set oracle_password='Z2xvZ2RiYQ=='
where data_source_gid='UNPOOLED_DBA';
commit;

ORA-06502: PL/SQL: numeric or value error: hex to raw conversion error ORA-06512: at "APPS.WF_EVENT", line 3600

February 4, 2013, 6:34 pm

≫ Next: Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)

≪ Previous: java.sql.SQLException: ORA-01017: invalid username/password; logon denied for glogdba after cloning OTM

$

0

0

Akhilesh pinged me today and told me that autoconfig was not working on a newly upgraded R12 instance.  It was failing with these errors:

AutoConfig could not successfully execute the following scripts:

   Directory: /r12ascp/erpapp/10.1.3/perl/bin/perl -I /r12ascp/erpapp/10.1.3/perl/lib/5.8.3 -I /r12ascp/erpapp/10.1.3/perl/lib/site_perl/5.8.3 -I /usnapss

1/erpapp/appl/au/12.0.0/perl -I /r12ascp/erpapp/10.1.3/Apache/Apache/mod_perl/lib/site_perl/5.8.3/sun4-solaris-thread-multi /r12ascp/erpapp/inst/apps/usnaps

s1_r12ascp/admin/scripts/adexecsql.pl sqlfile=/r12ascp/erpapp/inst/apps/r12ascp_r12ascp/admin/install

     afwebprf.sql            INSTE8_PRF         1

     amscmprf.sql            INSTE8_PRF         1

     amswebprf.sql           INSTE8_PRF         1

     cncmprf.sql             INSTE8_PRF         1

     csfadmprf.sql           INSTE8_PRF         1

     oksfrmprf.sql           INSTE8_PRF         1

   Directory: /r12ascp/erpapp/inst/apps/r12ascp_r12ascp/admin/install

     ibywebprf.sh            INSTE8_PRF         1

If we tried running any of the above scripts, they failed with this error:

SQLPLUS Executable : /r12ascp/erpapp/10.1.2/bin/sqlplus


SQL*Plus: Release 10.1.0.5.0 - Production on Mon Feb 4 12:04:27 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Enter value for 1: Enter value for 2: Enter value for 3: Connected.
declare
*
ERROR at line 1:
ORA-06502: PL/SQL: numeric or value error: hex to raw conversion error
ORA-06512: at "APPS.WF_EVENT", line 3600
ORA-06512: at "APPS.FND_WF_ENGINE", line 1025
ORA-06512: at "APPS.FND_PROFILE", line 3064
ORA-06512: at "APPS.FND_PROFILE", line 3514
ORA-06512: at "APPS.ADX_PRF_PKG", line 203
ORA-06512: at line 167
ORA-06512: at line 217

So I checked the code on line 3600 for wf_event package:

select line,text

from dba_source

where name='WF_EVENT'

and line between 3550 and 3610;

 3600
 wf_event.local_system_guid := hextoraw(wf_core.translate('WF_SYSTEM_GUID'));

I did another code search:

select line,text

from dba_source

where text like '%WF_SYSTEM_GUID%';

This was the result of the query:

   select text into l_result from wf_resources where name='WF_SYSTEM_GUID' and

language='US';

SQL> select text from wf_resources where name='WF_SYSTEM_GUID' and

 2  language='US';

TEXT

--------------------------------------------------------------------------------

CAFECAFE-0013-0001-0029-ABCDEFABCDEF

That was weird as that is the GUID used by Oracle JRE plugin 1.3.1.29.

I checked the actual GUID:

SQL> select guid from wf_systems;

GUID

--------------------------------

D37180AE23A4479AE04400212846C6CE

Updated the correct GUID:

SQL> update wf_resources

 2  set text='D37180AE23A4479AE04400212846C6CE'

 3  where name='WF_SYSTEM_GUID' and

 4  language='US';

1 row updated.

SQL> commit;

Re-ran $INST_TOP/admin/install/adwebprf.sql apps appspassword ORACLE_SID

New error this time:

SQL> @afwebprf.sql apps appspassword r12ascp
Connected.
declare
*
ERROR at line 1:
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at "APPS.WF_EVENT", line 3602
ORA-06512: at "APPS.FND_WF_ENGINE", line 1025
ORA-06512: at "APPS.FND_PROFILE", line 3064
ORA-06512: at "APPS.FND_PROFILE", line 3514
ORA-06512: at "APPS.ADX_PRF_PKG", line 203
ORA-06512: at line 167
ORA-06512: at line 217

Checked the code again:


    3602
 wf_event.local_system_status := wf_core.translate('WF_SYSTEM_STATUS');

SQL> select text from wf_resources where name='WF_SYSTEM_STATUS' and

 2  language='US';

TEXT

--------------------------------------------------------------------------------

CAFECAFE-0013-0001-0029-ABCDEFABCDEF

So I checked with query:

select count(*) from wf_resources where 

text='CAFECAFE-0013-0001-0029-ABCDEFABCDEF';

1207 rows

select count(*) from wf_resources;

1775 rows

So I inquired about an existing good instance.  No R12 instance was available, so I copied from an 11i instance with sqlplus COPY command:

First put the tnsnames.ora entry for the good instance in the DB $TNS_ADMIN/tnsnames.ora file
Then used the sqlplus copy command, which is the fastest way to copy data between two oracle databases:

SQL> copy from apps/****@11iascp to apps/****@r12ascp create wf_resources_d1 using select * from wf_resources

Array fetch/bind size is 15. (arraysize is 15)
Will commit when done. (copycommit is 0)
Maximum long size is 80. (long is 80)
Table WF_RESOURCES_11i created.

  1717 rows selected from apps@usnapsd1.
  1717 rows inserted into WF_RESOURCES_11i.
  1717 rows committed into WF_RESOURCES_11i at apps@r12ascp.


Since there was a difference of 58 rows, I first copied those rows in a different table:

create table wf_resources_more as
( select * from wf_resources
where (name,type) in
(
select name,type from wf_resources
minus
select name,type from wf_resources_11i
);

Which created table wf_resources_more with 58 rows.

Then I delete those 58 rows from the wf_resources table:
delete wf_resources

where (name,type) in
(
select name,type from wf_resources
minus
select name,type from wf_resources_11i
);

Then I updated the table with 11i values, through a correlated update statement:

update wf_resources a

set a.text=(select text from wf_resources_11i b

where a.type||a.name = b.type||b.name)

That updated the 1717 common rows with correct values.

Then I inserted the extra 58 rows new in R12

insert into wf_resources (select * from wf_resources_more);

commit;

I ran adwebprf.sql again and it gave a new error:

SQL> @afwebprf.sql apps appspass r12ascp

Connected.

declare

*

ERROR at line 1:

ORA-01403: no data found

ORA-06512: at "APPS.WF_EVENT", line 3604

ORA-06512: at "APPS.FND_WF_ENGINE", line 1025

ORA-06512: at "APPS.FND_PROFILE", line 3064

ORA-06512: at "APPS.FND_PROFILE", line 3514

ORA-06512: at "APPS.ADX_PRF_PKG", line 203

ORA-06512: at line 167

ORA-06512: at line 217

Again I checked code on line 3604 of WF_EVENT:

 select name into wf_event.local_system_name

     3605

 from wf_systems

     3606

 where guid = wf_event.local_system_guid;

Then I realized that after the correlated update, the wf_resources table now had the GUID of 11i system:

SQL> select name,text

 2  from wf_resources

 3  where name='WF_SYSTEM_GUID';

NAME

------------------------------

TEXT

--------------------------------------------------------------------------------

WF_SYSTEM_GUID

CA43363B52162863E04400212846C6CE

So I updated it again:

I checked the actual GUID:

SQL> select guid from wf_systems;

GUID

--------------------------------

D37180AE23A4479AE04400212846C6CE

Updated the correct GUID:

SQL> update wf_resources

 2  set text='D37180AE23A4479AE04400212846C6CE'

 3  where name='WF_SYSTEM_GUID' and

 4  language='US';

1 row updated.

SQL> commit;

Re-ran $INST_TOP/admin/install/adwebprf.sql apps appspassword ORACLE_SID

SQL> @afwebprf.sql apps appsapss1 usnapss1

Connected.

[ APPS_WEB_AGENT ]

Application Id : 0

Profile Value  : http://usnapss1.tsg.ge.com:8004/pls/usnapss1

Level Name: SITE

INFO           : Updated/created profile option value.

.

Deleted : 0

[ APPS_SERVLET_AGENT ]

Application Id : 0

Profile Value  : http://usnapss1.tsg.ge.com:8004/OA_HTML

Level Name: SITE

INFO           : Updated/created profile option value.

.

Deleted : 0

[ APPS_JSP_AGENT ]

Application Id : 0

Profile Value  : http://usnapss1.tsg.ge.com:8004

Level Name: SITE

INFO           : Updated/created profile option value.

.

Deleted : 0

[ APPS_FRAMEWORK_AGENT ]

Application Id : 0

Profile Value  : http://usnapss1.tsg.ge.com:8004

Level Name: SITE

INFO           : Updated/created profile option value.

.

Deleted : 0

[ WF_MAIL_WEB_AGENT ]

Application Id : 0

Profile Value  :

Level Name: SITE

INFO           : Updated/created profile option value.

.

.

.

It succeeded.

I told Akhilesh about the result and asked him to run autoconfig.  We tailed the log at

$INST_TOP/admin/log/02042015/adconfig.log

Adconfig succeeded without errors this time:

AutoConfig is exiting with status 0

AutoConfig execution completed on Mon Feb  4 20:17:58 2013

Time taken for AutoConfig execution to complete : 2 mins  38 secs

I was overjoyed, and so was Akhilesh.  We still had 53 rows which had the value of text as:

CAFECAFE-0013-0001-0029-ABCDEFABCDEF

select count(*) from wf_resources

where text='CAFECAFE-0013-0001-0029-ABCDEFABCDEF';

53 rows.

We would investigate further, how this had happened.  I suspect human error.  Human error makes troubleshooting more interesting, just like human error makes chess more interesting.  If it was a computer playing chess, it would not make any mistakes and win the game.  It is the human errors which make chess and troubleshooting, an interesting pursuit.

Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)

March 2, 2013, 8:13 pm

≫ Next: 550 5.7.1 Relaying denied. IP name lookup failed

≪ Previous: ORA-06502: PL/SQL: numeric or value error: hex to raw conversion error ORA-06512: at "APPS.WF_EVENT", line 3600

$

0

0

In OTM FTI you may get an error like this:

OTM > Transportation Intelligence > Dashboard

Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)


Error Details

Error Codes: U9KP7Q94    

From OTM install guide In the $OBIEE_DATA/web/config/ directory, edit the instanceconfig.xml as follows:

a. Change the CatalogPath to point the new “aa” directory that was created when you extracted the aa_webcat.zip file. The template shows the following line.

vi instanceconfig.xml


Search for string CatalogPath and you'll find this
$OBIEE_DATA/web/catalog/samplesales
Edit this line so that it looks like this:
$OBIEE_DATA/web/catalog/aa


Note: Do not put in the $OBIEE_DATA only add the reference to the “aa” directory.

The entry may look like this:

/opt/oraclebi/oraclebidata/web/catalog/aa   Save.
Restart services.

oc4j -start
./run-sa.sh start64
./run-saw.sh start64
./run-sch.sh start64

550 5.7.1 Relaying denied. IP name lookup failed

April 15, 2013, 8:12 pm

≫ Next: R12.2.2 is now available on edelivery.oracle.com

≪ Previous: Path not found (/shared/FTI Reports/_Portal/Main Dashboard/dashboard layout)

$

0

0

Recently we configured a BigIP virtual for SMTP called smtp.justanexample.com

Whenever I tried sending mail through this it would complain:

# telnet smtp.justanexample.com 25


Trying 192.168.10.201...

Connected to smtp.justanexample.com
Escape character is '^]'.

220 smtp1.justanexample.com ESMTP Sendmail 8.13.8/8.13.8; Mon, 15 Apr 2013 22:34:54 -0400

mail from:vikram.das@justanexample.com

250 2.1.0 vikram.das@justanexample.com... Sender ok

rcpt to:vikram.das@ge.com

550 5.7.1 vikram.das@justanexample.com... Relaying denied. IP name lookup failed [192.168.10.2]

Doing an nslookup on this name failed.  If I did https://192.168.10.2 it would go to the BigIP user interface.  This IP address was somewhere related to the BigIP device, and did not have a hostname in DNS.  I have sent a mail to the network team for further investigation about this IP.

The usual practice for allowing relay from certain hosts is to add those hosts to /etc/mail/local-host-names.  However since this IP was not present in DNS, I added the following line to /etc/mail/access on all smtp nodes :

Connect:192.168.10.2             RELAY

and bounced sendmail on Linux

/etc/init.d/sendmail restart

The Relaying Denied is not coming anymore.

R12.2.2 is now available on edelivery.oracle.com

September 19, 2013, 5:51 am

≫ Next: adpreclone.pl fails with segmentation fault, bus error, out of memory

≪ Previous: 550 5.7.1 Relaying denied. IP name lookup failed

$

0

0

At last, R12.2.2 is now available on edelivery.oracle.com

adpreclone.pl fails with segmentation fault, bus error, out of memory

November 20, 2013, 9:51 pm

≫ Next: R12.2 Apache and Weblogic log file locations

≪ Previous: R12.2.2 is now available on edelivery.oracle.com

$

0

0

Manoj pinged me about this issue last night.  Whenever he was trying to do adpreclone.pl dbTier or AppsTier, when he entered the apps password, the process would crash with either segmentation fault, bus error or out of memory.

I tried executing top and it was also ending up with segmentation fault some of the time.  At the times it would start, it would give a flashing

I checked the server memory and it had 73 GB free memory.

Eventually  I opened up the adpreclone.pl script in vi to study what it was doing.  At one place, I noticed that it was looking for the environment variable $USER.  When I checked, on the server, echo $USER, did not return anything.  I checked on other working servers and echo $USER returned the unix applmgr username.

So I manually set

USER=applmgr
export USER
echo $USER

perl adpreclone.pl appsTier

Entered apps password, and it worked fine.

This was a new server. Seems like unix team missed something during this build.  I tried looking up /etc/profile, but wasn't able to find out why the environment variable USER was not set.  I'll update this post, when I discover the reason.

R12.2 Apache and Weblogic log file locations

December 20, 2013, 9:25 am

≫ Next: R12.2 Failure of server APACHE bridge

≪ Previous: adpreclone.pl fails with segmentation fault, bus error, out of memory

$

0

0

Apache Logs

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_*/*log

OPMN Log
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*

Weblogic Logs

$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*
$EBS_DOMAIN_HOME/servers/forms*/logs/*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*
$EBS_DOMAIN_HOME/sysman/log/*

Source: Oracle® E-Business Suite
Maintenance Guide
Release 12.2

---

R12.2 Failure of server APACHE bridge

December 20, 2013, 7:19 pm

≫ Next: weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

≪ Previous: R12.2 Apache and Weblogic log file locations

$

0

0

Failure of server APACHE bridge:

---

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

If there is an issue with the oacore managed server, meaning apache is not able to connect to the oacore managed server, you'll see this error.

It is possible that the managed server oacore is not yet started, and you try to access the EBS URL.
It is possible that the managed server for oacore is not able to start up because of a port conflict.

weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

January 7, 2014, 6:47 pm

≫ Next: R12.2 upgrade - icxr12pd.sql takes a very long time

≪ Previous: R12.2 Failure of server APACHE bridge

$

0

0

After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
        at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
        at weblogic.store.io.file.Heap.open(Heap.java:325)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

A search on support.oracle.com for keywords
resulted in this hit:

E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)

WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.

Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
  
Specifically for EBS R12.2 I did this:

cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
 cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT

adadminsrvctl.sh start

After keying in apps, system and weblogic passwords, it finally started up.

Have to see if this is a recurrent workaround or fixes the issue for good.

R12.2 upgrade - icxr12pd.sql takes a very long time

February 8, 2014, 3:19 pm

≫ Next: Product Restrictions during R12.2 online patching

≪ Previous: weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

$

0

0

During an R12.2 upgrade, the worker icxr12pd.sql takes very long time.  This is document in My Oracle Support Article ICXR12PD.Sql Runs Slowly During Upgrade to 12.1.1 (Doc ID 864898.1)
 
Cause: The pre-upgrade steps for iProcurement were not implemented as documented in the upgrade guide
Though this article talks about R12.1.1, it also applies to R12.2.  If you read the 11i to R12.2 upgrade guide ( http://docs.oracle.com/cd/V39571_01/current/acrobat/122upg11i.pdf ). it has specifically mentioned on page 285:

iProcurement
Completing these tasks could substantially reduce the downtime required for your
upgrade.
The Catalog Data Pre-upgrade Process
This pre-upgrade process is strongly recommended if you are upgrading iProcurement
from 11.5.9 or 11.5.10 to Release 12. It pre-processes bulk-loaded content to reduce the
actual time required for the upgrade and to ensure the upgrade process runs smoothly.
You can run it multiple times. If exceptions are found, then make corrections and re-run
the program until no exceptions are noted. Running this program does not require users
to log off the system. Note: Exceptions noted and not fixed before the upgrade will not
be available in the iProcurement Catalog. Specifically, this program shortens the time it
takes to run these upgrade scripts:
• -icxiftug.sql: updates POR_TEMPLATE_INFO
• -icxr12in.sql: general setup upgrade script needed for iProcurement
• -poxujpoh.sql:updates PO_HEADERS_ALL (for example, create_language)
• -poxukpol.sql: updates PO_LINES_ALL (for example, ip_category_id)
• -poxukrt.sql: updates PO_REQEXPRESS_LINES_ALL (for example, ip_catetory_id)
• -icxr12ug.sql: migrates iProcurement data model to PO
• -icxr12rt.sql: populates the requisition templates in iProcurement intermedia index
tables
• -icxr12pd.sql: populates the Purchasing documents in iProcurement intermedia
index tables: blanket purchase agreements (BPAs), global blanket agreements
(GBPAs), and quotes
• -icxr12mi.sql: populates the master items in iProcurement intermedia index tables
• -poxukfi.sql: purchasing script to approve the GBPAs created during iProcurement
migration
• -poxukat.sql: purchasing script to update the attachments
• -icxr12fi.sql: final upgrade script, which upgrades favorite lists, purges the BPAs
and GBPAs that are not approved, and creates the intermedia index


Make sure that you complete the tasks described in the upgrade guide, during the pre-upgrade phase, much before actual cutover downtime starts.

If you haven't done this and proceed to upgrade, it is going to take a long time for all the workers which are going to run icxr12pd.sql.  Based on my analysis of v$sql_longops, this is the insert query most fired during this worker's execution:

INSERT INTO AP_INV_DISTS_TARGET T1 (BATCH_ID, INVOICE_ID, INVOICE_LINE_NUMBER,
INVOICE_DISTRIBUTION_ID, DISTRIBUTION_LINE_NUMBER, LINE_TYPE_LOOKUP_CODE,
DESCRIPTION, DIST_MATCH_TYPE, ORG_ID, DIST_CODE_COMBINATION_ID, ACCOUNTING_DATE,
PERIOD_NAME, ACCRUAL_POSTED_FLAG, CASH_POSTED_FLAG, AMOUNT_TO_POST,
BASE_AMOUNT_TO_POST, POSTED_FLAG, ACCOUNTING_EVENT_ID, SET_OF_BOOKS_ID, AMOUNT,
BASE_AMOUNT, EXCHANGE_DATE, ROUNDING_AMT, QUANTITY_VARIANCE, BASE_QUANTITY_VARIANCE,
MATCH_STATUS_FLAG, ENCUMBERED_FLAG, PACKET_ID, USSGL_TRANSACTION_CODE,
USSGL_TRX_CODE_CONTEXT, REVERSAL_FLAG, PARENT_REVERSAL_ID, CANCELLED_FLAG, INCOME_TAX_REGION,
TYPE_1099, STAT_AMOUNT, CHARGE_APPLICABLE_TO_DIST_ID, PREPAY_AMOUNT_REMAINING,
PREPAY_DISTRIBUTION_ID, PREPAY_TAX_PARENT_ID, PARENT_INVOICE_ID, CORRECTED_QUANTITY,
PO_DISTRIBUTION_ID, RCV_TRANSACTION_ID, UNIT_PRICE, MATCHED_UOM_LOOKUP_CODE, QUANTITY_INVOICED,
FINAL_MATCH_FLAG, RELATED_ID, ASSETS_ADDITION_FLAG, ASSETS_TRACKING_FLAG, PROJECT_ID, TASK_ID, EXPENDITURE_T

In our test environment, we have 25 million + rows in ap_inv_dists_target.  We have already waited for 8 hours for all the workers to complete.  Here's the status of the queries:

set pages 300 lines 300
col "Username" for a12
col "QC/Slave" for A8
col "Slaveset" for A8
col "Slave INST" for A9
col "QC SID" for A6
col "QC INST" for A6
col "operation_name" for A30
col "target" for A30

select
decode(px.qcinst_id,NULL,username,
' - '||lower(substr(pp.SERVER_NAME,
length(pp.SERVER_NAME)-4,4) ) )"Username",
decode(px.qcinst_id,NULL, 'QC', '(Slave)') "QC/Slave" ,
to_char( px.server_set) "SlaveSet",
to_char(px.inst_id) "Slave INST",
substr(opname,1,30) operation_name,
substr(target,1,30) target,
sofar,
totalwork,
units,
start_time,
timestamp,
decode(px.qcinst_id, NULL ,to_char(s.sid) ,px.qcsid) "QC SID",
to_char(px.qcinst_id) "QC INST"
from gv$px_session px,
gv$px_process pp,
gv$session_longops s
where px.sid=s.sid
and px.serial#=s.serial#
and px.inst_id = s.inst_id
and px.sid = pp.sid (+)
and px.serial#=pp.serial#(+)
order by
decode(px.QCINST_ID, NULL, px.INST_ID, px.QCINST_ID),
px.QCSID,
decode(px.SERVER_GROUP, NULL, 0, px.SERVER_GROUP),
px.SERVER_SET,
px.INST_ID
/

Username QC/Slave SlaveSet Slave INS OPERATION_NAME TARGET SOFAR TOTALWORK UNITS START_TIME TIMESTAMP QC SID QC INS
------------ -------- -------- --------- ------------------------------ ------------------------------ ---------- ---------- -------------------------------- --------------- --------------- ------ ------
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Gather Table's Index Statistic 0 0 Indexes 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335

7 rows selected.

Product Restrictions during R12.2 online patching

February 20, 2014, 9:34 pm

≫ Next: ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table. ORA-06512: at "APPS.AD_ZD_ADOP", line 523

≪ Previous: R12.2 upgrade - icxr12pd.sql takes a very long time

$

0

0

This information is from page 75 of Oracle® E-Business Suite Maintenance Guide Release 12.2 Part No. E22954-13, published in July 2013 available here  http://docs.oracle.com/cd/V39571_01/current/acrobat/122ebsmt.zip

For patches that have manual steps, the patch readme file instructs you to use Oracle Patch Application Assistant (PAA) to create customized instructions for your system. PAA consolidates and displays only the relevant manual steps for all the patches you want to apply, including steps that you have completed. It also automatically merges the contents of individual patch readme files for a merged patch.

Create Checklist of Product Functionality Disabled in Online Patching Cycle:
During an online patching cycle, the following product restrictions will apply. Before you commence patching, you should therefore ensure there will be no requirement for any these actions or features until the cycle is complete.

• Payroll
• Users will not be able to define Fast Formulas or use the Fast Formula Assistant.
• Users will not be able to perform dynamic trigger maintenance.
• Users will not be able to create, update, or delete US Cities.
• Data Pump meta-mapper generator will be disabled.
• The Japanese Balance Dimensions concurrent program will be deferred to after the cutover phase is complete.
• Pension Calculation Setup cannot be used.
• US localization earnings and deduction setup cannot be used.
• Tax Withholding Rules Setup cannot be used.
• Wage Attachment Earnings Rules Setup cannot be used.
• Garnishment Rules Setup cannot be used.
• Quick Paint Reports cannot be used.
• Quantum Program Update Installer execution is unavailable.
• Order Management:
• Creation of a new Defaulting Condition in the Attribute Defaulting Rules form is disabled, unless the same seeded condition already exists for a given attribute.
• Warehouse Management:
• WMS Rule creation is restricted.
• Inventory:
• Concurrent program "Generate Stock Locator Flexfield Definition for Mobile Transactions" will be disabled.
• Public Sector Financials International:
• Users will not be able to run the following concurrent programs:
• Subledger Security: Apply Security
• Subledger Security: Import/Export Data Fix
• Subledger Accounting:
• Users will not be able to Validate the Application Accounting definitions.
• Accounts Receivable:
• Users will not be able to create new Transaction Sources.
• Incentive Compensation:
• Transaction collection process for new mappings will not be available and any changed mapping will continue to use previous mapping rules.
• Users will not be able to run the "Synchronize Classification Rulesets" program.
• Users will not be able to use the "Formula Generation" feature.
• Users will not be able to specify new formulas or changes to compensation rules.
• Oracle Demand Planning:
• Demand plans will not be available for users.

ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table. ORA-06512: at "APPS.AD_ZD_ADOP", line 523

February 26, 2014, 7:37 pm

≫ Next: R12.2 Dictionary Corrupted error during adop phase=prepare

≪ Previous: Product Restrictions during R12.2 online patching

$

0

0

I reported this error to Oracle on an SR:

The adop phase=prepare fails before generating any log files. Whatever we get on the screen is what we have. I'll paste it for you again:

$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Please wait. Validating credentials...


RUN file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml
PATCH file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml


************* Start of session *************
version: 12.2.0
started at: Sun Feb 23 2014 20:55:38

APPL_TOP is set to $APPL_TOP
[STATEMENT] Using 48 workers (Default: 48, Recommended maximum limit: 696)
[EVENT] [START 2014/02/23 20:55:48] Verify SSH
Logfile location $INST_TOP/logs/appl/rgf/TXK/verifyssh.log
xml output = $INST_TOP/logs/appl/rgf/TXK/out.xml
[EVENT] [END 2014/02/23 20:55:54] Verify SSH
[ERROR] Failed to execute sql statement :
begin
AD_ZD_ADOP.LOCK_SESSIONS_TABLE('server1',60,0);
end;

[ERROR] SQLPLUS error: buffer=
SQL*Plus: Release 10.1.0.5.0 - Production on Sun Feb 23 20:55:59 2014
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> SQL> Connected.
SQL> begin
*
ERROR at line 1:
ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table.
ORA-06512: at "APPS.AD_ZD_ADOP", line 523
ORA-06512: at line 2
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

[UNEXPECTED]Unable to acquire lock on sessions table

adop exiting with status = 2 (Fail)

Oracle asked me to confirm whether I had run adconfig on both run and patch environments. I had forgotten to run adconfig on patch environment.  After running adconfig on the patch environment, I retried and it worked.

R12.2 Dictionary Corrupted error during adop phase=prepare

March 10, 2014, 10:07 pm

≫ Next: R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade

≪ Previous: ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table. ORA-06512: at "APPS.AD_ZD_ADOP", line 523

$

0

0

Do not follow the document 1576086.1.  I followed it and increased my invalid counts from 1000 to 100,000 and wasted 10 hours to get back to where I started.  Instead follow these instructions:

Reference :Ora-04023 Reported while Accessing Valid Objects (Doc ID 1610514.1)

Dictionary Corrupted:

Details are as follows:

OWBSYS          HAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          WB_RTI_OBJECT_CATALOG                          OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          UAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

PUBLIC          WB_OLAP_AW_PRECOMPUTE                          OWBSYS          WB_OLAP_AW_PRECOMPUTE                          TS mismatch: 23-SEP-10 05:17:34 26-NOV-12 18:48:25

OWBSYS          ALL_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          RAB_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

OWBSYS          HAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

please contact Oracle Support and request them to open a bug against Oracle Application Install(166), Component Online Patching(OP)

        [ERROR]     Prepare phase completed with errors/warnings. Please check logfiles

        Log file: /erpodpa1/erpapp/fs_ne/EBSapps/log/adop/3/adop_20140306_005850.log

select owner,object_type,object_name

from dba_objects

where object_name in

(

'HAB_RT_EXEC_PROC_RUN_METRICS',

'WB_RTI_OBJECT_CATALOG',

'UAB_RT_EXEC_PROCESS',

'UAB_RT_EXEC_PROCESS_RUNS',

'UAB_RT_EXEC_PROC_RUN_COUNTS',

'UAB_RT_EXEC_PROC_RUN_METRICS',

'WB_OLAP_AW_PRECOMPUTE',

'ALL_RT_INSTALLATIONS',

'RAB_RT_INSTALLATIONS',

'HAB_RT_EXEC_PROCESS',

'HAB_RT_EXEC_PROCESS_RUNS',

'HAB_RT_EXEC_PROC_RUN_COUNTS'

);

Select against a view  fails with following error.

SQL> Select * from . ;

ORA-04023: could not be validated or authorized

This is issue is due to a timestamp inconsistency in the data dictionary. This timestamp inconsistency will lead to the problem when we  populate the relevant cached objects and later try to select from the affected views.

Run following script to find out the objects which are having timestamp discrepencies.

Connect as / as sysdba

SQL> @?/rdbms/admin/utldtchk.sql

This script is available from 11g onwards.

Recompile all the objects manually returned by this script.

Conn owbsys/owbsys

alter view ALL_RT_INSTALLATIONS compile;

alter view RAB_RT_INSTALLATIONS compile;

alter view HAB_RT_EXEC_PROCESS compile;

alter view HAB_RT_EXEC_PROCESS_RUNS compile;

alter view HAB_RT_EXEC_PROC_RUN_COUNTS compile;

alter view HAB_RT_EXEC_PROC_RUN_METRICS compile;

alter view UAB_RT_EXEC_PROCESS compile;

alter view UAB_RT_EXEC_PROCESS_RUNS compile;

alter view UAB_RT_EXEC_PROC_RUN_COUNTS compile;

alter view UAB_RT_EXEC_PROC_RUN_METRICS compile;

alter function WB_OLAP_AW_PRECOMPUTE compile;

drop public synonym WB_OLAP_AW_PRECOMPUTE;

CREATE PUBLIC SYNONYM WB_OLAP_AW_PRECOMPUTE

FOR OWBSYS.WB_OLAP_AW_PRECOMPUTE;

It will remove the timestamp inconsistency in data dictionary and resolve this issue.

Verify by re-running the script, it should return zero rows.

SQL> @?/rdbms/admin/utldtchk.sql

If you still see occurances, bounce the database and rerun the above script.

R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade

March 11, 2014, 6:51 pm

≫ Next: R12.2 URL Validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists,

≪ Previous: R12.2 Dictionary Corrupted error during adop phase=prepare

$

0

0

This problem occurs during rapidwiz where it fails to create apache instance and the error is logged in deploytech.log:

from deploytech.log:

ERROR: Failed to create the EBS domain.
Failed to deploy the E-Business Suite domain

Oracle would usually advise you to re-install. However I found a way to do it by tracking the rapidwiz log files to see what all scripts it calls to create it manually:   

Create missing apache instance, not created by rapidwiz

Reference :

. ./EBSapps.env run

cd $FMW_HOME/opmnctl/bin

./opmnctl createinstance -oracleInstance $FMW_HOME/webtier/instances/EBS_web_ebs122_OHS1

-adminhost admin.justanexample.com -adminPort 7002 -adminRegistration OFF

Creating Oracle Instance directories...Done

Recording OPMN ports reservations...Done

Bootstrapping OPMN configuration files...Done

Instantiating opmnctl for direct usage...Done

Skipping instance registration

Command succeeded.

./opmnctl createcomponent -componentType OHS -componentName EBS_web_ebs122

-oracleInstance $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1

Creating empty component directories...Done

Provisioning OHS files for EBS_web_ebs122

  Copying OHS files from ORACLE_HOME to ORACLE_INSTANCE locations

  Customizing httpd.conf

  Adding component's process control to OPMN...Done

Skipping EBS_web_ebs122 component registration.

Command succeeded.

cd $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/bin

./opmnctl registerinstance –adminhost admin.justanexample.com -adminPort 7002

Command requires login to weblogic admin server (alpcmfapq11.comfin.ge.com):

  Username: weblogic

  Password:

Deploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl admin.justanexample.com:7002 -username weblogic -name NonJ2EEManagement -source $FMW_HOME/webtier/opmn/applications/NonJ2EEManagement.ear -nostage -deploy -upload -noexit

Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.

Target state: deploy completed on Server AdminServer

Done

Registering instance

Command succeeded.

Copy these *.conf files to $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/config/OHS/EBS_web_ebs122 obtained from Vision instance :

admin.conf
apps.conf
custom.conf
httpd.conf
mod_wl_ohs.conf
oracle_apache.conf
oracle_apache_ssl.conf
security2.conf
security2_dmz.conf
ssl.conf
ssl_terminator.conf
trusted.conf
url_fw.conf
url_fw_ws.conf
webgate.conf

Global replace the server name and port in the files above.

Run adconfig to regenerate these files

Instantiate the Oracle HTTP Server configuration files admin.conf, ssl.conf, mod_wl_ohs.conf on Run File System.
Note: Since the templates are being instantiated, all customizations will be lost and have to be restored explicitly.
Perform the instantiation with the following commands (the backslash at the end of each line is being used to divide each command across several screen lines for readability). When prompted for the APPS password, press .

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/ssl_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/mod_wl_ohs_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/admin_conf_FMW.tmp

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-outfile=$INST_TOP/logs/appl/rgf/TXK/test.log

apps.conf generated successfully

mod_wl_ohs.conf generated successfully
 

 

---

R12.2 URL Validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists,

August 4, 2014, 11:32 am

≫ Next: adopreports utility in R12.2

≪ Previous: R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade

$

0

0

Sravan came across this error while using SSO enabled URL for R12.2


URL validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists, Please contact system administrator.


This occurs for any user that is newly created. The user_guid column is not populated. Here is the Solution:

ssoreg_r12_linux erppgai1 DEV appspassword cleanup
ssoreg_r12_linux erppgai1 DEV appspassword dereg

./oamreg_linux erppgai1 DEV appspassword undeployag

./oamreg_linux erppgai1 DEV appspassword deployag

./oamreg_linux erppgai1 DEV appspassword reg

adopreports utility in R12.2

August 28, 2014, 10:01 am

≫ Next: oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

≪ Previous: R12.2 URL Validation failed. The error could have been caused through the use of the browser's navigation buttons ( the browser Back button or refresh, for example). If the error persists,

$

0

0

I discovered a utility in R12.2 when I was looking for the directory of adop :

which adop
$NE_BASE/EBSapps/appl/ad/bin/adop

cd $NE_BASE/EBSapps/appl/ad/bin/
$ ls
adop  adopreports

Curious I executed adopreports
$ adopreports

Enter the APPS username: apps
Enter the APPS Password:



    Online Patching Diagnostic Reports Main Menu
    --------------------------------------------

    1.  Run edition reports
    2.  Patch edition reports
    3.  Other generic reports
    4.  Exit

    Enter your choice [4]: 3




    Other Generic Reports Sub Menu
    ------------------------------

    1.  Editions summary
    2.  Editioned objects summary
    3.  Free space in important tablespaces
    4.  Status of critical AD_ZD objects
    5.  Actual objects in current edition
    6.  Objects dependencies
    7.  Objects dependency tree
    8.  Editioning views column mappings
    9.  Index details for a table
    10.  Inherited objects in the current edition
    11.  All log messages
    12.  Materialized view details
    13.  Database sessions by edition
    14.  Table details (Synonyms, EV, etc.)
    15.  Count and status of DDL execution by phase
    16.  Back to main menu

This is a great utility for R12.2

oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

September 27, 2014, 9:40 am

≫ Next: Mystery of java.sql.SQLRecoverableException: IO Error: Socket read timed out during adop/adpatch

≪ Previous: adopreports utility in R12.2

$

0

0

Yesterday, the users of an EBS R12.2 instance got this error when they logged in:

Error Page
You have Encountered an unexpected error.  Please contact the System Administrator for assistance.

On checking the $EBS_DOMAIN_HOME/servers/oacore_server1/logs/oacore_server1.out, we found this error:

oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full.
       at oracle.ias.cache.CacheHandle.findObject(CacheHandle.java:1680)
       at oracle.ias.cache.CacheHandle.locateObject(CacheHandle.java:1118)
       at oracle.ias.cache.CacheAccess.get(CacheAccess.java:877)
       at oracle.apps.jtf.cache.IASCacheProvider.get(IASCacheProvider.java:771)
       at oracle.apps.jtf.cache.CacheManager.getInternal(CacheManager.java:4802)
       at oracle.apps.jtf.cache.CacheManager.get(CacheManager.java:4624)
       at oracle.apps.fnd.cache.AppsCache.get(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Grant.getGrantArray(Unknown Source)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenusForGrantee(Authorization.java:829)
       at oracle.apps.fnd.functionSecurity.Authorization.getFunctionSecurityGrantedMenus(Authorization.java:744)
       at oracle.apps.fnd.functionSecurity.Authorization.getFuncSecGrants(Authorization.java:251)
       at oracle.apps.fnd.functionSecurity.Authorization.testMenuTreeFunction(Authorization.java:499)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:254)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:279)
       at oracle.apps.fnd.functionSecurity.Navigation.getMenuTree(Navigation.java:160)

We tried bouncing services and deleting $EBS_DOMAIN_HOME/servers/oacore_server1/cache.  None of those actions helped.  Things got back to normal only after the Xmx, Xms,and permsize startup parameters for the oacore JVM were changed in weblogic console on Gary's suggestion:

-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

I also changed it in the context file:

Old: s_oacore_jvm_start_options">-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m

New: s_oacore_jvm_start_options">-XX:PermSize=512m -XX:MaxPermSize=512m –Xms4096m –Xmx4096m

The oacore_server1 and oacore_server2 were bounced after this.  We haven't seen that error ever since.

There is a support.oracle.com article: Receive Intermittent Error You Have Encountered An Unexpected Error. Please Contact Your System Administrator (Doc ID 1519012.1)

Cause

There are user accounts having extremely high numbers of FUN_ADHOC_RECI_XXXXXXX / FUN_ADHOC_INIT_XXXXXXX assigned.

Users have an extremely high number of (ADHOC) ROLES assigned to them, so when these attempt to login this fills the JOC ( Java object cache ) and causes it to run into it's limits resulting in the errors reported. Once bounce is done all is working fine until such an user logs in again.

While working in AGIS and creating and progressing batches, in the workflow there are several ad hoc roles created which remain on the system and do not get end dated or deleted. This can cause performance issues.
Ad Hoc Roles in WF_LOCAL_ROLES starting with FUN_ADHOC_RECI_XXXXXXX ; FUN_ADHOC_INIT_XXXXXXX with no expiration_date.

A. Run the following SQL to verify if there are accounts having extreme numbers of roles assigned

SQL> SELECT user_name, count(*) FROM wf_user_roles WHERE role_name <> user_name GROUP BY user_name ORDER BY 2;


B. Run following for particular user

SQL> SELECT distinct role_name FROM wf_user_roles
WHERE user_name = cp_user_name
or (user_name = (SELECT name FROM wf_local_roles wlr, fnd_user fusr
WHERE fusr.person_party_id = wlr.orig_system_id
AND wlr.orig_system = 'HZ_PARTY'
AND fusr.user_name = cp_user_name
AND rownum < 2))
AND role_name <> user_name;

Note: Replace cp_user_name with name of user having high number of ADHOC roles

Solution

To implement the solution, please execute the following steps:

1. Ensure that you have taken a backup of your system before applying the recommended solution.

2. Follow the steps given in document to purge the WF_LOCAL_ROLES for the AGIS transactions in 'COMPLETE' status.

AGIS: HOW TO DELETE AD HOC ROLES CREATED IN WORKFLOW (Doc ID 1446561.1)

3. If you are satisfied that the issue is resolved, migrate the solution as appropriate to other environments.

We had also logged SR with Oracle where they pointed us to the very same article and also asked us to do the following:

Action Plan
===========

1. How to find out the existing adhoc roles?

select name, start_date, start_date, expiration_date
from wf_local_roles
where orig_system = 'WF_LOCAL_ROLES'
order by name;

2. Define an expiration date for the ad hoc role:

exec WF_DIRECTORY.SetAdHocRoleExpiration (role_name=> >,expiration_date=>sysdate-1);


3. Periodically, purge expired users and roles in order to improve performance.

exec WF_PURGE.Directory(end_date);

This purges all users and roles in the WF_LOCAL_ROLES,WF_LOCAL_USER_ROLES, and WF_USER_ROLE_ASSIGNMENTS tables whose expiration date is less than or
equal to the specified end date and that are not referenced in any notification.

Parameter: end_date Date to purge to.

For more information, please refer to Oracle Workflow API Reference on page 2 – 128.
Use the workflow API's to purge the ad hoc roles:

NOTE:
After end dating the adhoc roles, the expired adhoc roles can also be purged by running the Purge Obsolete Workflow Runtime Data concurrent program. Make sure the "Core Workflow Only" parameter set to N.

Oracle also shared 3 open bugs (unpublished, can be read only Oracle employees) for this issue:

Bug 19025537 : ORACLE.IAS.CACHE.CACHEFULLEXCEPTION: J2EE JOC-017 THE CACHE IS FULL.
Bug 11772304 : JOC INVESTIGATION WITH 12.2
Bug 19582421 : R12.2 THE CACHE IS FULL; EXCEPTION IN OACORE_SERVER1 LOG.

Mystery of java.sql.SQLRecoverableException: IO Error: Socket read timed out during adop/adpatch

November 11, 2014, 7:19 pm

≫ Next: Table TXK_TCC_RESULTS needs to be installed by running the EBS Technology Codelevel Checker (available as patch 17537119).

≪ Previous: oracle.ias.cache.CacheFullException: J2EE JOC-017 The cache is full

$

0

0

While applying the R12.2 upgrade driver, we faced the issue of WFXLoad.class failing in adworker log but showing up as running on adctrl

        Control
Worker  Code      Context            Filename                    Status
------  --------  -----------------  --------------------------  --------------
     1  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     2  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     3  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     4  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     5  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     6  Run       AutoPatch R120 pl                              Wait        
     7  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     8  Run       AutoPatch R120 pl  WFXLoad.class               Running      
     9  Run       AutoPatch R120 pl  WFXLoad.class               Running      
    10  Run       AutoPatch R120 pl                              Wait        

adworker log shows:

Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Socket read timed out

        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:482)

        at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:678)

        at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:238)

        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)

        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)

        at java.sql.DriverManager.getConnection(DriverManager.java:571)

        at java.sql.DriverManager.getConnection(DriverManager.java:215)

        at oracle.apps.ad.worker.AdJavaWorker.getAppsConnection(AdJavaWorker.java:1041)

        at oracle.apps.ad.worker.AdJavaWorker.main(AdJavaWorker.java:276)

Caused by: oracle.net.ns.NetException: Socket read timed out

        at oracle.net.ns.Packet.receive(Packet.java:341)

        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:308)

        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1222)

        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:330)

        ... 8 more

This was happening again and again. The DBAs were suspecting network issue, cluster issue, server issue and all the usual suspects.  In Database alert log we saw these errors coming every few seconds:

Fatal NI connect error 12537, connecting to:

 (LOCAL=NO)

  VERSION INFORMATION:

        TNS for Linux: Version 11.2.0.3.0 - Production

        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production

        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production

  Time: 11-NOV-2014 19:58:19

  Tracing not turned on.

  Tns error struct:

    ns main err code: 12537

TNS-12537: TNS:connection closed

    ns secondary err code: 12560

    nt main err code: 0

    nt secondary err code: 0

    nt OS err code: 0

opiodr aborting process unknown ospid (26388) as a result of ORA-609

We tried changing the parameters in sqlnet.ora and listener.ora as instructed in the article:

Troubleshooting Guide for ORA-12537 / TNS-12537 TNS:Connection Closed (Doc ID 555609.1)

Sqlnet.ora: SQLNET.INBOUND_CONNECT_TIMEOUT=180

Listener.ora: INBOUND_CONNECT_TIMEOUT_listener_name=120

However, the errors continued.  To rule out any issues in network, I also restarted the network service on Linux:

service network restart

One thing which I noticed was the extra amount of time that the connect was taking 4 seconds:

21:17:38 SQL> conn apps/apps

Connected.

21:17:42 SQL> 

Checked from remote app tier and it was same 4 seconds.

Stopped listener and checked on DB server that uses bequeath protocol:

21:18:47 SQL> conn / as sysdba

Connected.

21:18:51 SQL> conn / as sysdba

Connected.

Again it took 4 seconds.

A few days back, I had seen that connect time had increased after turning setting the DB initialization parameter pre_page_sga to true in a different instance.  On a hunch, I checked this and indeed pre_page_sga was set to true.  I set this back to false:

alter system set pre_page_sga=false scope=spfile;

shutdown immediate;

exit

sqlplus /nolog

conn / as sysdba

startup

SQL> set time on

22:09:46 SQL> conn / as sysdba

Connected.

22:09:49 SQL>

The connections were happening instantly.  So I went ahead and resumed the patch after setting:

update fnd_install_processes 

set control_code='W', status='W';

commit;

I restarted the patch and all the workers completed successfully.  And the patch was running significantly faster.  So I did a search on support.oracle.com to substantiate my solution with official Oracle documentation.  I found the following articles:

Slow Connection or ORA-12170 During Connect when PRE_PAGE_SGA init.ora Parameter is Set (Doc ID 289585.1)

Health Check Alert: Consider setting PRE_PAGE_SGA to FALSE (Doc ID 957525.1)

The first article (289585.1) says:

PRE_PAGE_SGA can increase the process startup duration, because every process that starts must access every page in the SGA. This approach can be useful with some applications, but not with all applications. Overhead can be significant if your system frequently creates and destroys processes by, for example, continually logging on and logging off. The advantage that PRE_PAGE_SGA can afford depends on page size.

The second article (957525.1) says:

Having the PRE_PAGE_SGA initialization parameter set to TRUE can significantly increase the time required to establish database connections.

The golden words here are "Overhead can be significant if your system frequently creates and destroys processes by, for example, continually logging on and logging off.".  That is exactly what happens when you do adpatch or adop.

Keep this in mind, whenever you do adpatch or adop, make sure that pre_page_sga is set to false.  It is possible that you may get the error "java.sql.SQLRecoverableException: IO Error: Socket read timed out" if you don't.  Also the patch will run significantly slower if pre_page_sga is set to true.  So set it to false and avoid these issues.

Table TXK_TCC_RESULTS needs to be installed by running the EBS Technology Codelevel Checker (available as patch 17537119).

November 19, 2014, 10:06 am

≫ Next: ERROR: The following required ports are in use: 6801 : WLS OAEA Application Port

≪ Previous: Mystery of java.sql.SQLRecoverableException: IO Error: Socket read timed out during adop/adpatch

$

0

0

I got this error while trying to apply a patch in R12.2:

 [EVENT]     [START 2014/11/19 09:18:39] Performing database sanity checks
   [ERROR]     Table TXK_TCC_RESULTS needs to be installed by running the EBS Technology Codelevel Checker (available as patch 17537119).

This table TXK_TCC-RESULTS is created in APPLSYS schema, by the latest version of the checkDBpatch.sh script delivered by 17537119.

So go ahead, download patch 17537119.  

Login as oracle user on your database node.

Source environment

cd $ORACLE_HOME/appsutil

unzip p17537119*

$ ./checkDBpatch.sh 

+===============================================================+ 

| Copyright (c) 2005, 2014 Oracle and/or its affiliates. | 

| All rights reserved. | 

| EBS Technology Codelevel Checker | 

+===============================================================+ 

Executing Technology Codelevel Checker version: 120.18 

Enter ORACLE_HOME value : /exampler122/oracle/11.2.0 

Enter ORACLE_SID value : exampler122

Bugfix XML file version: 120.0.12020000.16 

Proceeding with the checks... 

Getting the database release ... 

Setting database release to 11.2.0.3 

DB connectivity successful. 

The given ORACLE_HOME is RAC enabled. 

NOTE: For a multi-node RAC environment 

- run this tool on all non-shared ORACLE_HOMEs. 

- run this tool on one of the shared ORACLE_HOMEs. 

Created the table to store Technology Codelevel Checker results. 

STARTED Pre-req Patch Testing : Wed Nov 19 10:53:00 EST 2014 

Log file for this session : ./checkDBpatch_7044.log 

Got the list of bug fixes to be applied and the ones to be rolled back. 

Checking against the given ORACLE_HOME 

Opatch is at the required version. 

Found patch records in the inventory. 

All the required one-offs are present in Oracle Database Home 

Stored Technology Codelevel Checker results in the database successfully. 

FINISHED Pre-req Patch Testing : Wed Nov 19 10:53:03 EST 2014 

========================================================= 

1 select owner,table_name from dba_tables 

2* where table_name='TXK_TCC_RESULTS' 

SQL> / 

OWNER TABLE_NAME 

------------------------------ ------------------------------ 

APPLSYS TXK_TCC_RESULTS 

SQL> 

Once you have done this, restart your patch with adop with additional parameter restart=yes