Friday, April 19, 2013

Upgrade from OEM 11g to 12c: EM Console Service & EM Jobs Service have status Pending, Database System discovered had no members and status Pending.

We created a new OEM 12c [12.1.0.2.2], using the 2-system upgrade approach from our OEM 11g.
The first agent I installed, in the machine where I installed OMS and repository database had every target discovered with status Up, except the Database System, which had status Pending.
Also, EM Console Service & EM Jobs Service had status Pending.

Strangely, both issues had the same solution.
Although, in our old 11g repository database the parameter job_queue_processes had a value of 1000, some where in the upgrade process, in the new 12c repository database this changed to 0.
Probably, that happened during the database upgrade from 11.2.0.2 to 11.2.0.3 using dbua.

So the solution was:
Run as SYS:
SQL> show parameter job_queue_processes

NAME                TYPE     VALUE
------------------- -------- -----
job_queue_processes integer  0

SQL> ALTER SYSTEM SET job_queue_processes=1000 SCOPE=BOTH;
System altered.

SQL> show parameter  job_queue_processes

NAME                TYPE     VALUE
------------------- -------- -----
job_queue_processes integer  1000


Run as SYSMAN:
SQL> exec  emd_maintenance.remove_em_dbms_jobs;
PL/SQL procedure successfully completed.

SQL> exec emd_maintenance.submit_em_dbms_jobs;
PL/SQL procedure successfully completed.

Wednesday, April 10, 2013

Upgrade from OEM 11g to 12c: ORA-06532 during repository upgrade

In our 2-system upgrade, during the repository upgrade step, we got an ORA-06532 error:
2013-04-08 13:19:07,497 [Thread-514] ERROR - ERROR:ORA-06532: Subscript outside of limit
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 576
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 898
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 1686
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 2526
ORA-06512: at line 5
File:/oms/sysman/admin/emdrep/sql/core/11.2.0.1/notification/em_notification_data_upgrade.sql
Statement:DECLARE
l_upgrade_type VARCHAR2(64);
BEGIN
-- Migrate notification rules
NOTIFICATION_MIGRATION_HELPER.MIGRATE_RULES(p_batch_size => 50);

-- Find the upgrade type
l_upgrade_type := em_upgsoln_util.get_upgrade_property(EM_UPGSOLN_UTIL.G_UPGRADE_TYPE);

-- Perform pending violation specific upgrade for 1-system upgrade
IF l_upgrade_type IS NOT NULL AND
l_upgrade_type = EM_UPGSOLN_UTIL.G_UPG_ONE_SYSTEM
THEN
-- Migrate pending violations from aq$mgmt_notify_qtable
NOTIFICATION_MIGRATION_HELPER.MIGRATE_DELIVERY_BUS_DATA(p_batch_size => 25);

-- Migrate pending violations from mgmt_notify_requeue table
NOTIFICATION_MIGRATION_HELPER.MIGRATE_REQUEUE_DATA(p_batch_size => 25);

-- Migrate pending rule engine entries from aq$mgmt_notify_input_qtable
NOTIFICATION_MIGRATION_HELPER.MIGRATE_RULE_BUS_DATA(p_batch_size => 25);
END IF;
END;

Bug 15904251 : UPGRADE OF 11G NOTIFICATION RULES FAIL FOR RULE WITH LARGE NUMEBRS OF CONDITIONS was relevant, but applying it, did not solve the issue.
post_config.sh failed, with error: "No OMS is found to be associated with the current Oracle Home"

To continue, first, run these commands as SYSMAN to the new repository database:

ALTER TYPE EM_RULE_EXPRESSION_ARRAY MODIFY LIMIT 100000 CASCADE;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile body;
ALTER TYPE EM_RULE_DETAILS_OBJ compile body;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile body;
ALTER PACKAGE EM_RULE_BAM_INTG compile;
ALTER PACKAGE EM_INCIDENT_RULES compile;
ALTER PACKAGE MGMT_RCA compile;
ALTER PACKAGE NOTIFICATION_MIGRATION_HELPER compile;
ALTER PACKAGE EM_RULE_BAM_INTG compile body;
ALTER PACKAGE EM_INCIDENT_RULES compile body;
ALTER PACKAGE MGMT_RCA compile body;
ALTER PACKAGE NOTIFICATION_MIGRATION_HELPER compile body;


After that, run:

$OMS_HOME/oui/bin/runConfig.sh ORACLE_HOME=$OMS_HOME ACTION=configure MODE=perform


At the end, your OMS processes will be running ["emctl status oms" to verify] and you may login to your new 12c OEM.

Monday, April 1, 2013

Upgrade from OEM 11g to 12c: Clustered agents reported as "agents with Invalid Inventory"

We are in the process of upgrading our OEM 11g installation to 12c.
In the "Upgrade Console", agents installed to monitor a RAC were reported as "agents with Invalid Inventory".
These agents were installed using this method, in a custom Oracle inventory.
Additionally, from the "Setup", "Agents" tab, you can see a particular agent's page. There, by pressing the "Oracle Home" link, I was getting the following error:

Could not find Oracle Home <ORACLE_HOME> in the inventory collected for <HOSTNAME> on <TIMESTAMP>


As described in Oracle note 735952.1 - "Error Could not find Oracle Home in the inventory collected for on ", although some changes were necessary, to fix this problem:
  • Edit $AGENT_HOME/sysman/config/OUIinventories.add
  • Put an entry for your custom oraInst.loc, for example: inventory: /var/opt/oragrid/oraInst.loc
  • Restart the agent
  • From the OEM, go to the particular host's page, "Configuration" tab and press the "Refresh Host Configuration" button.
  • Wait for it to complete and then go to your "Upgrade Console" and press the "Refresh Agents and Targets List" button.
Now, those agents are removed from the "agents with Invalid Inventory" list.