Upgrade from OEM 11g to 12c: Step unzipAndDeploySoftware fails

We created a new OEM 12c [12.1.0.2.2], using the 2-system upgrade approach from our OEM 11g.

I tried to run the "Deploy and Configure Agent" step on a Linux 64-bit machine and unzipAndDeploySoftware step failed with error:

Check complete: Failed <<<<
Problem: The 32bit software cannot be installed on a 64 bit platform. Similarly the vice-versa is also true where 64bit product cannot be installed on a 32bit machine.
Recommendation: Please install the software on a 32-bit machine if the product is 32bit, otherwise install the product on the 64bit machine if it is a 64bit software.

This is Bug 15995299.
The problem is the OS has been discovered as a Linux 32-bit by the upgrade process.

SELECT TARGET_NAME, PLATFORM_ID, PLATFORM_NAME, CURRENT_VERSION
FROM SYSMAN.PRE_UPGC_AGT_STAT_MGMT
WHERE TARGET_NAME='your_server:3872';

TARGET_NAME	PLATFORM_ID	PLATFORM_NAME	CURRENT_VERSION
your_server:3872	46	Linux x86	11.1.0.1.0

Also, the targets which are monitored by that agent have wrong information:

SELECT EMD_URL, AGENT_NAME, TARGET_NAME, TARGET_TYPE, PLATFORM_ID, PLATFORM_NAME
FROM SYSMAN.PRE_UPGC_TGT_SW
WHERE AGENT_NAME='your_server:3872';

EMD_URL	AGENT_NAME	TARGET_NAME	TARGET_TYPE	PLATFORM_ID	PLATFORM_NAME
https://your_server:3872/emd/main/	your_server:3872	your_server:3872	oracle_emd	46	Linux x86
https://your_server:3872/emd/main/	your_server:3872	your_server	host	46	Linux x86

You have to update these tables with the correct OS information:

UPDATE SYSMAN.PRE_UPGC_AGT_STAT_MGMT
    SET PLATFORM_ID='226', PLATFORM_NAME='Linux x86-64'
WHERE TARGET_NAME ='your_server:3872';

UPDATE SYSMAN.PRE_UPGC_TGT_SW SW
SET (PLATFORM_ID,PLATFORM_NAME,OS_VERSION ) =
(SELECT DISTINCT AGT.PLATFORM_ID,AGT.PLATFORM_NAME,AGT.OS_VERSION 
FROM SYSMAN.PRE_UPGC_AGT_STAT_MGMT AGT, SYSMAN.PRE_UPGC_TGT_SW TGT
WHERE TGT.TARGET_GUID=AGT.TARGET_GUID AND SW.EMD_URL=TGT.EMD_URL);

COMMIT;

 Rerun the "Deploy and Configure Agent" step for the particular agent.

You may also check Oracle Note 1519366.1: EM 12c: PreUpgrade Console shows Linux 64-bit Host As 32-bit for RHEL / OEL 5 64-bit servers.

Upgrade from OEM 11g to 12c: EM Console Service & EM Jobs Service have status Pending, Database System discovered had no members and status Pending.

We created a new OEM 12c [12.1.0.2.2], using the 2-system upgrade approach from our OEM 11g.
The first agent I installed, in the machine where I installed OMS and repository database had every target discovered with status Up, except the Database System, which had status Pending.
Also, EM Console Service & EM Jobs Service had status Pending.

Strangely, both issues had the same solution.
Although, in our old 11g repository database the parameter job_queue_processes had a value of 1000, some where in the upgrade process, in the new 12c repository database this changed to 0.
Probably, that happened during the database upgrade from 11.2.0.2 to 11.2.0.3 using dbua.

So the solution was:
Run as SYS:

SQL> show parameter job_queue_processes

NAME                TYPE     VALUE
------------------- -------- -----
job_queue_processes integer  0

SQL> ALTER SYSTEM SET job_queue_processes=1000 SCOPE=BOTH;
System altered.

SQL> show parameter  job_queue_processes

NAME                TYPE     VALUE
------------------- -------- -----
job_queue_processes integer  1000

Run as SYSMAN:

SQL> exec  emd_maintenance.remove_em_dbms_jobs;
PL/SQL procedure successfully completed.

SQL> exec emd_maintenance.submit_em_dbms_jobs;
PL/SQL procedure successfully completed.

Upgrade from OEM 11g to 12c: ORA-06532 during repository upgrade

In our 2-system upgrade, during the repository upgrade step, we got an ORA-06532 error:

2013-04-08 13:19:07,497 [Thread-514] ERROR - ERROR:ORA-06532: Subscript outside of limit
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 576
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 898
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 1686
ORA-06512: at "SYSMAN.NOTIFICATION_MIGRATION_HELPER", line 2526
ORA-06512: at line 5
File:/oms/sysman/admin/emdrep/sql/core/11.2.0.1/notification/em_notification_data_upgrade.sql
Statement:DECLARE
l_upgrade_type VARCHAR2(64);
BEGIN
-- Migrate notification rules
NOTIFICATION_MIGRATION_HELPER.MIGRATE_RULES(p_batch_size => 50);

-- Find the upgrade type
l_upgrade_type := em_upgsoln_util.get_upgrade_property(EM_UPGSOLN_UTIL.G_UPGRADE_TYPE);

-- Perform pending violation specific upgrade for 1-system upgrade
IF l_upgrade_type IS NOT NULL AND
l_upgrade_type = EM_UPGSOLN_UTIL.G_UPG_ONE_SYSTEM
THEN
-- Migrate pending violations from aq$mgmt_notify_qtable
NOTIFICATION_MIGRATION_HELPER.MIGRATE_DELIVERY_BUS_DATA(p_batch_size => 25);

-- Migrate pending violations from mgmt_notify_requeue table
NOTIFICATION_MIGRATION_HELPER.MIGRATE_REQUEUE_DATA(p_batch_size => 25);

-- Migrate pending rule engine entries from aq$mgmt_notify_input_qtable
NOTIFICATION_MIGRATION_HELPER.MIGRATE_RULE_BUS_DATA(p_batch_size => 25);
END IF;
END;

Bug 15904251 : UPGRADE OF 11G NOTIFICATION RULES FAIL FOR RULE WITH LARGE NUMEBRS OF CONDITIONS was relevant, but applying it, did not solve the issue.
post_config.sh failed, with error: "No OMS is found to be associated with the current Oracle Home"

To continue, first, run these commands as SYSMAN to the new repository database:

ALTER TYPE EM_RULE_EXPRESSION_ARRAY MODIFY LIMIT 100000 CASCADE;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile;
ALTER TYPE EM_RULE_COND_ACTIONS_OBJ compile body;
ALTER TYPE EM_RULE_DETAILS_OBJ compile body;
ALTER TYPE EM_RULE_SET_DETAILS_OBJ compile body;
ALTER PACKAGE EM_RULE_BAM_INTG compile;
ALTER PACKAGE EM_INCIDENT_RULES compile;
ALTER PACKAGE MGMT_RCA compile;
ALTER PACKAGE NOTIFICATION_MIGRATION_HELPER compile;
ALTER PACKAGE EM_RULE_BAM_INTG compile body;
ALTER PACKAGE EM_INCIDENT_RULES compile body;
ALTER PACKAGE MGMT_RCA compile body;
ALTER PACKAGE NOTIFICATION_MIGRATION_HELPER compile body;

After that, run:

$OMS_HOME/oui/bin/runConfig.sh ORACLE_HOME=$OMS_HOME ACTION=configure MODE=perform

At the end, your OMS processes will be running ["emctl status oms" to verify] and you may login to your new 12c OEM.

Upgrade from OEM 11g to 12c: Clustered agents reported as "agents with Invalid Inventory"

We are in the process of upgrading our OEM 11g installation to 12c.
In the "Upgrade Console", agents installed to monitor a RAC were reported as "agents with Invalid Inventory".
These agents were installed using this method, in a custom Oracle inventory.
Additionally, from the "Setup", "Agents" tab, you can see a particular agent's page. There, by pressing the "Oracle Home" link, I was getting the following error:

Could not find Oracle Home <ORACLE_HOME> in the inventory collected for <HOSTNAME> on <TIMESTAMP>

As described in Oracle note 735952.1 - "Error Could not find Oracle Home in the inventory collected for on ", although some changes were necessary, to fix this problem:

• Edit $AGENT_HOME/sysman/config/OUIinventories.add
• Put an entry for your custom oraInst.loc, for example: inventory: /var/opt/oragrid/oraInst.loc
• Restart the agent
• From the OEM, go to the particular host's page, "Configuration" tab and press the "Refresh Host Configuration" button.
• Wait for it to complete and then go to your "Upgrade Console" and press the "Refresh Agents and Targets List" button.

Now, those agents are removed from the "agents with Invalid Inventory" list.