8 Upgrading Enterprise Manager Grid Control

Oracle recommends reading this to get a better understanding of the requirements for an Enterprise Manager upgrade.

Oracle Enterprise Manager Grid Control consists of three major components:

Repository Database
Management Service
Management Agent

With the option of deploying the Grid Control environment across the enterprise and in any number of permutations, upgrading the entire environment becomes a very complex activity involving updating of software and configurations in different levels (tiers) located in different machines.

The Enterprise Manager Upgrade process aims at simplifying this entire operation and render it as seamless and error-free as possible.

The following topics are covered in this chapter:

Software Prerequisites
Checks to be Performed Before Starting the Upgrade
Upgrade Scenarios
Upgrading Enterprise Manager

Software Prerequisites

Grid Control 10.2 installer only upgrades the corresponding Enterprise Manager 10.1 components. The following table lists the supported component versions:

Table 8-1 Supported Component Versions that Correspond to the Install Type

Install Type	OMS Version	RDBMS Version
Enterprise Manager Using New Database	10.1 (with Application server 9.0.4)	9.0.1.5
Enterprise Manager Using Existing Database	10.1 (with Application server 9.0.4)	9.2.0.6 or higher
Additional Management Service	10.1 (with Application server 9.0.4)	containing the 10.1. OMS schema
Additional Agent	Not Applicable	Not Applicable

Checks to be Performed Before Starting the Upgrade

This section lists the checks that you must perform before starting any of the Enterprise Manager 10.2 upgrade scenarios discussed in the following sections.

Shutdown Enterprise Manager Before Upgrade

You must ensure that the existing Enterprise Manager processes are shut down before starting the upgrade process.

To shut down Enterprise Manager, execute the following commands:

<OMS Oracle_Home>/bin/emctl stop oms
<OMS Oracle_Home>/bin/emctl stop em
<OMS Oracle_Home>/opmn/bin/opmnctl stopall

These commands usually successfully shut down all Enterprise Manager processes that are running. However, there may be instances where some processes might still be left running. Such instances may cause the upgrade to fail. To avoid this, you must ensure the following processes are not running before attempting to start the upgrade process:

OPMN
DCM

Check for Symbolic Links

The <Oracle_Home>/Apache/<component> configuration files must be examined to ensure only hard links (and no symbolic links) were referenced.

As part of the Enterprise Manager 10g R2 (10.2) Upgrade from 10.1, the underlying Oracle iAS 9.0.4 stack is also upgraded. During this phase, the upgrade utility attempts to replace the 10g R1 Oracle home path that is referenced in these configuration files with the new Enterprise Manager 10g R2 ORACLE_HOME path.

If you have customized the Oracle iAS 9.0.4 stack using symbolic links to the Oracle home, then the configuration files may also contain these references as symbolic links, instead of the complete path of the Oracle home.

For example, <Oracle_Home>/Apache/modplsql/conf/plsq.conf contains references to the plsql_module. If this path is a symbolic link, you must modify this path to be a hard link.

Customizations and User Permissions

If there have been any mid-tier customizations files that cannot be accessed using the the upgrade user's credentials, ensure such customizations are removed or commented out before starting the upgrade. You can reapply these customizations after the upgrade is successfully completed.

Select an Agent that is Not Secure

During the upgrade process, if you have selected an agent that is not secure, you must decide whether or not to specify the Agent Registration password (that will secure this agent).

You can also choose not to specify the Agent Registration password. This leaves the upgraded agent insecure. The Installer displays an error message when the agent's REPOSITORY_URL does not match with that of the OMS, indicating the agent is not bound to the OMS.

This can happen if multiple OMSs are grouped together via a load balancer, and the agent is using this load balancer to communicate with the OMSs.You can also choose to secure the upgraded agent manually after the agent has been upgraded.

Shutdown Existing Database Listener

During the Oracle database upgrade process, ensure the existing database listener is shut down just before you execute allroot.sh.

ATTENTION:

The database listener must be running during the upgrade process until the point where you are prompted to execute allroot.sh. If you shut down the listener before this point, the upgrade process fails.

In an Enterprise Manager 10g R1 (10.1) installation, the Install Enterprise Manager Using New Database installation type performs a 9.0.1.5 RDBMS installation along with the other Enterprise Manager products. When you upgrade Enterprise Manager 10.1 to 10.2, the RDBMS instance is upgraded to a 10.1.0.4 version before the Enterprise Manager migration configuration is performed.

To successfully migrate your existing Enterprise Manager to 10.2, the new 10.1.0.4 RDBMS listener has to be started. If during the upgrade, the 9.0.1.5 RDBMS listener is still running after you have executed allroot.sh, the new listener is not started successfully, resulting in an unsuccessful migration of Enterprise Manager.

Upgrade Scenarios

The following upgrade scenarios are discussed in this section:

Upgrading Oracle Management Service that is Installed Using a New Database
Upgrading Oracle Management Service that is Installed Using an Existing Database
Upgrading the Oracle Management Service
Upgrading the Management Agent

ATTENTION:

During the upgrade process, you must ensure the monitoring agent is also upgraded along with the OMS and repository database.

Upgrading Oracle Management Service that is Installed Using a New Database

If the Enterprise Manager 10g R1 (10.1) was of this installation type, then the 10g R2 (10.2) installer performs an out-of-place upgrade of the 10.1 Management Service (OMS), the repository database, and the agent Oracle homes. During the upgrade, it creates a new Oracle home for the OMS, database, and agent. The upgrade assistants upgrade the datafiles and SYSMAN schema, and then configure the new Oracle homes.

Upgrading Oracle Management Service that is Installed Using an Existing Database

If the earlier Enterprise Manager installation was done using an existing database, the 10g R2 installer automatically checks the remote database version (must be 9.2.0.6 or higher). If the minimum database version requirement is met, the installer performs an upgrade of both the Management Service as well as the database Oracle homes.

Note:

If the database version is lower than 9.2.0.6, this database is not upgraded as part of the Enterprise Manager upgrade process. You are required to upgrade the database separately.

IMPORTANT:

If you are upgrading an Enterprise Manager target (for example, database) using an upgrade mechanism other than the Oracle Enterprise Manager Installer, ensure you manually modify the central inventory information by updating the targets.xml file of the monitoring agent to reflect the configuration parameters of the upgraded Oracle home. See AppendixA, "Monitoring Agent Does Not Discover Upgraded Targets" for more information.

Upgrading the Oracle Management Service

If the previous Enterprise Manager installation was for an additional Management Service, the 10g R2 installer checks the remote database (must be 9.2.0.6 or higher). If this minimum database version requirement is met, the installer performs the OMS upgrade.

If you are upgrading only the Management Service to 10g R2 (10.2) without upgrading the monitoring agent, you may encounter a metric collection error. To resolve this issue, you must upgrade the monitoring agent to 10g R2 as well.

Note:

If the database version is lower than 9.2.0.6, this database is not upgraded as part of the Enterprise Manager upgrade process. You are required to upgrade the database separately.

ATTENTION:

After successfully upgrading the 10g R1 (10.1.0.4) OMS to (10.2), the 10.1.0.4 agents that are uploading data to this OMS may show an incorrect OMS version when you execute ./emctl status agent.

To resolve this issue, you must stop the 10.1.0.4 agents (./emctl stop agent) and restart them (./emctl start agent).

Upgrading the Management Agent

All Agent homes that need to be upgraded are detected and displayed in the Select Install or Upgrade page of the installer. You can select the agent homes that you want to upgrade and proceed with the process.

Note:

The agent homes that are installed as part of the first two installation types are not upgraded automatically along with the OMS Oracle homes. You must select the agent home that you want to upgrade. This agent home can be independent of the Management Service home.

Upgrading 10.1.0.4 Agents That Monitor User-Defined Metrics

Since the Management Agent upgrade process is an out-of-place installation, upgrading an agent will create a new Oracle home directory. If you are using OS-based user-defined metrics that are referencing scripts located in an agent Oracle home, then such scripts will not be copied over during the upgrade process. Specifically, if a 10.1.0.4 agent is being upgraded to 10.2 agent version, any user-defined metric scripts that may have existed in the 10.1.0.4 agent Oracle home will not be copied into the new 10.2 agent Oracle home.

In order to ensure the user-defined metrics continue to work the same, you must manually copy all user-defined metrics scripts into another directory (outside any Oracle home), and then update the user-defined metric definitions to reflect the new script location.

For example, if the user-defined script is called myScript.sh and is located in the 10.1.0.4 agent Oracle home (for example, /u1/oracle/sysman/admin/scripts directory), copy this script over to a new directory (for example, /u1/scripts). Now, in the definition of the user-defined metric, you must change the command line from /u1/oracle/sysman/admin/scripts/myScript.sh to /u1/scripts/myScript.sh.

Note:

Ensure you do not delete the original Oracle home (that is 10.1.0.4 Oracle home in the above-example) until you have changed the user-defined metric script location.

Upgrading Enterprise Manager

When you invoke Oracle Universal Installer to perform an upgrade, it automatically detects the existing Enterprise Manager installation type of the target Oracle homes and displays the components that need to be upgraded. You can select the component that you want to upgrade and proceed with the process.

Caution:

Oracle recommends that you perform a backup of the repository before starting the upgrade process.

Complete the instructions detailed below to perform the Enterprise Manager upgrade.

Start the Oracle Universal Installer by running the runInstaller script in Linux (go to the top-level folder in the contents copied from the DVD and execute ./runInstaller) from the top directory of Disk1.
When you invoke Oracle Universal Installer, you will be presented with two choices:
1. Perform a New Enterprise Manager Installation
  
  See Chapter3, "Enterprise Manager Installation Types" for more information.
2. Products Upgrade
  
  Select this option to continue with the upgrade process.
Click Products Upgrade. The Select Install or Upgrade page appears.

Figure 8-1 Select Install or Upgrade
The installer automatically detects the Grid Control components that require an upgrade and lists them in this page.
Here, you can choose to perform the following upgrades:
1. Management Service and the associated (monitoring) Management Agent
2. Only the Management Service
3. Only the Management Agent
  
  IMPORTANT:
  
  If you are upgrading both Management Service and Agent, ensure the agent you have selected is the one that is communicating with selected Management Service.
Select the Oracle homes that you want to upgrade and click Next. The Specify Installation Location page appears.

Figure 8-2 Specify Installation Location

Here, specify a parent directory (base directory), for example, /scratch/OracleHomes, for the new installation. Since the installer is going to perform an out-of-place^Foot 1 upgrade, all the Oracle homes created during the upgrade will be placed as sub-directories under this parent directory.
Click Next. The Product-Specific Prerequisite Checks page appears.

Figure 8-3 Product-Specific Prerequisite Checks
1. At this point, the Installer runs some prerequisite checks to verify whether the environment meets the minimum requirements for a successful Enterprise Manager installation.
  
  Early detection of problems with the system setup reduces the chances of you encountering problems during installation; for instance, problems with insufficient disk space, missing patches, inappropriate hardware, and so on.
  
  This page displays the check name, type, and status for all prerequisite checks designed for the installation. Automatic checks are run first, followed by optional and manual checks.
  
  Depending on the status of the automatic checks, you must verify all warning and manual checks. At some point, if you have stopped the prerequisite check and want to re-run these checks, select the checks that you want to re-run and click Retry. As each check runs, a progress bar is shown, and test details (expected results, actual results, error messages, instructions) are displayed in the details section, at the bottom of the page.
2. To stop all prerequisite checks, click Stop. At any point of time, click a prerequisite check to view its corresponding details, including the recommended user actions.
  
  Note:
  You must manually verify and confirm all checks that were flagged with a warning, skipped (stopped by user), or failed.
3. To continue with the installation without retrying, click Next.
The Specify Configuration page appears.

Figure 8-4 Specify Configuration
1. Here, you must specify the existing Database and Secure Management Service passwords. The Database Password (SYS) is required to access the configuration files of the existing database repository that is associated with the Management Service that you have selected for upgrade.
2. In the Database password section, specify the SYS Password (the default Super Administrator account) for the existing database repository that is associated with the selected OMS.
3. In the Management Service Security section, specify a password that will be used to secure the communications between the OMS and its agents.
  
  Note:
  Management Service Security: This section is enabled only when the existing Management service that you are upgrading is not secure.

Click Next. If you are upgrading both Management Service and Agent, but only the existing Management Service is secure, then the Agent Registration Password page is displayed. Here, you must provide the correct password to enable communications between the secure Management Service and the Agent that you are upgrading.

Figure 8-5 Specify Agent Registration Password

IMPORTANT:

If you do not know the password and choose to leave the Password field blank, you must do the following after installation, to enable communication between the agent and secure OMS:

Find out the correct password for the secure OMS environment. If you do not know the password, obtain it from the user who configured the Management Service for SSL.
In the Agent's <Oraclehome>/bin directory, execute the following command:
```
/emctl secure agent <password>
```
where <password> is the Agent Registration Password.

The Summary page appears. This page displays all the Oracle homes that will be created. Depending on the type of upgrade you have selected, this page will display all/any of the following details:
- Global Settings
- Product Languages
- Space Requirements
- New Installations
Verify the choices that you have made, and click Install to start the upgrade.

Upgrading Management Agent Using the Agent Deploy Application

Agent Deploy is a J2EE application that is used for mass deployment of Management Agents.

The Upgrade Agent option in this application will help you upgrade an existing Agent installation to a 10.2 Management Agent.

To upgrade Management Agent using Agent Deploy, follow the instructions documented below:

On the Agent Deploy application home page, click Agent Upgrade to perform the Management Agent upgrade.
In the Installation Details page that is displayed, specify the source agent information, destination hosts, OS Credentials, and base directory details that are required for the Management Agent Upgrade.

Note:
While performing an agent upgrade, you must ensure that all the specified hosts have the same operating system credentials and file system structure.
In the Source Agent Information section, specify the full path to an existing Agent home location. This source Agent installation will be used to perform the upgrade.

Note:
The path of the source agent should be the same on all the remote hosts.
Select the Platform on which you want to perform this installation.
In the Provide Host List text box, specify all the hosts (hostnames or IP addresses) on which you want to perform the Agent upgrade. Alternatively, click Get Host Names From File to select the file that contains a list of all the required hostnames.

Note:
You can use a comma (,) or just white space as a separator when specifying multiple hosts.
In the OS Credentials section, specify the appropriate operating system user credentials. Select Run root.sh if you want Agent Deploy to execute this script. Agent Deploy will use sudo to run this script. You must specify the invoking user's password here. You must also ensure that the targetpw is not set in the /etc/sudoers file.
In the Destination section, specify the absolute path for the Installation Base Directory. This directory will be created on all the specified hosts, and the Agent Oracle home directory will be created as a sub-directory under this directory.
In the Additional Parameters text box, specify any additional parameters that you want to pass during installation.
In the Additional Scripts section, specify any pre and/or post-installation scripts that you want to execute.
Click Continue to start the installation process.

Click Help, in the Agent Deploy application for more information.

Possible Parameters that You Can Specify During Agent Upgrade

The important parameters for Agent Upgrade are -o, -b and optionally -i, -t.

The following table lists all the possible parameters that you can specify:

Table 8-2 Possible parameters that you can specify during Agent Upgrade

Parameters	Description
-t	Do NOT start the Agent after installation/upgrade. No value required.
-b	Install Base Dir location. For example, `-b /home/OracleHomes/agent/`
-d	Do not initiate automatic target discovery. No value required.
-i	Inventory pointer location file. For example, `-i /etc/oraInst.loc`
-o	Existing Agent home to upgrade. For example, `-o /home/OracleHomes/oldAgent/`
-p	File location for static port for Agent. For example, `-p /home/config/staticports.ini`

Note:

If the parameters that you specify here are also specified independently through command-line options, the value of the parameters that you specify here will take precedence over the others. For example, if installation Base Dir has been specified independently, and the -b option is specified here, the value of the latter (-b) will be used during the upgrade.

Upgrading Management Agent Using agentDownload Script

To upgrade Management Agent using the agentDownload Script, follow the instructions documented below:

Invoke the agendDownload script using -u option.

The OLD_ORACLE_HOME that you want to upgrade should be specified either by passing the -o option, or setting the OLD_ORACLE_HOME Env Variable.

Invoke the agentdownload script using the following command:
```
./agentDownload.linux -u -o <OLD_ORACLE_HOME_PATH>
```
Note:
The first two options (-u and -o) are mandatory. The -b option can be skipped if this has already been specified in the response files.
To construct the new Oracle home name, either pass the -b option, or specify these values for BASEDIR in the agent_download.rsp file.

Note:
You can specify these options in the command line even though they are present in the response file. The command line options will have a higher precedence over the ones in response file.
During the upgrade, if you want to secure the Agent, specify the appropriate password using AGENT_INSTALL_PASSWORD environment variable.

You can use the following options to execute the agentDownload script:

Table 8-3 Options that you can use to execute the agentDownload script

Options	Description
-b	baseDirectory of the Agent Oracle home
-l	local (pass -local to runInstaller)
-n	Cluster name
-o	OLD_ORACLE_HOME during Upgrade
-u	Upgrade
-c	CLUSTER_NODES (to specify the cluster nodes)
-p	`staticports.ini` file
-s	Installer stage directory
-t	Do NOT start the Agent
-i	Inventory pointer location file
-d	Do NOT initiate automatic target discovery

Note:

When performing an Agent upgrade using the agentDownload script, ensure you are using the agentDownload script from the Oracle home of the pointing Management Service (OMS).

For example, if you are upgrading Agent1 that is pointing to OMS A, then you must use the agentDownload script that is located in the OMS A Oracle home only.

ATTENTION:

During the upgrade, if you want to secure the existing unsecured agent, you must specify the password using the AGENT_INSTALL_PASSWORD environment variable, or by executing the following command after the installation is complete:

<Agent_Home>/bin/emctl secure agent <password>

General System Installation Requirements for Real Application Clusters

Each node that is going to be part of your Real Application Clusters installation must meet the following hardware and software requirements. You will perform step-by-step tasks for hardware and software verification for the platform-specific pre-installation procedures.

Hardware Requirements for Real Application Clusters Setup

Each node in a cluster requires the following hardware:

External shared disks for storing the Oracle Clusterware files.

Refer to the respective Real Application Clusters install guides for information on the disk configuration options that are available. Review these options before you decide which storage option to use in your Real Application Clusters environment.
One private internet protocol (IP) address for each node to serve as the private interconnect. The following must be true for each private IP address:

– It must be separate from the public network– It must be accessible on the same network interface on each node– It must have a unique address on each nodeThe private interconnect is used for inter-node communication by both Oracle Clusterware and Real Application Clusters. If the private address is available from a network name server (DNS), then you can use that name. Otherwise, the private IP address must be available in each node's /etc/hosts file.

During Oracle Clusterware installation, the information you enter as the private IP address determines which private interconnects are used by Real Application Clusters database instances. They must all be in an up state, just as if their IP addresses were specified in the initialization parameter, CLUSTER_INTERCONNECTS. Real Application Clusters does not fail over between cluster interconnects; if one is down then the instances using them will not start.Oracle recommends that you use a logical IP address that is available across all private networks, and that you take advantage of any available operating system-based failover mechanism by configuring it according to your third-party vendor's instructions for using their product to support failover.
One public IP address for each node, to be used as the Virtual IP address for client connections and for connection failover.

This public Virtual IP address (VIP) must be associated with the same interface name on every node that is part of your cluster. In addition, the IP addresses that you use for all of the nodes that are part of a cluster must be from the same subnet. If you have a domain name server (DNS), then register the host names for the VIP with DNS. The Virtual IP address should not be in use at the time of the installation, because this is a Virtual IP address that Oracle manages.
One public fixed hostname address for each node, typically assigned by the system administrator during operating system installation. If you have a DNS, then register both the fixed IP and the VIP address with DNS. If you do not have DNS, then you must make sure that both public IP addresses are in the node hostfile.

Software Requirements for Real Application Clusters Setup

Each node in a cluster requires a supported interconnect software protocol to support Cache Fusion, and to support Oracle Clusterware polling. Your interconnect must be certified by Oracle for your platform. You should also have a Web browser, both to enable Oracle Enterprise Manager, and to view online documentation. For Oracle Database 10g requirements, Oracle Clusterware provides the same functionality as third-party vendor clusterware. Using Oracle Clusterware also reduces installation and support complications. However, you may require third-party vendor clusterware if you use a non-ethernet interconnect, or if you have deployed clusterware-dependent applications on the same cluster where you deploy Real Application Clusters.

See Appendix C, "Installing Enterprise Manager on Real Application Clusters" for more information on the pre-installation tasks.

ATTENTION:

If you are upgrading a 10.1.0.2 agent to 10.1.0.3 version, you must execute the following command before starting the upgraded agent:

emctl resetTZ agent

This command is required to correct the agent timezone.

This command will correct the agent side timezone, and specify an additional command to be run against the repository to correct the value there.

IMPORTANT:

Before you change the timezone, check if there are any blackouts that are currently running or scheduled to run on any of the targets that are monitored by the upgraded agent. Do the following to check this:

In the Grid Control console, go to the All Targets page under Targets and locate the Agent in the list of targets. Click the agent name link. The Agent home page appears.
The list of targets monitored by the agent will be listed in the Monitored Targets section.
For each target in the list, click the target name to view the target home page.
Here, in the Related Links section, click Blackouts to check any blackouts that are currently running or may be scheduled to run int he future.
If such blackouts exist, you must stop all the blackouts that are running on all the targets monitored by this agent.
From the console, stop all the targets that are scheduled to run on any of these monitored targets.
Now, run the following command from the agent home to reset the timezone;
```
emctl resetTZ agent
```
After the timezone is reset, you can create new blackouts on the targets.

Footnote Legend

Footnote 1: Out-of-place upgrade refers to the process of installing the software in a new Oracle home. All the data and configuration files from an existing Oracle home are retrieved and migrated to the new Oracle home. After the upgrade is complete, the old Oracle home is discarded and the software can be started from the new Oracle home.