13 OracleAS Disaster Recovery

13.1.1 OracleAS Disaster Recovery Requirements

To ensure that your implementation of the OracleAS Disaster Recovery solution performs as designed, the following requirements must be adhered to:

• On each host in the standby site, make sure the following is identical to its equivalent peer in the production site:

  • For the middle-tier hosts, physical hostnames.

    
    

    Note: If you already have installed systems, you only need to modify the physical names for the middle-tier systems at the standby site and then create a virtual hostname for the physical hostname of the OracleAS Infrastructure (see the next bullet). See Section 13.2.1, "Planning and Assigning Hostnames" for information about how to change these physical hostnames and the virtual hostname.

    
    

  • Virtual hostname for the OracleAS Infrastructure. The virtual hostname can be specified in the Specify Virtual Hostname screen presented by the installer.

  • Hardware platform

  • Operating system release and patch levels

• All installations conform to the requirements listed in the Oracle Application Server Installation Guide to install Oracle Application Server.

• Oracle Application Server software is installed in identical directory paths between each host in the production site and its equivalent peer in the standby site.

• The following details must be the same between a host in the production site and a peer in the standby site:

  • User name and password of the user who installed Oracle Application Server must be the same between a host in the production site and its peer in the standby site.

  • Numerical user ID of the user who installed Oracle Application Server on that particular node

  • Group name of the user who installed Oracle Application Server on that particular node

  • Numerical group ID of the group for the user who installed Oracle Application Server on that particular node

  • Environment profile

  • Shell (command-line environment)

  • Directory structure, Oracle home names, and path of the Oracle home directory for each OracleAS installation on a node. Do not use symbolic links anywhere in the path.

  • Oracle Application Server installation types (Any instance installed on the standby system must be identical to that installed on the production system):

    • Middle Tier: J2EE and Web Cache, and Portal and Wireless

    • OracleAS Infrastructure: Metadata Repository (MR) and Identity Management (IM)

13.1.2 Supported Oracle Application Server Releases and Operating Systems

OracleAS Guard supports Oracle Application Server releases 10g (9.0.4) and 10g (10.1.2.0.0 and 10.1.2.0.2). The OracleAS Guard kit located on the 10g (10.1.2.0.2) Utility media #2 must be installed on all systems with Oracle homes or Oracle Application Server instances in the topology. See the OracleAS Disaster Recovery installation information in Oracle Application Server Installation Guide for more information.

OracleAS Guard supports a mixed OracleAS 10g (9.0.4) and OracleAS 10g (10.1.2) or higher release environment, such as may occur during an upgrade scenario. In this case, you may have upgraded the standby site to OracleAS 10g (10.1.2) Oracle homes in the middle tier but the Infrastructure is still an OracleAS 10g (9.0.4) Infrastructure. This mixed release environment will work as long as the OracleAS Disaster Recovery peer home for the given production middle tier Oracle homes are also upgraded to the same release OracleAS 10g (10.1.2) and as long as its Infrastructure is still an OracleAS 10g (9.0.4) Infrastructure. So the rule of thumb is that all Oracle home peer middle tiers within the topology must match exactly in release number on both the standby and production sites. Also the Infrastructures must match exactly in release number on both the standby and production sites and Oracle AS Guard must be the same version on both sites. However, as an upgrade based requirement, the Infrastructure can be at a lower release than the middle tiers because this happens during an upgrade scenario.

13.1.3 Supported Topologies

OracleAS Disaster Recovery supports a number of basic topologies for the configuration of the Infrastructure and middle tier on production and standby sites. OracleAS Disaster Recovery supports these basic topologies:

• Symmetrical Topologies - Strict Mirror of the Production Site with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure

• Asymmetrical Topologies - Simple Asymmetric Standby Topology with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure

• Separate OracleAS Metadata Repository for OracleAS Portal with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure (the Departmental Topology)

• Distributed Application OracleAS Metadata Repositories with Non Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure

13.1.3.1 Symmetrical Topologies - Strict Mirror of the Production Site with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure

For OracleAS Disaster Recovery Release 10.1.2.0.1, only the OracleAS Disaster Recovery symmetrical topology environment was supported. This OracleAS Disaster Recovery environment has two major requirements:

• The deployment must use a single default Infrastructure install that contains a collocated OracleAS Metadata Repository and Oracle Identity Management.

• The standby site has to be a strict mirror of the production site with the same number of instances (symmetrical topology).

Figure 13-1 depicts an example OracleAS Disaster Recovery solution having a symmetrical topology with a Cold Failover Cluster on the primary site. This is considered a symmetrical topology because from an Oracle Application Server perspective both sites contain two OracleAS middle tiers and one Infrastructure.

Figure 13-1 Example Oracle Application Server Site-to-Site Disaster Recovery Solution (Load Balancer Appliance Is Optional If Only One Middle-Tier Node Is Present)

Description of "Figure 13-1 Example Oracle Application Server Site-to-Site Disaster Recovery Solution (Load Balancer Appliance Is Optional If Only One Middle-Tier Node Is Present)"

The procedures and steps for configuring and operating the OracleAS Disaster Recovery solution support 1 to n number of middle-tier installations in the production site. The same number of middle-tier installations must exist in the standby site. The middle tiers must mirror each other in the production and standby sites.

For the OracleAS Infrastructure, a uniform number of installations is not required (names or instances must be equal) between the production and standby sites. For example, the OracleAS Cold Failover Cluster (Infrastructure) solution can be deployed in the production site, and a single node installation of the OracleAS Infrastructure can be deployed in the standby site as shown in Figure 13-1. This way, the production site's OracleAS Infrastructure has protection from host failure using an OracleAS Cold Failover Cluster. This solution provides hardware redundancy by utilizing a virtual hostname. Refer to the section Section 6.2.2, "Active-Passive High Availability Topologies" for more information on OracleAS Cold Failover Clusters.

The OracleAS Disaster Recovery solution is an extension to various single-site Oracle Application Server architectures. Examples of such single-site architectures include the combination of OracleAS Cold Failover Cluster (Infrastructure) and active-active Oracle Application Server middle-tier architecture. For the latest information on what single-site architectures are supported, check the Oracle Technology Network (OTN) Web site for the latest certification matrix.

http://www.oracle.com/technology/products/ias/hi_av/index.html

The following are important characteristics of the OracleAS Disaster Recovery solution:

• Middle-tier installations are identical between the production and standby sites. In other words, each middle-tier installation in the production site has an identical installation in the standby site. More than one middle-tier node is recommended because this enables each set of middle-tier installations on each site to be redundant. Because they are on multiple machines, problems and outages within a site of middle-tier installations are transparent to clients.

• The OracleAS Disaster Recovery solution is restricted to identical site configuration to ensure that processes and procedures are kept the same between sites, making operational tasks easier to maintain and execute. Identical site configuration also allows for a higher success rate for manually maintaining the synchronization of Oracle Application Server component configuration files between sites.

• When the production site becomes unavailable due to a disaster, the standby site can become operational within a reasonable time. Client requests are always routed to the site that is operating in the production role. After a failover or switchover operation occurs due to an outage, client requests are routed to another site that assumes the production role. For a symmetric topology, the quality of service offered by the new production site should be the same as that offered by the original production site before the outage.

• The sites are set up in active-passive configuration. An active-passive setup has one primary site used for production and one secondary site that is initially passive (on standby). The secondary site is made active only after a failover or switchover operation is performed. Since the sites are symmetrical, after failover or switchover, the original standby site can be kept active as the new production site. After repairing or upgrading the original production site, it can be made into the new standby site as long as the OracleAS Disaster Recovery site requirements are maintained. Either site should offer the same level of service to clients as the other.

• The site playing the standby role contains a physical standby of the Oracle Application Server Infrastructure coordinated by Oracle Data Guard, OracleAS Guard automates the configuration and use of Oracle Data Guard together with procedures for backing up and restoring OracleAS Infrastructure configuration files and provides configuration synchronization between the production and standby sites. Switchover and failover operations allow the roles to be traded between the OracleAS Infrastructures in the two sites. Refer to Section 13.8, "OracleAS Guard Operations -- Standby Site Cloning of One or More Production Instances to a Standby System", Section 13.9, "OracleAS Guard Operations -- Standby Instantiation and Standby Synchronization", Section 13.10, "Runtime Operations -- OracleAS Guard Switchover and Failover Operations", and Section 13.13, "Using OracleAS Guard Command-Line Utility (asgctl)" for information about using the asgctl command-line interface to perform OracleAS Guard administrative tasks of cloning, instantiation, synchronization, switchover, and failover in the OracleAS Disaster Recovery solution.

13.1.3.3 Separate OracleAS Metadata Repository for OracleAS Portal with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure (the Departmental Topology)

This topology (Figure 13-4), consists of an OracleAS Infrastructure with two OracleAS Metadata Repositories and multiple middle tiers. One OracleAS Metadata Repository is used by Oracle Identity Management components, such as Oracle Internet Directory and OracleAS Single Sign-On. All middle tiers use this OracleAS Metadata Repository for Oracle Identity Management services, as well as any additional middle tiers that might be added to this topology as it expands. The other OracleAS Metadata Repository is used for product metadata by the OracleAS Portal and OracleAS Wireless middle tier components. With two metadata repositories, this deployment can best be described as two DCM production farms.

An OracleAS Disaster Recovery standby configuration could be set up as either a symmetrical topology as described in Section 13.1.3.1, "Symmetrical Topologies - Strict Mirror of the Production Site with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure", thereby requiring two DCM standby farms be configured or as a simple asymmetric topology as described in Section 13.1.3.2, "Asymmetrical Topologies - Simple Asymmetric Standby Topology with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure", with service guaranteed requiring minimally that a single DCM standby farm be configured.

Figure 13-4 Collocated Oracle Identity Management and OracleAS Metadata Repository with a Separate OracleAS Metadata Repository

Description of "Figure 13-4 Collocated Oracle Identity Management and OracleAS Metadata Repository with a Separate OracleAS Metadata Repository"

13.1.3.4 Distributed Application OracleAS Metadata Repositories with Non Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure

The topologies in Section 13.1.3.1, "Symmetrical Topologies - Strict Mirror of the Production Site with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure", Section 13.1.3.2, "Asymmetrical Topologies - Simple Asymmetric Standby Topology with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure", and Section 13.1.3.3, "Separate OracleAS Metadata Repository for OracleAS Portal with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure (the Departmental Topology)" describe a deployment for a default database repository collocated for both the Oracle Identity Management and OracleAS Metadata Repository Infrastructure, while Section 13.1.3.3, "Separate OracleAS Metadata Repository for OracleAS Portal with Collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure (the Departmental Topology)" also describes a topology with a separate OracleAS Metadata Repository.

In a topology with distributed application OracleAS Metadata Repositories and non collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure, the Oracle Identity Management Infrastructure and one OracleAS Metadata Repository Infrastructure are installed on separate hosts, and other OracleAS Metadata Repositories are installed to reside with respective applications on different hosts. Thus, one OracleAS Metadata Repository can be the result of a deployment using a single default Infrastructure install, while one or more OracleAS Metadata Repositories can be the result of an OracleAS user using a tool, such as the OracleAS Metadata Repository Creation Assistant, to install one or more application OracleAS Metadata Repositories on one or more systems with the application data, for management or policy reasons, or both.

Figure 13-5 shows an example OracleAS Disaster Recovery solution having non collocated Oracle Identity Management and OracleAS Metadata Repository Infrastructure and distributed OracleAS Metadata Repositories.

Figure 13-5 Non-Collocated Oracle Identity Management (IM) and OracleAS Metadata Repository (MR) Infrastructure Topology with Distributed OracleAS Metadata Repositories

Description of "Figure 13-5 Non-Collocated Oracle Identity Management (IM) and OracleAS Metadata Repository (MR) Infrastructure Topology with Distributed OracleAS Metadata Repositories"

13.2.1 Planning and Assigning Hostnames

Before performing the steps to set up the physical and network hostnames, plan the physical and network hostnames you wish to use with respect to the entire OracleAS Disaster Recovery solution. The overall approach to planning and assigning hostnames must meet the following goals:

• OracleAS components in the middle tier and OracleAS Infrastructure must use the same physical hostnames in their configuration settings regardless of whether the components are in the production or standby site. In addition, you must also create a virtual hostname for the physical hostname of the OracleAS Infrastructure.

  For example, if a middle-tier component in the production site uses the name "asmid1" to reach a host in the same site, the same component in the standby site must use the same name to reach asmid1's equivalent peer in the standby site. Likewise, if the virtual hostname of the OracleAS Infrastructure on the production site uses the name "infra", the virtual hostname for the physical hostname of the OracleAS Infrastructure on the standby site must be named "infra".

• No changes to hostnames (physical, network, or virtual) are required when the standby site takes over the production role. However, a DNS switchover must be performed, see Section 13.12, "Wide Area DNS Operations" for more information.

  
  

  Note: Although the physical hostnames in the production and standby sites must remain uniform between the two sites, the resolution of these physical hostnames to the correct hosts can be different. Section 13.2.2, "Configuring Hostname Resolution" explains hostname resolution.

  
  

Figure 13-6 illustrates the process of planning and assigning hostnames.

Figure 13-6 Name Assignment Example in the Production and Standby Sites

Description of "Figure 13-6 Name Assignment Example in the Production and Standby Sites"

In Figure 13-6, two middle-tier nodes exist in the production site. The OracleAS Infrastructure can be a single node or an OracleAS Cold Failover Cluster solution (represented by a single virtual hostname and a virtual IP, as for a single node OracleAS Infrastructure). The common names in the two sites are the physical hostnames of the middle-tier nodes and the virtual hostname of the OracleAS Infrastructure. Table 13-2 details what the physical, network, and virtual hostnames are in the example:

asmid1

- 

prodmid1, standbymid1

asmid2

- 

prodmid2, standbymid2

- Foot 1 

infra

prodinfra, standbyinfra

• Cohosting non OracleAS applications

  If the hosts in the production site are running non OracleAS applications, and you wish to cohost OracleAS on the same hosts, changing the physical hostnames of these hosts may break these applications. In such a case, you can keep these hostnames in the production site and modify the physical hostnames in the standby site to match those in the production site. The non OracleAS applications can then also be installed on the standby hosts so that they can act in a standby role for these applications.

As explained in Section 1.2.1, "Terminology", physical, network, and virtual hostnames have different purposes in the OracleAS Disaster Recovery solution. They are also set up differently. The following sections provide information about how the three types of hostnames are set up.

prodmid1.oracle.com        IN     A     123.1.2.333
prodmid2.oracle.com        IN     A     123.1.2.334
prodinfra.oracle.com       IN     A     123.1.2.111
standbymid1.oracle.com     IN     A     213.2.2.443
standbymid2.oracle.com     IN     A     213.2.2.444
standbyinfra.oracle.com    IN     A     213.2.2.210

13.2.2 Configuring Hostname Resolution

In the OracleAS Disaster Recovery solution, you can configure hostname resolution in one of two ways to resolve the hostnames you planned and assigned in Section 13.2.1, "Planning and Assigning Hostnames". These are:

• Section 13.2.2.1, "Using Local Hostnaming File Resolution"

• Section 13.2.2.2, "Using DNS Resolution"

In UNIX, the order of the method of name resolution can be specified using the "hosts" parameter in the file /etc/nsswitch.conf. The following is an example of the hosts entry:

hosts:     files dns nis

In the previous statement, local hostnaming file resolution is preferred over DNS and NIS (Network Information Service) resolutions. When a hostname is required to be resolved to an IP address, the /etc/hosts file (UNIX) or C:\WINDOWS\system32\drivers\etc\hosts file is consulted first. In the event that a hostname cannot be resolved using local hostnaming resolution, DNS is used. (NIS resolution is not used for the OracleAS Disaster Recovery solution.) Refer to your UNIX system documentation to find out more about name resolution using the file /etc/nsswitch.conf.

123.1.2.333 asmid1.oracle.com asmid1
123.1.2.334 asmid2.oracle.com asmid2
123.1.2.111 infra.oracle.com infra
213.2.2.210 remoteinfra.oracle.com remoteinfra

123.1.2.334 asmid2.oracle.com asmid2
123.1.2.333 asmid1.oracle.com asmid1
123.1.2.111 infra.oracle.com infra
213.2.2.210 remoteinfra.oracle.com remoteinfra

123.1.2.111 infra.oracle.com infra
123.1.2.333 asmid1.oracle.com asmid1
123.1.2.334 asmid2.oracle.com asmid2
213.2.2.210 remoteinfra.oracle.com remoteinfra

213.2.2.443 asmid1.oracle.com asmid1
213.2.2.444 asmid2.oracle.com asmid2
213.2.2.210 infra.oracle.com infra
123.1.2.111 remoteinfra.oracle.com remoteinfra

213.2.2.444 asmid2.oracle.com asmid2
213.2.2.443 asmid1.oracle.com asmid1
213.2.2.210 infra.oracle.com infra
123.1.2.111 remoteinfra.oracle.com remoteinfra

213.2.2.210 infra.oracle.com infra
213.2.2.443 asmid1.oracle.com asmid1
213.2.2.444 asmid2.oracle.com asmid2
123.1.2.111 remoteinfra.oracle.com remoteinfra

13.2.2.2 Using DNS Resolution

To set up the OracleAS Disaster Recovery solution to use DNS hostname resolution, you must set up site-specific DNS servers in the production and standby sites in addition to the overall corporate DNS servers (usually more than one DNS server exists in a corporate network for redundancy). Figure 13-7 provides an overview of this setup.

See Also: Chapter 17, "Setting Up a DNS Server" for instructions on how to set up a DNS server in a UNIX environment.

Figure 13-7 DNS Resolution Topology Overview

Description of "Figure 13-7 DNS Resolution Topology Overview"

For the topology in Figure 13-7 to work, the following requirements and assumptions must be made:

• The DNS servers for the production and standby sites must not be aware of each other. They make non authoritative lookup requests to the overall corporate DNS servers if they fail to resolve any hostnames within their specific sites.

• The production site and standby site DNS servers must contain entries for middle-tier physical hostnames and OracleAS Infrastructure virtual hostnames. Each DNS server contains entries of only the hostnames within its own site. The sites have a common domain name that is different from that of the overall corporate domain name.

• The overall corporate DNS servers contain network hostname entries for the middle-tier hosts and OracleAS Infrastructure hosts of both production and standby sites.

• In UNIX, the /etc/hosts file in each host does not contain entries for the physical, network, or virtual hostnames of any host in either the production or standby site. In Windows, this applies to the file C:\WINDOWS\system32\drivers\etc\hosts.

To set up the OracleAS Disaster Recovery solution for DNS resolution, follow these steps:

1. Configure each of the overall corporate DNS servers with the network hostnames of all the hosts in the production and standby sites. Using the example presented in Figure 13-6, the following entries are made:

   prodmid1.oracle.com      IN    A    123.1.2.333
   prodmid2.oracle.com      IN    A    123.1.2.334
   prodinfra.oracle.com     IN    A    123.1.2.111
   standbymid1.oracle.com   IN    A    213.2.2.443
   standbymid2.oracle.com   IN    A    213.2.2.444
   standbyinfra.oracle.com  IN    A    213.2.2.210
   
   

2. For each site, production and standby, create a unique DNS zone by configuring a DNS server as follows:

   1. Select a unique domain name to use for the two sites that is different from the corporate domain name. As an example, use the name "oracleas" for the domain name for the two sites in Figure 13-6. The high level corporate domain name is oracle.com.

   2. Configure the DNS server in each site to point to the overall corporate DNS servers for unresolved requests.

   3. Populate the DNS servers in each site with the physical hostnames of each middle-tier host and the virtual hostname of each OracleAS Infrastructure host. Include the domain name selected in the previous step.

      For the example in Figure 13-6, the entries are as follows:

      For the DNS server on the production site:

      asmid1.oracleas      IN    A    123.1.2.333
      asmid2.oracleas      IN    A    123.1.2.334
      infra.oracleas       IN    A    123.1.2.111
      
      

      For the DNS server on the standby site:

      asmid1.oracleas      IN    A    213.2.2.443
      asmid2.oracleas      IN    A    213.2.2.444
      infra.oracleas       IN    A    213.2.2.210
      
      

      
      

      Note: If you are using the OracleAS Cold Failover Cluster solution for the OracleAS Infrastructure in either site, enter the cluster's virtual hostname and virtual IP address. For example, in the previous step, infra is the virtual hostname and 123.1.2.111 is the virtual IP of the cluster in the production site. For more information on the OracleAS Cold Failover Cluster solution, see Section 6.2.2, "Active-Passive High Availability Topologies".

      
      

13.2.2.2.1 Additional DNS Server Entries for Oracle Data Guard

Because OracleAS Guard automates the use of Oracle Data Guard technology, which is used to synchronize the production and standby OracleAS Infrastructure databases, the production OracleAS Infrastructure must be able to reference the standby OracleAS Infrastructure and conversely.

For this to work, the IP address of the standby OracleAS Infrastructure host must be entered in the production site's DNS server with a hostname that is unique to the production site. Similarly, the IP address of the production OracleAS Infrastructure host must be entered in the standby site's DNS server with the same hostname. These DNS entries are required because Oracle Data Guard uses TNS Names to direct requests to the production and standby OracleAS Infrastructures. Hence, the appropriate entries must also be made to the tnsnames.ora file. Additionally, OracleAS Guard asgctl command-line commands must reference the network hostnames.

Using the example in Figure 13-6 and assuming that the selected name for the remote OracleAS Infrastructure is "remoteinfra," the entries for the DNS server in the production site are:

asmid1.oracleas        IN    A    123.1.2.333
asmid2.oracleas        IN    A    123.1.2.334
infra.oracleas         IN    A    123.1.2.111
remoteinfra.oracleas   IN    A    213.2.2.210

And, in the standby site, the DNS server entries should be as follows:

asmid1.oracleas        IN    A    213.2.2.443
asmid2.oracleas        IN    A    213.2.2.444
infra.oracleas         IN    A    213.2.2.210
remoteinfra.oracleas   IN    A    123.1.2.111

13.4.1 Overview of asgctl

The asgctl command-line utility greatly simplifies the complexity and magnitude of the steps involved in setting up and managing OracleAS Disaster Recovery. This utility provides a distributed solution that consists of a client component and a server component. The client component (OracleAS Guard client) can be installed on a system on the topology. The server component (OracleAS Guard server) is installed by default on the systems hosting the primary and standby Oracle homes that comprise the OracleAS Disaster Recovery environment.

13.4.2 OracleAS Guard Client

The OracleAS Guard client is installed on every OracleAS install type. The OracleAS Guard client attempts to open and maintain a connection to the OracleAS Guard server.

The OracleAS Guard client provides an asgctl command-line interface (CLI) (see Chapter 14, "OracleAS Guard asgctl Command-line Reference") consisting of a set of commands to perform administrative tasks described in Section 13.4.4, "asgctl Operations".

13.4.3 OracleAS Guard Server

The OracleAS Guard server is a distributed server (installed by default) that runs on all the systems in an OracleAS Disaster Recovery configuration. The OracleAS Guard client maintains an active connection to the OracleAS Guard server on one system that has network connectivity in the OracleAS Disaster Recovery configuration. This coordinating server communicates to the OracleAS Guard servers on the other systems in the OracleAS Disaster Recovery configuration as necessary to complete processing during standby site cloning, instantiation, synchronization, verification, switchover, and failover operations. The OracleAS Guard server carries out asgctl commands issued directly by the OracleAS Guard client or issued on behalf of the OracleAS Guard client by another OracleAS Guard server in the network for the client session. The steps to complete an operation will execute throughout all systems in both the production and standby topologies. Most operational steps will be executed either in parallel or sequentially (as required) on these systems throughout the OracleAS Disaster Recovery configuration by the OracleAS Guard server.

13.4.4 asgctl Operations

Major asgctl operations using the asgctl commands belong in the following categories of operations:

• Authentication -- Identify the OracleAS Infrastructure database on the primary topology (set primary database command). If there are topologies with multiple Infrastructures, each must be identified using this command prior to performing an operation involving both production and standby topologies.

  Identify the new OracleAS Infrastructure database on the standby topology (set new primary database command) prior to a failover operation.

  Set the credentials (set asg credentials command) used to authenticate the OracleAS Guard client connections to OracleAS Guard servers and the connections between OracleAS Guard servers to a specific host. See the set asg credentials command for an example, and see Section 13.14.1.1, "Setting asgctl Credentials" for more information.

  When OracleAS Guard discovers the topology (discover topology command), it requires you provide Oracle Internet Directory authentication credentials (Oracle Internet Directory password) in order to query Oracle Internet Directory to obtain instance information for the production site.

• Discover the topology -- Discover (discover topology command) by querying Oracle Internet Directory all instances within the topology that share the same Oracle Internet Directory for a production site and generate a topology XML file that describes the topology and replicates this file to all instances in the topology. See Section 13.6, "Discovering, Dumping, and Verifying the Topology" for more information.

  The command discover topology within farm discovers the topology using OPMN at a production site for special cases where Oracle Internet Directory is not available.

• Standby site cloning -- Clone a single production instance to a standby system (clone instance command) or clone two or more production instances to standby systems (clone topology command). See Section 13.8, "OracleAS Guard Operations -- Standby Site Cloning of One or More Production Instances to a Standby System" for more information. The standby site cloning operation eliminates the task of having to install these Oracle instances on the standby middle tier systems and perform an instantiate operation.

• Standby site instantiation -- Creates the disaster recovery environment. It establishes the relationship between standby and production instances, mirrors the configuration, create the standby Infrastructure, then synchronize the standby site with the primary site (instantiate topology command). See Section 13.9.1, "Standby Instantiation" for more information.

• Standby site synchronization -- Applies database redo logs for OracleAS Infrastructures to the standby site in conjunction with synchronizing external configuration files across the topology (sync topology command). See Section 13.9.2, "Standby Synchronization" for more information.

• Switchover -- Switch from the production site to the standby site after the standby site is synchronized with the production site with the application of the database redo logs (switchover topology command). See Section 13.10.1.1, "Scheduled Outages" for more information.

• Failover -- Make the standby site the production site after restoring configuration files and restoring the OracleAS server environment to the point of the last successful sync operation (failover command). See Section 13.10.1.2, "Unplanned Outages" for more information.

• Verification -- Validate that the primary topology is running and the configuration is valid (verify topology command) or if a standby topology is specified, compare the primary topology to which the local host system is a member with the standby topology to validate that they are consistent with one another and conform to the requirements for OracleAS Disaster Recovery. See Section 13.11.1, "Verifying the Topology" for more information.

• Using a policy file -- Used as a filter to filter out unnecessary instances for supporting asymmetric topologies. The dump policies command writes detailed, default policy information to respective XML formatted files for a select set of asgctl commands. You can then edit each respective XML policy file and use it in the using policy <file> parameter with any one of these select set of asgctl commands: dump topology, verify topology, clone topology, failover, instantiate topology, switchover topology, and sync topology to define by instance the domain of execution operations that are permitted for each of these asgctl commands. Each instance list entry in an XML policy file logically tags a production-standby peer combination with a particular attribute that defines the success requirement for its successful operation. For example, you may want to omit a node in a symmetric topology while performing one of the operations previously mentioned. Use the policy file to specify the node to be ignored. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

• Instance management -- Enables you to shut down (shutdown topology command) and start up the topology (startup topology command).

• Troubleshooting -- Uses the dump topology command to write detailed information about the topology to the screen or to a file. Lets you determine the current operations that are running (show operation command) and stop any operations that need to be halted (stop operation command).

Table 13-4 describes the OracleAS Disaster Recovery production and standby site environment before and after performing an asgctl clone, instantiate, sync, failover, and switchover operation.

13.4.5 OracleAS Guard Integration with OPMN

A typical Oracle Application Server site has multiple farms. OracleAS Guard server and its ias-component DSA process is not started by default by OPMN because it is only necessary in the context of disaster recovery sites. You must start this ias-component DSA process in all Oracle homes as described later in this section. To check the status of this component and determine if the component is running, run the following opmnctl command on each system in your topology:

On UNIX systems
> <ORACLE HOME>/opmn/bin/opmnctl status

On windows systems
> <ORACLE HOME>\opmn\bin\opmnctl status

Because there is no way an OracleAS Guard client nor OPMN on the production site can start OracleAS Guard services on the standby site, OracleAS Guard must be started directly using opmnctl on the Infrastructure node in the standby topology. Connect to a node and run the following OPMN command on UNIX systems:

> <ORACLE HOME>/opmn/bin/opmnctl startproc ias-component=DSA

On Windows systems, issue the following OPMN command to start OracleAS Guard if your Oracle home is located on drive C:.

C:\<ORACLE HOME>\opmn\bin\opmnctl startproc ias-component=DSA

After the OracleAS Guard server is started it is non transient, while the remaining OracleAS Guard servers in the standby topology are transient servers. This configuration allows cross-topology communication.

13.4.6 Supported OracleAS Disaster Recovery Configurations

For OracleAS 10g release (10.1.2), OracleAS Guard supports not only the default OracleAS Infrastructure configuration supported on Oracle Application Server Cold Failover Cluster and single instance, but also the topologies described in Section 13.1.3, "Supported Topologies".

13.4.7 Configuring OracleAS Guard and Other Relevant Information

By default, OracleAS Guard and asgctl, the command-line utility for OracleAS Guard, are installed for all install types with the following default configuration information, which includes:

• The following OracleAS Guard parameters are configurable. The value is described and the default value is indicated. The OracleAS Guard readme.txt file in the <ORACLE_HOME>\dsa\doc directory also lists these OracleAS Guard parameters that are configurable.

  • port - the TCP/IP port for OracleAS Guard server and client. OracleAS Guard uses a default port (port) number of 7890; for example, port=7890. If there is a second Oracle home installed on a system, this second Oracle home must have a different OracleAS Guard port number, usually incremented by one, for example, port=7891, and so on.

    Value: integer, any valid TCP/IP port number. Default is 7890.

  • exec_timeout_secs - timeout value for executing operating system command.

    Value: integer, number of seconds. Default is 60 seconds.

  • trace_flags - trace flags to be turned on.

    Value: string list, separated by ",". Default is none.

  • backup_mode - indicates whether to perform a full or incremental backup.

    Value: String, "full" or "incremental". Default is "incremental".

  • backup_path - the backup directory path to be used by OracleAS Guard server.

    Value: string, a directory path. Default is <ORACLE_HOME>/dsa/backup.

  • ha_path - the High Availability directory path where the backup scripts are located.

    Value: string, a directory path. Default is <ORACLE_HOME>/backup_restore.

  • port.<host> - the TCP/IP port for a given host.

    Value: integer, any valid TCP/IP port number.

    
    

    Note: If the port number must be changed for some reason (it must be unique for each OracleAS Guard server in each Oracle home on a machine, which is automatically handled during installation), you can change its value in the <ORACLE_HOME>/dsa/dsa.conf file. Then, stop the OracleAS Guard server(<ORACLE_HOME>/opmn/bin/opmnctl stopproc ias-component=DSA) and start the OracleAS Guard server (<ORACLE_HOME>/opmn/bin/opmnctl startproc ias-component=DSA) to activate the change. See Section 13.4.5, "OracleAS Guard Integration with OPMN") for more information.

    
    

  • copyfile_buffersize - the buffer size for copy file operation, in kilobytes.

    Value: integer, maximum buffer size is 500K.

  • server_inactive_timeout - the number of seconds server will wait before shutting down due to inactivity.

    Value: integer, number of seconds. Default value is 600 seconds (10 minutes).

  • inventory_location - the alternative Oracle Inventory location

    Value: string, a directory path.

• OracleAS Guard command-line utility asgctl is installed in the <ORACLE_HOME>/dsa/bin directory on UNIX systems and <ORACLE_HOME>\dsa\bin directory on Windows systems on all nodes in the topology production and standby topologies.

• OracleAS Guard starts up the OracleAS component services across the production topology.

• The OracleAS Guard operation status information for a topology (from either an asgctl show operation full or show operation history command) remains available for the life of the current OracleAS Guard client asgctl connect session only. When the OracleAS Guard client disconnects from the OracleAS Guard server, this topology's operation history information becomes unavailable.

• After you start an asgctl operation, you cannot run another asgctl command on the same OracleAS Guard server until the previous command that is running completes or is forced to stop (see the asgctl stop operation command for more information.) In addition, you cannot run an asgctl operation in background and then quit or exit the asgctl utility.

13.8.1 Cloning a Single Production Instance to a Standby System

As an example, you want to add a production instance to a standby system. The clone instance operation eliminates the task of having to install the Oracle instance on the standby middle-tier system and then perform an instantiate operation.

The production instance to be cloned cannot exist on the standby system.

The following are prerequisites for performing the clone instance operation to the standby site system:

• The OracleAS Guard standalone kit must be installed on the standby system.

• Backup and Restore must be installed in the OracleAS Guard home on the standby system.

• A Java development kit with its jar utility must be installed on the standby system.

• For Windows systems, the services kit (sc.exe) must be installed on the standby system.

The basic procedure consists of the following pre-clone, clone, and post-clone steps.

Pre-Clone Steps

For each instance on the production and standby sites, perform the following steps:

1. Log in as su - root on UNIX systems or as Administrator on Windows systems.

2. CD to the instance home.

3. Shut down any OracleAS Guard servers.

   On UNIX systems:
   > <ORACLE HOME>/opmn/bin/opmnctl stopproc ias-component=DSA
   
   On Windows systems:
   C:\<ORACLE HOME>\opmn\bin\opmnctl stopproc ias-component=DSA
   
   

4. On UNIX systems only: make sure dsaServer.sh in <ORACLE_HOME>/dsa/bin is executable by everyone. If it is not, record the permission, then change the executable permission by issuing the following command:

   chmod +x dsaServer.sh
   chmod u+x asgexec
   
   

5. Invoke asgctl and issue the startup command.

   >On UNIX systems from the <ORACLE_HOME>/dsa/bin directory
   > asgctl.sh startup
   
   On Windows systems fom the <ORACLE_HOME>\dsa\bin directory
   C:\> asgctl startup 
   
   

6. Log out as root on UNIX systems.

Clone Steps

From any instance on the production site, perform the following steps:

1. Log in as user (non root user on UNIX systems).

2. CD to the production instance home.

3. Invoke asgctl and run the clone instance command to clone the instance to the standby topology host system.

   > asgctl.sh
   Application Server Guard: Release 10.1.2.0.2
   (c) Copyright 2004, 2005 Oracle Corporation. All rights reserved
   ASGCTL> connect asg prodinfra ias_admin/adminpwd
   Successfully connected to prodinfra:7890
   ASGCTL> set primary database sys/testpwd@asdb
   Checking connection to database asdb
   ASGCTL> clone instance portal_2 to asmid2
   Generating default policy for this operation
   .
   .
   .
   ASGCTL> disconnect
   ASGCTL> exit
   >
   
   

4. Log out of the system.

Post-Clone Steps

For the instance on the production and standby sites, perform the following steps:

1. Log in as su - root on UNIX systems or as Administrator on Windows systems.

2. CD to the instance home.

   • On the production site systems, CD to the instance home.

   • On the standby site systems, CD to the OracleAS Guard standalone home.

3. Perform an asgctl shutdown command.

   >On UNIX systems from the <ORACLE_HOME>/dsa/bin directory
   > asgctl.sh shutdown
   
   On Windows systems fom the <ORACLE_HOME>\dsa\bin directory
   C:\> asgctl shutdown
   
   

4. Log out as root on UNIX systems.

5. On UNIX systems only: Restore the permission for dsaServer.sh to what you recorded it as in Pre-Clone Step 4.

6. On the standby site only, CD to the newly cloned home.

7. Start up OracleAS Guard using the following opmnctl command:

   On Unix systems:
   > <ORACLE HOME>/opmn/bin/opmnctl startproc ias-component=DSA
   
   On Windows systems:
   C:\<ORACLE HOME>\opmn\bin\opmnctl startproc ias-component=DSA
   
   

The last step completes the cloning instance operation and brings the systems back to where they were before you started the operation. At this point, you could invoke asgctl, connect to a production system, discover the topology, and then perform a verify operation to determine if the production and standby topologies were valid and consistent with one another as you would expect them to be.

13.8.2 Cloning Multiple Production Instances to Standby Systems

As an example, you want to add two or more production instances to a standby middle-tier host system. The clone topology operation eliminates the task of having to install these Oracle instances on the standby middle-tier systems and then perform an instantiate operation.

As part of the clone topology operation, the production instances are cloned and the OracleAS Metadata Repository is instantiated. However, for a OracleAS Metadata Repository configuration created using OracleAS Metadata Repository Creation Assistant, no instantiate operation is performed.

The production instances to be cloned cannot exist on the standby systems.

If you want to use a policy file, edit and use the clone policy file (clone_policy.xml). Specify this file in the using policy <file> parameter of the clone topology command to clone a standby topology for only those instances specified accordingly. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

The following are prerequisites for performing the clone topology operation to standby site systems:

• The OracleAS Guard standalone kit must be installed on each standby system.

• Backup and Restore must be installed on each OracleAS Guard home on each standby system.

• A Java development kit with its jar utility must be installed on each standby system.

• For Windows systems only, the services kit (sc.exe) must be installed on each standby system.

The basic procedure consists of the following pre-clone, clone, and post-clone steps.

Pre-Clone Steps

For each instance on the production and standby sites, perform the following steps:

1. Log in as su - root on UNIX systems or as Administrator on Windows systems.

2. CD to the instance home.

3. Shut down any OracleAS Guard servers.

   On UNIX systems:
   > <ORACLE HOME>/opmn/bin/opmnctl stopproc ias-component=DSA
   
   On Windows systems:
   C:\<ORACLE HOME>\opmn\bin\opmnctl stopproc ias-component=DSA
   
   

4. On UNIX systems only: make sure dsaServer.sh in <ORACLE_HOME>/dsa/bin is executable by everyone. If it is not, record the permission, then change the executable permission by issuing the following command:

   chmod +x dsaServer.sh
   chmod u+x asgexec
   
   

5. Invoke asgctl and issue the startup command.

   >On UNIX systems from the <ORACLE_HOME>/dsa/bin directory
   > asgctl.sh startup
   
   On Windows systems fom the <ORACLE_HOME>\dsa\bin directory
   C:\> asgctl startup 
   
   

6. Log out as root on UNIX systems.

Clone Steps

From any instance on the production site, perform the following steps:

1. Log in as user (non root user on UNIX systems).

2. CD to any production instance home.

3. Invoke asgctl and run the clone topology command to clone the topology to the standby topology host system.

   > asgctl.sh
   Application Server Guard: Release 10.1.2.0.2
   (c) Copyright 2004, 2005 Oracle Corporation. All rights reserved
   ASGCTL> connect asg prodinfra ias_admin/adminpwd
   Successfully connected to prodinfra:7890
   ASGCTL> set primary database sys/testpwd@asdb
   Checking connection to database asdb
   ASGCTL> 
   # Command to use if you are using a policy file where <file>
   # is the full path and file spec of the clone policy file.
   ASGCTL> clone topology to standbyinfra using policy <file>
   Generating default policy for this operation
   .
   .
   .
   ASGCTL> disconnect
   ASGCTL> exit
   >
   
   

4. Log out from the system.

Post-Clone Steps

For each instance on the production and standby sites, perform the following steps:

1. Log in as su - root on UNIX systems or as Administrator on Windows systems.

2. CD to the instance home.

   • On the production site systems, CD to the instance home.

   • On the standby site systems, CD to the OracleAS Guard standalone home.

3. Perform an asgctl shutdown command.

   >On UNIX systems from the <ORACLE_HOME>/dsa/bin directory
   > asgctl.sh shutdown
   
   On Windows systems fom the <ORACLE_HOME>\dsa\bin directory
   C:\> asgctl shutdown
   
   

4. Log out as root on UNIX systems.

5. On UNIX systems only: Restore the permission for dsaServer.sh to what you recorded it as in Pre-Clone Step 4.

6. On the standby site only, CD to the newly cloned homes.

7. Start up OracleAS Guard using the following opmnctl command:

   On Unix systems:
   > <ORACLE HOME>/opmn/bin/opmnctl startproc ias-component=DSA
   
   On Windows systems:
   C:\<ORACLE HOME>\opmn\bin\opmnctl startproc ias-component=DSA
   
   

The last step completes the cloning topology operation and brings the systems back to where they were before you started the operation. At this point, you could invoke asgctl, connect to a production system, discover the topology, and then perform a verify operation to determine if the production and standby topologies were valid and consistent with one another, as you would expect them to be.

13.8.3 Cloning When There Are Multiple Instances on One System

When you are cloning a topology and there are two or more instances on a production system, multiple DSA ports are configured, with one DSA port uniquely configured for each production instance in each respective dsa.conf file; however, there is only one configured DSA port on the standby site configured for the single standby instance. How does OracleAS Guard resolve this problem?

To answer this question, lets consider the following example. Assume that on the production site host1 there are two production instances, instance_1 using DSA port 7890 and instance_2 using DSA port 7891. Then, lets assume that on the standby site host2, the OracleAS Guard standalone kit is installed there and is using DSA port 7890.

By default, instance_2 on production site host1 will try to connect to the standby site host2 using DSA port 7891. Because there is no standby OracleAS Guard server on the standby site using DSA port 7891, the dsa.conf file on production site host1 for instance_2 needs an entry in its dsa.conf file to resolve to DSA port 7890 before performing the clone operation as follows:

port.host2 = 7890

Making this entry in the instance_2 dsa.conf file must precede the first pre-clone step (see Pre-Clone Steps in Section 13.8.2).

Then, after the clone operation completes, immediately following step 2 (see Post-Clone Steps in Section 13.8.2), this edited entry must be removed from the dsa.conf file for instance_2 and the OracleAS Guard server stopped (see Step 3) and restarted (see Step 7) on production site host1 for instance_2.

13.9.1 Standby Instantiation

The standby instantiation operation performs a number of operations to set up and maintain a logical mirror of the production site at the standby site. OracleAS Guard is used to coordinate the distributed operations across the production and standby sites to ensure the disaster recovery functionality is enabled. The setup operations are:

• Uses a previous topology file created by performing a discovery topology operation.

• Verifies the topology definitions to ensure they comply with the rules of the OracleAS Disaster Recovery environment.

• Configures Oracle Data Guard to maintain the OracleAS Disaster Recovery environment for the database repository.

• Mirrors the configuration information of all the Oracle homes in the OracleAS topology to the corresponding Oracle home at the standby site.

• If you want to use a policy file, edit and use the instantiate policy file (instantiate_policy.xml). Specify this file in the using policy <file> parameter of the instantiate topology command to instantiate a standby topology for only those instances specified accordingly. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

• Reports any errors found for correction.

The procedure to perform a standby instantiation operation uses the following example, which assumes that you have invoked the OracleAS Guard client and performed a discover topology command to create a topology file.

See Section 14.2.1.1, "Special Considerations for Running Instantiate and Failover Operations in CFC Environments" if you have an OracleAS Disaster Recovery configuration in a CFC environment and are about to perform an instantiate operation.

1. Connect to the OracleAS Guard server.

   ASGCTL > connect asg prodinfra ias_admin/<adminpwd>
   Successfully connected to prodinfra:7890 
   ASGCTL>
   
   

2. Specify the primary OracleAS Metadata Repository database. See Section 13.13.1.2, "Specifying the Primary Database" for more information. If you have multiple OracleAS Metadata Repositories in your topology, you must authenticate each one using the set primary database command.

   ASGCTL> set primary database sys/testpwd@asdb
   
   

3. Dump the policies (dump policies command), then edit and use the verify policy file (verify_policy.xml) and the instantiate policy file (instantiate_policy.xml) to specify the success requirement attribute for each instance in the file. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

   ASGCTL> dump policies
   Generating default policy for this operation
   Creating policy files on local host in directory "/private1/OraHome2/asr1012/dsa/conf/"
   
   

4. Verify the topology. The network hostname standbyinfra is used.

   ASGCTL> verify topology with standbyinfra 
   
   

5. Instantiate the topology at the secondary site. The network hostname standbyinfra is used. This command assumes that all Oracle homes have been installed using Oracle installer software. Specify the using policy <file> parameter where <file> represents the path and file specification for the instantiate_policy.xml file.

   ASGCTL> instantiate topology to standbyinfra using policy <file>
   
   

Whenever a standby instantiation is performed using the asgctl instantiate topology command a synchronization operation is also performed. Thus, you do not need to perform another synchronization operation immediately following the instantiation operation. If a period of time had passed following an instantiate operation, ensure that both the primary and standby sites are consistent. Then, perform a sync topology operation to ensure any changes that occurred on the primary site are applied to the secondary site.

13.9.2 Standby Synchronization

The OracleAS Guard synchronization operation synchronizes the standby site with the primary site to ensure that the two sites are logically consistent. This operation is necessary whenever any of the following circumstances exist:

• Deploy a new application or redeploy an existing application - Both the deployment of a new application and the redeployment of an existing application require changes to schema-based information in the metadata repository as well as component configuration information distributed among the Oracle homes in an OracleAS topology. This information has to be uniformly deployed at the standby site.

• Configuration changes - Specific changes, small to large, to a configuration, must be reflected at the standby site.

• User Provisioning - The default Infrastructure installation maintains the database for Oracle Internet Directory. As new users are added to the database, they should be synchronized to the standby site on a schedule that fulfills the business availability requirements.

• Periodic full synchronization - By default, the synchronization operations synchronizes only the pieces of configuration that have changed since the last synchronization operation. During test cycles or occasional complex configuration changes, administrators may want to fully refresh of their configuration information to the standby site to ensure mirroring of these changes.

You can specify a full or incremental synchronization. By default, an incremental synchronization is performed, which offers the best performance. However, in the following three circumstances a full synchronization should be specified:

• When you want to force a full synchronization to happen for some reason, such as synchronizing the standby site completely with the primary site.

• When you know there are many transactional changes over a short period of time on the primary site that must be synchronized with the secondary site.

• When you know that there is a large accumulation of transactional changes over a long period of time on the primary site that must be synchronized with the secondary site.

As part of the synchronization operation, a verify operation is performed to ensure the required OracleAS Disaster Recovery environment is maintained. Additionally, if new OracleAS instances are installed into the OracleAS topology, OracleAS Guard will discover these installations.

If you want to use a policy file, edit and use the synchronization policy file (sync_policy.xml). Specify this file in the using policy <file> parameter of the sync topology command for synchronizing a standby topology for only those instances specified accordingly. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

The following example assumes that you have invoked the OracleAS Guard client and performed a discover topology command to create a topology file.

The procedure to perform standby synchronization is as follows:

1. Connect to the OracleAS Guard server.

   ASGCTL > connect asg prodinfra ias_admin/<adminpwd>
   Successfully connected to prodinfra:7890 
   ASGCTL>
   
   

2. Specify the primary database. See Section 13.13.1.2, "Specifying the Primary Database" for more information.

   ASGCTL> set primary database sys/testpwd@asdb
   
   

3. Synchronize the secondary site with the primary site.

   ASGCTL> sync topology to standbyinfra
   
   

13.10.1 Outages

Outages fall into two categories scheduled and unplanned.

The following subsections describe these outages.

13.11.1 Verifying the Topology

To validate that the primary topology is running and the configuration is valid, enter the following asgctl command at the asgctl prompt.

ASGCTL> connect asg ias_admin/iastest2
Successfully connected to prodinfra:7890
ASGCTL> discover topology oidpassword=oidpwd
ASGCTL> verify topology
Generating default policy for this operation
prodinfra:7890
     HA directory exists for instance asr1012.infra.us.oracle.com
asmid2:7890
     HA directory exists for instance asmid2.asmid2.us.oracle.com
asmid1:7890
     HA directory exists for instance asmid1.asmid1.us.oracle.com
ASGCTL>

If you want to use a policy file, edit and use the verify policy file (verify_policy.xml) to specify the success requirement attribute for each instance in the file. Then specify the using policy <file> parameter in the verify command where <file> represents the path and file specification for the verify_policy.xml file. See Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information.

To compare a primary topology to which the local host is a member with a standby topology and ensure that they are consistent with one another and that both topologies conform to OracleAS Disaster Recovery requirements, enter the following asgctl command at the asgctl prompt and specify the name of the standby host system.

ASGCTL> dump policies
Generating default policy for this operation
Creating policy files on local host in directory "/private1/OraHome2/asr1012/dsa/conf/"

ASGCTL> verify topology with standbyinfra
Generating default policy for this operation
prodinfra:7890
     HA directory exists for instance asr1012.infra.us.oracle.com
asmid2:7890
     HA directory exists for instance asmid2.asmid2.us.oracle.com
asmid1:7890
     HA directory exists for instance asmid1.asmid1.us.oracle.com
standbyinfra:7890
     HA directory exists for instance asr1012.infra.us.oracle.com
asmid2:7890
     HA directory exists for instance asmid2.asmid2.us.oracle.com
asmid1:7890
     HA directory exists for instance asmid1.asmid1.us.oracle.com
prodinfra:7890
    Verifying that the topology is symmetrical in both primary and standby configuration
ASGCTL>

# Command to use if you want to use a policy file
# verify topology with standbyinfra using policy <file>

13.11.2 Displaying the Current Operation

To display the status of all the current operations running on all nodes of the topology to which the OracleAS Guard client is connected, enter the following asgctl command at the asgctl prompt:

ASGCTL> show operation
*************************************
OPERATION: 19
  Status: running
  Elapsed Time: 0 days, 0 hours, 0 minutes, 28 secs
  TASK: syncFarm
    TASK: backupFarm
      TASK: fileCopyRemote
      TASK: fileCopyRemote
    TASK: restoreFarm
      TASK: fileCopyLocal

13.11.3 Displaying a List of Completed Operations

To display only operations that have completed (are not running on any nodes of the topology to which the OracleAS Guard client is connected for the current session), enter the following asgctl command at the asgctl prompt:

ASGCTL> show operation history
*************************************
OPERATION: 7
  Status: success
  Elapsed Time: 0 days, 0 hours, 0 minutes, 0 secs
  TASK: getTopology
    TASK: getInstance
*************************************
OPERATION: 16
  Status: success
  Elapsed Time: 0 days, 0 hours, 0 minutes, 0 secs
  TASK: getTopology
    TASK: getInstance
*************************************
OPERATION: 19
  Status: success
  Elapsed Time: 0 days, 0 hours, 1 minutes, 55 secs
  TASK: syncFarm
    TASK: backupFarm
      TASK: fileCopyRemote
      TASK: fileCopyRemote
    TASK: restoreFarm
      TASK: fileCopyLocall

13.11.4 Stopping an Operation

To stop a specific operation that is running on the server, enter the following asgctl command at the asgctl prompt and specify the operation number you want to stop. You can obtain the operation number you want to stop by entering a asgctl show operation full command.

ASGCTL> show operation full
*************************************
OPERATION: 19
  Status: running
  Elapsed Time: 0 days, 0 hours, 0 minutes, 28 secs
  Status: running
.
.
.
ASGCTL> stop operation 19

13.11.5 Tracing Tasks

To set a trace flag for a specific event and to log the output to the asgctl log files, enter the following asgctl command at the asgctl prompt and specify the on keyword and enter the trace flags to be enabled. In this case, the trace flag DB indicates that trace information regarding processing in the Oracle Database environment will be displayed. See the set trace command for more information about other trace flags that can be enabled. See the set trace command for a complete list of the trace flags that can be set.

ASGCTL> set trace on db

13.11.6 Writing Information About the Topology to a File

To write detailed information about the topology to which the local host is connected, enter the following asgctl command at the asgctl prompt and specify the path name and file name where the detailed output is to be written. The output is the same as the display shown in the dump topology command, except it is written to a file that you can save for future reference.

ASGCTL> dump topology to c:\dump_mid_1.txt

13.11.7 Error Messages

Appendix C, "OracleAS Guard Error Messages" categorizes and describes the error messages that may appear while using the OracleAS Disaster Recovery solution.

13.12.1 Using a Wide Area Load Balancer

When a wide area load balancer (global traffic manager) is deployed in front of the production and standby sites, it provides fault detection services and performance-based routing redirection for the two sites. Additionally, the load balancer can provide authoritative DNS name server equivalent capabilities.

During normal operations, the wide area load balancer can be configured with the production site's load balancer name-to-IP mapping. When a DNS switchover is required, this mapping in the wide area load balancer is changed to map to the standby site's load balancer IP. This allows requests to be directed to the standby site, which now has the production role.

This method of DNS switchover works for both site switchover and failover. One advantage of using a wide area load balancer is that the time for a new name-to-IP mapping to take effect can be almost immediate. The downside is that an additional investment needs to be made for the wide area load balancer.

13.12.2 Manually Changing DNS Names

This method of DNS switchover involves the manual change of the name-to-IP mapping that is originally mapped to the IP address of the production site's load balancer. The mapping is changed to map to the IP address of the standby site's load balancer. Follow these instructions to perform the switchover:

1. Make a note the current time-to-live (TTL) value of the production site's load balancer mapping. This mapping is in the DNS cache and it will remain there until the TTL expires. As an example, let's assume that the TTL is 3600 seconds.

2. Modify the TTL value to a short interval (for example, 60 seconds).

3. Wait one interval of the original TTL. This is the original TTL of 3600 seconds from Step 1.

4. Ensure that the standby site is switched over to receive requests.

5. Modify the DNS mapping to resolve to the standby site's load balancer giving it the appropriate TTL value for normal operation (for example, 3600 seconds).

This method of DNS switchover works for planned site switchover operations only. The TTL value set in Step 2 should be a reasonable time period where client requests cannot be fulfilled. The modification of the TTL is effectively modifying the caching semantics of the address resolution from a long period of time to a short period. Due to the shortened caching period, an increase in DNS requests can be observed.

13.13.1 Typical OracleAS Guard Session Using asgctl

A typical OracleAS Guard session using asgctl involves the following tasks, which are described in the following subsections:

• Section 13.13.1.1, "Getting Help"

• Section 13.13.1.2, "Specifying the Primary Database"

• Section 13.13.1.3, "Discovering the Topology"

One of the advantages of supporting an asgctl command-line interface is that you can place these asgctl commands in a proper sequence in a script as described in Section 13.13.1.4, "Creating and Executing an asgctl Script" and then execute the script as described in Section 13.13.2, "Periodic Scheduling of OracleAS Guard asgctl Scripts" and Section 13.13.3, "Submitting OracleAS Guard Jobs to the Enterprise Manager Job System".

ASGCTL> help
    connect asg [<host>] [ias_admin/<password>]
    disconnect 
    exit
    quit
    clone topology to <standby_topology_host> [using policy <file>]
    clone instance <instance> to <standby_topology_host> 
    discover topology [oidhost=<host>] [oidsslport=<sslport>] [oiduser=<user>] oidpassword=<pass>
    discover topology within farm
    dump farm [to <file>]  (Deprecated)
    dump topology  [to <file>] [using policy <file>]
    dump policies
    failover [using policy <file>]
    help [<command>]
    instantiate farm to <standby_farm_host> (Deprecated)
    instantiate topology to <standby_topology_host> [using policy <file>]
    set asg credentials <host> ias_admin/<password> [for topology]
    set asg credentials <host> ias_admin/<password> [for farm] (Deprecated)
    set primary database <username>/<password>@<servicename> [pfile <filename> | spfile <filename>]
    set new primary database <username>/<password>@<servicename> [pfile <filename> | spfile <filename>]
    set noprompt
    set trace on|off <traceflags>
    sync farm to <standby_farm_host> [full | incr[emental]] (Deprecated)
    sync topology to <standby_topology_host> [full | incr[emental]] [using policy <file>]
    startup
    startup farm (Deprecated)
    startup topology
    shutdown [local]
    shutdown farm (Deprecated)
    shutdown topology
    show op[eration] [full] [[his]tory]
    show env
    stop op[eration] <op#>
    switchover farm to <standby_farm_host> (Deprecated)
    switchover topology to <standby_topology_host> [using policy <file>]
    verify farm [with <host>](Deprecated)
    verify topology [with <host>] [using policy <file>] 
ASGCTL>

ASGCTL> set primary database sys/testpwd@asdb 
Checking connection to database asdb
ASGCTL>

ASGCTL> discover topology oidpassword=oidpwd
Discovering topology on host "infra" with IP address "123.1.2.111" prodinfra:7890
    Connecting to the OID server on host "infra.us.oracle.com" using SSL port "636" and username "orcladmin"
    Getting the list of databases from OID
    Gathering database information for SID "asdb" from host "infra.us.oracle.com"
    Getting the list of instances from OID
    Gathering instance information for "asr1012.infra.us.oracle.com" from host "infra.us.oracle.com"
    Gathering instance information for "asmid1.asmid1.us.oracle.com" from host "asmid1.us.oracle.com"
    Gathering instance information for "asmid2.asmid2.us.oracle.com" from host "asmid2.us.oracle.com"
The topology has been discovered. A topology.xml file has been written to each home in the topology.

ASGCTL>

> ASGCTL @myasgctlscript.txt

13.13.2 Periodic Scheduling of OracleAS Guard asgctl Scripts

For OracleAS Guard operations that you want to run periodically, such as a periodic sync topology operation to keep the standby topology synchronized with the primary topology, you can automate the periodic running of an OracleAS Guard asgctl script.

On UNIX systems, you can set up a cron job to run the asgctl script. Copy your asgctl script into the appropriate /etc subdirectory cron.hourly, cron.daily, cron.weekly, or cron.monthly. It will run either hourly, daily, weekly, or monthly, depending on the name of the subdirectory in which you choose to place your script. Or you can edit a crontab and create an entry that will be specific for the time on which you want to run the asgctl script. See the one or two manpages on cron and crontab for more information.

On Windows systems, you can use the task scheduler or scheduled tasks from the Control Panel to choose the time to run the asgctl script, daily, weekly, monthly, or at specific times. You can also purchase additional scheduler software with more options from a third party and then set the time and frequency to run the asgctl script. See the Windows operating system help for more information.

13.13.3 Submitting OracleAS Guard Jobs to the Enterprise Manager Job System

You can use the Enterprise Manager Job System to automate the execution of any asgctl script to be run at a specified time interval or at a specified time and date, or both, in addition to setting other custom settings. To do this, access the EM Job Activity page and create your own host command job to execute your asgctl script, which is called a job task. Your job task (script) will invoke asgctl to run the asgctl commands in the order in which they are listed. After you create your OracleAS Guard job, save it to the EM Job Library, which is a repository for frequently used jobs, where it can be executed based on the custom settings and time specifications you selected. See the Enterprise Manager online help and Oracle Enterprise Manager Concepts for more information.

13.14.1 Special Considerations for Multiple OracleAS Metadata Repository Configurations

By default, the credentials you specified in the asgctl connect command are used whenever one OracleAS Guard server connects to another OracleAS Guard server. However, there may be cases where you want to do either of the following:

• Use different credentials for each system on a given site, see Section 13.14.1.1, "Setting asgctl Credentials".

• Use a common set of credentials in the standby topology that are the same as the credentials used in the primary topology, see Section 13.14.1.2, "Specifying the Primary Database".

If the credentials for any host system are not the same as those used in the asgctl connect command, you must set the OracleAS Guard credentials so that the OracleAS Guard server can connect to each host system in the configuration.

ASGCTL> set asg credentials standbyinfra ias_admin/<iasadminpwd> for topology

ASGCTL> set asg credentials standbyinfra ias_admin/<iasadminpwd>

ASGCTL> set primary database sys/testpwd@asdb
Checking connection to database asdb
ASGCTL>

13.14.2 Special Considerations for OracleAS Metadata Repository Configurations Created Using OracleAS Metadata Repository Creation Assistant

The following items are special considerations for an OracleAS Metadata Repository configuration created using OracleAS Metadata Repository Creation Assistant. These Metadata Repository databases are installed in Oracle homes with schemas containing user data. For this reason, there are some special considerations regarding OracleAS Disaster Recovery.

• On the standby site, no Metadata Repository is created by OracleAS Disaster Recovery. The System Administrator must use the OracleAS Metadata Repository Creation Assistant on the standby site and create this Metadata Repository.

• During a clone topology operation to the standby site no instantiate operation is performed on the Metadata Repository.

• Warning: Do not perform a clone operation to a standby system containing an existing Oracle home because it will get overwritten. Only perform a clone operation to a standby system where there is no Oracle home installed.

• The OracleAS Disaster Recovery solution assumes that user schemas are already configured for Oracle Data Guard.

• The OracleAS Disaster Recovery solution assumes that when using Oracle Data Guard, that the Metadata Repository is not in managed recovery mode.

• OracleAS Disaster Recovery will not change the recovery mode of Oracle Data Guard for the Metadata Repository if it is found to be in managed recovery mode; instead, OracleAS Guard will issue a warning indicating that the database is in managed recovery mode and this feature must be set differently.

• OracleAS Guard must be installed in every Oracle home on every system that is part of your production and standby topology configured for the OracleAS Disaster Recovery solution. OracleAS Guard can be installed as a standalone install a kit located on OracleAS Utility media #2. See the OracleAS Disaster Recovery installation information in Oracle Application Server Installation Guide for more information.

13.15.1 Some Special Considerations That Must Be Taken When Setting Up Some OracleAS Disaster Recovery Sites

Some special considerations must be taken when setting up OracleAS Disaster Recovery for sites that include:

• Middle-tier CFC configurations

• OracleAS Guard release 10g (9.0.4) cloning

In both cases, the instance name stored in Oracle Internet Directory is comprised of the original host name on which the production site installation was performed. In the case of an OracleAS Disaster Recovery site having a symmetric topology, the standby OracleAS Disaster Recovery peer must be installed identically to the production site and for an OracleAS Guard Release 10.1.2.0.2 clone instance or clone topology operation, the operation must be performed to mirror the configuration.

In an asymmetric standby topology, where the production site physical host does not exist at the standby site, the instance name should be filtered out of the topology using the policy file capabilities (see Section 13.7, "Dumping Policy Files and Using Policy Files With Some asgctl Commands" for more information). The hosts file of the host on which a discover topology operation is performed must map the original host name to the corresponding IP of the new host system on which it was cloned.

13.15.2 Handling ons.conf and dsa.conf Configuration Files for Asymmetric Topologies

The OracleAS Guard operation synchronizes the configuration files of the standby site with those of the production site through a backup operation on the primary site and restores them to the standby site.

For asymmetric topologies the standby site has fewer nodes, thus node name list in the ons.conf configuration file is different from the one on the production site. Therefore, the ons.conf configuration file should be excluded from the backup list of files so it is not restored on the standby site. If not excluded, the nodes listed in the ons.conf configuration file will reflect the node list of the production site and not the actual node list of the standby site. This will cause inefficiencies as OPMN will continue to ping non existing nodes.

Additionally, for asymmetric topologies the dsa.conf configuration file for an Oracle home may contain special settings on the production site that are different from the standby site. For example, the inventory_location parameter setting may be different on the standby site than it is on the primary site. In this case, you should also exclude the dsa.conf configuration file from the backup list of files so it is not restored on the standby site. Otherwise, in this example, the location of the OraInventory will not be correct on the standby site following a switchover or failover operation.

In both these cases, you should modify the Backup and Restore exclusion file as follows to exclude both of these configuration files from the backup list of files so neither is then restored to the standby site:

# Exclude Files
# - Add additional files to this list that you want to be ignored
# - during the configuration file backup/restore
c:\oracle\ias1012\opmn\conf\ons.conf
c:\oracle\ias1012\dsa\dsa.conf

If the directives set in the dsa.conf file are necessary at the site that currently functions as the production site, it may be desirable to include the dsa.conf file for synchronization and add a post switchover or failover step to edit physical site specific directives.

13.15.3 Other Special Considerations for OracleAS Disaster Recovery Environments

See Section 14.2, "Information Specific to a Small Set of OracleAS Guard Commands" for information describing some additional special considerations.