10 Cloning Application Server Middle-Tier Instances

10.3.1 Source Preparation Phase

At the source, you run the script called prepare_clone.pl. This is a Perl script that prepares the source for cloning. It takes a snapshot of the information required for cloning.

During this phase, prepare_clone.pl parses files in the source Oracle home to extract and store required values. It also creates an archive using Distributed Configuration Management (DCM), backs up required files, and runs the OracleAS Wireless clone assistant, if OracleAS Wireless is installed.

Then, you tar the Oracle home directories.

See Section 10.4.2, "Preparing the Source" for specific instructions for preparing the source instance.

10.3.2 Cloning Phases

At the destination, you extract the Oracle home from the tar file. Then, you run the script called clone.pl. This is a Perl script that performs all parts of the cloning operation automatically, calling various other utilities and Oracle Universal Installer, as needed. This script makes use of the cloning functionality in Oracle Universal Installer. When you invoke the clone.pl script, it goes through the following three phases:

1. Pre-cloning phase

   During this phase, the clone.pl script lays the groundwork necessary to ensure that cloning can be done.

2. Cloning phase

   During this phase, the clone.pl script invokes Oracle Universal Installer in clone mode with the necessary arguments to perform the Oracle Universal Installer home cloning. This re-instantiates all files (after creating backups of the existing instantiated files), sets environment variables, updates links, and so on. In other words, it repeats all actions that were performed at installation time, with the exception of the file copying.

3. Post-cloning phase

   The postinstallation configuration assistants are not designed to be run again at clone time. Consequently, some of the instance-specific configuration files that should be updated by the configuration assistants are not updated at the end of the Oracle Universal Installer cloning session. Instead, Oracle has created a set of post-cloning scripts that update those files to bring the cloned home to a working state.

   The post-cloning steps performed by the script are:

   1. Sets the new Oracle home in DCM.

   2. Updates configuration files. In this step, many configuration files that have been re-instantiated by Oracle Universal Installer during the cloning phase are restored from their backups. Those files are then updated with the new values that reflect the new environment, if needed. For example, if a file has a reference to the source Oracle home, that reference is updated to reflect the destination Oracle home.

   3. Calls the home's chgiphost command to change the host name and IP number information in the cloned home. Before calling chgiphost, the script must collect the following required information to invoke chgiphost in silent mode:

      • Source host name

      • Source IP address

      • Destination host name

      • Destination IP address

      Note that when chgiphost is run as part of cloning, it does not run all of the configuration tools that are run when chgiphost is run standalone (as when you change hostname or domain name).

   4. If the source instance is connected to Oracle Internet Directory, adds information about the clone to Oracle Internet Directory.

   5. After all the operations for cloning are completed, starts services, as well as the Application Server Control Console, to verify the success of the cloning operation.

Note that you do not need to perform each of these phases manually, because the clone.pl script takes care of all three phases automatically. The information is provided only for conceptual understanding.

See Section 10.4.3, "Cloning the Instance" for specific instructions on the tasks you do at the destination.

Files Updated During the Post-Cloning Phase

During the post-cloning phase, a set of important configuration files are restored from their backup versions and updated. Typical changes made to the files include updating environment-specific variables such as host name, Oracle home, and port numbers to their new values.

The following list shows some of the key files that are updated. Note that this is not an exhaustive list of the files being updated.

• Oracle_Home/config/ias.properties

• Oracle_Home/sysman/emd/targets.xml

• Oracle_Home/sysman/j2ee/application-deployments/em/emd/orion-web.xml

• Oracle_Home/diagnostics/config/dcmPlugins.xml

• Oracle_Home/Apache/Apache/conf/httpd.conf

• Oracle_Home/Apache/Apache/conf/mod_oc4j.conf

• Oracle_Home/Apache/Apache/conf/oracle_apache.conf

• Oracle_Home/Apache/modplsql/conf/dads.conf

• Oracle_Home/Apache/modplsql/conf/plsql.conf

• Oracle_Home/Apache/modplsql/conf/cache.conf

• Oracle_Home/Apache/oradav/conf/moddav.conf

• Oracle_Home/opmn/conf/opmn.xml

The format of the paths are shown in UNIX format. For Windows, invert the slashes.

10.4.1 Prerequisites for Cloning

For cloning, Perl 5.6.1 or higher must be installed on your system. Before running the cloning Perl scripts, you must set the PERL5LIB environment variable to the path of the Perl directory in the Oracle home. This path must be the first one listed in the variable definition. For example:

• On UNIX

  setenv PERL5LIB $Oracle_Home/perl/lib/5.6.1:$Oracle_Home/perl/lib/site_perl/5.6.1:$PERL5LIB
  
  

• On Windows:

  set PERL5LIB=%Oracle_Home%\perl\5.6.1\lib;%OH%\perl\
  5.6.1\lib\MsWin32-x86;%Oracle_Home%\perl\site\5.6.1\lib;%PERL5LIB%
  

10.4.2 Preparing the Source

To prepare the source Oracle home to be cloned, take the following steps at the source instance:

1. Change to the following directory:

   (UNIX) ORACLE_HOME/clone/bin
   (Windows) ORACLE_HOME\clone\bin
   
   

2. Run the script prepare_clone.pl. This script prepares the source to be cloned.

   The command line for the script has the following format:

   perl prepare_clone.pl [ORACLE_HOME=OH_dir]
                         [-oid_password OIDPassword] 
                         [-oid_admin OIDadmin]
                         [-silent]
                         [-debug]
                         [-help]
   
   

   Table 10-1 describes the parameters and options for the prepare_clone.pl script.

   Table 10-1 Parameters and Options for the prepare_clone.pl Script

   Parameter or Option	Description
   ORACLE_HOME	The complete directory specification for the source Oracle home. If you do not supply this parameter, the script uses the ORACLE_HOME environment variable, if it exists. If the environment variable does not exist, the script assumes that ORACLE_HOME is the directory from which the script is being run. Do not use a trailing slash (UNIX) or backslash (Windows) when specifying the Oracle home. Use the value that was provided during installation; do not use symbolic links. If ORACLE_HOME is invalid, the script exits and logs an error to standard output (STDOUT).
   -oid_password	The Oracle Internet Directory password. If the original installation required that the user specify an Oracle Internet Directory password (such as an instance connected to Oracle Identity Management), this option is required. If you do not supply the option, but one is needed, the script prompts the user for the password.
   -oid_admin	The Oracle Internet Directory admin value. If you do not specify this option, the script uses a default value of cn=orcladmin.
   -silent	Runs the script in silent mode. If the command line does not contain the required password-related options, the script exits.
   -debug	Runs the script in debug mode.
   -help	Prints the usage for the script.

   
   

3. Archive and compress the source Oracle home, using your preferred tool for archiving. For example, you can use WinZip on Windows and tar and gzip on UNIX. Make sure that the tool you are using preserves the permissions and timestamps of the files. The following example shows how to archive and compress the source on UNIX:

   cd Source_Oracle_Home 
   tar cf - * | gzip > oracleas.tar.gz
   
   

   The tar utility may issue warnings if the sticky bit is set on some files. You can safely ignore these warnings.

   Note that you should not use the jar utility to archive and compress the Oracle home.

10.4.3 Cloning the Instance

At the destination, to clone the source instance, take the following steps:

1. Copy the compressed Oracle home from the source machine to the destination machine.

2. Extract the compressed Oracle home into a directory, which will become the new Oracle home at the destination location. Use your preferred tool to extract the compressed files. For example, you can use WinZip on Windows and tar and gunzip on UNIX. Make sure that the tool you are using preserves the permissions and timestamps of the files. The following example shows how to extract the files on UNIX:

   mkdir -p Destination_Oracle_Home 
   gunzip  <  Dir_Containing_Tar/oracleas.tar.gz | tar xf -
   

   
   

   Note: Make sure that the tar and gzip/gunzip versions on the source and destination machines are compatible. You may encounter problems unzipping the archive if these versions differ.

   
   

3. Change to the following directory:

   (UNIX) ORACLE_HOME/clone/bin
   (Windows) ORACLE_HOME\clone\bin
   
   

4. Run the clone.pl script. You must have write permission to the directory containing the Oracle inventory file. (See Section 10.4.4, "Locating and Viewing Log Files" for information about the location of the Oracle inventory directory.)

   The command line for the script has the following format:

   perl clone.pl ORACLE_HOME=OH_dir
                 ORACLE_HOME_NAME=OH_Name
                 -instance Instance_Name
                 [-ias_admin_old_pwd Old_Ias_Admin_Password]
                 [-ias_admin_new_pwd New_Ias_Admin_Password]
                 [-oid_password OIDPassword]
                 [-dcm_schema_pwd DCMPassword]
                 [-lbr {true|false}]
                 [-Ostring]
                 [-silent]
                 [-debug]
                 [-help]
   
   

   Table 10-2 describes the parameters and options for the clone.pl script.

   Table 10-2 Parameters and Options for the clone.pl Script

   Parameter or Option	Description
   ORACLE_HOME	Required. The complete directory specification for the destination Oracle home. This parameter is required. If you do not supply this parameter, or if the value is invalid, the script exits. Do not use a trailing slash (UNIX) or backslash (Windows) when specifying the Oracle home.
   ORACLE_HOME_NAME	Required. The name for the destination Oracle home (the Oracle home of the clone.)
   -instance	Required. The instance name for the clone. The instance name must be different from the source instance and any other instances that use the same OracleAS Infrastructure or that are part of the same OracleAS Cluster or farm.
   -ias_admin_old_pwd	Required. The administrator password for Oracle Application Server for the source instance. If you do not supply this option and the script is not running in silent mode, the script prompts the user for the password.
   -ias_admin_new_pwd	Required. A new password for administrator for Oracle Application Server for the cloned instance. If you do not supply this option and the script is not running in silent mode, the script prompts the user for the password.
   -oid_password	The Oracle Internet Directory password. If the original installation required that the user specify an Oracle Internet Directory password (such as an instance connected to Oracle Identity Management), this option is required. If you do not supply this option, but one is needed, the script prompts the user for the password.
   -dcm_schema_pwd	The Distributed Configuration Management (DCM) schema password. If the original installation required that the user specify a DCM schema password (such as a J2EE instance connected to a OracleAS Metadata Repository, but not to Oracle Identity Management), this option is required. If you do not supply this option, but one is needed, the script prompts the user for the password. To find the current encrypted password, use the following command: SELECT password FROM dba_users WHERE username='DCM';
   -lbr	Whether or not a Load Balancing Router is used. The default is true. If a Load Balancing Router is not used, you must specify false. If you are cloning a Portal and Wireless instance and you specify false, the cloning process overwrites the configuration entries for the source instance, which are stored in the OracleAS Metadata Repository. See Section 10.5.6 and Section 10.7.2 for more information.
   -O	Specifies that any text following the option is passed to the Oracle Universal Installer command line. For example, you can use this option to pass the location of the oraparam.ini file to be used by Oracle Universal Installer, by using the following code: '-O-paramFile C:\OraHome_1\oui\oraparam.ini' Note that if the text you want to pass contains spaces or other delimiting characters, you must enclose the option in double quotation marks ("). To pass multiple parameters to Oracle Universal Installer using this option, you can either pass all parameters with a single -O option or pass individual parameters using multiple -O options.
   -silent	Runs the script in silent mode. If the command line does not contain the required password-related options, the script exits.
   -debug	Runs the script in debug mode.
   -help	Prints the usage for the script.

   
   

   For example:

   perl clone.pl ORACLE_HOME=/opt/oracle/Ora_1012_B
                 ORACLE_HOME_NAME=OH_1012_B
                 -instance Portal_B
                 -ias_admin_old_pwd my_old_ias_pass
                 -ias_admin_new_pwd my_new_ias_pass
                 -oid_password my_oidpwd
                 -dcm_schema_pwd my_DCM_pass
                 -lbr true
                 '-O-paramFile /var/opt/oracle/oui/oraparam.ini'
                 -silent
   
   

5. On UNIX, run the root.sh script in the Oracle home so that the cloned instance works properly. You must be logged in as the root user to run the script. The script is located in the cloned instance's Oracle home directory.

   For example:

   $ORACLE_HOME/root.sh
   
   

Now, the cloned instance's configuration is identical to that of the source instance. Application Server Control Console and OPMN are able to start and stop all processes in the cloned instance, including any OC4J custom instances. All applications deployed should be visible and able to run as expected.

10.4.4 Locating and Viewing Log Files

The cloning script invokes multiple tools, each of which generates its own log files. However, the following log files, which are generated by Oracle Universal Installer and the cloning scripts, are the key ones of interest for diagnostic purposes:

• Oracle_inventory/logs/cloneActionstimestamp.log: Contains a detailed log of the actions that occur during the Oracle Universal Installer part of the cloning.

• Oracle_inventory/logs/oraInstalltimestamp.err: Contains information about errors that occur when Oracle Universal Installer is running.

• Oracle_inventory/logs/oraInstalltimestamp.out: Contains other miscellaneous messages generated by Oracle Universal Installer.

• Oracle_Home/clone/logs/clonetimestamp.log: Contains a detailed log of the actions that occur during the precloning and cloning operations.

• Oracle_Home/clone/logs/errortimestamp.log: Contains information about errors that occur during the precloning and cloning operations. In addition, it contains all messages written to standard error (STDERR) by the multiple tools that are invoked by the cloning script. Depending upon the tool, some of these messages may be informational messages or error messages.

The format of the path is shown in UNIX format. For Windows, invert the slashes.

After the clone.pl script finishes running, consult these log files to get more information about the cloning process. To view the log files from Application Server Control Console, take the following steps:

1. Select Logs from the Home page.

2. In the View Logs page, select ASClone from the Available Components box. Click Move to move the selection to the Selected Components box.

3. Click Search.

   The log files are displayed in the Results table.

4. To view the log, click the log name in the Log File column.

10.4.5 Cloning Instances That Are Members of a Farm or OracleAS Cluster

You can clone a middle-tier instance that is a member of a farm or an OracleAS Cluster. However, you must remove the instance from the OracleAS Cluster before beginning the cloning operation.

The following information describes how cloning works when the source instance is a member of a farm or an OracleAS Cluster:

• File-based repository

  If the source instance is connected to a file-based repository in a separate instance, the cloned instance will be a member of the same farm as the source instance. If the source instance is connected to a file-based repository in the same instance (that is, it is the host of the file-based repository), the cloned instance will be the host of a new file-based repository.

• File-based repository and Oracle Identity Management

  If the source instance is connected to a file-based repository and Oracle Identity Management in a separate instance, the cloned instance will be a member of the same farm as the source instance and will be connected to the same Oracle Identity Management as the source instance. If the source instance is connected to a file-based repository and Oracle Identity Management in the same instance (that is, it is the host of the file-based repository), the cloned instance will be the host of a new file-based repository and will it be connected to the same Oracle Identity Management.

• OracleAS Metadata Repository

  If the source instance is connected to OracleAS Metadata Repository, the cloned instance will be connected to the same OracleAS Metadata Repository. The cloned instance will be a member of the same farm as the source instance.

• OracleAS Metadata Repository and Oracle Identity Management

  If the source instance is connected to OracleAS Metadata Repository and Oracle Identity Management, the cloned instance will be connected to the same OracleAS Metadata Repository and Oracle Identity Management. The cloned instance will be a member of the same farm as the source instance.

  One example of this scenario is a Portal and Wireless installation, which requires OracleAS Metadata Repository and Oracle Identity Management.

• No repository, but Oracle Identity Management

  If the source instance is not connected to a repository, but is connected to Oracle Identity Management in a separate instance, the cloned instance will be connected to that same Oracle Identity Management.

If you remove an instance from an OracleAS Cluster, you can add it, and the cloned instance, to the OracleAS Cluster after you complete the cloning operation.

10.5.1 General Considerations and Limitations for Cloning

For this release, you cannot clone the following:

• OracleAS Infrastructure components (Oracle Identity Management and OracleAS Metadata Repository)

• Developer Kits, including Oracle Content Management SDK (Oracle CM SDK)

• Installations that include Oracle Application Server Integration B2B, Oracle Application Server Integration InterConnect, Oracle BPEL Process Analytics

• Adapters, including Oracle Application Server Adapters and Oracle Application Server Integration InterConnect Adapters

• Installations that include Oracle Workflow

Note the following important additional considerations about cloning:

• The user may need to update the security certificates to match the new host name and to set up the certificates. See Chapter 15 for information about managing wallets and certificates.

• If an instance is part of an OracleAS Cluster, you must remove the instance from the cluster before performing the cloning operation.

  See Section 10.4.5, "Cloning Instances That Are Members of a Farm or OracleAS Cluster" for more information.

• Cloning an Oracle Application Server middle tier that is connected to an OracleAS Infrastructure will result in a new Oracle Application Server middle tier connected to the same OracleAS Infrastructure as the source instance. To join a different OracleAS Infrastructure, you can reassociate the middle tier with a different OracleAS Infrastructure, as described in Section 9.5, "Changing the Metadata Repository Used by a Middle-Tier Instance".

• If you have changed the default file permissions for configuration files, those file permissions are not preserved by cloning.

• User customizations for the following components are not preserved. The status of these components are reset to the default:

  • Log Loader

  • Oracle Application Development Framework

  • Port tunneling

  • UIX

  • XDK

• Cloning does not carry over all the dependencies of the source Oracle home, such as loadable modules or application-specific libraries to the cloned home, because cloning proceeds by copying the entire source Oracle home to the destination Oracle home. Any files outside the source Oracle home are not automatically copied. Hence, any applications that refer to files outside the source Oracle home may not work properly in the cloned home.

  You may need to copy the files manually to the destination host after extracting the archived source Oracle home, but before running the clone.pl script.

• If you created symbolic links to files or applications outside the source Oracle home (for example, to Oracle Wallet files that are not stored in the default location), you must re-create the link manually in the cloned home for your applications to work properly.

• The cloning operation generates default ports for the cloned instance. To specify other ports, you can use the staticports.ini file as described in Section 10.6.2, "Assigning Custom Ports". If you specify ports less than 1024 on UNIX, the cloned instance will not start during the cloning operation. After the cloning process is completed, you must run the root.sh script with root privileges, then start the processes.

• The cloning process does not configure a Load Balancing Router to recognize the cloned instance. If you use a Load Balancing Router in your environment, you must manually configure the Load Balancing Router, including any invalidation port.

• If a cloning operation fails, but it results in the Oracle home being registered with Oracle Inventory, you cannot use the same Oracle home in subsequent cloning operations. Either use another directory and name for the Oracle home in subsequent cloning operations or deinstall the Oracle home before attempting another cloning operation.

10.5.2 Considerations for Cloning Oracle HTTP Server

The following describes important information about cloning Oracle HTTP Server:

• All configuration information in the following files is updated:

  • Oracle_Home/Apache/Apache/conf/httpd.conf

  • Oracle_Home/Apache/Apache/conf/oracle_apache.conf

  • Oracle_Home/Apache/Apache/conf/mod_oc4j.conf

  • Oracle_Home/Apache/modplsql/conf/dads.conf

  • Oracle_Home/Apache/modplsql/conf/plsql.conf

  • Oracle_Home/Apache/modplsql/conf/cache.conf

  • Oracle_Home/Apache/oradav/conf/moddav.conf

  The format of the paths are shown in UNIX format. For Windows, invert the slashes.

  The cloning script preserves the source settings and updates these files with new environment parameters.

  Note that cloning only updates the files it knows about, that is, files that are part of the original installation. In particular, cloning does not update configuration files that the user added to the "include" list in files such as httpd.conf, oracle_apache.conf, dads.conf, plsql.conf, olap.conf, or moddav.conf. You can, however, explicitly add the "include" files to the list of files that cloning will update. See Section 10.6.3, "Updating Custom Data" for details on how to update custom settings.

• Cloning preserves all VirtualHost directives in httpd.conf. It replaces any references to the source home inside these directives. However, cloning does not change the IP numbers or port numbers that these virtual hosts listen to.

  If these values are not valid for the destination environment, then you must do one of the following:

  • Register these changes with the clone script to be updated during cloning. See Section 10.6.3, "Updating Custom Data" for more information.

  • Update them manually in httpd.conf after cloning.

• If you changed the port number in httpd.conf to use the Load Balancing Router port rather than the local Oracle HTTP Server or OracleAS Web Cache port, that change is lost after cloning. You must edit the httpd.conf file in the cloned home to change the port number to the Load Balancing Router port.

• Cloning is not supported if you are using Oracle HTTP Server based on Apache 1.3 or Apache 2.0. (These are not installed by default, but are included in the companion CD-ROM.)

10.5.3 Considerations for Cloning Oracle Application Server Containers for J2EE (OC4J)

This following lists considerations in cloning OC4J and describes which OC4J components are preserved:

• On the source Oracle home during the prepare phase of the cloning process, do not attempt to undeploy OC4J applications or perform other administrative work for OC4J applications while the prepare_clone.pl script is running.

• You must manually register files that contain environment-specific information in custom OC4J instances so that those files are updated during cloning. An example of such a file is oc4j.properties. See Section 10.3, "Understanding the Cloning Process" for details.

The following describes which OC4J components are preserved:

• All default OC4J instances are preserved.

• Custom OC4J instances that you created, as well as applications deployed in them, are preserved. However, external dependencies for these applications that are not in the Oracle home are not copied to the cloned home and will be lost.

• If the OC4J instance uses an Oracle HTTP Server instance that is not part of the source Oracle home, the cloning procedure does not update the mod_oc4j.conf file for the Oracle HTTP Server. You must manually add the instance to the mod_oc4j.conf file.

• OPMN can manage all default and custom cloned OC4J instances.

• Grid Control Console on the cloned home can manage default and custom OC4J instances.

• Data source information in data-sources.xml is preserved.

• User configurations in jms.xml, java2.policy, jazn.xml, jazn-data.xml, global-web-application.xml, and application.xml are preserved.

10.5.4 Considerations for Cloning OracleAS Web Cache

The following describes important information about cloning OracleAS Web Cache:

• Settings in webcache.xml and internal.xml are preserved. Only the Oracle home location, host name, instance name, and port numbers are updated to reflect the new environment.

• If the source OracleAS Web Cache instance is a cache cluster member, you must reconfigure the cache cluster after cloning. The cloned home contains the reference to the other nodes in the cache cluster whereas the other nodes do not know about the new instance. After cloning, you need to either add the cloned instance to the cache cluster or remove the cloned cache from the cluster. See the Oracle Application Server Web Cache Administrator's Guide for information about adding caches to a cache cluster.

• If the source OracleAS Web Cache instance is configured to forward requests to more than one origin server (application Web server or proxy server) on the same host as the source OracleAS Web Cache instance, the cloning process will fail.

  To work around this limitation, remove all but one of the origin servers from the source OracleAS Web Cache configuration, using OracleAS Web Cache administration tools. See Oracle Application Server Web Cache Administrator's Guide for information about removing the origin server.

  Note that because the cloning process does not modify definitions of origin servers on remote hosts, you do not need to remove references to those origin servers from the OracleAS Web Cache configuration.

10.5.5 Considerations for Cloning Application Server Control

The Application Server Control Console can manage the same instances that were managed at the source.

The SSL settings of the source instance are preserved, by preserving the emd.properties and emd-web-site.xml files. In other words, if the source Application Server Control Console was configured for HTTPS, then the cloned Application Server Control Console will be as well.

10.5.6 Considerations for Cloning OracleAS Portal

The following describes important considerations about cloning OracleAS Portal:

• If the cache for OracleAS Portal is located outside the Oracle home, you must manually specify the cache location in the cache.conf file. (Any files used by the source instance that are located outside the source Oracle home's directory structure are not moved to the destination location.) The cache.conf file is located in the following directory:

  (UNIX) ORACLE_HOME/Apache/modplsql/conf
  (Windows) ORACLE_HOME\Apache\modplsql\conf
  
  

  In this file, make sure that the location for the PlsqlCacheDirectory directive is valid.

• The settings in the following file are not preserved during the cloning operation:

  (UNIX) Oracle_Home/portal/pdkjava/providerGroups/iasProviders.xml
  (Windows) Oracle_Home\portal\pdkjava\providerGroups\iasProviders.xml
  
  

  If you added providerGroup entries to the file in the source instance, you must add those entries to the file in the cloned instance.

• OracleAS Portal does not support two instances connected to the same infrastructure unless they are front-ended by a Load Balancing Router.

  Because OracleAS Portal stores only a single set of configuration information, the cloning process overwrites the configuration entries stored in the repository if you do not specify that there is a Load Balancing Router. If you specify that there is a Load Balancing Router, the cloning process does not perform any configuration steps, and therefore it does not overwrite the configuration settings stored in the Portal schemas in the OracleAS Metadata Repository. See Section 10.7.2, "Cloning a Portal and Wireless Instance Front-Ended by a Load Balancing Router" for information about cloning a Portal and Wireless instance that is front-ended by a Load Balancing Router.

  Note that you can reconfigure OracleAS Portal by using the ptlconfig command, as shown in the following example:

  $ORACLE_HOME/portal/config/ptlconfig -dad portal
  
  

• When you are cloning a Portal and Wireless instance in an environment where the instances are front-ended by a Load Balancing Router, you must make sure that the cloned instance uses the same ports as the other existing instances. To guarantee that the cloning process will assign the same ports for the cloned instance, you can use the staticports.ini file, as described in Section 10.6.2, "Assigning Custom Ports".

• OracleAS Portal creates Oracle Internet Directory users only once, during the first installation of OracleAS Portal. If you clone a Portal and Wireless installation, you use the original password of the ias_admin user of the first OracleAS Portal instance associated with the OracleAS Metadata Repository when you log in to OracleAS Portal on the cloned instance. This is true even if you changed the password of the ias_admin user of the cloned instance. The password used when Portal users and groups are created in Oracle Internet Directory remains the same even if multiple instances are later connected to this infrastructure.

• Any external applications are stored in the OracleAS Single Sign-On server. If OracleAS Portal is not front-ended by a Load Balancing Router and if there is any change in the URL of an external application, you must update it manually in the external application. To do this, log in directly to the OracleAS Single Sign-On server as a user who has single sign-on Administrator privileges:

  http://SSO_host:SSO_port/pls/orasso/
  
  

  Then, select the Administer External Applications link, select the application, and update the Login URL. Alternatively, you can use the Administer External Applications link in the SSO Administration portlet in the portal. This portlet requires the same privileges as the OracleAS Single Sign-On Server.

  For more information, see the chapter "Configuring and Administering External Applications" in the Oracle Application Server Single Sign-On Administrator's Guide.

• If OracleAS Portal is not front-ended by a Load Balancing Router, you must manually update the registration URL of custom-built Web providers and provider groups that are local to the source instance. A custom-built provider or provider group is any Web provider or provider group that is not seeded by the OracleAS Portal installation. To update the registration, take the following steps:

  For a Web provider:

  1. Click Edit Registration for the provider on the Providers tab of the Navigator, under Registered Providers.

  2. Select the Connection tab, and change the host and port part of the provider registration URL from the source host and port to the cloned instance host and port. For example:

     http://clone_host:clone_port
     
     

  3. Verify that the custom-built provider works properly on the cloned instance, by using the test pages at the URL:

     http://clone_host:clone_port/webApp/providers/providerName
     
     

  For a provider group:

  1. Click Edit Registration for the provider group on the Providers tab of the Navigator, under Providers Groups.

  2. Change the host and port part of the URL from the source host and port to the cloned instance host and port. For example:

     http://clone_host:clone_port
     

10.5.7 Considerations for Cloning OracleAS Wireless

The following describes important information about cloning OracleAS Wireless:

• If the Portal and Wireless middle tier is not front-ended by a Load Balancing Router and you do not specify the -lbr option when cloning, you must reconfigure the OracleAS Portal and OracleAS Wireless integration as described in "Manually Reconfiguring the Mobile Setup" section of the Oracle Application Server Portal Configuration Guide.

10.5.8 Considerations for Cloning OracleBI Discoverer

The following describes important information about cloning OracleBI Discoverer:

• Values set by users after installation in configuration.xml are not preserved in the cloned Oracle home. The file is located at:

  (UNIX) Oracle_Home/discoverer/config/configuration.xml
  (Windows) Oracle_Home\discoverer\config\configuration.xml
  
  

  To work around this problem, reset the overwritten values by copying them from the configuration.xml file in the source Oracle home to the configuration.xml file in the cloned Oracle home.

10.5.9 Considerations for Cloning OracleAS Forms Services

The following describes important information about cloning OracleAS Forms Services:

• Cloning preserves the users configuration information in the OracleAS Forms Services default configuration files and makes the necessary modifications (such as changing the host name, re-deploying the Forms EAR file and registering the OracleAS Forms Services in the new instance with Oracle Internet Directory) to reflect the new environment.

• Cloning preserves the user settings in the following files:

  • Oracle_Home/forms/forms.conf

  • Oracle_Home/forms/server/default.env

  • Oracle_Home/forms/server/formsweb.cfg

  • Oracle_Home/forms/converter.properties

  • Oracle_Home/bin/frmcmp.sh

  • Oracle_Home/bin/frmcmp_batch.sh

  • Oracle_Home/bin/graphicsrun.sh

  • Oracle_Home/bin/frmplsqlconv.sh (UNIX)

  • Oracle_Home\bin\frmplsqlconv.bat (Windows)

  Unless otherwise specified, the format of the paths are shown in UNIX format. For Windows, invert the slashes.

• Cloning does not preserve the user settings in the following file because the Forms EAR file (formsapp.ear) is redeployed in the cloned instance:

  ORACLE_HOME/j2ee/OC4J_BI_Forms/applications/formsapp/formsweb/WEB-INF/web.xml
  
  

• If the OracleAS Forms Services executable files (.fmx) are residing on the local drive on the source machine but not in the Oracle home, then you must copy these files to the machine hosting the cloned instance. If the .fmx files are residing on a shared network drive or residing inside the Oracle home, then no action is required.

• If you have created customized configuration files to replace the default Forms configuration files (formsweb.cfg, default.env, and template HTML files) and if these files are in the Oracle home, then these files will be present in the cloned Oracle home instance. However, any references to the source Oracle home will not be automatically replaced with references to the cloned Oracle home settings. You should manually make the changes in the cloned Oracle home.

• If you have redeployed the Forms EAR file (formsapp.ear) using Enterprise Manager, you must repeat this action in the cloned Oracle home.

10.5.10 Considerations for Cloning OracleAS Reports Services

The following describes important information about cloning OracleAS Reports Services:

• Cloning preserves the settings in the following files: rwservlet.properties, rwserver.template, rwbridge.template, rwnetwork.template, network-files.conf, server-name.conf, cgicmd.dat, proxyinfo.xml, xmlpds.conf, jdbcpds.conf, and textpds.conf.

• Cloning preserves the port settings in the following files: rwbridge.template, rwnetwork.template, network-files.conf, bridge-network-files.conf, server-name.conf, tsnnames.ora.

• Cloning preserves OracleAS Reports Services server settings in opmn.xml and targets.xml.

• The following files are not cloned:

  • All files in the cache subdirectory

  • The data (.dat) files in the server subdirectory

• The name of the in-process server in the cloned instance will not be the same as the name of the source instance. The name takes the following form:

  rep_hostname_OracleHomeName
  
  

  If you have registered the in-process server in OracleAS Portal, you must manually change the name of the in-process server in OracleAS Portal and associate the new name with reports that use the in-process server.

• The user name and password in the identifier element in the server configuration files in the cloned instance are randomly generated; they are not the same as the source instance. You do not need to change them because they are used only by Enterprise Manager.

  However, if you want to change the user name and password, take the following steps:

  1. In the OracleAS Reports Services configuration file, servername.conf, modify the <identifier> element to specify the user name and password, and set the encrypted attribute to no. For example:

     <identifier confidential="yes" encrypted="no"> scott/tiger</identifier>
     
     

  2. Restart the OracleAS Reports Services server.

  3. Edit the ORACLE_HOME/sysman/emd/targets.xml.

  4. Search for the target with TYPE="oracle_repserv" and DISPLAY_NAME="Reports Server" server_name.

  5. In the entry, set the Username and Password properties to the same values as you used in the <identifier> element of the OracleAS Reports Services configuration file and set the ENCRYPTED attribute to FALSE.

  6. Restart Enterprise Manager.

     When Enterprise Manager restarts, it sets the ENCRYPTED attribute to TRUE, encrypting the password.

10.5.11 Considerations for Cloning OracleAS Forms and Reports Services

The following describes important information about cloning OracleAS Forms and Reports Services (installed from a separate CD):

• Before you run the prepare_clone.pl script, you must update the ias.properties file in the source Oracle home, changing the value of DatabaseManagedClusterSupport from false to true, as shown in the following example:

  DatabaseManagedClusterSupport=true
  
  

• On Windows, before you run the prepare_clone.pl script, you must update the ias.properties file in the source Oracle home, changing the value of installType from FRServices to Business, as shown in the following example:

  installType=Business
  
  

For information about other considerations for cloning OracleAS Forms Services and OracleAS Reports Services, see the following sections:

• Section 10.5.9, "Considerations for Cloning OracleAS Forms Services"

• Section 10.5.10, "Considerations for Cloning OracleAS Reports Services"

10.6.1 Specifying Oracle Universal Installer Parameters

When cloning an instance, you do not directly invoke Oracle Universal Installer. However, you can still pass information to Oracle Universal Installer indirectly, by specifying any Oracle Universal Installer parameters that you normally specify on the command line in the configuration file cs.properties. This file is located in the following directory:

(UNIX) ORACLE_HOME/clone/ias/config
(Windows) ORACLE_HOME\clone\ias\config

For example, to specify a nondefault location for the Oracle inventory file on UNIX, you can add the following line to the cs.properties file:

clone_command_line= -invptrloc /private/oracle/oraInst.loc

To specify multiple arguments, separate each argument with a space.

In addition, you can specify information to be passed to the Oracle Universal Installer command line by using the -Ostring option. For example, you can use this option to pass the location of the oraparam.ini file to be used by Oracle Universal Installer, by using the following code:

'-O-paramFile C:\OraHome_1\oui\oraparam.ini'

10.6.2 Assigning Custom Ports

By default, the cloning script automatically assigns free ports and lists them in the file portlist.ini. The algorithm for assigning default ports when cloning is the same as that used when installing Oracle Application Server.

When installing a new Oracle Application Server instance, you can specify the ports to be used by listing them in a staticports.ini file. Then, this file is passed as the value of a parameter when calling Oracle Universal Installer. For more information on how ports are assigned and on using the staticports.ini file, please refer to the Oracle Application Server Installation Guide for your platform.

When cloning an instance, you do not directly invoke Oracle Universal Installer. Hence, you cannot assign custom ports by specifying a staticports.ini file on the command line. However, you can still pass port information to Oracle Universal Installer indirectly, by specifying the location of the staticports.ini file in the configuration file cs.properties.

For example, if you want to use ports less than 1024, you can specify them in the staticports.ini file and specify the location of the staticports.ini file in the cs.properties file.

To assign custom ports during cloning:

1. List the port numbers in the staticports.ini file, as explained in the Oracle Application Server Installation Guide for your platform.

2. Specify the location of the staticports.ini file by adding the following line to the cs.properties file:

   clone_command_line = 
   oracle.iappserver.iapptop:szl_PortListSelect="{\"YES\", \"/tmp/staticports.ini\"}" 
   
   

The ports listed in the staticports.ini file are read during cloning, and Oracle Universal Installer assigns the port numbers accordingly.

If you specify ports less than 1024 on UNIX, the cloned instance will not start during the cloning operation. After the cloning process is completed, you must run the root.sh script with root privileges, then start the processes.

oracle.iappserver.iapptop:szl_PortListSelect="{\"NO\", \"\"}" 

10.6.3 Updating Custom Data

By default, the cloning scripts update key configuration files in the Oracle home so they contain the correct information for the destination environment. Section 10.4.3, "Cloning the Instance" contains a partial listing of the files that are updated.

You can modify the default cloning process to update custom data that is not updated by default. Information about which files to update during cloning and which entries to update in those files is contained in another set of files, which are read by the cloning scripts. By editing these files, you can:

• Preserve changes you have made to files present in the source Oracle home that are not updated by default during cloning

• Preserve changes you have made to files that are updated by default during cloning, but which are not normally preserved by the cloning process

These changes are made by a Java utility called FileFixer, which searches for specific text strings in a file by matching regular expressions, and updates them to their new values. Note that FileFixer searches for patterns one line at a time. It cannot match patterns across lines.

The changes that you can make include the following:

• Change the host name in a file.

  To do this, for the file in which the host name needs to be changed, add the path name for the file, relative to the Oracle home, to the following file:

  (UNIX) ORACLE_HOME/chgip/config/hostname.lst
  (Windows) ORACLE_HOME\chgip\config\hostname.lst
  
  

• Update all occurrences of the Oracle home in a file from the old to the new value.

  To do this, add a replace tag in the XML configuration file, fixup_script.xml.tmpl. This file is located in the following directory:

  (UNIX) ORACLE_HOME/clone/ias/config
  (Windows) ORACLE_HOME\clone\ias\config
  
  

  The value of the file_name attribute specifies the name and location of the file in which the replacement should occur. For example, the following tag updates the Oracle home value in the file dcmPlugins.xml.

  <cfw:operation>
     <replace file_name="%NEW_HOME%/sysman/config/dcmPlugins.xml">
           <cfw:replaceCommand>
              <cfw:pattern>(%OLD_HOME%)</cfw:pattern>
              <cfw:value_ref>1</cfw:value_ref>
              <cfw:new_value>%NEW_HOME%</cfw:new_value>
           </cfw:replaceCommand>
     </replace>
  </cfw:operation>
  
  

• Extract a value from file1 and use it to replace a value in file2.

  To do this, add an alter tag in fixup_script.xml.tmpl. For example, adding the following tag extracts the new HTTP port number from the portlist.ini file in the cloned home and uses it to replace the old port number in targets.xml, as shown in the following example:

  <cfw:operation>
      <alter cluster="false" alter_file_name="%NEW_HOME%/sysman/emd/targets.xml" 
  reference_file_name="%NEW_HOME%/install/portlist.ini">
         <cfw:alterCommand>
            <cfw:pattern>(Oracle HTTP Server port)([ ]*)(=)([ ]*)([0-9]*)
           </cfw:pattern>
            <cfw:value_ref>5</cfw:value_ref>
            <cfw:subst>(Property NAME=&quot;HTTPPort&quot;
                        VALUE=&quot;)([0-9]*)(&quot;)</cfw:subst>
            <cfw:subst_ref>2</cfw:subst_ref>
         </cfw:alterCommand>
      </alter>
  </cfw:operation>
  

10.7.1 Using Cloning to Expand an OracleAS Cluster

A common use of cloning is to expand the size of an OracleAS Cluster. Consider a OracleAS Cluster consisting of multiple J2EE and Web Cache middle tiers, with identical configuration and application deployment. To expand the OracleAS Cluster, you want to create a new middle-tier instance that has the same configuration as the other instances and add it to the cluster. This example assumes that the source instance is connected to a file-based repository and is a member of a farm and an OracleAS Cluster.

The strategy for expanding an OracleAS Cluster consists of the following steps:

1. Remove the source instance from the OracleAS Cluster.

   To do this, issue the following commands at the source instance:

   • On UNIX:

     cd $ORACLE_HOME/dcm/bin
     ./dcmctl leaveCluster
     
     

   • On Windows:

     cd ORACLE_HOME\dcm\bin
     dcmctl leaveCluster 
     
     

2. Prepare the source instance for cloning, using the steps described in Section 10.4.2:

   1. Perform Step 1, as described, in Section 10.4.2.

   2. Perform Step 2 in Section 10.4.2.

      In that step, use a command similar to the following:

      perl prepare_clone.pl ORACLE_HOME=/opt/oracle/Ora_1012
      
      

   3. Perform Step 3 as described, in Section 10.4.2.

3. Clone the source instance to create a new instance, using the steps described in Section 10.4.3:

   1. Perform Steps 1, 2, and 3 as described, in Section 10.4.3.

   2. Perform Step 4 in Section 10.4.3.

      In that step, you must change the name of the cloned instance, by specifying it on the command line, as shown in the following example:

      perl clone.pl ORACLE_HOME=/opt/oracle/Ora_1012_B
                    ORACLE_HOME_NAME=OH_1012_B
                    -instance J2EE_B
                    -ias_admin_old_pwd my_old_ias_pass
                    -ias_admin_new_pwd my_new_ias_pass
      
      

      In this example, the instance name for the cloned instance is J2EE_B.

4. Add the source instance to the OracleAS Cluster, using the following command:

   • On UNIX:

     cd $ORACLE_HOME/dcm/bin
     ./dcmctl joinCluster -cl cluster_name
     
     

   • On Windows:

     cd ORACLE_HOME\dcm\bin
     dcmctl joinCluster -cl cluster_name
     
     

5. Add the cloned instance to the OracleAS Cluster, using the following command:

   • On UNIX:

     cd $ORACLE_HOME/dcm/bin
     ./dcmctl joinCluster -cl cluster_name
     
     

   • On Windows:

     cd ORACLE_HOME\dcm\bin
     dcmctl joinCluster -cl cluster_name
     
     

The cloned instance will be a member of the same farm as the source instance.

10.7.2 Cloning a Portal and Wireless Instance Front-Ended by a Load Balancing Router

Consider an environment where you have configured a Portal and Wireless middle-tier instance to be front-ended by a Load Balancing Router. Now, you want to add more instances with identical configuration and application deployment. The source instance is connected to OracleAS Infrastructure (Oracle Identity Management and OracleAS Metadata Repository). You want the new instances to be connected to the same OracleAS Infrastructure as the source instance and the same farm as the source instance.

Before you start the cloning, note the following:

• Make sure that the source instance is configured to work with a Load Balancing Router. See "Configuring Multiple Middle-Tiers with a Load Balancing Router" in the Oracle Application Server Portal Configuration Guide for more information.

• Review the limitations and considerations listed in Section 10.5, "Considerations and Limitations for Cloning", especially those in Section 10.5.2, Section 10.5.4, and Section 10.5.6.

• You must use the same port numbers for the cloned instance as for the source instance. To review the port numbers used by the source instance, navigate to the Application Server Control Console Home page for the source instance and click Ports. To guarantee that the cloning process will assign the same ports for the cloned instance, you can use the staticports.ini file, as described in Section 10.6.2.

The strategy for cloning a Portal and Wireless middle tier connected to OracleAS Infrastructure and configured with a Load Balancing Router consists of the following steps:

1. Prepare the source instance for cloning, using the steps described in Section 10.4.2:

   1. Perform Step 1 as described, in Section 10.4.2.

   2. Perform Step 2 in Section 10.4.2.

      In this scenario, you must specify the Oracle Internet Directory password on the command line, as shown in the following example:

      perl prepare_clone.pl ORACLE_HOME=/opt/oracle/Ora_1012
                            -oid_password my_oidpwd
      
      

   3. Perform Step 3, as described, in Section 10.4.2.

2. To use the same ports as the source instance, use the staticports.ini file to specify the port numbers for the cloned instance. See Section D.1.2 for a list of the default ports used by a Portal and Wireless instance.

   See Section 10.6.2 for details on using the staticports.ini file.

3. Clone the source instance to create a new instance using the steps described in Section 10.4.3:

   1. Perform Steps 1, 2, and 3, as described, in Section 10.4.3.

   2. Perform Step 4 in Section 10.4.3. However, note the following:

      • You must specify the Oracle Application Server administration password and the Oracle Internet Directory password on the command line.

      • Because OracleAS Portal stores only a single set of configuration information, the cloning process overwrites the configuration entries stored in the repository if you do not specify that there is a Load Balancing Router. If you specify that there is a Load Balancing Router, the cloning process does not perform any configuration steps, and therefore it does not overwrite the configuration settings stored in the Portal schemas in the OracleAS Metadata Repository. To specify that there is a Load Balancing Router, use the -lbr option with the value of true in the clone.pl command line.

      The following example clones a Portal and Wireless instance and names the cloned instance as Portal_B:

      perl clone.pl ORACLE_HOME=/opt/oracle/Ora_1012_B
                    ORACLE_HOME_NAME=OH_1012_B
                    -instance Portal_B
                    -ias_admin_old_pwd my_old_ias_pass
                    -ias_admin_new_pwd my_new_ias_pass
                    -oid_password my_oidpwd
                    -lbr true
      
      

   When the cloning process is finished, both the source instance and the cloned instance are part of the farm.

   Note that when you log in to OracleAS Portal on the cloned instance, you use the original ias_admin password of the first OracleAS Portal instance associated with the OracleAS Metadata Repository. See Section 10.5.6 for more information.

4. The cloning process does not configure the Load Balancing Router to recognize the cloned instance. You must manually configure the Load Balancing Router. To configure OracleAS Portal with a Load Balancing Router, see the section, "Configuring Multiple Middle-Tiers with a Load Balancing Router" in Oracle Application Server Portal Configuration Guide.

5. From the source instance, using either Application Server Control Console or OracleAS Web Cache Manager, take the following steps:

   1. Add the cloned instance to the OracleAS Web Cache cluster.

   2. Add the cloned origin server as an origin server for the cluster.

   3. Save the OracleAS Web Cache changes and propagate the changes to the cluster.

   See the Oracle Application Server Web Cache Administrator's Guide for information configuring OracleAS Web Cache, particularly Chapter 10 about adding caches to a cache cluster and Chapter 8 for information about specifying origin servers. See the section "Configuring Multiple Middle-Tiers with a Load Balancing Router" in the Oracle Application Server Portal Configuration Guide for an example of these tasks in an OracleAS Portal environment with a Load Balancing Router.

6. Configure the Load Balancing Router to accept invalidation requests from OracleAS Metadata Repository on a separate port, so that requests are forwarded to OracleAS Web Cache. See the section, "Configuring Multiple Middle-Tiers with a Load Balancing Router" in the Oracle Application Server Portal Configuration Guide for more information.

7. Configure the Load Balancing Router to balance the load between the OracleAS Web Cache cluster members.

8. Re-register mod_osso for the source instance with the Load Balancing Router. See the section, "Configuring Multiple Middle-Tiers with a Load Balancing Router" in the Oracle Application Server Portal Configuration Guide.

9. Edit the httpd.conf file to change the port number to use the Load Balancing Router port rather than the local Oracle HTTP Server port.

10.7.3 Cloning a Business Intelligence Instance

Consider an environment where you have one Business Intelligence middle-tier instance and you want to add more instances with identical configuration and application deployment. The instance is connected to OracleAS Infrastructure.

Review the limitations and considerations listed in Section 10.5, "Considerations and Limitations for Cloning", especially those in Section 10.5.9 and Section 10.5.10.

The strategy for cloning a Business Intelligence instance consists of the following steps:

1. Prepare the source instance for cloning, using the steps described in Section 10.4.2:

   1. Perform Step 1, as described, in Section 10.4.2.

   2. Perform Step 2 in Section 10.4.2.

      In this scenario, you must specify the Oracle Internet Directory password on the command line, as shown in the following example:

      perl prepare_clone.pl ORACLE_HOME=/opt/oracle/Ora_1012_BI
                            -oid_password my_oidpwd
      
      

   3. Perform Step 3, as described, in Section 10.4.2.

2. Clone the source instance to create a new instance using the steps described in Section 10.4.3:

   1. Perform Steps 1, 2, and 3, as described, in Section 10.4.3.

   2. Perform Step 4 in Section 10.4.3.

      In that step, you must change the name of the cloned instance, by specifying it on the command line, and you must specify the Oracle Application Server administration password and the Oracle Internet Directory password on the command line, as shown in the following example:

      perl clone.pl ORACLE_HOME=/opt/oracle/Ora_1012_BI2
                    ORACLE_HOME_NAME=OH_1012_BI2
                    -instance BI2
                    -ias_admin_old_pwd my_old_ias_pass
                    -ias_admin_new_pwd my_new_ias_pass
                    -oid_password my_oidpwd
      
      

      In this example, the instance name for the cloned instance is BI2.

3. During cloning, the data (.dat) files in the OracleAS Reports Services server subdirectory are not cloned. As a result, the past, current, and scheduled jobs information is not available. If you want to preserve the scheduled jobs information, take the following steps:

   1. Copy the .dat file from the source instance to the cloned instance. The files are located in:

      (UNIX) ORACLE_HOME/reports/server/server_name.dat
      (Windows) ORACLE_HOME\reports\server\server_name.dat
      
      

      For the in-process server, the .dat file name is rep_sourceHostName_sourceOracleHomeName, unless you have changed the in-process server name in the rwservlet.properties file.

   2. Ensure that any external dependencies for running reports in the source Oracle home is made the same in the cloned Oracle home. Some of the dependencies include:

      • The location of OracleAS Reports Services on the host

      • The REPORTS_PATH setting in the registry on Windows

      • The location of the destination

      • Distribution details

   3. Enter the .dat file name in the server configuration file:

      • For the in-process server in the cloned instance, the configuration file name is rep_cloneHostName_cloneOracleHomeName, unless you have changed the in-process server name in the rwservlet.properties file. You can add the .dat file name to the configuration file in the persistFile element, as shown in the following example:

        <persistFile filename="source_server_name.dat"/>
        
        

        In the example, source_server_name is rep_sourceHostName_sourceOracleHomeName, unless you have changed the in-process server name in the rwservlet.properties file.

      • For the Standalone server, if you want to run the server with the same name in the cloned instance, the server will use the copied .dat file with the name server_name.dat.

        However, if you want to run a server with a different name in the cloned instance, and you want to use the job information stored in the .dat file in the source home, you must add the .dat file name to the server configuration file in the persistFile element, as shown in the following example:

        <persistFile filename="source_server_name.dat"/>