Skip Headers
Oracle® Enterprise Manager Advanced Configuration
10g Release 2 (10.2)
B16242-01
  Go To Table Of Contents
Contents
Go To Documentation Library
Home
Go To Product List
Solution Area
Go To Index
Index

Previous
Previous
Next
Next
 

12 Configuring Notifications

The notification system allows you to notify Enterprise Manager administrators of alerts, policy violations, and the status changes of job executions. In addition to notifying administrators, the notification system can perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when an alert is triggered. This capability allows you to implement automatically specific IT practices under particular alert conditions. For example, if an alert is generated when monitoring the operational (up/down) status of a database, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.

By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.

This chapter covers the following:

12.1 Setting Up Notifications

All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.

12.1.1 Setting Up a Mail Server for Notifications

Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, other Enterprise Manager administrators.

You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 12-1). Display the Notification Methods page by clicking Setup on any page in the Grid Control Console and clicking Notification Methods in the vertical navigation bar.


Note:

You must have Super Administrator privileges in order to set up SMTP servers.

Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 12-1 shows sample notification method entries.

Example 12-1 Mail Server Settings

  • Outgoing Mail (SMTP) Server - smtp01.mycorp.com:587, smtp02.mycorp.com

  • User Name - myadmin

  • Password - ******

  • Confirm Password - ******

  • Identify Sender As - Enterprise Manager

  • Sender's E-mail Address - mgmt_rep@mycorp.com

Figure 12-1 Defining a Mail Server

Mail Server fields on Notification Methods page.
Description of "Figure 12-1 Defining a Mail Server"


Note:

The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Preferences General page.

After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.

Defining multiple mail servers will improve the reliability of e-mail notification delivery and spread the load across multiple systems. The Management Service makes use of each mail server to send e-mails and the behavior is controlled by the following parameters found in the $ORACLE_HOME/sysman/config/emoms.properties file.

Example 12-2 Management Service Parameters

# The maximum number of emails that can be sent in a single connection to an
# email server
# em.notification.emails_per_connection=20
#
# The maximum number of emails that can be sent in a minute
# em.notification.emails_per_minute=250

Based on the defaults in Example 12-2, the first mail server is used to send 20 e-mails before the Management Service switches to the next mail server which is used to send another 20 e-mails before switching to the next mail server. This prevents one mail server from becoming overloaded and should improve overall reliability and throughput.

12.1.2 Setting Up E-mail for Yourself

If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the General page under the Preferences link in the Grid Control console. In addition to defining notification e-mail addresses, you associate the notification message format (long or short) to be used for your e-mail address.

Setting up e-mail involves three steps:

Step 1: Define e-mail addresses.

Step 2: Set up a Notification Schedule.

Step 3: Subscribe to receive e-mail for notification rules.

12.1.2.1 Defining E-mail Addresses

An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.

To add an e-mail address:

  1. From the Grid Control Console, click Preferences. By default the General page is selected.

  2. Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.

  3. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.

    For example, user1@oracle.com

    Select the message format for your e-mail address. The Long Format sends a HTML formatted e-mail that contains detailed information. Example 12-3 shows a typical notification that uses the long format.

    The Short Format (Example 12-4) sends a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body. For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message.

  4. Click Apply to save your e-mail address.

Example 12-3 Long E-mail Notification for Alerts

Name=myhost.com
Type=Host
Host=myhost.com
Metric=Filesystem Space Available (%)
Mount Point =/usr
Timestamp=06-OCT-2006 16:27:05 US/Pacific
Severity=Warning
Message=Filesystem / has only 76.07% available space
Rule Name=Host Availability and Critical States
Rule Owner=SYSMAN

Example 12-4 Short E-mail Notification for Alerts

Subject is :
EM:Unreachable Start:myhost
Body is :
Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) 
but the host is UP

More about Short E-mail Format

Enterprise Manager does not directly support paging, SMS, etc., but instead relies on external gateways to, for example, perform the conversion from e-mail to page.

Entries in the emoms.properties file define the size and format of the short e-mail.

You must establish the maximum size your device can support and whether the message is sent in subject, body or both

emoms.properties Entries for a Short E-mail Format

# The maximum size of a short format email
# em.notification.short_format_length=155
# The format of the short email. It can be set to subject, body or both.
#
# When set to subject the entire message is sent in the subject i.e.
#  EM:<severity>:<target>:<message>:<timestamp>
# When set to body the entire message is sent in the body i.e.
#  EM:<severity>:<target>:<message>:<timestamp>
# When set to both the message is split i.e. the subject contains
#  EM:<severity>:<target>
# and the body contains
#  <message>:<timestamp>
# In all cases the message is truncated to the length specified in the
# em.notification.short_format_length parameter
# em.notification.short_format=both

12.1.2.2 Setting Up a Notification Schedule

Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule.


Note:

When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.

A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.

If you are a regular Enterprise Manager administrator and are defining your own notification schedule:

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page. By default the General page is selected.

  2. Click Notification Schedule in the vertical navigation bar. Your Notification Schedule page appears.

  3. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.

12.1.2.3 Subscribe to Receive E-mail for Notification Rules

A notification rule is a user-defined rule that defines the criteria by which notifications should be sent for alerts, policy violations, corrective action execution status, and job execution status. Specifically, in each rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. During notification rule creation, you specify criteria such as the targets you are interested in, their monitored metrics, associated alert severity conditions (clear, warning, critical), policy violations, corrective action execution status, or job execution status, and the associated notification method.

To subscribe to a notification rule you create, while creating the rule, go to the Methods page and check the "Send Me E-mail" option.

Out-of-Box Notification Rules

Enterprise Manager Grid control comes with out-of-box notification rules that cover the most common alert conditions. When you install the Oracle Management Service, you are given the option to receive e-mail notifications for critical alerts. If you choose this option, and if an e-mail address for the SYSMAN user was specified, then some default notification rules are created that cover the availability and critical states for common target types and would also be configured to send e-mail notifications to the SYSMAN e-mail address for the conditions defined in the notification rules.

You can access the out-of-box notification rules by clicking on Preferences on any page in the Enterprise Manager console and clicking Public Rules in the vertical navigation bar. If the conditions defined in the out-of-box notification rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule by clicking on Subscribe column in the row of the Public Rules table that corresponds to the notification rule that you are interested in. Click Apply to save your changes.

Table 12-1 lists all the default notification rules. These are all owned by the SYSMAN user and are public rules.

Table 12-1 Default Notification Rules

Name Description Applies to Targets of the Type Send Notification on the Following Availability States Send Notification if Any of the Metrics is at CRITICAL Alert Severity

Agent Upload Problems

System-generated notification rule for monitoring Agents that may have problems uploading data to the Management Service.

Oracle Management Service and Repository

N/A

Count of targets not uploading data

Agents Unreachable

System-generated notification rule for monitoring Agents that lose contact with the Management Service due to network problems, host problems or Agents going down.

Agents

Agent UnreachableAgent Unreachable Resolved

N/A

Application Server Availability and Critical States

System-generated notification rule for monitoring Application Servers' availability, and critical metric statuses.

Application Servers

Down

CPU Usage (%)

Database Availability and Critical States

System-generated notification rule for monitoring Databases' availability, and critical metric statuses.

Databases (single instance only)

Down

Process Limit Usage (%)

Session Limit Usage (%)

Blocking Session Count All Objects

Archiver Hung Alert Log Error Status

Data Block Corruption Alert Log Error Status

Generic Alert Log Error Status

Media Failure Alert Log Error Status

Session Terminated Alert Log Error Status

Archive Area Used (%) All Objects

Segments Not Able to Extend Count All Objects

Segments Approaching Maximum Extents Count All Objects

Tablespace Space Used (%) All Objects

Wait Time (%)

HTTP Server Availability and Critical States

System-generated notification rule for monitoring HTTP Server's availability, and critical metric statuses.

Oracle HTTP Server

Down

CPU Usage (%)

Percentage of Busy Processes

Active HTTP Connections

Request Processing Time (seconds)

Host Availability and Critical States

System-generated notification rule for monitoring Hosts' availability, and critical metric statuses.

Hosts

Agent Unreachable

Agent Unreachable Resolved

Average Disk I/O Service Time (ms)

Disk Device Busy (%)

Filesystem Space Available (%)

CPU in I/O Wait (%)

Run Queue Length (5 minute average)

CPU Utilization (%)

Memory Utilization (%)

Memory Page Scan Rate (per second)

Swap Utilization (%)

Network Interface Combined Utilization (%)

Listener Availability

System-generated notification rule for monitoring database Listeners' availability, and critical metric statuses.

Listeners

Down

N/A

OC4J Availability and Critical States

System-generated notification rule for monitoring OC4J instance's availability, and critical metric statuses.

OC4J

Down

CPU Usage (%)

OC4J Instance - Request Processing Time (seconds)

OC4J Instance - Active Sessions

Repository Operations Availability

System-generated notification rule for monitoring the availability of the DBMS jobs that are part of the Management Repository.

Oracle Management Service and Repository

Critical

DBMS Job UpDown

Violation Notification for Database Security Policies

System-generated notification rule for monitoring the secureness of the database configuration.

Databases

Critical

N/A

Web Cache Availability and Critical States

System-generated notification rule for monitoring Web Cache's instance's availability, and critical metric statuses.

Oracle Web Cache

Down

Hits (% of requests)

Web Cache CPU Usage (%)


Creating Your Own Notification Rules

If you find that the default notification rules do not meet your needs, you can define your own custom rules. The following procedure documents the process of notification rule creation for non-Super Administrators.

To create your own notification rule:

  1. From the Enterprise Manager Grid Control, click Preferences.

  2. Click My Rules in the vertical navigation bar.

    If you are not logged in as an administrator with Super Administrator privileges, you will see a link for My Rules instead of Rules as in the case of an administrator with Super Administrator privileges.

  3. Click Create.

    Enterprise Manager displays the Create Notification Rule pages. Enter the requisite information on each page to create your notification rule.

    When you specify the notification rule properties, check Make Public in the General page if you want other non-privileged users to be able to view and share that rule. For example, it allows other administrators to later specify that they should receive e-mail for this rule.

    When you specify the notification rule, you will only be able to choose from e-mail and SNMP traps. Specifying custom commands and PL/SQL procedures is an option that is only available to Super Administrators. To receive e-mail notifications for conditions defined in the rule, go to the Methods page and select the Send Me E-Mail option.

12.1.3 Setting Up E-mail for Other Administrators

If you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you must need to:

Step 1: Ensure Each Administrator Account has an Associated E-mail Address

Each administrator to which you want to send e-mail notifications must have a valid e-mail address.

  1. Click Setup.

  2. Click Administrators from the vertical navigation bar.

  3. For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified.

Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.

Step 2: Define Administrators' Notification Schedules

Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specified the e-mail addresses for the first time, you should review and edit the notification schedule as needed.

  1. Click Setup.

  2. From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.

  3. Specify the administrator who's notification schedule you wish to edit and click Change.

  4. Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.

  5. Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.

  6. Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.

  7. Click Finish when you are done.

  8. Repeat steps three through seven for each administrator.

Step 3: Assign Notification Rules to Administrators

With the notification schedules set, you now need to assign the appropriate notification rules for each designated administrator.

  1. Click Setup.

  2. From the vertical navigation bar, click Administrators.

  3. Select the desire administrator.

  4. Click Subscribe to Rules. The Subscribe admin to Public Notification Rules page appears.

  5. Select the desired notification rules and click Subscribe.

  6. Click OK when you are finished.

  7. Repeat steps three through six for each administrator.

12.2 Extending Notification Beyond E-mail

Notification Methods are the mechanisms by which alerts are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been setup as part of the Oracle Management Service installation.Enterprise Manager Super Administrators can also define other custom notification methods. For example, alerts may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once that is defined, any time an administrator needs to send alerts to the trouble-ticketing system, he merely needs to invoke the now globally available notification method called "Log trouble ticket". Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps.

Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) to their notification rules.

Through the Notification Methods page, you can:

12.2.1 Custom Notification Methods Using Scripts and SNMP Traps

You can create other custom notification methods based on OS scripts, PL/SQL procedures, or SNMP traps. Any administrator can then use these methods in Notification Rules.

12.2.1.1 Adding a Notification Method based on an OS Command or Script

Complete the following four steps to define a notification method based on an OS command or script.


Note:

Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.

Step 1: Define your OS command or script.

You can specify an OS command or script that will be called by the notification system. You can use target and alert or policy violation context information, corrective action execution status and job execution status within the body of the script. Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to OS commands/scripts allows you to customize automated responses to alerts or policy violations. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see

Step 2: Deploy the script on each Management Service host.

You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service.

The OS Command or Script should be deployed on the same location on each Management Service host machine. The OS Command should be an absolute path, for example, /u1/bin/logSeverity.sh. The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Example 12-5 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This is to prevent OS Commands from running for an inordinate length of time and blocking the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed.

Example 12-5 Parameter in emoms.properties File

# The amount of time in seconds after which an OS Command started by the
# Notification System will be killed if it has not exited
# em.notification.os_cmd_timeout=30

Step 3: Register your OS Command or Script as a new Notification Method.

Add this OS command as a notification method that can be called in Notification Rules. Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on the 'OS Command' type. See "Adding a Notification Method based on an OS Command or Script" .

The following information is required for each OS command notification method:

  • Name

  • Description

    Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.

  • OS Command

You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).

Example 12-6 shows information required for the notification method.

Example 12-6 OS Command Notification Method

Name Trouble Ticketing
Description Notification method to log trouble ticket for a severity occurrence
OS Command /private/mozart/bin/logTicket.sh

Note:

There can be more than one OS Command configured per system.

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

Passing Alert and Policy Violation Information to an OS Command or Script

The notification system passes severity information to an OS script or executable using system environment variables.

Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • Windows:%ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 12-2 Environment Variables

Environment Variable Description

TARGET_NAME

Name of the target on which the severity occurred.

TARGET_TYPE

Type of target on which the severity occurred. Targets are defined as any monitorable entity, such as Host, Database, Listener, or Oracle HTTP Server. You can view the type of a monitored target on the All Targets page.

HOST

Name of the machine on which the target resides.

METRIC

Metric generating the severity. This variable is not set for policy violations.

METRIC_VALUE

The value of the metric when the threshold was exceeded. Not set for policy violations

POLICY_RULE

The name of the policy when the threshold was exceeded. Not set for metric severities

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that shows the alert context for a policy violation.

TIMESTAMP

Time when the severity occurred.

SEVERITY

Type of severity. For example, severity for a target's (availability) status metric are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

MESSAGE

Message for the alert that provides details about what triggered the condition.

RULE_NAME

Name of the notification rule to which the OS Command notification method was assigned.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Your script may reference some or all of these variables.

The sample OS script shown in Example 12-7 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification

Example 12-7 Sample OS Command Script

#!/bin/ksh

LOG_FILE=/net/myhost/logs/severity.log
if test -f $LOG_FILE
then
echo $TARGET_NAME $MESSAGE $TIMESTAMP >> $LOG_FILE
else
   exit 100
fi

Example 12-8 shows an OS script that logs alert information to the file 'alertmsg.txt'. The file is saved to the /u1/results directory.

Example 12-8 Alert Logging Script

#!/usr/bin/sh
echo "Alert logged:" > /u1/results/alertmsg.txt
echo "\n" >> /u1/results/alertmsg.txt
echo "target name is " $TARGET_NAME >> /u1/results/alertmsg.txt
echo "target type is " $TARGET_TYPE >> /u1/results/alertmsg.txt
echo "target is on host " $HOST >> /u1/results/alertmsg.txt
echo "metric in alert is " $METRIC >> /u1/results/alertmsg.txt
echo "metric index is " $KEY_VALUE >> /u1/results/alertmsg.txt
echo "timestamp is " $TIMESTAMP >> /u1/results/alertmsg.txt
echo "severity is " $SEVERITY >> /u1/results/alertmsg.txt
echo "message is " $MESSAGE >> /u1/results/alertmsg.txt
echo "notification rule is " $RULE_NAME >> /u1/results/alertmsg.txt
echo "rule owner is " $RULE_OWNER >> /u1/results/alertmsg.txt
exit 0

Example 12-9 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Grid Control. When a metric alert is triggered, the Enterprise Manager Grid Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.

Example 12-9 HP OpenView Script

/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET"

12.2.1.2 Adding a Notification Method Based on a PL/SQL Procedure

Complete the following four steps to define a notification method based on a PL/SQL procedure.

Step 1: Define the PL/SQL procedure.

The procedure must have one of the following signatures depending on the type of notification that will be received.

For alerts and policy violations:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

For job execution status changes:

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

For corrective action status changes:

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

Note:

The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a notification rule.

For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:

"Passing Alert and Policy Violation Information to a PL/SQL Procedure"

"Passing Corrective Action Status Change Information"

"Passing Job Execution Status Information"

Step 2: Create the PL/SQL procedure on the Management Repository.

Create the PL/SQL procedure on the repository database using one of the following procedure specification:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)
PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)
PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)

If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user defined exception that uses the error code -20000. See Example 12-11, "PL/SQL Procedure Using a Severity Code". The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Step 3: Register your PL/SQL procedure as a new notification method.

Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on 'PL/SQL Procedure'. See "Adding a Notification Method Based on a PL/SQL Procedure".

Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.

Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:

  • Name

  • Description

  • PL/SQL Procedure

You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.

An example of the required information is shown in Example 12-10.

Example 12-10 PL/SQL Procedure Required Information

Name Open trouble ticket
Description Notification method to open a trouble ticket in the event
PLSQL Procedure ticket_sys.ticket_ops.open_ticket

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.

Information about the severity types that relate to a target's availability, and how metric severity and policy violation information is passed to the PLSQL procedure is covered in the next section.

Passing Alert and Policy Violation Information to a PL/SQL Procedure

Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to PL/SQL procedures allows you to customize automated responses to alerts or policy violations.

The notification system passes information about metric severities or policy violations to a PL/SQL procedure using the MGMT_NOTIFY_SEVERITY object. An instance of this object is created for every alert or policy violation. When an alert or policy violation occurs, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_SEVERITY object that has been passed to it.

The following table lists all metric severity attributes that can be passed:

Table 12-3 Metric Severity Attributes

Attribute Datatype Additional Information

TARGET_NAME

VARCHAR2(256)

Name of the target on which the severity occurred.

TARGET_TYPE

VARCHAR2(64)

Type of target on which the severity occurred. Targets are defined as any monitorable service.

TIMEZONE

VARCHAR2(64)

The target's regional timezone

HOST_NAME

VARCHAR2(128)

Name of the machine on which the target resides.

METRIC_NAME

VARCHAR2(64)

Metric or policy generating the severity.

METRIC_DESCRIPTION

VARCHAR2(128)

Meaningful description of the metric that can be understood by other administrators.

METRIC_COLUMN

VARCHAR2(64)

For table metrics, the metric column contains the name of the column in the table that is being defined. If the metric that is being defined is not a table metric, the value in this column is a single space. This attribute is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric.

KEY_VALUE

VARCHAR2(1290)

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE_GUID

VARCHAR2(256)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the alert context.

COLLECTION_TIMESTAMP

DATE

The time when the target status change was last detected and logged in the management repository.

SEVERITY_CODE

NUMBER

Numeric code identifying the severity level. See Severity Code table below.

MESSAGE

VARCHAR2(4000)

An optional message that is generated when the alert is created that provides additional information about the alert condition.

SEVERITY_GUID

RAW(16)

Severity global unique identifier.

METRIC_GUID

RAW(16)

Metric global unique identifier.

TARGET_GUID

RAW(16)

Target global unique identifier.

RULE_OWNER

VARCHAR2(64)

Name of the Enterprise Manager administrator who owns the rule.

RULE_NAME

VARCHAR2(132)

Name of the notification rule that resulted in the severity.


When a severity occurs for the target, the notification system creates an instance of the MGMT_NOTIFY_SEVERITY object and populates it with values from the severity. The severity codes in Table 12-4 have been defined as constants in the MGMT_GLOBAL package and can be used to determine the type of severity in the severity_code field of the MGMT_NOTIFY_SEVERITY object.

Table 12-4 Severity Codes

Name Datatype Value

G_SEVERITY_COMMENT

NUMBER(2)

10

G_SEVERITY_CLEAR

NUMBER(2)

15

G_SEVERITY_WARNING

NUMBER(2)

20

G_SEVERITY_CRITICAL

NUMBER(2)

25

G_SEVERITY_UNREACHABLE_CLEAR

NUMBER(3)

115

G_SEVERITY_UNREACHABLE_START

NUMBER(3)

125

G_SEVERITY_BLACKOUT_END

NUMBER(3)

215

G_SEVERITY_BLACKOUT_START

NUMBER(3)

225

G_SEVERITY_ERROR_END

NUMBER(3)

315

G_SEVERITY_ERROR_START

NUMBER(3)

325

G_SEVERITY_NO_BEACONS

NUMBER(3)

425

G_SEVERITY_UNKNOWN

NUMBER(3)

515


Example 12-11 PL/SQL Procedure Using a Severity Code

CREATE TABLE alert_log (target_name VARCHAR2(64),
alert_msg VARCHAR2(4000),
occured DATE);

PROCEDURE LOG_CRITICAL_ALERTS(severity IN MGMT_NOTIFY_SEVERITY)
IS
BEGIN
-- Log all critical severities
   IF severity.severity_code = MGMT_GLOBAL.G_SEVERITY_CRITICAL
   THEN
   	BEGIN
   		INSERT INTO alert_log (target_name, alert_msg, occured)
   		VALUES (severity.target_name, severity.message,
   				severity.collection_timestamp);
   	EXCEPTION
   	WHEN OTHERS
   	THEN
   	-- If there are any problems then get the notification retried
   		RAISE_APPLICATION_ERROR(-20000, 'Please retry');
  		END;
  		COMMIT;
   END IF;
END LOG_CRITICAL_ALERTS;

12.2.1.3 Adding a Notification Method Based on an SNMP Trap

Enterprise Manager supports integration with third-party management tools through the SNMP. For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.

The trap is an SNMP Version 1 trap and is described by the MIB definition shown at the end of this chapter. See "Management Information Base (MIB)".

For more comprehensive configuration information, see the documentation specific to your platform; SNMP configuration differs from platform to platform.


Note:

Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a notification rule.

Step 1: Define a new notification method based on an SNMP trap.

Log in to Enterprise Manager as a Super Administrator. Click Setup and then click Notification Method from the vertical navigation bar to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.

You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. In Example 12-12, the SNMP host will receive your SNMP traps.

Example 12-12 SNMP Trap Required Information

Name HP OpenView Console
Description Notification method to send trap to HP OpenView console
SNMP Trap Host Name machine1.us.oracle.com
SNMP Host Port 162
SNMP Community public
This SNMP host will receive your SNMP traps.

Note:

A Test Trap button exists for you to test your setup.

Metric severity information will be passed as a series of variables in the SNMP trap.

An example SNMP Trap is shown in Example 12-13. Each piece of information is sent as a variable embedded in the SNMP Trap.

Example 12-13 SNMP Trap

Tue Oct 28 05:00:02 2006

Command: 4
   Enterprise: 1.3.6.1.4.1.111.15.2
   Agent: 138.1.6.200
   Generic Trap: 6
   Specific Trap: 1
   Time Stamp: 8464:39.99
   Count: 11

Name: 1.3.6.1.4.1.111.15.1.1.1.2.1
   Kind: OctetString
   Value: "mydatabase"

Name: 1.3.6.1.4.1.111.15.1.1.1.3.1
   Kind: OctetString
   Value: "Database"

Name: 1.3.6.1.4.1.111.15.1.1.1.4.1
  Kind: OctetString
  Value: "myhost.com"

Name: 1.3.6.1.4.1.111.15.1.1.1.5.1
   Kind: OctetString
   Value: "Owner's Invalid Object Count"

Name: 1.3.6.1.4.1.111.15.1.1.1.6.1
   Kind: OctetString
   Value: "Invalid Object Owner"

Name: 1.3.6.1.4.1.111.15.1.1.1.7.1
   Kind: OctetString
   Value: "SYS"

Name: 1.3.6.1.4.1.111.15.1.1.1.8.1
   Kind: OctetString
   Value: "28-OCT-2006 04:59:10 (US/Eastern GMT)"

Name: 1.3.6.1.4.1.111.15.1.1.1.9.1
   Kind: OctetString
   Value: "Warning"

Name: 1.3.6.1.4.1.111.15.1.1.1.10.1
   Kind: OctetString
   Value: "12 object(s) are invalid in the SYS schema."

Name: 1.3.6.1.4.1.111.15.1.1.1.11.1
   Kind: OctetString
   Value: "Database Metrics"

Name: 1.3.6.1.4.1.111.15.1.1.1.12.1
   Kind: OctetString
   Value: "SYSMAN"

Step 2: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

12.3 Passing Corrective Action Status Change Information

Passing corrective action status change attributes (new status, job name, job type, notification rule, or rule owner, etc.) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case you will want to pass status (Problems, Aborted, etc.) to the script to open a trouble ticket and escalate the problem.

12.3.1 Passing Corrective Action Execution Status to an OS Command or Script

The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 12-5 Environment Variables

Environment Variable Description

JOB_NAME

The name of the corrective action.

JOB_OWNER

The owner of the corrective action.

JOB_TYPE

The type of corrective action.

JOB_STATUS

The corrective action status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target on which the corrective action ran. Example: TARGET_NAME1, TARGET_NAME2.

METRIC

The name of the metric in the alert that caused the corrective action to run. Not set for policy violations.

POLICY_RULE

The name of the policy rule in the alert that caused the corrective action to run. Not set for metric severities.

METRIC_VALUE

The value of the metric column in the alert that caused the corrective action to run.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that show the policy violation context.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

SEVERITY

Type of alert severity. For example, severity types that relate to a target's availability are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

RULE_NAME

Name of the notification rule that resulted in the execution of the corrective action.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


12.3.2 Passing Corrective Action Execution Status to a PLSQL Procedure

The notification system passes corrective action status change information to a PL/SQL procedure via the MGMT_NOTIFY_CORRECTIVE_ACTION object. An instance of this object is created for every status change. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_CORRECTIVE_ACTION object that has been passed to it.

Table 12-6 lists all corrective action status change attributes that can be passed:

Table 12-6 Corrective Action Status Attributes

Attribute Datatype Additional Information

JOB_NAME

VARCHAR2(128)

The corrective action name.

JOB_OWNER

VARCHAR(256)

The owner of the corrective action.

JOB_TYPE

VARCHAR2(32)

The type of the corrective action.

JOB_STATUS

NUMBER

The new status of the corrective action. See Table 12-7, "Corrective Action Status Codes" for a list of possible status conditions.

STATE_CHANGE_GUID

RAW(16)

The GUID of the state change record.

JOB_GUID

RAW(16)

The unique id of the corrective action.

EXECUTION_ID

RAW(16)

The unique id of the corrective action execution.

TARGETS

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the corrective action runs on.

METRIC_NAME

VARCHAR2(256)

The name of the metric/policy rule in the alert that caused the corrective action to run.

METRIC_COLUMN

VARCHAR2(64)

The name of the metric in the alert that caused the corrective action to run. This is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric in the alert that caused the corrective action to run.

SEVERITY_CODE

NUMBER

The severity code of the alert that caused the corrective action to run. See Table 12-4, "Severity Codes".

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

VARCHAR2(1290)

For table metrics, this column contains the value of the key column for the row in the table whose thresholds are being defined. If the thresholds are not for a table metric, or the thresholds apply for all rows in the metric column, then the value in this column will contain a single space.

KEY_VALUE_GUID

RAW(16)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the corrective action status change context.

RULE_OWNER

VARCHAR2(64)

The owner of the notification rule that caused the PL/SQL notification to be sent.

RULE_NAME

VARCHAR2(132)

The name of the notification rule that caused the PL/SQL notification method to be invoked.

OCCURRED_DATE

DATE

The time and date when the status change happened.


The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.

Table 12-7 Corrective Action Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 12-14 PL/SQL Procedure Using a Status Code

CREATE TABLE ca_log (jobid RAW(16),		      occured DATE);

CREATE OR REPLACE PROCEDURE LOG_PROBLEM_CAS(status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)
IS
BEGIN
	-- Log all failed corrective actions
   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS
   THEN
   	BEGIN
   		INSERT INTO ca_log (jobid, occured)
   		VALUES (status_change.job_guid, SYSDATE);
   	EXCEPTION
   	WHEN OTHERS
   	THEN
   	-- If there are any problems then get the notification retried
   		RAISE_APPLICATION_ERROR(-20000, 'Please retry');
  		END;
  		COMMIT;
   END IF;
END LOG_PROBLEM_CAS;

12.4 Passing Job Execution Status Information

Passing job status change attributes (new status, job name, job type, notification rule, or rule owner, etc.) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (Problems, Aborted, etc.) to the script to open a trouble ticket and escalate the problem.

12.4.1 Passing Job Execution Status to a PLSQL Procedure

The notification system passes job status change information to a PL/SQL procedure via the MGMT_NOTIFY_JOB object. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_JOB object that has been passed to it.

Table 12-8 lists all corrective action status change attributes that can be passed:

Table 12-8 Job Status Attributes

Attribute Datatype Additional Information

JOB_NAME

VARCHAR2(128)

The job name.

JOB_OWNER

VARCHAR2(256)

The owner of the job.

JOB_TYPE

VARCHAR2(32)

The type of the job.

JOB_STATUS

NUMBER

The new status of the job.

STATE_CHANGE_GUID

RAW(16)

The GUID of the state change record.

JOB_GUID

RAW(16)

The unique id of the job.

EXECUTION_ID

RAW(16)

The unique id of the execution.

TARGETS

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the job runs on.

RULE_OWNER

VARCHAR2(64)

The name of the notification rule that cause the notification to be sent.

RULE_NAME

VARCHAR2(132)

The owner of the notification rule that cause the notification to be sent.

OCCURRED_DATE

DATE

The time and date when the status change happened.


When a job status change occurs for the job, the notification system creates an instance of the MGMT_NOTIFY_JOB object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the MGMT_NOTIFY_JOB object.

Table 12-9 Job Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 12-15 PL/SQL Procedure Using a Status Code (Job)

CREATE TABLE job_log (jobid RAW(16),		      occured DATE);

CREATE OR REPLACE PROCEDURE LOG_PROBLEM_JOBS(status_change IN MGMT_NOTIFY_JOB)
IS
BEGIN
	-- Log all failed jobs
   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS
   THEN
   	BEGIN
   		INSERT INTO job_log (jobid, occured)
   		VALUES (status_change.job_guid, SYSDATE);
   	EXCEPTION
   	WHEN OTHERS
   	THEN
   	-- If there are any problems then get the notification retried
   		RAISE_APPLICATION_ERROR(-20000, 'Please retry');
  		END;
  		COMMIT;
   END IF;
END LOG_PROBLEM_JOBS;

12.4.2 Passing Job Execution Status to an OS Command or Script

The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 12-10 Environment Variables

Environment Variable Description

JOB_NAME

The name of the job.

JOB_OWNER

The owner of the job.

JOB_TYPE

The type of job.

JOB_STATUS

The job status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target. For example, TARGET_NAME1, TARGET_NAME2.

TARGET_TYPEn

The type of the nth target. For example TARGET_TYPE1, TARGET_TYPE2.

RULE_NAME

Name of the notification rule that resulted in the severity.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


12.5 Assigning Methods to Rules

For each notification rule, you can assign one or more notification methods to be called when any of the criteria in the notification rule is met.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. Perform your assignments.

Figure 12-2 Assigning Methods to Rules

This graphic shows the Assign Methods to Multiple rule page
Description of "Figure 12-2 Assigning Methods to Rules"

12.6 Assigning Rules to Methods

For each notification method, you can associate one or more notification rules that will use that method to send notifications.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. From the View menu, select By Method.

  5. Perform your assignments.

Figure 12-3 Assign Rules to Methods

Description of Figure 12-3  follows
Description of "Figure 12-3 Assign Rules to Methods"

12.7 Management Information Base (MIB)

Enterprise Manager Grid Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.

12.7.1 About MIBs

A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).

The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.

12.7.2 Reading the MIB Variable Descriptions

This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 2, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.

12.7.2.1 Variable Name

Syntax

Maps to the SYNTAX element of SNMP MIB definition, Version 2.

Max-Access

Maps to the MAX-ACCESS element of SNMP MIB definition, Version 2.

Status

Maps to the STATUS element of SNMP MIB definition, Version 2.

Explanation

Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 2.

Typical Range

Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.

Significance

Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.

Related Variables

Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.

Suggested Presentation

Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.

12.7.2.2 MIB Definition

Example 12-16 shows a typical MIB definition used by Enterprise Manager.

Example 12-16 MIB Definition

ORACLE-ENTERPRISE-MANAGER-4-MIB DEFINITIONS ::= BEGIN
IMPORTS
    TRAP-TYPE
        FROM RFC-1215
    DisplayString
        FROM RFC1213-MIB
    OBJECT-TYPE
        FROM RFC-1212
    enterprises
        FROM RFC1155-SMI;
oracle OBJECT IDENTIFIER ::= { enterprises  111 }
oraEM4 OBJECT IDENTIFIER ::= { oracle  15 }
oraEM4Objects OBJECT IDENTIFIER ::= { oraEM4  1 }
oraEM4AlertTable OBJECT-TYPE
    SYNTAX  SEQUENCE OF OraEM4AlertEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
     "Information on alerts generated by Oracle Enterprise Manager. This table is
 not queryable; it exists only to document the variables included in the
 oraEM4Alert trap.  Each trap contains a single instance of each variable in the
 table."
    ::= { oraEM4Objects  1 }
oraEM4AlertEntry OBJECT-TYPE
    SYNTAX  OraEM4AlertEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
     "Information about a particular Oracle Enterprise Manager alert."
    INDEX   { oraEM4AlertIndex }
    ::= { oraEM4AlertTable  1 }
OraEM4AlertEntry ::=
    SEQUENCE {
        oraEM4AlertIndex
            INTEGER,
        oraEM4AlertTargetName
       DisplayString,
        oraEM4AlertTargetType
       DisplayString,
        oraEM4AlertHostName
       DisplayString,
        oraEM4AlertMetricName
       DisplayString,
        oraEM4AlertKeyName
       DisplayString,
        oraEM4AlertKeyValue
       DisplayString,
        oraEM4AlertTimeStamp
       DisplayString,
        oraEM4AlertSeverity
       DisplayString,
        oraEM4AlertMessage
       DisplayString,
        oraEM4AlertRuleName
       DisplayString
        oraEM4AlertRuleOwner
       DisplayString
    oraEM4AlertMetricValue
           DisplayString,
        oraEM4AlertContext
           DisplayString
    }
oraEM4AlertIndex OBJECT-TYPE
    SYNTAX  INTEGER
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "Index of a particular alert, unique only at the moment an alert is generated."
    ::= { oraEM4AlertEntry  1 }
oraEM4AlertTargetName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the target to which this alert applies."
    ::= { oraEM4AlertEntry  2 }
oraEM4AlertTargetType OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The type of the target to which this alert applies."
    ::= { oraEM4AlertEntry  3 }
oraEM4AlertHostName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the host on which this alert originated."
    ::= { oraEM4AlertEntry  4 }
oraEM4AlertMetricName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the metric or policy which generated this alert."
    ::= { oraEM4AlertEntry  5 }
oraEM4AlertKeyName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the key-column, if present, for the metric which generated this alert."
    ::= { oraEM4AlertEntry  6 }
oraEM4AlertKeyValue OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The value of the key-column, if present, for the metric which generated this alert."
    ::= { oraEM4AlertEntry  7 }
oraEM4AlertTimeStamp OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The time at which this alert was generated."
    ::= { oraEM4AlertEntry  8 }
oraEM4AlertSeverity OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The severity of the alert e.g. Critical."
    ::= { oraEM4AlertEntry  9 }
oraEM4AlertMessage OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The message associated with the alert."
    ::= { oraEM4AlertEntry  10 }
oraEM4AlertRuleName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the notification rule that caused this notification."
    ::= { oraEM4AlertEntry  11 }
oraEM4AlertRuleOwner OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The owner of the notification rule that caused this notification."
    ::= { oraEM4AlertEntry  12 }
oraEM4AlertMetricValue OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The value of the metric which caused this alert to be generated."
    ::= { oraEM4AlertEntry  13 }
oraEM4AlertContext OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "A comma separated list of metric column names and values associated with the metric that caused this alert to be generated."
    ::= { oraEM4AlertEntry  14 }
oraEM4Traps OBJECT IDENTIFIER ::= { oraEM4  2 }
oraEM4Alert TRAP-TYPE
    ENTERPRISE  oraEM4Traps
    VARIABLES   { oraEM4AlertTargetName, oraEM4AlertTargetType,
                  oraEM4AlertHostName, oraEM4AlertMetricName,
                  oraEM4AlertKeyName, oraEM4AlertKeyValue, oraEM4AlertTimeStamp,
                  oraEM4AlertSeverity, oraEM4AlertMessage,
                  oraEM4AlertRuleName, oraEM4AlertRuleOwner,
                  oraEM4AlertMetricValue, oraEM4AlertContext }
    DESCRIPTION
     "The variables included in the oraEM4Alert trap."
    ::= 1
oraEM4JobAlertTable OBJECT-TYPE
    SYNTAX  SEQUENCE OF OraEM4JobAlertEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
     "Information on alerts generated by Oracle Enterprise Manager. This table is
 not queryable; it exists only to document the variables included in the
 oraEM4JobAlert trap.  Each trap contains a single instance of each variable in
 the table."
    ::= { oraEM4Objects  2 }
oraEM4JobAlertEntry OBJECT-TYPE
    SYNTAX  OraEM4AlertEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
     "Information about a particular Oracle Enterprise Manager alert."
    INDEX   { oraEM4JobAlertIndex }
    ::= { oraEM4JobAlertTable  1 }
OraEM4JobAlertEntry ::=
    SEQUENCE {
        oraEM4JobAlertIndex
            INTEGER,
        oraEM4JobAlertJobName
       DisplayString,
        oraEM4JobAlertJobOwner
       DisplayString,
        oraEM4JobAlertJobType
       DisplayString,
        oraEM4JobAlertJobStatus
       DisplayString,
        oraEM4JobAlertTargets
       DisplayString,
        oraEM4JobAlertTimeStamp
       DisplayString,
        oraEM4JobAlertRuleName
       DisplayString
        oraEM4JobAlertRuleOwner
       DisplayString,
    oraEM4JobAlertMetricName
           DisplayString,
    oraEM4JobAlertMetricValue
           DisplayString,
        oraEM4JobAlertContext
           DisplayString,
        oraEM4JobAlertKeyName
       DisplayString,
        oraEM4JobAlertKeyValue
       DisplayString,
        oraEM4JobAlertSeverity
       DisplayString
    }
oraEM4JobAlertIndex OBJECT-TYPE
    SYNTAX  INTEGER
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "Index of a particular alert, unique only at the moment an alert is
 generated."
    ::= { oraEM4JobAlertEntry  1 }
oraEM4JobAlertJobName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the job to which this alert applies."
    ::= { oraEM4JobAlertEntry  2 }
oraEM4JobAlertJobOwner OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The owner of the job to which this alert applies."
    ::= { oraEM4JobAlertEntry  3 }
oraEM4JobAlertJobType OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The type of the job to which this alert applies."
    ::= { oraEM4JobAlertEntry  4 }
oraEM4JobAlertJobStatus OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The status of the job to which this alert applies."
    ::= { oraEM4JobAlertEntry  5 }
oraEM4JobAlertTargets OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "A comma separated list of target to which this alert applies."
    ::= { oraEM4JobAlertEntry  6 }
oraEM4JobAlertTimeStamp OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The time at which this job status changed causing this alert."
    ::= { oraEM4JobAlertEntry  7 }
oraEM4JobAlertRuleName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the notification rule that caused this notification."
    ::= { oraEM4JobAlertEntry  8 }
oraEM4JobAlertRuleOwner OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The owner of the notification rule that caused this notification."
    ::= { oraEM4JobAlertEntry  9 }
oraEM4JobAlertMetricName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the metric or policy which caused the Corrective Action to run
 that caused this alert."
    ::= { oraEM4JobAlertEntry  10 }
oraEM4JobAlertMetricValue OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The value of the metric which caused the Corrective Action to run that
 caused this alert."
    ::= { oraEM4JobAlertEntry  11 }
oraEM4JobAlertContext OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "A comma separated list of metric column names and values associated with the
 metric which caused the Corrective Action to run that caused this alert."
    ::= { oraEM4JobAlertEntry  12 }
oraEM4JobAlertKeyName OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The name of the key-column, if present, for the metric which caused the
 Corrective Action to run that generated this alert."
    ::= { oraEM4JobAlertEntry  13 }
oraEM4JobAlertKeyValue OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The value of the key-column, if present, for the metric which caused the
 Corrective Action to run that generated this alert."
    ::= { oraEM4JobAlertEntry  14 }
oraEM4JobAlertSeverity OBJECT-TYPE
    SYNTAX  DisplayString
    ACCESS  read-only
    STATUS  mandatory
    DESCRIPTION
     "The severity of the metric which caused the Corrective Action to run that
 generated this alert e.g. Critical."
    ::= { oraEM4JobAlertEntry  15 }
oraEM4JobAlert TRAP-TYPE
    ENTERPRISE  oraEM4Traps
    VARIABLES   { oraEM4JobAlertJobName, oraEM4JobAlertJobOwner,
                  oraEM4JobAlertJobType, oraEM4JobAlertJobStatus,
                  oraEM4JobAlertTargets, oraEM4JobAlertTimeStamp,
                  oraEM4JobAlertRuleName, oraEM4JobAlertRuleOwner,
                  oraEM4JobAlertMetricName, oraEM4JobAlertMetricValue,
                  oraEM4JobAlertContext, oraEM4JobAlertKeyName,
                  oraEM4JobAlertKeyValue, oraEM4JobAlertSeverity }
    DESCRIPTION
     "The variables included in the oraEM4JobAlert trap."
    ::= 2
END

12.8 Troubleshooting Notifications

To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.

12.8.1 General Setup

The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.

OS Command, PL/SQL and SNMP Trap Notifications

Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called. For example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.

E-mail Notifications

  • Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail address is setup under General page of Preferences. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail schedule is defined under the Schedule page of Preferences. No e-mails will be sent unless a Notification Schedule has been defined.

  • Make sure a Notification Rule is defined to match the target, metric, severity and availability states you are interested and make sure e-mail and notification methods are assigned to the rule. A summary of the notification rule can be checked by going to the Rules page under Setup and clicking the rule name.

12.8.2 Notification System Errors

For any alerts involving problems with notifications, check the following for notification errors.

  • Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. These errors can be seen in the Errors page under Management Services and Repository under Setup.

  • Check for any delivery errors. From the Alerts section of a target home page, click on the alert message to access the metric details page. In the Alert History section, click on the Details icon for more information about the alert. The details will give the reason why the notification was not delivered. Delivery errors are stored in MGMT_NOTIFICATION_LOG with the DELIVERED column set to 'N'.

  • Severities will not be displayed in the Grid Control console if no metric values have been loaded for the metric associated with the severity.

12.8.3 Notification System Trace Messages

The Notification System can produce trace messages in sysman/log/emoms.trc file.

Tracing is configured by setting the following flag in sysman/config/emomslogging.properties file. You can set the trace level to INFO, WARN, DEBUG. For example,

log4j.em.notification=DEBUG

Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc file using the grep command. For example,

grep em.notification emoms.trc

What to look for in the trace file.

The following entries in the emoms.trc file are relevant to notifications.

Normal Startup Messages

When the OMS starts, you should see these types of messages.

2006-11-08 03:18:45,385 [Orion Launcher] INFO  em.notification init.1279 - Short format maximum length is 155

2006-11-08 03:18:45,386 [Orion Launcher] INFO  em.notification init.1297 - Short format is set to both subject and body

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.1010 - Waiting for connection to EM Repository...

2006-11-08 03:18:46,006 [NotificationMgrThread] INFO  em.notification run.1041 - Registering for Administrative Queue Name...

2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1078 - Administrative Queue is ADM21

2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1089 - Creating thread pool: min = 6 max = 24

2006-11-08 03:18:48,206 [NotificationMgrThread] INFO  em.notification handleAdminNotification.655 - Handling notifications for EMAIL1

Notification Delivery Messages

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.682 - Notification ready on EMAIL1

2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO  em.notification run.114 - Deliver to SYSMAN/admin@oracle.com

2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO  em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com

Notification System Error Messages

2006-11-08 07:26:30,242 [NotificationMgrThread] ERROR  em.notification getConnection.237 - Failed to get a connection Io exception: The Network Adapter could not establish the connection

12.8.4 E-mail Errors

The SMTP gateway is not set up correctly:

Failed to send e-mail to my.admin@oracle.com: For e-mail notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)

Invalid host name:

Failed to connect to gateway: badhost.us.oracle.com: Sending failed;
nested exception is:
javax.mail.MessagingException: Unknown SMTP 	host: badhost.us.oracle.com;

Invalid e-mail address:

Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed;
nested exception is:
javax.mail.MessagingException: 550 5.7.1 	<smpemailtest_ie@oracle.com>... Access denied

Always use the Test button to make sure the e-mail gateway configuration is valid. Check that an e-mail is received at the sender's e-mail address

12.8.5 OS Command Errors

When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.

Invalid path or no read permissions on file:

Could not find /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

No execute permission on executable:

Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule ) 

Timeout because OS Command ran too long:

Timeout occurred running /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.

Always use the Test button to make sure OS Command configuration is valid.

12.8.6 SNMP Trap Errors

Use the Test button to make sure SNMP Trap configuration is valid.

Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.

12.8.7 PL/SQL Errors

When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.

Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC

Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule) 

Procedure is not the correct signature. Example: PROCEDURE p(s IN MGMT_NOTIFY_SEVERITY)

Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)

Procedure has bug and is raising an exception.

Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)

Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Grid Control console.

Always use the Test button to make sure the PL/SQL configuration is valid.