14 Web Scraping

14.1 Overview of Web Scraping

The majority of applications available on the Web render content in format specific to certain types of devices. Web scraping allows applications developed for a particular markup language to be reformatted for use with other devices. Specifically, Web applications rendered in HTML and developed for PC browsers can be scraped to deliver to wireless devices through OracleAS Wireless.

To reuse WML applications, you can use the WML Translator.

14.2 Introducing Oracle Application Server Web Clipping

The Wireless Web Clipping server enables OracleAS Wireless administrators to clip, then scrape Web content and create Web Clipping applications that are stored persistently in the Wireless Web Clipping server repository. Then, the server serves Web Clipping applications based on these definitions, which are delivered by OracleAS Wireless.

The Wireless Web Clipping server architecture as it relates to OracleAS Wireless is shown in Figure 14-1.

Figure 14-1 Web Clipping Server Architecture Relative to OracleAS Wireless Core

Description of "Figure 14-1 Web Clipping Server Architecture Relative to OracleAS Wireless Core"

The Web Clipping definitions are comprised of URLs and scraping instructions so that they can be used to dynamically play back the scraping action to the repurposed Web sites. The Wireless Web Clipping server repository stores these definitions, while the Wireless Repository stores information to access the Wireless Web Clipping application.

The Wireless Web Clipping server serves Web Clipping applications based on these definitions, which are delivered by OracleAS Wireless.

When a request is initiated by a user using a mobile wireless device to view a particular Web Clipping application, that application is retrieved by the HTTP Adapter and delivered to OracleAS Wireless for processing and delivery to the mobile device, as shown in Figure 14-2.

Figure 14-2 Web Clipping Application

Description of "Figure 14-2 Web Clipping Application"

Web clipping content can be clipped and scraped from Internet or Intranet Web sites scattered throughout a large organization. Using OracleAS Wireless, you can generate Wireless Web Clipping applications using Web clippings and store them in the Wireless Repository.

In addition, you can create customized wireless applications that use Web clippings to their best advantage. As a developer, you first identify the development environment that best fits your needs. Depending upon the development environment you select, you must download and deploy the required archive files into that development environment. Next, you clip the desired content. Then, you download either the Java or JSP stub files to start developing.

Figure 14-3 shows a more detailed look at the Web Clipping server.

Figure 14-3 Detailed Web Clipping Server Architecture

Description of "Figure 14-3 Detailed Web Clipping Server Architecture"

OracleAS Wireless provides archive files for the following development environments are:

• J2EE—for developing in a J2EE (OC4J) environment using the Sun standard JCA Common Client Interface (CCI) instead of the Web Clipping Bean API, and for developing with an OC4J version that is release 9.0.3 or later. Download the JCA Resource Adaptor Archive (RAR) file, webclipping.rar from the Custom Applications page of the Web Clipping Manager.

• J2SE—for developing not only in a J2EE (OC4J) environment, but also in a standalone Java 2 SDK environment. Download the Java library (JAR) file, wcbean.jar from the Custom Applications page of the Web Clipping Manager.

Applications can be customized and deployed in J2EE or J2SE environments as standalone Java applications or part of other JavaServer Pages (JSP) applications, or these applications can be created into wireless applications and stored in the wireless repository, or this content can be clipped.

To create a Web Clipping application, the OracleAS Wireless administrator uses a Web browser to navigate to the Web page containing the desired content, then selects the portion of the page to clip and scrape, sets some attributes, and if the Web clipping uses form-based submission, exposes input parameters, saves the application, and tests the application.

Web Clipping applications support:

• Navigation through various styles of login mechanisms, including form- and JavaScript-based submission and HTTP Basic and Digest Authentication with cookie-based session management.

• Fuzzy matching of clippings. If a Web clipping gets reordered within the source page or if its character font, size, or style changes, it will still be identified correctly by the Web Clipping server and delivered as the Web Clipping application content.

• Reuse of a wide range of Web content, including support for pages written with HTML 4.0.1, JavaScript, applets, and plug-in enabled content, retrieved through HTTP GET and POST (form submission) methods.

• Clipping page content from HTML 4.0.1 pages, including:

  • Clipping of <applet>, <body>, <div>, <embed>, <img>, <object>, <ol>, <span>, <table>, and <ul> tagged content

  • Preservation of <head> styles and fonts, Cascading Style Sheets (CSS)

  • UTF-8 compliant character sets

  • Navigation through hyperlinks (HTTP GET), form submissions (HTTP POST), frames, URL redirection

• Navigation and clipping of HTTPS-based external Web sites, provided that appropriate server certificates are acquired.

• National Language Sets (NLS) from existing Web content. Web Clipping checks for NLS settings in the following way:

  1. Web Clipping checks the Content-Type in the HTTP header for the charset attribute. If this is present, it assumes that this is the character encoding of the HTML page.

  2. If the charset attribute is not present, it checks the HTML META tag on the page to determine the character encoding.

  3. If the HTML META tag is not found, Web Clipping uses the charset in the previous browsed page. If this is the first page, it defaults to the ISO-8859-1 character encoding.

  4. If the value of the charset for Content-Type or META tag is not supported (for example, if the charset was specified as NONE), Web clipping uses the default character set, ISO-8859-1, not the charset in the previously browsed page.

  In addition, Web Clipping supports NLS in URLs and URL parameters.

• Browsers including Netscape Navigator releases 6 and 7, Microsoft Internet Explorer releases 5.0, 5.5, 6.0, and higher.

• Clipping of most non-HTML elements, including applets, plug-in content (for example, embedded QuickTime video, Macromedia Flash presentation), and client-side JavaScript (v1.2).

The following are known limitations of Web clipping:

• NLS or internationalization limitations include the following:

  • The character set must be specified in the META tag or HTTP header; if not specified, Web Clipping Studio and the provider default to ISO-8859-1 character encoding.

  • Web Clipping outputs pages in UTF-8. Therefore, languages not supported under UTF-8 will not be shown correctly.

  • Characters from two or more character sets specified on a page will not be shown properly.

• If you use the Netscape Web browser and Japanese is your browser's first priority language, you will not be able to correctly view the Web clipping Help pages.

• The output may display in a language dissimilar to the language of your user profile (the language you use to log in to OracleAS Wireless Webtool). The Web Clipping Manager uses your Web browser language as the default.

  If you want the output of OracleAS Wireless Web clipping to appear in a specific language, change your browser's priority language and refer to the Help for your Web browser for information about setting the browser language.

• The action of the Back and Forward buttons in the Web Clipping Studio operate on the whole page, not on individual frames.

• Navigation from plug-in content (for example, Macromedia Flash).

• Customization of HTTP parameters with multiple values.

• JavaScript-based encryption of username/password for login pages.

• The following page content cannot be clipped by Web clipping:

  • Global JavaScript statements that contain doc.write()

  • JavaScript in sectioned Web clipping that writes additional JavaScript

• Scraping of applets

The Web Clipping server provides and renders clipped Web content as Web Clipping application content. The Web Clipping Studio allows OracleAS Wireless administrators and wireless application users to do the following:

• Browse for Web content

• Divide the chosen target page into sections

• Choose the exact portion of the Web content to clip

• Preview the clipped content

• Scrape the clipped content

• Set some Web Clipping application attributes, parameterize input parameters in form-based submission, and save the Web Clipping application

• Test the Web Clipping application and, if using form-based submission, modify initial input values and test these as well

By default, all Web Clipping application definitions are stored persistently in the OracleAS Infrastructure database. Any secure information, such as passwords, are stored in encrypted form, according to the Data Encryption Standard (DES), using Oracle encryption technology.

14.3 Creating an OracleAS Web Clipping Application

The Web Clipping server is automatically installed as part of OracleAS Wireless. As an OracleAS Wireless administrator, you can access the Web Clipping server in the repository under the Applications: Browse Folder: page in the OracleAS Wireless Service Manager.

You may need to perform some configuration tasks before getting started. See Section 14.6, "Administering Web Scraping" for more information.

To create a Web Clipping, perform the following steps:

1. Log into OracleAS Wireless Service Manager as an OracleAS Wireless administrator.

2. Select the Services tab.

3. On the Applications tab, click Create Application.

4. On the Select the Application To Be Created page, select Web Clipping Application, then click Create.

5. On the Manage Web Clippings page, click Add Web Clipping. The Find Web Clipping page in the Web Clipping Studio is displayed.

6. In the URL Location field on the Find Web Clipping page, enter the location of the first Web page that leads to the Web page you want to clip. This example starts with http://oraclestore.oracle.com, as shown in Figure 14-4.

   Figure 14-4 Find a Web Clipping

   
   Description of "Figure 14-4 Find a Web Clipping"
   
   
   
   

   Note: To import a WIDL file and create a Web clipping, select the Import from WIDL tab. See Section 14.4, "Migrating from Existing Transcoding Technologies" for more information.

   
   

7. Click Start.

   Web Clipping Studio displays the target page, which asks you to choose your store.

8. Select a country and language. For example, for United States, click English.

   The Oracle Store Web page for the country you chose appears in the Web Clipping Studio, as shown in Figure 14-5.

   Figure 14-5 Browse ORACLESTORE.ORACLE.COM in the Web Clipping Studio

   
   Description of "Figure 14-5 Browse ORACLESTORE.ORACLE.COM in the Web Clipping Studio"
   
   

   As you click hyperlinks in the Web page to browse to the content you want to clip, Web Clipping Studio records your navigation links.

   
   

   Note: Any browsing operations that do not contribute to the eventual Web clipping will be discarded. Thus, only the significant browsing operations are recorded for later playback during the test mode; any discarded links are not visited.

   
   
   
   

   Note: For any Web sites that require HTTP Basic or Digest Authentication, a form is displayed that requests user name and password information. This encoded authentication information is recorded as part of the browsing information.

   
   

   For this example, in the Quick Search field, enter J2EE and click Go. All products containing that name appear in the search results, as shown in Figure 14-6.

   Figure 14-6 Browse to the Content to Be Clipped

   
   Description of "Figure 14-6 Browse to the Content to Be Clipped"
   
   

9. In the Web Clipping Studio in browse mode, after you find the content you want to clip, click Section. This divides the target Web page into its clippable sections, as shown in Figure 14-7.

   Figure 14-7 Divide the Page into Clippable Sections

   
   Description of "Figure 14-7 Divide the Page into Clippable Sections"
   
   
   
   

   Note: After you click Section, you are no longer able to browse links in the displayed page. If you want to continue navigation, click Unsection.

   
   

10. In the Web Clipping Studio in section mode, find the section of the Web content you want to clip, then click Choose.

    To increase the number of sections available from which to choose, click Section Smaller. For example, you click Section Smaller to drill down one level of nested tables. To decrease the number of sections available from which to choose, click Section Larger.

    
    

    Note: You can click only one Choose section at a time. Only the content immediately below the Choose section icon you selected is chosen as a clipping. Continuing to preview mode will let you view the clipped content. Clicking Unselect will return you to section mode again to choose another section to clip if the section you had chosen was not the one you wanted.

    
    

    In the Web Clipping Studio, a preview of the Item section is displayed with the search results, as shown in Figure 14-8.

    Figure 14-8 Preview the Web Clipping in the Web Clipping Studio

    
    Description of "Figure 14-8 Preview the Web Clipping in the Web Clipping Studio"
    
    
    
    

    Note: Some sections may contain no data, only whitespace. For example, a Web page may contain an HTML <DIV> tag that contains no text or images. If you clicked Choose on a section that contains no data, Web Clipping displays a preview, but the preview correctly shows only whitespace. In this case, click Unselect to return to the sectioned page. Then, select a section containing data.

    
    

    If you do not want to use the section you clipped, click Unselect to return to the page containing the section. You can choose another section on the page, or click Unsection to navigate to another page.

11. If it is the section you want, click Scrape to continue to scrape mode.

12. In scrape mode, a check box following anything, such as text, indicates that you can select it as output that can be scraped. One or more items of output can make up a collection of similar output. In tables with cells, a special stacked check box is displayed to allow selection of all text in the cell as output.

    Click the check box at the end of the first output. For each output you select, a row of values appears under the Value and Name table in the bottom frame, where you can name your output. In the Name column, enter a meaningful name for your output, such as Books.

    
    

    Note: The name cannot contain any spaces.

    
    

    If the content you select contains an associated hyperlink, the value of the hyperlink will appear as an additional column in the resulting data source output. In addition, if you select an image, the value of the source (SRC) of the image will be output.

    After you select the output, you will have created a group that can be repeated as a whole as long as the first output you selected represents the first item in the collection of similar output. This means that you can only apply repeatability to a collection of similar output. Click More once to increase the repeat level to encompass the entire collection of similar output. Every item of output is checked, as shown in Figure 14-9.

    Figure 14-9 Scrape the Web Clipping

    
    Description of "Figure 14-9 Scrape the Web Clipping"
    
    

    Clicking More lets you quickly select the entire collection of similar output to be scraped. As you click More, check boxes of successive groups of selected output are checked and designated to be scraped. This is a quick way of selecting successive groups of similar output to be scraped as output, rather than individually checking each one.

    Click Continue.

13. In the Edit Clipping Attributes section of the Find Web Clipping page, you can set the following attributes of the Web Clipping application:

    • Title: The name of the Wireless Web Clipping application.

    • Description: A brief description of the Wireless Web Clipping application. Enter Search Oracle Store.

    • Time Out (seconds): The connection timeout or length of time in seconds that the Wireless Web Clipping server will wait to establish a connection to the source of the Web clipping application before it returns a timeout message. The default value is generated from the actual time measured to connect to the intranet Web site being clipped using the Web Clipping Studio, and it will be between 0 and 60 seconds.

    Figure 14-10 shows the top third of the Find Web Clipping page.

    Figure 14-10 Edit Clipping Attributes

    
    Description of "Figure 14-10 Edit Clipping Attributes"
    
    

14. If you went through form-based submission while clipping content for the Web Clipping application, a heading titled Select the Clipping Customizable Parameters appears in the Find Web Clipping page. A table lists the parameters.

    For each parameter that you want to page viewers to be able to customize, select the desired parameter from the drop-down box in the Parameters column, then select the Customizable column. Doing this lets OracleAS Wireless administrators customize the input value to personalize each parameter selected.

    
    

    Note: Do not make the user name and password fields into parameters that could be customized for Web sites that use form-based submission login. In addition, to enforce stronger security, the Web clipping does not support capturing JavaScript from Web sites that may perform additional JavaScript processing of the user name and password each time a login takes place.

    
    

    Figure 14-11 shows the Select the Clipping Customizable Parameters section of the Find Web Clipping page.

    Figure 14-11 Customize Form Input for a Web Clipping Application

    
    Description of "Figure 14-11 Customize Form Input for a Web Clipping Application "
    
    

15. In the row whose Index value is 2, select kw from the Parameters list. Then, change its Display Name from kw to keyword to make this name more recognizable for wireless application users.

16. Click the Customizable box to let wireless application users customize the parameter input value.

17. To test the Web clipping application, click Test in the Test the Web Clipping section as shown in Figure 14-12.

    Figure 14-12 Test the Web Clipping Application

    
    Description of "Figure 14-12 Test the Web Clipping Application"
    
    

    Results of the Web clipping test appear in a browser page as an XML document. Inspect the contents of the XML document to see if it contains the desired content. In a wireless device, this XML document would appear formatted.

18. Click Apply to save your changes when you have finished choosing the options.

    
    

    Note: Selecting Apply saves all edits and options for the Web Clipping application, which makes these changes immediately accessible to wireless application users while letting the OracleAS Wireless administrator make more edits as necessary on the Web Clipping application attributes page. On the other hand, selecting OK both saves all edits to the Web Clipping application attributes, and makes these changes immediately accessible to wireless application users, but exits the Web Clipping application attributes page and returns to the Applications page.

    
    

19. After you click Apply to save your changes, the Test the Web Clipping section of the Find Web Clipping page includes an Inputs heading, which lists the parameters that were customized, along with their initial input values, as shown in Figure 14-13.

    Figure 14-13 Test the Web Clipping Application Input Value

    
    Description of "Figure 14-13 Test the Web Clipping Application Input Value"
    
    

20. Notice the parameter named Keyword and its input value J2EE. To test another input value, replace the initial value and enter a new initial value, such as cache, as shown in Figure 14-14. Click Test to test the new input value.

    Figure 14-14 Change the Input Value

    
    Description of "Figure 14-14 Change the Input Value"
    
    

21. Results of this Web Clipping application test appear in a browser page. Inspect the contents of the XML document to see if it contains the desired content.

22. Click OK to save your changes and return to the Manage Web Clippings page as shown in Figure 14-15.

    Figure 14-15 Manage Web Clippings

    
    Description of "Figure 14-15 Manage Web Clippings"
    
    

    On the Manage Web Clipping page, you can:

    • Search for a Web clipping: In the Search field, enter a search string that matches any part of the name of a Web clipping, then click Go to initiate the search. The list of Web clippings whose names match this search string is displayed.

    • Edit a Web clipping: Select the Web clipping you want to edit by selecting it in the Select column. Then, click Edit. The Find Web Clipping page with the Edit Attributes section, as shown in Figure 14-10, appears. From this page, you can edit the Web clipping attributes.

    • Create a new Web clipping: Click Add Web Clipping. The Find Web Clipping page appears in which you can enter a starting-point URL and begin browsing a Web site, and looking for content to clip.

    • Delete a Web clipping: Select the Web clipping you want to delete by selecting it in the Select column and then click Delete. A confirmation message is displayed indicating the Web clipping is deleted.

    • Create a default application: Click Create Default Application to see a default rendering of your newly created Web clipping.

14.4 Migrating from Existing Transcoding Technologies

Previously in OracleAS Wireless, transcoding or Web content reuse was performed using Web Methods' Web Integration technologies. The recorded instructions on how and where to fetch external Web content was captured in a Web Integration Definition Language (WIDL) file. WIDL is a meta-language that implements a service-based architecture over the document-based resources of the World Wide Web. In this release, the responsibility of transcoding falls on Web Clippings.

You do not need to recreate the same set of instructions on Web Clippings terms. Web Clipping provides a migration path that allows for importing WIDL mapping files into Web Clippings so that you may reuse what you have captured previously. Figure 14-16 shows the augmented Web Clipping Server architecture, with the added WIDL Source as an alternate means of creating a Web Clipping.

Figure 14-16 Web Clipping Server Architecture Showing Importing a WIDL Definition File

Description of "Figure 14-16 Web Clipping Server Architecture Showing Importing a WIDL Definition File"

To migrate to Web Clipping:

1. To import a WIDL definition file and create a Web clipping from it, click the Web Clipping Studio tab. On the Find Web Clipping page, select the Import from WIDL subtab.

   Figure 14-17 shows the Import Web Clippings from WIDL page.

   Figure 14-17 Import from a WIDL Definition to a Web Clipping

   
   Description of "Figure 14-17 Import from a WIDL Definition to a Web Clipping"
   
   

2. At the Import Web Clippings from WIDL page, you can import an existing WIDL file, create a Web clipping, and save it in the repository.

   To import a WIDL definition file, you can either:

   • Enter the starting-point URL in the WIDL Location field and click Start.

   • Click Browse to browse to the directory on your system where your WIDL definition file is located and click the WIDL file to populate the WIDL Location field. Then, click Start to begin importing the WIDL definition file.

   The Import from WIDL page reappears, containing a new section Choose the WIDL service to import, as shown in Figure 14-18.

   Figure 14-18 Choose the WIDL Service to Import

   
   Description of "Figure 14-18 Choose the WIDL Service to Import"
   
   

3. If your WIDL file contains more than one service to be imported, select the service that you want to import from the Choose WIDL Service box.

4. If the WIDL service went through form-based submission (that is, it contains input bindings), the Import from WIDL page contains the section Set Parameters. In this section, you can specify default values for any parameter. In the Default Value column, and enter a default value for each chosen parameter, as shown in Figure 14-19.

   Figure 14-19 Specify Default Values for Parameters

   
   Description of "Figure 14-19 Specify Default Values for Parameters"
   
   

5. Click Continue. The WIDL service is imported and made into a Web clipping. The Find Web Clipping page appears.

6. In the Find Web Clipping page, you can edit the attributes of the new Web clipping, including customizing form input information if the Web clipping has form-based submission. Edit the clipping attributes, such as the Description field, then click OK to save your changes and return to the Manage Web Clippings page.

At the Manage Web Clippings page, you are ready to create a wireless application using the Web clipping you created.

14.5 Creating a Wireless Application

To create a wireless application from a Web clipping, you use either of the following approaches:

• Create a default application for development purposes to see a default rendering of your newly created Web clipping and to allow parameters to be customized. See Section 14.5.1, "Creating a Default Application" for more information.

• Use Java APIs to access the clipped data and customize the rendering. See Section 14.5.3, "Building a Custom Application" for more information.

14.5.1 Creating a Default Application

On the Manage Web Clippings page, as shown in Figure 14-15, you can create a default wireless application.

From the Select table, select the Web clipping you want to make into a default wireless application, and then click Create Default Application. A default wireless application is created to deliver a default rendering of the scraping just performed, in OracleAS Wireless XML.

After creating the default application, you are returned to the OracleAS Wireless Applications page, where you can create new Web Clipping applications.

Continuing with the example, the Oracle Store Search Result application is added to the list of applications, as shown in Figure 14-20.

Figure 14-20 Applications Page with New Wireless Application Listed

Description of "Figure 14-20 Applications Page with New Wireless Application Listed"

After you create a default application from a Web clipping, you can publish a new application. To create a new application link for a new wireless application, take the following steps:

1. Click the Content tab and then click Add Application Link to add this new application to the Applications Links page.

2. Follow the steps provided by the wizard. At the Application step, click the application on which you want to base this application. The page now shows a list of applications. Select your application, Oracle Store Search Result. Click Next.

3. At the General step, enter a name, such as ORACLESTORE, for your application, then click Next.

4. At the Input Parameters step, accept the default settings for your new application by clicking Next.

5. At the Async Application step, click Next.

6. At the Additional step, in the Description field, enter the word Books, then click Finish.

Your new application appears in the Applications Links list of applications, as shown in Figure 14-21. This enables an application to be published to user groups and made accessible to mobile users of wireless devices.

Figure 14-21 Application Links Page with New Application Published

Description of "Figure 14-21 Application Links Page with New Application Published"

From this page, you can also:

• Delete a Web Clipping application: Select the application to be deleted by selecting it in the Select column, and then click Delete. A confirmation message appears indicating that the selected Web Clipping application is deleted.

• Edit an existing Web Clipping application's attributes: Select the application to be edited by selecting it in the Select column, then click Edit in the Edit Clip in Web Clipping Studio section.

• Categorize an application: Selecting it in the Select column, then click Categorize.

14.5.3 Building a Custom Application

You can build a custom wireless application using the Manage Web Clippings page, as shown in Figure 14-15. (To navigate to this page, from the Services tab, click Web Clipping Manager.) Select the Custom Applications tab. The Custom Applications page appears, as shown in Figure 14-24.

Figure 14-24 Custom Applications Downloads Page

Description of "Figure 14-24 Custom Applications Downloads Page"

From the Downloads section of the Custom Applications page, you can:

• Download (for J2EE development) the Java Connector Architecture (JCA) Resource Adaptor Archive (RAR) for development within an instance of Oracle Application Server Containers for J2EE (OC4J) using JCA's Common Client Interface (CCI). Take the following steps:

  1. Click the RAR link in the J2EE Development section to download the JCA RAR file, webclipping.rar.

  2. Follow the instructions about deploying resource adapters in Oracle Application Server Containers for J2EE Services Guide to deploy an RAR file into your OC4J. See Oracle Application Server Containers for J2EE User's Guide for information about the correct version of OC4J.

     You may need to restart OC4J for the library to take effect.

  3. Return to the Web Clippings tab and find your desired Web clipping in the form of a JSP file to start developing using the RAR file you have just deployed.

• Download (for J2SE development) the Java library (JAR) for development using a standalone Java 2 SDK. Take the following steps:

  1. Click the JAR link in the J2SE Development section to download the Java library file, wcbean.jar.

  2. For a standalone Java 2 SDK environment, place the JAR file in the classpath during compilation and running of your code.

     For J2EE (OC4J) development, place this JAR file where your Web application can access it, either in the root classpath of OC4J, in the application's own library path, or in the WEB-INF/lib directory of the Web Archive (WAR) file containing your Web application. Depending on the resource adaptor you use, you may need to restart your OC4J instance to be able to use the Web Clipping Bean API.

     The other files that are needed in order to compile and run the stub Java files are http_client.jar, Javax-ssl-1_2.jar, jssl-1_2.jar, and xmlparserv2.jar. Look for these files in your installation of OC4J.

  3. Return to the Web Clippings tab and find your desired Web clipping in the form of a Java source file (.Java) to start developing using the JAR file you just deployed.

At the Web Clippings tab, you can do one of the following:

• Generate a Java file.

  Click Download Java (see Figure 14-15) to generate a Java (.Java) file from the Web clipping. This file can be compiled with other Java classes into an application, and deployed in J2SE as a standalone Java 2 SDK application.

• Generate a JSP file.

  Click Download JSP (see Figure 14-15) to generate a JSP (.jsp) file from the Web clipping. This file can be deployed into J2EE OC4J so you can execute the Web clipping as a JSP file.

• Create your own custom HTTP application using the downloaded Java file or JSP file.

  • Using the Java file

    Starting with the downloaded Java file, which contains the sample code of how to use the Web Clipping Bean APIs, you can create your own HTTP application using any standard J2EE Container (such as OC4J). You must first include the wcbean.jar file, as well as its dependency libraries, into the library path of the HTTP Server.

  • Using the JSP file

    With the downloaded JSP file, which contains the sample code of how to use the Java Connector Architecture APIs, you can create your own HTTP application using OC4J (this is a requirement as the RAR file is compliant with OC4J).

  • Building the Mobile UI

    To be compliant with OracleAS Wireless, the HTTP application must render its output in either OracleAS Wireless XML, XHTML/MP, or XHTML/XForms. You have full control over the Mobile UI of this HTTP application including the addition of mobile links or images. Note that the default JSP code renders the data as an HTML unordered list; you must change that rendering to one that is compliant with OracleAS Wireless.

  • Creating a Wireless Application

    After the creation of the HTTP application, you can use it to create a Wireless application using the HTTP Adaptor.

14.6 Administering Web Scraping

The administrative tasks related to Web Scraping that must be performed by the OracleAS Wireless administrator include:

• Configuring HTTP or HTTPS proxy settings. See "Configuring the Wireless Server" in Oracle Application Server Wireless Administrator's Guide for information about configuring the proxy server.

• Configuring security. See Section 14.6.1, "Configuring Security" for information.

• Rendering events to be logged and generating useful reports. See Section 14.6.2, "Rendering Events to Be Logged and Generating Reports" for information.

ORACLE_HOME/j2ee/OC4J_Wireless/applications/webclipping/ webclipping-web/WEB-INF/web.xml

ALTER SESSION SET CURRENT_SCHEMA=WCRSYS;

/**
* This describes a record type used to return a single clipping rendering
* event.
*
* This structure is used by the lookup APIs to encapsulate the 
* information that is retrieved from the wwwcp_render_log$ table. 
* It is used to describe the cursor that will be returned as an OUT 
* parameter of lookup_render_events.
*
* @field clip_id           The clip id that allows the fetching of the 
*                          other facets of the clipping definition to 
*                          populate the events table.
* @field clip_description  Textual description of the clip rendered.
* @field clip_title        Title of the clip that was rendered.
* @field clip_timeout      Timeout in milliseconds that allows the clipping
*                          that was rendered to be timed out. This could be
*                          an indication of the quality of service.
* @field effective_url     The url where the clip resides. This is usually
*                          the last url declared in the clipping definition
*                          clipping definition, where the clip would reside.
* @field render_status     A number that indicates either success or 
*                          failure of the rendering call.
* @field render_type       Tells what type of rendering is in question,
*                          whether it be for Portal Show Mode, for 
*                          Wireless default SimpleResult show mode, or 
*                          for the Wireless connector show mode.
* @field render_start      Starting time of the rendering.
* @field render_end        Ending time of the rendering.
* @field fuzzy_used        Indicates whether fuzzy match was kicked in.
* @field db_lookup         The time it takes in milliseconds to look up
*                          the clipping definition from the database.
* @field http_latency      The time it takes to reach the first byte to
*                          read from the http remote site.
* @field render_user       The user (together with a possible realm) for
*                          which the rendering was done.
* @field error_cause       The cause of the error if the status indicated
*                          a failure of the rendering.
* @field event_description This field provides more information about
*                          the logged event.
* @field clip_input        The input provided to render this clipping. It
*                          is usually provided in the form of an HTTP URL
*                          query string like abc=def.
* @field clip_output       The possible output (only if it's small) of 
*                          rendering, if it gives any more hints on the
*                          billing process.
*/
type rec_render_event is record (
    clip_id           integer,
    clip_description  varchar2(1024),
    clip_title        varchar2(512),
    clip_timeout      integer,
    effective_url     varchar2(2048),
    render_status     integer,
    render_type       integer,
    render_start      date,
    render_end        date,
    fuzzy_used        integer,
    db_lookup         integer,
    http_latency      integer,
    render_user       varchar2(512),
    error_cause       varchar2(256),
    event_description varchar2(256),
    clip_input        varchar2(128),
    clip_output       varchar2(256));

/**
 * This API looks up rows of render events from the wwwcp_render_log$
 * table, filtered by a specific clip id.
 *
 * The full rendering event information structure is returned as iterable
 * by the cursor returned.
 *
 * @param p_clip_id          The clip id as a filtering mechanism of the
 *                           events that were logged for this particular
 *                           clip.
 * @param p_cv_render_events The list of render events that are associated
 *                           with this clip.
 */
 procedure lookup_render_events(
     p_clip_id          in  integer,
     p_cv_render_events out render_event_cursor
 );

/**
 * This API looks up rows of render events from the wwwcp_render_log$
 * table, filtered by a specific effective url.
 *
 * This is useful for business queries that center on not the clip id but by
 * the actual location of where the clips are found. For example, if Yahoo's
 * partner, namely someone using Web Clippings, can do a search of all the
 * events that went to http://my.yahoo.com, it can find out how to charge
 * each user for what they have rendered.
 *
 * The full rendering event information structure is returned as iterable
 * by the cursor returned.
 *
 * @param p_effective_url    The effective url, starting with which points
 *                           to the Web page containing the clip.
 * @param p_cv_render_events The list of render events that are associated
 *                           with this clip.
 */
 procedure lookup_render_events(
     p_effective_url    in  varchar2,
     p_cv_render_events out render_event_cursor
 );
 
/**
 * This API looks up rows of render events from the wwwcp_render_log$
 * table, filtered by the domain.
 *
 * This is useful for business queries that center on not the clip id but by
 * the domain of where the clips are found. For example, if Yahoo can do a
 * search of all the events that went to *.yahoo.com, it can find out how to
 * charge each user for what they have rendered.
 *
 * The full rendering event information structure is returned as iterable
 * by the cursor returned.
 *
 * @param p_effective_domain The domain of the effective url, starting with
 *                          which points to the Web page containing the
 *                          clip. This domain is provided in the format
 *                          "oracle.com" or "yahoo.com".
 * @param p_cv_render_events The list of render events that are associated
 *                          with this clip.
 */
 procedure lookup_render_events(
     p_effective_domain in  varchar2,
     p_cv_render_events out render_event_cursor
 );
 
 /**
 * This API looks up rows of render events from the wwwcp_render_log$
 * table, filtered by the user that requested the render.
 *
 * This is useful for business queries that center on not the clip id but by
 * the user to visualize how much a user has rendered, regardless of where
 * the clipping is from, so it can charge for a flat fee based on for 
 * example, the number of render events for a particular user.
 *
 * The full rendering event information structure is returned as iterable
 * by the cursor returned.
 *
 * @param p_render_user      The user on behalf of whom the render request
 *                           is made.
 * @param p_cv_render_events The list of render events that are associated
 *                           with this clip.
 */
 procedure lookup_render_events(
     p_render_user      in  varchar2,
     p_cv_render_events out render_event_cursor
);

/**
 * This API removes all render events from the wwwcp_render_log$ table.
 *
 * This is useful as a cleansing measure that an organization can do between
 * the different phases of rolling out their Web Clipping usage initiative.
 * It can be used to reset the stage before going into development testing
 * phase, or when the rollout is going live and the company wishes to begin
 * charging its subscribers for their usages, or another possible cleansing
 * may occur at for example, every month end or year end when the data
 * recorded is no longer useful.
 */
 procedure remove_all_render_events;
 
/**
 * This API removes all render events from the wwwcp_render_log$ table for
 * a given rendering user.
 *
 * This is useful as a cleansing measure that an organization can do for
 * example, labeling the development user to be some constant user id and
 * then removing all rendering events when it goes live.
 *
 * @param p_render_user The user on behalf of whom the render request is
 *                      made.
 */
 procedure remove_render_events(p_render_user in varchar2);
 
/**
 * This API removes all render events from the wwwcp_render_log$ table for
 * a given starting or stopping time before or after the query time.
 *
 * This is useful as a cleansing measure that an organization can do for
 * example, all rendering events that started or stopped before a certain 
 * date are deemed development efforts and therefore should be disregarded.
 *
 * @param p_query_time   The date/time with which to query.
 * @param p_query_before Whether the query is for before or after. A value
 *                       of 1 signifies that the query is for comparing 
 *                       times before the p_query_time while a 0 value.
 *                       implies comparing times after the p_query_time.
 * @param p_query_start  Whether or not the query is for start time.
 *                       A value of 1 means true while 0 means false.
 */
 procedure remove_render_events(
     p_query_time   in date,
     p_query_before in number,
     p_query_start  in number
 );

14.7 Understanding the WML Translator

The OracleAS Wireless WML Translator reformats applications developed in WML for all wireless Web-enabled devices. At runtime, content developed in WML is converted to OracleAS Wireless XML, which is then transformed into the appropriate device specific markup language.

Generally, source documents in WML 1.x (up to WML 1.3) are supported. Each source document is processed in the following order:

• Validity of source WML is checked: Is it a valid XML? Is it a valid WML?

• All neighboring "p" elements under "card" elements are grouped. Grouped "p" elements are consolidated.

• Source WML is transformed into OracleAS Wireless XML.

• Navigation elements (if any, see Section 14.7.1, "Deploying and Configuring the WML Translator") are added to the result Wireless XML.

• URLs in result Wireless XML are converted to absolute URLs (if they are not), then further rewritten to point back to OracleAS Wireless Portal.

Table 14-1 shows a rough comparison between source elements in WML and translated elements in OracleAS Wireless XML.

The WML Translator does not support pass-through mode. Even if the end user's device supports WML, the WML Translator still translates source WML into OracleAS Wireless XML format.

URLs are generally rewritten to point to the Wireless and Voice Portal. However, URLs to images are not rewritten. Image adaptation is utilized in this release. For example, suppose you have the following WML element:

<img src="/images/yahooicon.wbmp" alt="Yahoo!"/>

It is translated into:

<SimpleImage src="http://wap.yahoo.com/images/yahooicon.wbmp" vspace="0" hspace="0" valign="bottom" alt="Yahoo!" addImageExtension="auto"/>

This image is adapted and rendered based on end user's device model.

There are certain constraints in current release of WML Translator:

• OracleAS Wireless XML has certain limitations that prevent flawless transformation from WML to Wireless XML. The most noticeable drawback is that Wireless XML does not have an event-processing model.

• The WML user agent equipped with WML translator is limited. It does not maintain a navigation history. It has very limited support of WML variables.

• WML script is not yet supported.

• Anchor submit: If there is an <anchor> element with a <go> child, only one <anchor> can appear in the card.

ORACLE_SERVICES_COMMERCE_TRANSLATOR_DEFAULT_CONNECTION
ORACLE_SERVICES_COMMERCE_TRANSLATOR_HELPER_WML
ORACLE_SERVICES_COMMERCE_TRANSLATOR_XSL_WML_FILENAME
 

ORACLE_SERVICES_COMMERCE_TRANSLATOR_SHOW_GOHOME
 

ORACLE_SERVICES_TRANSCODER_NAVIGATION
 

<?xml version="1.0" encoding="UTF-8"?>
<Navigation>
<NavigationItems>
<Item target="%value home.url%" 
label="Home"
showAs="Link" 
preferredLocation="Header" />
<Item target="%value service.parent.url%" 
label_prefix="Back"
showAs="Link" />
<Item target="http://www.oraclemobile.com" 
label="OracleMobile" 
showAs="Button" 
preferredLocation="Footer" />
</NavigationItems>
</Navigation>
 

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" 
elementFormDefault="qualified" attributeFormDefault="unqualified">
<xs:element name="Navigation">
<xs:complexType>
<xs:all>
<xs:element ref="NavigationItems" minOccurs="0"/>
</xs:all>
</xs:complexType>
</xs:element>
<xs:element name="NavigationItems">
<xs:complexType>
<xs:sequence>
<xs:element ref="Item" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Item">
<xs:complexType>
<xs:attribute name="target" type="xs:string" use="required"/>
<xs:attribute name="label" type="xs:string" use="optional"/>
<xs:attribute name="label_prefix" type="xs:string" use="optional"/>
<xs:attribute name="label_suffix" type="xs:string" use="optional"/>
<xs:attribute name="showAs" type="xs:string" use="optional"/>
<xs:attribute name="preferredLocation" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>

omp://oracle/services/commerce/translator?XLTORSITE=http%3A%2F%2Fwww.oraclemobile.com