D Globalization Support in the Directory

D.1.1 About Unicode

No single character set contains enough characters to meet the requirements of day-to-day e-business requirements. For example, no one national character set can represent all the languages in the European Union. Moreover, there are potential conflicts between character sets because the same character can be represented by different codes in different character sets.

To overcome these obstacles, a global character set, called Unicode, was developed. It is a universal encoded character set that can store information from any language including punctuation marks, diacritics, mathematical symbols, technical symbols, musical symbols, and so forth. As of version 3.2, the Unicode Standard supports over 95,000 characters from the world's alphabets, ideograph sets, and symbol collections. It includes 45,000 supplementary characters, most of which are Chinese, Japanese, and Korean characters that are rarely used but nevertheless need representation in electronic documentation.

Unicode has more than one implementation standard, and these are described in Table D-1.

D.1.2 About Oracle and UTF-8

Oracle began supporting Unicode as a database character set beginning with Oracle database version 7. With Oracle9i, Oracle added a new UTF-8 character set called AL32UTF8. This database character set supports the latest version of Unicode (3.2), including the latest supplementary characters. Oracle intends to enhance AL32UTF8 as necessary to support future versions of the Unicode standard.

D.1.3 Migration from UTF8 to AL32UTF8 when Upgrading Oracle Internet Directory

Oracle Internet Directorynow supports AL32UTF8. If you have upgraded Oracle Internet Directory from a version prior to 10g Release 2 (10.1.2), then, for better performance, Oracle recommends that you change the character set for the directory database from UTF8 to AL32UTF8. To do this:

1. Run the character set scanner (CSSCAN) to ensure that there are no invalid UTF8 characters inside your current database.

2. Run the CSALTER script to update the database to AL32UTF8.

D.4.1 An LDIF file Containing Only ASCII Strings

In this scenario, character strings for attribute values are also in ASCII.

Because all tools use the UTF-8 character set by default, and ASCII is a proper subset of UTF-8, all tools can interpret these files. The same is true of keyboard input of values that are simply ASCII strings.

D.4.2 An LDIF file Containing UTF-8 Encoded Strings

In this scenario, character strings for attribute values are also in UTF-8.

Because, by default, all tools use the UTF-8 character set, all tools can interpret these files. The same is true of keyboard input of values that are UTF-8 strings.

In such a file, some characters may be multibyte. Multibyte characters strings can be present in the LDIF files as attribute values or given as keyboard input. They can be encoded in their native character set or in UTF-8. They can also be BASE64 encoded representations of either the native or the UTF-8 string.

Consider the following cases:

• CASE 1: Native Strings (Non-UTF-8)

• CASE 2: UTF-8 Strings

• CASE 3: BASE64 Encoded UTF-8 Strings

• CASE 4: BASE64 Encoded Native Strings

Because the directory server understands and expects only UTF-8 encoded strings, cases 1, 3, and 4 need to undergo conversion to UTF-8 strings before they can be sent to the LDAP server.

ldapsearch -h my_host -p 389 -E ".ZHS16GBK" -b base_DN  -s base  "objectclass=*"

D.5.1 Specifying the -E Argument When Using Each Tool

The client tools always assume UTF-8 (the Oracle character set name is AL32UTF8) to be the character set unless otherwise specified by the -E argument. The BASE64-encoded values are decoded, and then the decoded buffer is converted to UTF-8 if the -E argument is specified. For example, if you specify -E ".ZHS16GBK", then the decoded buffer is converted from simplified Chinese GBK to Unicode UTF-8 before being sent to the directory server.

Specifying the -E argument ensures that proper character set conversion can occur from the character set you specify for the -E argument (-E ".character_set") to the AL32UTF8 character set.

The command-line tools use the -E argument to process the input in the character set specified for the -E argument. They display their output in the character set specified in the NLS_LANG environment variable.

For example, to add entries from an LDIF file encoded in the simplified Chinese character set (ZHS16GBK) by using ldapadd, type:

ldapadd -h myhost -p 389 -E ".ZHS16GBK" -f my_ldif_file

In this example, the ldapadd tool converts the characters from ".ZHS16GBK" (simplified Chinese character set) to ".AL32UTF8" before they are sent across the wire to the directory server.

D.5.2 Examples: Using the -E Argument with Command-Line Tools

Table D-3 provides additional examples of how to use the -E argument correctly for each command-line tool. In each example, the command converts data from simplified Chinese, as specified by the value ".ZHS16GBK", to AL32UTF8. For example, in each command, the values for the -D and -w options are in GBK. Specifying the -E argument converts them to UTF-8.

Note that, in the examples in Table D-3, we do not show any actual characters belonging to the .ZHS16GBK character set. These examples would, therefore, work without the -E argument. However, if the argument values contained actual characters in the .ZHS16GBK character set, then we would need to use the -E argument.

D.7.1 Using Globalization Support with bulkload

Add to the command the argument -encode "character_set" where the input LDIF file is encoded in "character_set".

For example:

bulkload.sh -connect connect_string -encode ".ZHS16GBK" my_ldif_file

D.7.2 Using Globalization Support with ldifwrite

The ldifwrite utility always writes BASE64 encoded values for multibyte strings.

The BASE64 encoding could be of the UTF-8 strings as they are stored in the directory server, or of native strings as specified by the NLS_LANG environment variable setting when running ldifwrite.

For example:

ldifwrite -c connect_string -b baseDN -f output_file

In this example, if the NLS_LANG environment variable is not set, or is set to language_territory.AL32UTF8, then the output LDIF file will contain BASE64-encoded UTF-8 strings for any multibyte characters.

To reload this LDIF file into the directory by using ldapaddmt, use the following syntax:

ldapaddmt -h my_host -p port_number -f output_file

In this case, the -E argument is not required because the decoded BASE64 strings are already UTF-8-encoded and can be readily sent to the server.

If the NLS_LANG environment variable is set to a character set other than AL32UTF8—for example, ".ZHS16GBK"—then the output LDIF file will contain a BASE64 encoded value of simplified Chinese GBK strings.

To reload this LDIF file into the directory using ldapaddmt, use the following syntax:

ldapaddmt -h host -p port -E ".ZHS16GBK" -f my_input_file.LDIF

In this case the -E argument is required because the decoded BASE64 strings are simplified Chinese GBK, which need to be converted to UTF-8 strings before being sent to the server.

D.7.3 Using Globalization Support with bulkdelete

Add -encode ".character_set" to the command.

For example:

bulkdelete.sh -connect connect_string -encode ".ZHS16GBK" \
              -base "ou=manufacturing,o=acme,c=us"

In this case the value for the -base option could be in the ZHS16GBK native character set, that is, simplified Chinese.

D.7.4 Using Globalization Support with bulkmodify

Add -E ".character_set" to the command the argument.

For example:

bulkmodify.sh -c my_service_name -E ".ZHS16GBK" \
              -b "ou=manufacturing,o=acme,c=us" -r title -v Foreman \
              -f "objectclass=*"

In this example, values for the -b, -v, and -f arguments can be specified using the simplified Chinese GBK character set.