Localization of Schema Languages

Internationalization, Localization and Locale

Internationalization is the process of making a product ready for its global use. Localization is the process of the actual adaptation of the product to a specific locale, that is a country, region or market. See the definition of Localization vs. Internationalization [i18n l10n] for further information on these terms.

Taking the area of schema languages, an example for schema internationalization is to provide markup to express directionality of text in scripts with mixed directionality. With such markup, document instances of the schema can be created by users who work with Arabic or Hebrew texts. The translation of element and attribute names mentioned in sec. “Introduction” can be part of the localization of the schema.

A key definition for this paper is the notion of a locale. LDML describes a locale as follows:

Unfortunately there is no general agreement about what a locale is. For example LDML and the POSIX locale model differ in many areas. Also, it is not clear whether language should be the center of the locale. Although this is often adequate, there are cases like time zone information, which are independent of a specific language or a distinguished set of languages.

This paper mainly follows LDML by using language as the center of a locale. Nevertheless, there are examples in the paper which do not rely on language like the description of locale specific currency names.

Locale Identification

Since a clear definition of the term locale is not possible, a problem arises: How is a locale identified? Or to put it differently: How can the choice of German versus English translated names, modified data types or other localization information like in fig. 1 be made explicit?

The solution is to use the locale identifiers defined in LDML. Since LDML puts language in the core of its local model, it is natural that it applies a standard for language identification and matching of so-called language tags: the IETF BCP 47 [Best Common Practice 47] [BCP 47]. Previously, BCP 47 was represented by RFC 3066 [RFC 3066]. Recently RFC 3066 was replaced by RFC 4646 [RFC 4646] and RFC 4647 [RFC 4647]. RFC 4646 describes the structure of a language tag, and RFC 4647 describes requirements for matching of language tags. Parts of the syntax of an RFC 4646 language tag and its application in LDML are introduced in fig. 2.

RFC 4646 SYNTAX (part)
Language-Tag  = langtag
                 / privateuse ; private use tag
                 / grandfathered; grandfathered 
   langtag       = (language
                    ["-" script]
                    ["-" region]
                    *("-" variant)
                    *("-" extension)
                    ["-" privateuse])
 
LDML LOCALE IDENTIFIER
locale_id := base_locale_id options?
base_locale_id := extended_RFC3066bis_identifiers
options := "@" key "=" type ("," key "=" type )*
SAMPLE IDENTIFIER
de_DE@collation=phonebook,currency=DDM

RFC 4646 defines a syntax for language tags. These consist of one or several subtags. The subtags provide information about language (language), script (script), region (region) and variants (variant). The values of these subtags are registered in the iana language subtag registry (http://www.iana.org/assignments/language-subtag-registry). In addition, there are extension (extension) subtags or subtags for private use (privateuse), which are not part of the subtag registry.

LDML extends such a language tag with zero or more keys. In fig. 2 there are keys for a collation (German phonebook order) and the currency (DDM which means "East German Ostmark"). Another difference is that LDML uses the delimiter _ instead of - between subtags.

The requirements of schema language localization described in sec. “Introduction” can make use of various parts of a locale identifier:

• Translation of names for elements and attributes: The translation might be specific to language, script or region. For example, the translation of the element name <city> can target the Japanese language identified via the locale_id "ja", that is a language subtag. An adequate translation would be 都市. If the translation should use a latin transliteration, the script Latn needs to be specified. The appropriate locale identifier should be locale_id "ja_Latn", the translation would be toshi. An example of the combination of language and region identifier in a locale is locale_id "en_US" versus locale_id "en_GB". These help to differentiate e.g. color from colour.
• Modification of data types: the CLDR data which will be used for data type modification mainly provides distinctions related to language, for example locale specific definitions for date values in Japanese locale_id "ja" versus German locale_id "de". Hence the language subtag will be used as a locale identifier for indicating the appropriate set of locale data.
  CLDR uses other subtags to specify items to be localized within each set of locale data. For example for the region, the items within the local data specified by these subtags are locale display names. In this paper, the locale display names for the region will be applied to localize the <country> element in fig. 1, and the locale display names for language will be used to localize the <language> element².
• Document type specific information about internationalization and localization: For this task, the whole set of subtags provided by RFC 4646 will be applicable. The reason is that it depends very much on the granularity of the localization project what locale identifier is appropriate. This is different to name translation and the data type modification tasks mentioned above.

Input from Existing Approaches to XML Localization

The following subsections describe existing approaches towards schema localization. These will be introduced here, modified and later used for a general schema language localization approach.

<define name="city-elem">
 <elementSpec ident="city">
  <altIdent xml:lang="de">Stadt</altIdent>
  <altIdent xml:lang="ja">都市</altIdent> ....</elementSpec>
</define>

<ldml>
 <identity> [...] <language type="en"/>
 </identity>
 <localeDisplayNames>
  <languages>
   <language type="de">German</language> [...] </languages>
  <scripts>
   <script type="Latn">Latin</script> [...] </scripts>
  <territories>
   <territory type="DE">Germany</territory> [...] </territories>
  <variants>
   <variant type="1901">Traditional German orthography</variant>
   <variant type="1996">German orthography of 1996</variant> 
 [...] </variants>
 </localeDisplayNames>
 <numbers>
  <currencyFormats>
   <currencyFormatLength>
    <currencyFormat>
     <pattern>¤#,##0.00</pattern>
    </currencyFormat>
   </currencyFormatLength>
  </currencyFormats>
  <currencies>
   <currency type="USD">
    <displayName>US Dollar</displayName>
   </currency>
  </currencies>
 </numbers>
 <dates> [...] <calendars>
   <calendar type="gregorian">
    <months>
     <monthContext type="format">
      <monthWidth type="wide">
       <month type="1">January</month>
      </monthWidth> [...] </monthContext>
    </months>
    <eras>
     <eraNames>
      <era type="0">Before Christ</era>
     </eraNames> [...] </eras>
    <dateFormats>
     <dateFormatLength type="full">
      <dateFormat>
       <pattern>EEEE, MMMM d, yyyy</pattern>
      </dateFormat>
     </dateFormatLength> [...] </dateFormats>
   </calendar>
  </calendars>
 </dates>
</ldml>

DOCUMENT 1 (local ITS markup):
<help xmlns:its="http://www.w3.org/2005/11/its" its:version="1.0">
 <head>
  <title>Building the Zebulon Toolkit</title>
 </head>
 <body>
  <p>To re-compile all the modules of the Zebulon toolkit
you need to go in the
<path its:translate="no">\Zebulon\Current Source\binary</path> directory.
Then from there, run batch file
<cmd its:translate="no">Build.bat</cmd>.</p>
 </body>
</help>
DOCUMENT 2 (global ITS rules):
<help>
 <head>
  <title>Building the Zebulon Toolkit</title>
  <its:rules version="1.0"  
   xmlns:its="http://www.w3.org/2005/11/its" its:version="1.0">
   <its:translateRule selector="//path | //cmd" translate="no"/>
  </its:rules>
 </head>
 <body>
  <p>To re-compile all the modules of the Zebulon toolkit
you need to go in the
<path>\Zebulon\Current Source\binary</path> directory.
Then from there, run batch file <cmd>Build.bat</cmd>.</p>
 </body>
</help>

Summary on Input to Schema Localization and Potential Implementation Mechanisms

The following table summarizes the input of existing approaches to (schema) localization and their modification made within this paper.

In the following section, a general approach towards schema language localization will be introduced. Before, some potential implementation mechanisms will be discussed here.

DSRL [Document Schema Renaming Language] [ISO/IEC 19757-8] provides a means to rename names of elements, attributes, processing instructions etc. from a source into a target vocabulary. DSRL basically provides the functionality of Architectural Forms [ISO/IEC 10744]. An example of DSRL is given in Fig. 6.

<dsrl:element-name-map
 target="po:purchaseOrder">poloc:Kaufbestellung</dsrl:element-name-map>
<dsrl:attribute-name-map 
 target="po:purchaseOrder[@orderDate]">Lieferdatum</dsrl:attribute-name-map>

The fragment of a DSRL document shows the renaming of the <purchaseOrder> element to <Kaufbestellung>, and the orderDate attribute to Lieferdatum.

An approach which can be used for the implementation of data type definitions and data type modification is DTLL [ Data Type Library Language] [ISO/IEC 19757-5], see fig. 7.

<datatype name="monthNamesGerman">
  <super type="month" />
  <parse name="month">
    <enumeration code="@name"
      values="document('months.xml')/months/month"/>
  </parse>
  <property name="Januar" select="$month/@january" />
  <property name="Februar" select="$month/@february" />
  <property name="März" select="$month/@march" />
</datatype>

The DTLL document specifies the relation of locale specific month names like März to their general counterpart march. DTLL allow for specifying much more complex relations than simply value mapping and could also be used for implementing other data type modifications described above.

In summary, both DSRL and DTLL could be used to implement some requirements for schema localization, but they are not used in this paper. The reason is that the goal of this paper is to show one framework for specifying localization and internationalization information in a schema. The implementation described in sec. “Implementation” goes a direct way to XSLT, which seems to be simpler than taking a step through DSRL or DTLL.

From a different point of view, the approach of this paper may look like a danger of getting in the way of the customization mechanisms already built into a schema language or related technologies. Both perspectives are understandable: the need to reduce the numer of involved technologies as much as possible, and the need to have one framework for a specific purpose. This paper puts an emphasis on the later perspective, taking the position of localization workers into account who might benefit from a single framework for their needs.

Outline

The outline of the approach is demonstrated in fig. 8.

<loc:localInfo locale="de_DE">
[...]
</loc:localInfo>
regular expression for locale value testing:
(
((([a-z]|[A-Z]){2,3})|(([a-z]|[A-Z]){5,8}))
(_(([a-z]|[A-Z]){4}))?
(_(([a-z]|[A-Z]){2}|\d{3}))?
(_(([a-z]|[A-Z]|\d){5,8})|(\d{1}([a-z]|[A-Z]|\d){3}))?
)
|
(
(([a-z]|[A-Z]){1,3})((_([a-z]|[A-Z]){2,8}){1,2})?
)"/>

The container for locale related information is an <localInfo> element with a mandatory locale attribute. The value of that attribute specifies the target locale. The figure contains the regular expression which is used for testing the locale value. It is based on the ABNF in sec. 2.1 of RFC 4646. The difference is that the delimiter between sub tags - is replaced with _, to follow the LDML convention mentioned in sec. “Locale Identification”.

Locale information can be applied as schema annotation or in a separate document. The former usage is applicable for XML Schema or RELAX NG and will be exemplified in most of the following sections. The latter usage is used to apply such a description to XML DTDs⁶. It is exemplified in fig. 9.

<loc:localInformation xmlns:loc="http://example.com/schemalocalization"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xs="http://www.w3.org/2001/XMLSchema" 
 xsi:schemaLocation="http://example.com/schemalocalization localInfo.xsd">
 <loc:localInfo locale="de"
  targetDeclaration="xs:element[@name='purchaseOrder']"
  generalization="Kaufbestellung"
  xmlns:po="http://example.com/purchaseOrder">
  <loc:altIdent>Kaufbestellung</loc:altIdent>
 </loc:localInfo>
 <loc:localInfo locale="de"
  targetDeclaration="xs:element[@name='purchaseOrder']"
  generalization="Kaufbestellung">
  <loc:altDocumentation>Schema zu Kaufbestellungen</loc:altDocumentation>
 </loc:localInfo>
 <loc:localInfo locale="de" 
 targetDeclaration="xs:attribute[@name='orderDate']"
 generalization="Kaufbestellung/@Lieferdatum">
  <loc:dateInfo calendarType="Gregorian" dateFormatLengthType="long"/>
 </loc:localInfo>
 <loc:localInfo locale="de" 
 targetDeclaration="xs:element[@name='language']" generalization="Sprache">
  <loc:localeDisplayNames type="languages"/>
 </loc:localInfo>
 <loc:localInfo targetDeclaration="xs:element[@name='purchaseOrder']"
  translate="no" locale="de">
  <loc:locNote>Localization already available in German. Make sure that the
   exising localization can be reused as much as possible. </loc:locNote>
 </loc:localInfo>
</loc:localInformation>

The document contains all locale information which will be discussed in the following sections. To be able to apply this information to global or local declarations in an XML Schema document, the XPath expressions in the targetDeclaration attribute can be exploited. An additional, optional generalization attribute provides information for the transformation to an instance of the general schema.

During the development of this approach, XML Schema: Component Designators [SCD] have been considered as an alternative means to XPath for selecting declarations. This idea was dropped since XPath is applicable to various schema languages, whereas component designators are specific to XML Schema.

Adaptation of Names

An example of this functionality is given in fig. 10:

<xs:element name="purchaseOrder">
    <xs:annotation>
        <xs:appinfo>
            <loc:localInfo locale="de_DE">
                <loc:altIdent>Kaufbestellung</loc:altIdent>
            </loc:localInfo>
        </xs:appinfo>
    </xs:annotation> ... </xs:element>

The <localInfo> element contains an <altIdent> element which fulfils a similar functionality as the <altident> element in the TEI localization approach described in sec. “The TEI Approach towards Markup Language Localization”. The difference is that the <altIdent> element here can be applied to locally declared elements and global elements in an XML Schema.

The mapping to a locale unspecific representation (i.e. from <Kaufbestellung> to <purchaseOrder> is realized by exploiting the generalization attributes in fig. 9. The XPath expressions can be used to transform an instance of the localized schema to an instance of the general schema.

Translation of Documentation

This functionality is demonstrated in fig. 11.

<xs:element name="purchaseOrder">
    <xs:annotation>
        <loc:localInfo locale="de_DE">
            <loc:altDocumentation>Schema zu
Kaufbestellungen</loc:altDocumentation>
        </loc:localInfo> ... </xs:annotation> ... </xs:element>

This approach is again very similar to the TEI localization approach described in sec. “The TEI Approach towards Markup Language Localization”. The difference is again that it can be applied both for global and local markup declarations.

Adaptation of Simple Data Types

<xs:attribute name="orderDate" type="xs:date">
    <xs:annotation>
        <xs:appinfo>
            <loc:localInfo locale="de">
                <loc:dateInfo calendarType="Gregorian" 
 dateFormatLengthType="long"/>
            </loc:localInfo>
        </xs:appinfo>
    </xs:annotation>
</xs:attribute>

<xs:element name="language">
    <xs:annotation>
        <xs:appinfo>
            <loc:localInfo locale="de">
                <loc:localeDisplayNames type="languages"/>
            </loc:localInfo>
        </xs:appinfo>
    </xs:annotation> ... </xs:element>

Document Type specific Information about Internationalization and Localization

Fig. 14 demonstrates how "Translate" and "Localization Note" related information can be provided for the <purchaseOrder> element.

<xs:element name="purchaseOrder">
    <xs:annotation>
    <loc:localInfo targetDeclaration="xs:element[@name='purchaseOrder']" translate="no"
    locale="de">
        <loc:locNote>Localization already available in German.
Make sure that the existing localization can be reused as much as possible.</loc:locNote>
    </loc:localInfo>
    </xs:annotation> [...] </xs:element>

The <localInfo> element contains the translate attribute and the <locNote> element. Their function is identical to the markup described in ITS 1.0. The difference is that in the case described in fig. 14 they apply to all instances of the markup which they are attached to. In the case of the "standoff" usage described in fig. 9, they apply to all element or attribute declarations selected by the targetDeclarations attribute.

It depends on the application (e.g. a translation or localization tool), how this information should be processed. Example applications are given in sec. 2 of ITS 1.0.

Example of a Localized Schema

The example schema in fig. 15 implements the requirements formulated in sec. “Introduction”, using the existing approaches described in sec. “Background”. It can be generated relying on the markup described in sec. “Localization of Schema Languages: Realization”.

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
 targetNamespace="http://example.com/purchaseOrderLocalized"
 xmlns:loc="http://example.com/schemalocalization"
 xmlns:poloc="http://example.com/purchaseOrderLocalized">
 <xs:annotation>
  <xs:appinfo>
   <loc:baseSchema uri="purchaseOrderExample.xsd"/>
  </xs:appinfo>
 </xs:annotation>
 <xs:element name="Name" type="xs:string"/>
 <xs:element name="Straße" type="xs:string"/>
 <xs:element name="Stadt" type="xs:string"/>
 <xs:element name="Bundesland" type="xs:string"/>
 <xs:element name="PLZ" type="xs:string"/>
 <xs:element name="Land">
  <xs:simpleType>
   <xs:restriction base="xs:string">
    <xs:enumeration value="Vereinigte Staaten"/>
    <xs:enumeration value="Deutschland"/>
    <xs:enumeration value="Japan"/>
    <!-- ... -->
   </xs:restriction>
  </xs:simpleType>
 </xs:element>
 <xs:element name="Sprache">
  <xs:simpleType>
   <xs:restriction base="xs:string">
    <xs:enumeration value="Englisch"/>
    <xs:enumeration value="Deutsch"/>
    <xs:enumeration value="Japanisch"/>
    <!-- ... -->
   </xs:restriction>
  </xs:simpleType>
 </xs:element>
 <xs:element name="Preis">
  <xs:complexType>
   <xs:simpleContent>
    <xs:extension base="xs:decimal">
     <xs:attribute name="Waehrung">
      <xs:simpleType>
       <xs:restriction base="xs:string">
        <xs:enumeration value="US Dollar"/>
        <xs:enumeration value="Yen"/>
        <xs:enumeration value="Europäische Währungseinheit (XBB)"/>
        <!-- ... -->
       </xs:restriction>
      </xs:simpleType>
     </xs:attribute>
    </xs:extension>
   </xs:simpleContent>
  </xs:complexType>
 </xs:element>
 <xs:element name="Kommentar">
  <xs:complexType mixed="true">
   <xs:sequence>
    <xs:any processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
   </xs:sequence>
  </xs:complexType>
 </xs:element>
 <xs:element name="Lieferaddresse">
  <xs:complexType mixed="false">
   <xs:sequence>
    <xs:element ref="poloc:Name"/>
    <xs:element ref="poloc:Straße"/>
    <xs:element ref="poloc:Stadt"/>
    <xs:element ref="poloc:Bundesland" minOccurs="0"/>
    <xs:element ref="poloc:PLZ"/>
    <xs:element ref="poloc:Land"/>
    <xs:element ref="poloc:Sprache"/>
    <xs:element ref="poloc:Preis"/>
    <xs:element ref="poloc:Kommentar" minOccurs="0"/>
   </xs:sequence>
  </xs:complexType>
 </xs:element>
 <xs:element name="Kaufbestellung">
  <xs:complexType mixed="false">
   <xs:sequence>
    <xs:element ref="poloc:Lieferaddresse"/>
   </xs:sequence>
   <xs:attribute name="Lieferdatum" use="required">
    <xs:simpleType>
     <xs:restriction base="xs:string">
      <xs:pattern
       value="\d{2}\.\s+(Januar|Februar|März|April|Mai|Juni|Juli|August
 |September|Oktober|November|Dezember)\s+\d{4}"
      />
     </xs:restriction>
    </xs:simpleType>
   </xs:attribute>
  </xs:complexType>
 </xs:element>
</xs:schema>

As an example target locale, a German user will be assumed. For such a user, all element names are translated into German, and the types of elements are modified (see for example the Land element). The elements contain no information about the localization of document instances, e.g. the translate attribute. This information will be used not for the localized schema, but instance documents of the locale unspecific schema.

The pattern of the data type for the <Lieferdatum> element is based on the pattern d. MMMM yyyy. However, this pattern is relaxed regarding whitespace, to allow the user for a convenient input of data.