Ripper is not all that complicated a program, though its character-by-character parsing logic isn't particularly delightful. For the most part, it trudges through the document, keeps track of both lexical and structural context, and reports what it finds. There are a few cases where markup is so badly wrong that it can't be processed even at Ripper's relatively simple level, and these are reported as errors.

The only particularly difficult part of writing Ripper was created by Namespaces in XML. Prior to namespaces, an XML document could be parsed directly in sequence. Everything needed to know to parse a given part of a document, if anything, came from earlier parts of the document. Because of namespaces, however, the parser frequently needs to read to the end of the start tag to interpret the element name at the beginning of the tag. Namespaced attributes often have the same problem, with namespace declarations that come after the prefix has already been used.

Solving this problem requires parsing the start tag twice. The first parse is used to set the context, including namespace context, and the second parse is used to report the text to the application. Using this approach, the application will have the namespace context it needs to interpret element and attribute names as they arrive.

Unfortunately this double-parse has created some duplicate code, both for the double-parse itself and for ampersand and entity handling. Attribute values may of course include entities and character references, even attribute values which happen to be used by namespace declarations. For purposes of context, Ripper resolves these entities, but it then reports the unresolved entities separately during the reporting phase.

Ripper is distributed as part of the Gorille project, all of which is licensed under the Mozilla Public License (MPL). Gorille includes Ripper, rules-based character testing code, and some common code used to create shorthand descriptions of document structures. Ripper (and all of Gorille) is written in Java, and requires at least Java 1.2.

While most of the arguments about XML 1.1 focus on the NEL character and whether or not change of any kind is a good thing for the core of XML, XML 1.0 certainly helped create its own versioning problem. The list of characters included in XML 1.0 was illuminating and useful, but it was also built so deeply into parsers that changing it now is difficult.

Ripper can't fix all of the old parsers, but it does offer an approach that may be useful in the future. Ripper builds only its expectations for the markup characters themselves into the parsing logic, and leaves determination of whitespace and other acceptable characters to the application. As Gorille integration with Ripper proceeds, this information will become available through the Context object and applicable through a dedicated layer or processing.

This separation of context from the parsing logic also means that it is possible to configure context and then process document fragments within that context. This greatly simplifies the processing of external entities which themselves contain entity references, to give one common XML 1.0 example. It may also help in the processing of document fragments which are missing their namespace declarations.

The most recently prominent use case involves the current set of issues surrounding character entities, where DTDs, particularly the internal subset, are used to provide entity declarations in otherwise schema-centric (or completely unvalidated) environments. Some developers would prefer not to deal with DTDs at all, and there are now a fair number of environments (BEEP's core vocabulary, SOAP messages) where the DOCTYPE declaration is prohibited. This creates problems for some developers, notably those using MathML with its many frequently-used entities.

Because Ripper reserves entity processing to the application, applications can solve problems like these with entity resolvers focused on their particular needs rather than the expectations of a given parser. An application could even resolve entities based on their current namespace or element scope, making it possible to create entity vocabularies which are associated with particular structural vocabularies rather than with a single document. This could potentially reduce name collisions between the entities used by different vocabularies, a problem avoided today by copying and coordination.

While Ents only provides support for character entities, Ripper's foundations are flexible enough that applications can summon a new instance of Ripper to parse an external entity and integrate it with the existing document, if desired. If the application takes care to preserve context objects, those can be combined to provide support for complex cases like nested entities which rely on namespace declarations from the parent.

Another use case involves situations where otherwise unimportant details of an XML document are used to mark content which needs special treatment - a transformation that only applies to attributes with single quotes, for example. While such work doesn't necessarily accord with an Infoset view of XML, and has often "flown under the radar" it can still be a practical means of combining and massaging information from different sources.

Unfortunately for the "Desperate Perl Hacker", XML is not exactly conducive to simple manipulation with regular expressions. Entity references and namespace prefixes both serve as abbreviations for information declared elsewhere, and default attributes can also make such processing difficult. Also, while transformations are a critical part of XML processing, such transformations may throw away information, notably comments, processing instructions, and whitespace - which are actually useful to developers manipulating documents as text.

Ripper doesn't solve these problems automatically, but it provides a framework within which developers can combine textual processing and an understanding of the markup context.

Developers can also build their own parsing or storage logic on top of this API. Because the entire document will be reported verbatim, it's possible to use this information to report an XML document to an application while preserving its original form more precisely than is possible with approaches like SAX or DOM. (Encoding issues may still keep it from being byte-for-byte identical, but character-for-character is plausible.)

There are a number of uses for this kind of processing. Environments like MOE, which support more features than are provided by SAX or DOM, have always been limited by the kinds of parsers available to them. A Ripper application could easily create MOE events, effectively building a MOE parser. Lexical analysis programs might also use Ripper as a front end to their own work.