[RFC] Add Unicode support

[nojb]

This is a first try to adding Unicode support to ocaml-re. The approach is the one suggested at the end of #24. Namely, translate unicode regular expressions into byte-oriented regular expressions that match UTF-8-encoded strings.

This patchset adds a new module, Re_unicode which defines the type of unicode regular expressions. A new type is needed because it is not safe to mix unicode and byte-oriented regular expressions. The interface is a slightly modified version of the existing interface in Re. There is also a corresponding findlib library re.unicode. The module Re_unicode uses uutf to decode UTF-8 strings, but this is not a hard dependency and could easily be swapped out later to keep dependencies to a minumum.

On the implementation side, Re_unicode uses the same implementation as Re, but overloading the Set variant to mean sets of unicode code points. Before compilation the unicode regular expression is traversed (see handle_unicode in re_unicode.ml) to translate unicode character sets into suitable byte-oriented regular expression. After this step, everything goes through as before.

The same POSIX character classes (alnum, digit, etc.) present in Re are offered by this module, but implemented in terms of unicode as suggested by UTS #18. More fine-grained unicode character sets can be added easily. The definition of unicode character sets depends on static data contained in the module Unicode_groups. This module can be regenerated from the Unicode Character Database by a tool called gen_unicode_groups found in the tools/ directory. This tool depends on the uucd library.

There are some difficulties in integrating unicode elegantly to the current code base arising from the fact that this library makes some assumptions about the character set when compiling and interpreting REs:

• The bol, eol, eow, bow combinators are ASCII-specific and handled specially in the RE engine. These are not currently present in the Re_unicode interface.
• (BREAKING CHANGE) The case, no_case combinators have different semantics for unicode because a single unicode character can change case to a sequence of unicode characters, so it is no longer true that applying one of these combinators to a character set gives a character set. In the present patchset the result of applying one of these combinators to a character set is no longer considered to be a character set (which is the right behaviour for unicode, I think). This can be reverted back to the old behaviour for the Re module with a little bit of refactoring. This is solved by using the simple_case_folding property of unicode characters which is 1:1, and so does not suffer from the above problem. UPDATE The case, no_case combinators are now working with Unicode.
• The ~pos and ~len arguments in the exec*, all*, split* functions require indexing into a UTF-8 string which takes time linear on the length of the string. These arguments are not present in the current changeset, but could be easily added.

In summary, I think that, long-term, the best course of action would be to refactor the library so that the core is completely independent of the character set used and deals only with arbitrary bytes. Both ASCII and unicode engines should be added on top. Right now, there is some special handling of the bol, eol, eow, bow combinators in the RE engine that would have to be factored out, but someone more knowledgeable about the internals needs to explain what is involved in this.

Any and all comments welcome!

[vouillon]

This is great! Thank a lot!

I agree the library will need to be refactored.

Regarding the ~pos and ~len arguments, I think we can use byte values for them. We could experiment with a substring algebra to abstract them (see A Subsequence Algebra: First Class Values for Substrings, Wilfred J Hansen).

The bol, eol, eow and bow combinators are going to be tricky to implement. I need to think about that.

[nojb]

Great! Regarding the problematic combinators (bol, eol, eow, bow), it looks like they are special cases of lookahead/lookbehind. How hard would it be to add general lookahead/lookbehind to the RE engine?

[vouillon]

Indeed, these combinators are special cases of lookahead and lookbehind, and I think that's how they should be implemented.

The way to implement a lookbehind subexpression (?<=r) is by matching with .*(r) in parallel. Then, the subexpression (?<=r) matches at a given position in the string if we have a match for .*(r) ending at this position.

The implementation of ocaml-re is DFA-based. Intuitively, each state of the DFA is the disjunction of the positions we may have reached in the regular expression. In fact, we have a tree structure rather than simply a disjunction to deal with the matching semantics (longest, shortest or first match). To deal with lookahead, we will need something like a disjunction of conjunctions (both the main regular expression and the lookahead expression(s) must match).

We need a couple of restrictions on lookbehind and lookahead expression to make this work:

• the lengths of the set of strings matched by a lookbehind expression must be bounded, so that one knows how many bytes before the start position one should start parsing to get enough context (this is a standard restriction);
• we cannot have lookahead expressions inside lookbehind expressions (as we must be able to decide whether a lookbehind expression matches right before a given position without looking beyond this position).

And it's going to be much simpler to implement if we do not allow group matching inside lookahead and lookbehind expressions.

[m2ym]

👍

[nojb]

A fun hack, but no more than that.