-
-
Notifications
You must be signed in to change notification settings - Fork 570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add core:text/regex
#3962
Add core:text/regex
#3962
Conversation
class_data := vm.class_data[operand] | ||
next_rune := vm.next_rune | ||
|
||
check: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOVELY use of a named block!
Simplified error checking while I was at it, too.
The `original_pattern` introduced a tenuous dependency to the expression value as a whole, and after some consideration, I decided that it would be better for the developer to manage their own pattern strings. In the event you need to print the text representation of a pattern, it's usually better that you manage the memory of it as well.
This should hopefully avoid any issues with loading operands greater than 8 bits on alignment-sensitive platforms.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's not too much, can you provide some documentation to public procedures and types before merging? I see that you did a great job documenting the packages themselves. A lot of procedures are obvious as to what they do, but other procedures would like to have some clarifications in usage.
Also some of the features that might be wanted:
I'm not sure if any of these are implemented, but I'm seeing you didn't mention them on your PR. I haven't had the time to check the code yet beyond some light skimming |
They've all been documented already.
Which ones did you have trouble understanding?
Already implemented.
Could be implemented.
Already implemented ( |
We do have |
That's certainly doable with the individual shorthands, in isolation: that we could use special opcodes for the various Unicode classes. The real issue arises when someone wants to have a class with one of those shorthands and some additional characters. I.e. Any special class-based opcodes will also require an extra step in the optimizer to check if a user has supplied a pattern that matches any of what these opcodes represent, i.e. if I wrote some class that had every character checked by Character class shorthands are one of the more maintenance-heavy parts of this implementation, as can be seen at the three |
Good points. |
We don't directly support printing these. To prevent future issues being raised about the pattern being missing if someone tries to print one, hide everything.
I thought about this for a few days, and I've decided to leave the shorthand classes as they are: ASCII only. It's simpler this way. I've written my rationale in the documentation along with a workaround if someone wants their own shorthand. If there's a great need or provable utility for expanding them, we can look at that another time. Also added some more documentation in general, along with an extra test. No major changes since I last checked in. Ready to go. |
@Feoramund I have been using your regex package since 2 weeks now, it's very solid, thanks! |
So this PR is amazing, but I am just contemplating whether this package should be in core or not simply because: Should people be using regex in the first place when other better alternatives exist? If it exists in core, people will use and abuse it, and think because it is in core, it is the recommended way without looking for things like |
This is a very reasonable cautionary consideration, given the usage history of regular expressions. If you end up not wanting it in Odin, I can move it to a repo of my own, but I'm glad to make any more technical changes you might like if that ends up not being the case. However, I'll say that I started on this work because I saw it as a candidate listed in #978 and thoroughly read through the writings of Russ Cox you linked back then. I know it's a discussion of almost 4 years old now, but it looked like the most interesting target at the time, for learning for myself and utility to others. That said, I'm not arguing for or against its inclusion into Odin. |
Regular Expressions
I've been testing, benchmarking, and optimizing this over the past couple weeks, and now I'm finally ready to share it with everyone. I realize a package like this is going to be opening a Pandora's box of sorts, given the reputation that regular expressions have for being misused, but I've included some helpful tips on when and when not to use RegEx in the documentation. With that said, let's get into it.
First of all, this is a complete implementation of a multi-threaded virtual machine based off of the writings of Russ Cox with designs attributed to Ken Thompson and Rob Pike. The VM dispatches runes, one at a time, running each thread in lock step with no look-behind, guaranteeing linear time execution.
Feature Set
[0-9]
.
apple|cherry
a*
a+
a?
(a)
(?:a)
^$
\b\B
Compile-Time Flags
g
m
^
and$
to behave as if they also match newlines.i
x
u
n
-
Construction
Regular expressions may be compiled either by specifying the pattern and a
bit_set
of flags, or a delimited string.The following three ways are equivalent.
For
create_by_user
, the delimiter is determined by the first rune in the string and has the only requirement that it not be\
.Matching
Regular expressions are matched against strings after compilation. No API is provided to match an arbitrary pattern; they must be compiled first.
Benchmark Results
Here are the results from the included benchmark, output modified only for readability.
Command:
odin test . -define:ODIN_TEST_THREADS=1 -o:speed -disable-assert -no-bounds-check
Implementation Details
The construction stage starts with a tokenizer for the pattern string, goes to a Pratt parser to resolve precedence parsing, then to a bytecode-based compiler for maximum compactness. The original implementation of the compiler and virtual machine used an array of unions to structs. The current bytecode design is much smaller and even slightly faster.
All opcodes are 8 bits, and no operand is greater than 32 bits. Most opcodes take no operands.
The compiler also comes with an expression optimizer that works on the AST given to it by the parser. It can turn simple conjunctions into more performant constructions.
Optimizer Feature Summary
[aab]
=>[ab]
[a]
=>a
[abc]
=>[a-c]
[aa-c]
=>[a-c]
[a-cc-e]
=>[a-e]
a|
=>a?
|a
=>a??
a|a
=>a
a|b
=>[ab]
[a0]|[b1]
=>[a0b1]
a|.
=>.
blueberry|strawberry
=>(?:blue|straw)berry
abi|abe
=>ab(?:i|e)
.*$
=><special opcode>
The optimizer runs an indefinite number of passes until it sees no more changes in the AST.
There are also a few post-compilation optimizations done at the end, such as turning Jumps to Jumps into Jumps to their final calculated destination and resolving relative Jumps into absolute Jumps.
Of note, I've used a bitmap to keep track of which threads occupy which PCs. No method was explicitly described in Russ's articles of how to do this, but I found this to be the simplest way.
Limitations
Due to the size of the
Jump
andSplit
operands (16 bits), the VM is limited to a program size no larger than 32,767 bytes. This should be enough for any sane pattern.Due to the size of the
Rune_Class
operands, no more than 256[abc]
-type class specifiers can be stored in a Regular Expression. Note that both the negated and regularRune_Class
share whichRune_Class_Data
specifiers they refer to, so[a-c]
and[^a-c]
are stored the same but evaluated separately. The optimizer is also able to reduce how many classes are used, to a limited extent, by collapsing fundamentally identical classes that are written differently, i.e.[0-3]
versus[0123]
.For arbitrary reasons, it is not possible to save more than 10 capture groups, including the implicit expression-wide capture group. Thus an expression can only have up to 9 distinct
(groups)
.Testing and Debugging
A complete test suite of 64 test cases is included, covering standard regular expressions, erroneous expressions, invalid expressions, and a few edge cases with the optimizer.
There is also an
ODIN_DEBUG_REGEX
config that enables output toSTDERR
of differences between the unoptimized and optimized AST and each thread's execution in the virtual machine.Final Notes
This is a sizeable package with much going on inside. I've reviewed it myself a few times now, but that doesn't mean I haven't missed something. I intend to let this sit as a draft to gather feedback for a few days. Otherwise, it is feature-complete and fully functional.
If you happen to give the package a go, I'm on the lookout for bugs, particularly misconstructed patterns (so a pattern that shouldn't compile but does, or a pattern that matches in an unexpected way). I'd also like to hear about API ideas. I have a note to consider possibly removing
groups
from theCapture
struct and relying only onpos
instead (or alternatively turning them into multi-pointers to conserve space), but this is more of a matter of taste, to see what the community might like.With the variety of parsers out there, there's bound to be a feature I haven't implemented. If it's simple enough, I can see to it, but for now, this is a rather basic implementation of Regular Expressions with serviceable speed.
I just hope someone doesn't use this package a few years in the future to write an HTML parser...