Skip to content

An EcmaScript lexical scanner that is as correct as one can be and produces a token stream that any valid EcmaScript parser will unambiguously interpret.

Notifications You must be signed in to change notification settings

mikesamuel/es5-lexer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ES5 Lexer

A lexical scanner for EcmaScript 5 that is as correct as one can be (given the ambiguity between regular expressions and division ops) and produces a token stream that will be treated as equivalent by any valid EcmaScript parser that parses the output.

This is meant to efficiently allow token level transformations of valid EcmaScript 5 programs from untrusted sources that will have same meaning as that assumed by the transformer when parsed by a full EcmaScript interpreter. For this to hold, the output token stream must not contain any ambiguity.

The resulting token stream will not have semicolons inserted into it, and is not guaranteed to specify a valid EcmaScript program.

Usage

This code is divided into several separate JavaScript files which are meant to be combined before use.

Run

$ make

from a UNIX shell to build the combined JS bundle which will be avilable at build/es5_lexer_combined.js.

Once you've got an HTML page that loads your code, you can use

var scanner = es5Lexer.makeScanner(sourceCode);

to create a token stream which is just a function that will return the next token each time it is called or null if none are available.

 var unambiguousTokenStream = es5Lexer.disambiguateTokenStream(
    scanner);

produces a token stream that is unambiguous.

See src/exports.js for the full public API that incude other utility functions for dealing with EcmaScript tokens.

Other usage examples are available in the unit tests under the tests/ directory.

About

An EcmaScript lexical scanner that is as correct as one can be and produces a token stream that any valid EcmaScript parser will unambiguously interpret.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published