Support ES2015 unicode escape sequence additions

[p-bakker]

ES6 added the \u{...} escape sequence format to allow escaping unicode values > FFFF

See https://exploringjs.com/es6/ch_unicode.html for example

[tuchida]

I am implementing this and have noticed a few things.

• In Rhino, Unicode escaped identifier does not throw an error when it is the same as a reserved keyword, but this is a violation of the specification.
  

  rhino/testsrc/org/mozilla/javascript/tests/ParserTest.java

  Lines 1122 to 1126 in c275c35

  	public void testParseUnicodeReservedKeywords1() {
  	AstRoot root = parse("\\u0069\\u0066");
  	AstNode first = ((ExpressionStatement) root.getFirstChild()).getExpression();
  	assertEquals("i\\u0066", first.getString());
  	}

  • Here is the cause.
    

    rhino/src/org/mozilla/javascript/TokenStream.java

    Lines 755 to 761 in c275c35

    	} else if (isKeyword(
    	str,
    	parser.compilerEnv.getLanguageVersion(),
    	parser.inUseStrictDirective())) {
    	// If a string contains unicodes, and converted to a keyword,
    	// we convert the last character back to unicode
    	str = convertLastCharToHex(str);

• In Rhino, an incomplete Unicode escape does not throw an error, but it is a violation of the specification.
  

  rhino/testsrc/tests/ecma/LexicalConventions/7.7.4.js

  Line 172 in c275c35

  new TestCase( SECTION, "\\u", "u", "\u" );

  • Here is the cause.
    

    rhino/src/org/mozilla/javascript/TokenStream.java

    Line 985 in c275c35

    continue strLoop;

I can be modified as per specifications, but it would be breaking changes. It might be better to control it with some flag. These may be different issues.

[p-bakker]

I've implemented this as well as part of trying to implement the 'u'/Unicode option for regexes and ran into the same problems, see
#958 (comment)

Also see that above comment for my thoughts on fixing this: enable the spec compliant behavior in a new ECMASCRIPT language version

[p-bakker]

Also, the commit that implemented that specific behavior for keyword is 323f151

The commit references an issue 490, but I cannot find any reference to that, so no idea why exactly this was implemented this way

[p-bakker]

Think I found what the '490' referred to: issue report against Closure Compiler on the archived Google Code project, see http://blickly.github.io/closure-compiler-issues/#490

[tuchida]

• Support unicode codepoint escape #1383

This has been supported.