Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scopes tokenizer: new implementation #5628

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

Scopes tokenizer: new implementation #5628

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0ff5751
scopes tokenizer: new implementation
mkslanc Aug 20, 2024
b1ab920
changes to make snippets tokenizer work
mkslanc Sep 2, 2024
1b7d9c3
refactor due to code review
mkslanc Sep 2, 2024
b52aea3
fix: strict equal checks/tests
mkslanc Sep 7, 2024
92db310
make it work with scopes
mkslanc Sep 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions ace-internal.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1512,3 +1512,25 @@ declare module "./src/mouse/default_gutter_handler" {
export interface GutterHandler {
}
}

declare module "./src/scope" {

export interface Scope extends String {
name: string;
children: { [name: string]: Scope };
parent?: Scope;
data: any;

get(name: any, extraId: string): Scope;

find(states): Scope | undefined;

hasParent(states): boolean;

count(): number;

getAllScopeNames(): string[];

toStack(): any[];
}
}
32 changes: 25 additions & 7 deletions demo/kitchen-sink/token_tooltip.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,25 @@ class TokenTooltip extends Tooltip {
return;
}

var tokenText = token.type;
if (token.state)
tokenText += "|" + token.state;
if (token.merge)
tokenText += "\n merge";
if (token.stateTransitions)
tokenText += "\n " + token.stateTransitions.join("\n ");
var tokenText = "";
var scope = token.type;
if (scope.name !== undefined) {
let firstScope = true;
do {
tokenText += firstScope ? scope.name + "\n" :" " + scope.name + "\n";
firstScope = false;
if (!scope.parent)
tokenText += "\ntoken count:" + count(scope);
} while(scope = scope.parent)
} else {
tokenText += token.type;
if (token.state)
tokenText += "|" + token.state;
if (token.merge)
tokenText += "\n merge";
if (token.stateTransitions)
tokenText += "\n " + token.stateTransitions.join("\n ");
}

if (this.tokenText != tokenText) {
this.setText(tokenText);
Expand Down Expand Up @@ -121,4 +133,10 @@ class TokenTooltip extends Tooltip {

}

function count(root) {
return Object.keys(root.children).reduce(function (n, key) {
return n + count(root.children[key]);
}, 1);
}

exports.TokenTooltip = TokenTooltip;
4 changes: 2 additions & 2 deletions src/background_tokenizer.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,8 @@ class BackgroundTokenizer {
var state = this.states[row - 1];
// @ts-expect-error TODO: potential wrong argument
var data = this.tokenizer.getLineTokens(line, state, row);

if (this.states[row] + "" !== data.state + "") {
if (this.states[row] !== data.state) {
this.states[row] = data.state;
this.lines[row + 1] = null;
if (this.currentLine > row + 1)
Expand Down
2 changes: 1 addition & 1 deletion src/edit_session/folding.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,7 +710,7 @@ function Folding() {
if (token && /^comment|string/.test(type)) {
type = type.match(/comment|string/)[0];
if (type == "comment")
type += "|doc-start|\\.doc";
type += "|doc-start|\\.doc|empty";
var re = new RegExp(type);
var range = new Range();
if (dir != 1) {
Expand Down
2 changes: 1 addition & 1 deletion src/layer/text_util.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
const textTokens = new Set(["text", "rparen", "lparen"]);

exports.isTextToken = function(tokenType) {
return textTokens.has(tokenType);
return textTokens.has(tokenType.toString());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without it simple_tokenizer is not working as expected

};
2 changes: 1 addition & 1 deletion src/mode/folding/mixed.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ oop.inherits(FoldMode, BaseFoldMode);


this.$getMode = function(state) {
if (typeof state != "string")
if (Array.isArray(state))
state = state[0];
for (var key in this.subModes) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add .syntax property to Scope to make this simpler?

if (state.indexOf(key) === 0)
Expand Down
144 changes: 69 additions & 75 deletions src/mode/lua_highlight_rules.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ var LuaHighlightRules = function() {

var keywords = (
"break|do|else|elseif|end|for|function|if|in|local|repeat|"+
"return|then|until|while|or|and|not"
"return|then|until|while|or|and|not"
);

var builtinConstants = ("true|false|nil|_G|_VERSION");

var functions = (
// builtinFunctions
// builtinFunctions
"string|xpcall|package|tostring|print|os|unpack|require|"+
"getfenv|setmetatable|next|assert|tonumber|io|rawequal|"+
"collectgarbage|getmetatable|module|rawset|math|debug|"+
Expand All @@ -34,9 +34,9 @@ var LuaHighlightRules = function() {
"setupvalue|getlocal|getregistry|getfenv|setn|insert|getn|"+
"foreachi|maxn|foreach|concat|sort|remove|resume|yield|"+
"status|wrap|create|running|"+
// metatableMethods
// metatableMethods
"__add|__sub|__mod|__unm|__concat|__lt|__index|__call|__gc|__metatable|"+
"__mul|__div|__pow|__len|__eq|__le|__newindex|__tostring|__mode|__tonumber"
"__mul|__div|__pow|__len|__eq|__le|__newindex|__tostring|__mode|__tonumber"
);

var stdLibaries = ("string|package|os|io|math|debug|table|coroutine");
Expand Down Expand Up @@ -64,92 +64,86 @@ var LuaHighlightRules = function() {
this.$rules = {
"start" : [{
stateName: "bracketedComment",
onMatch : function(value, currentState, stack){
stack.unshift(this.next, value.length - 2, currentState);
return "comment";
onMatch2 : function(value, scope){
var parent = scope.get("bracketedComment" + (value.length - 2))
parent.meta = (value.length - 2);
return parent.get(this.next).get("comment");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use scope.get(this.next, value.length - 2).get("comment");

},
regex : /\-\-\[=*\[/,
next : [
{
onMatch : function(value, currentState, stack) {
if (value.length == stack[1]) {
stack.shift();
stack.shift();
this.next = stack.shift();
onMatch2 : function(value, scope) {
if (scope.parent && value.length == scope.parent.meta) {
return scope.parent.parent.get("comment");
} else {
this.next = "";
return scope.get("comment");
}
return "comment";
},
regex : /\]=*\]/,
next : "start"
}, {
defaultToken: "comment.body"
mkslanc marked this conversation as resolved.
Show resolved Hide resolved
defaultToken : "comment"
}
]
},

{
token : "comment",
regex : "\\-\\-.*$"
},
{
stateName: "bracketedString",
onMatch : function(value, currentState, stack){
stack.unshift(this.next, value.length, currentState);
return "string.start";
{
token : "comment",
regex : "\\-\\-.*$"
},
regex : /\[=*\[/,
next : [
{
onMatch : function(value, currentState, stack) {
if (value.length == stack[1]) {
stack.shift();
stack.shift();
this.next = stack.shift();
} else {
this.next = "";
}
return "string.end";
},

regex : /\]=*\]/,
next : "start"
}, {
defaultToken : "string"
}
]
},
{
token : "string", // " string
regex : '"(?:[^\\\\]|\\\\.)*?"'
}, {
token : "string", // ' string
regex : "'(?:[^\\\\]|\\\\.)*?'"
}, {
token : "constant.numeric", // float
regex : floatNumber
}, {
token : "constant.numeric", // integer
regex : integer + "\\b"
}, {
token : keywordMapper,
regex : "[a-zA-Z_$][a-zA-Z0-9_$]*\\b"
}, {
token : "keyword.operator",
regex : "\\+|\\-|\\*|\\/|%|\\#|\\^|~|<|>|<=|=>|==|~=|=|\\:|\\.\\.\\.|\\.\\."
}, {
token : "paren.lparen",
regex : "[\\[\\(\\{]"
}, {
token : "paren.rparen",
regex : "[\\]\\)\\}]"
}, {
token : "text",
regex : "\\s+|\\w+"
} ]
{
stateName: "bracketedString",
onMatch2 : function(value, scope){
var parent = scope.get("bracketedString" + value.length);
parent.meta = value.length;
return parent.get(this.next).get("string.start");
},
regex : /\[=*\[/,
next : [
{
onMatch2 : function(value, scope) {
if (scope.parent && value.length == scope.parent.meta) {
return scope.parent.parent.get("string.end");
} else {
return scope.get("string.end");
}
},

regex : /\]=*\]/,
}, {
defaultToken : "string"
}
]
},
{
token : "string", // " string
regex : '"(?:[^\\\\]|\\\\.)*?"'
}, {
token : "string", // ' string
regex : "'(?:[^\\\\]|\\\\.)*?'"
}, {
token : "constant.numeric", // float
regex : floatNumber
}, {
token : "constant.numeric", // integer
regex : integer + "\\b"
}, {
token : keywordMapper,
regex : "[a-zA-Z_$][a-zA-Z0-9_$]*\\b"
}, {
token : "keyword.operator",
regex : "\\+|\\-|\\*|\\/|%|\\#|\\^|~|<|>|<=|=>|==|~=|=|\\:|\\.\\.\\.|\\.\\."
}, {
token : "paren.lparen",
regex : "[\\[\\(\\{]"
}, {
token : "paren.rparen",
regex : "[\\]\\)\\}]"
}, {
token : "text",
regex : "\\s+|\\w+"
} ]
};

this.normalizeRules();
};

Expand Down
15 changes: 14 additions & 1 deletion src/mode/text.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,20 @@ Mode = function() {
this.getTokenizer = function() {
if (!this.$tokenizer) {
this.$highlightRules = this.$highlightRules || new this.HighlightRules(this.$highlightRuleConfig);
this.$tokenizer = new Tokenizer(this.$highlightRules.getRules());
var modeName;
if (this.$id) {
var chunks = this.$id.split('/');
mkslanc marked this conversation as resolved.
Show resolved Hide resolved
if (chunks.length > 1) {
modeName = chunks[chunks.length - 1];
}
else {
modeName = chunks;
}
}
else {
modeName = "root";
}
this.$tokenizer = new Tokenizer(this.$highlightRules.getRules(), modeName);
}
return this.$tokenizer;
};
Expand Down
10 changes: 7 additions & 3 deletions src/mode/text_highlight_rules.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,16 @@ TextHighlightRules = function() {
};

var pushState = function(currentState, stack) {
if (currentState != "start" || stack.length)
stack.unshift(this.nextState, currentState);
if (typeof currentState.get == "function") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep only the scope variant here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would need to make significant changes in normalizeRules logic in that case

return currentState.get(this.nextState);
}
if (currentState !== "start" || stack.length) stack.unshift(this.nextState, currentState);
return this.nextState;
};
var popState = function(currentState, stack) {
// if (stack[0] === currentState)
if (typeof currentState.get == "function") {
return (currentState.parent.parent) ? currentState.parent : currentState.get("start");
}
stack.shift();
return stack.shift() || "start";
};
Expand Down
Loading
Loading