Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Vim Script support #1541

Merged
merged 2 commits into from
Aug 23, 2024
Merged

feat: Add Vim Script support #1541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
51940a9
feat: Add Vim Script support
ianlewis Aug 23, 2024
72333dd
chore: remove comment
ianlewis Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Support for [MATLAB](https://www.mathworks.com/products/matlab.html) files
was added.
- Support was added for
[MATLAB](https://www.mathworks.com/products/matlab.html) and
[Vim Script](https://vimdoc.sourceforge.net/htmldoc/usr_41.html).

## [0.9.0] - 2024-08-08

Expand Down
3 changes: 2 additions & 1 deletion SUPPORTED_LANGUAGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Supported Languages

43 languages are currently supported.
44 languages are currently supported.

| File type | Extension | Supported comments |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
Expand Down Expand Up @@ -44,6 +44,7 @@
| TypeScript | `.ts`, `.cts`, `.mts` | `//`, `/* */` |
| Unix Assembly | `.s`, `.ms` | `;`, `/* */` |
| VBA | `.bas`, `.cls`, `.frm`, `.vba` | `'` |
| Vim Script | `.vim`, `.vba`, `.vimrc`, `.vmb` | `"` |
| Visual Basic .NET | `.vb`, `.vbhtml` | `'` |
| XML | `.xml`, `.adml`, `.admx`, `.ant`, `.axaml`, `.axml`, `.builds`, `.ccproj`, `.ccxml`, `.clixml`, `.cproject`, `.cscfg`, `.csdef`, `.csl`, `.csproj`, `.ct`, `.depproj`, `.dita`, `.ditamap`, `.ditaval`, `.dll.config`, `.dotsettings`, `.filters`, `.fsproj`, `.fxml`, `.glade`, `.gml`, `.gmx`, `.grxml`, `.gst`, `.hzp`, `.iml`, `.ivy`, `.jelly`, `.jsproj`, `.kml`, `.launch`, `.mdpolicy`, `.mjml`, `.mm`, `.mod`, `.mojo`, `.mxml`, `.natvis`, `.ncl`, `.ndproj`, `.nproj`, `.nuspec`, `.odd`, `.osm`, `.pkgproj`, `.pluginspec`, `.proj`, `.props`, `.ps1xml`, `.psc1`, `.pt`, `.qhelp`, `.rdf`, `.res`, `.resx`, `.rs`, `.rss`, `.sch`, `.scxml`, `.sfproj`, `.shproj`, `.srdf`, `.storyboard`, `.sublime-snippet`, `.sw`, `.targets`, `.tml`, `.ts`, `.tsx`, `.typ`, `.ui`, `.urdf`, `.ux`, `.vbproj`, `.vcxproj`, `.vsixmanifest`, `.vssettings`, `.vstemplate`, `.vxml`, `.wixproj`, `.workflow`, `.wsdl`, `.wsf`, `.wxi`, `.wxl`, `.wxs`, `.x3d`, `.xacro`, `.xaml`, `.xib`, `.xlf`, `.xliff`, `.xmi`, `.xml.dist`, `.xmp`, `.xproj`, `.xsd`, `.xspec`, `.xul`, `.zcml` | `<!-- --!>` |
| YAML | `.yml`, `.mir`, `.reek`, `.rviz`, `.sublime-syntax`, `.syntax`, `.yaml`, `.yaml-tmlanguage`, `.yaml.sed`, `.yml.mysql` | `#` |
29 changes: 22 additions & 7 deletions internal/scanner/languages.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,26 +54,41 @@ type Config struct {
Strings []StringConfig `yaml:"strings,omitempty"`
}

type escapeFunc func(s *CommentScanner, st *stateString) (bool, error)
type escapeFunc func(s *CommentScanner, st *stateString) ([]rune, error)

var escapeFuncs = map[string]escapeFunc{
BackslashEscape: backslashEscape,
NoEscape: noEscape,
DoubleEscape: doubleEscape,
}

func noEscape(_ *CommentScanner, _ *stateString) (bool, error) {
return false, nil
func noEscape(_ *CommentScanner, _ *stateString) ([]rune, error) {
return nil, nil
}

func backslashEscape(s *CommentScanner, st *stateString) (bool, error) {
return s.peekEqual(append([]rune{'\\'}, s.config.Strings[st.index].End...))
func backslashEscape(s *CommentScanner, st *stateString) ([]rune, error) {
b := append([]rune{'\\'}, s.config.Strings[st.index].End...)
eq, err := s.peekEqual(b)
if err != nil {
return nil, err
}
if eq {
return b, nil
}
return nil, nil
}

func doubleEscape(s *CommentScanner, st *stateString) (bool, error) {
func doubleEscape(s *CommentScanner, st *stateString) ([]rune, error) {
b := append([]rune{}, s.config.Strings[st.index].End...)
b = append(b, s.config.Strings[st.index].End...)
return s.peekEqual(b)
eq, err := s.peekEqual(b)
if err != nil {
return nil, err
}
if eq {
return b, nil
}
return nil, nil
}

// LanguagesConfig is a map of language names to their configuration. Keys are
Expand Down
19 changes: 9 additions & 10 deletions internal/scanner/languages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,16 +446,15 @@ Unix Assembly:
- start: "'"
end: "'"
escape: none
# TODO(#460): Support Vim Script
# Vim Script:
# line_comment_start: ['"']
# strings:
# - start: '"'
# end: '"'
# escape: backslash
# - start: "'"
# end: "'"
# escape: backslash
Vim Script:
line_comment_start: ['"']
strings:
- start: '"'
end: '"'
escape: backslash
- start: "'"
end: "'"
escape: backslash
VBA:
line_comment_start: ["'"]
strings:
Expand Down
125 changes: 109 additions & 16 deletions internal/scanner/scanner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,6 @@
if s.err != nil {
return false
}

switch st := s.state.(type) {
case *stateCode:
s.state, s.err = s.processCode(st)
Expand All @@ -218,6 +217,12 @@
if _, ok := s.state.(*stateLineComment); !ok {
return true
}
case *stateLineCommentOrString:
var hasComment bool
hasComment, s.state, s.err = s.processLineCommentOrString(st)
if hasComment {
return true
}
case *stateMultilineComment:
s.state, s.err = s.processMultilineComment(st)
if _, ok := s.state.(*stateMultilineComment); !ok {
Expand All @@ -243,14 +248,18 @@

if len(m) > 0 || len(mm) > 0 {
if len(m) >= len(mm) {
for i, stringStart := range s.config.Strings {
if string(stringStart.Start) == string(m) {
return &stateLineCommentOrString{
index: i,
}, nil
}
}

return &stateLineComment{}, nil
}

if !s.config.MultilineCommentAtLineStart || s.atLineStart {
// Discard the opening. It will be added by processMultilineComment.
if _, errDiscard := s.reader.Discard(len(s.config.MultilineCommentStart)); errDiscard != nil {
return st, fmt.Errorf("parsing code: %w", errDiscard)
}
return &stateMultilineComment{
line: s.line,
}, nil
Expand All @@ -264,10 +273,6 @@
return st, err
}
if eq {
// Discard the string opening.
if _, err := s.reader.Discard(len(strs.Start)); err != nil {
return st, fmt.Errorf("parsing code: %w", err)
}
return &stateString{
index: i,
}, nil
Expand Down Expand Up @@ -309,15 +314,21 @@

// processString processes strings and returns the next state.
func (s *CommentScanner) processString(st *stateString) (state, error) {
// Discard the string start characters.
if _, err := s.reader.Discard(len(s.config.Strings[st.index].Start)); err != nil {
return st, fmt.Errorf("parsing string: %w", err)

Check warning on line 319 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L319 

Added line #L319 was not covered by tests
}

for {
// Handle escaped characters.
escaped, err := s.config.Strings[st.index].EscapeFunc(s, st)
if err != nil {
// There may still be characters to process so continue if we get EOF.
if err != nil && !errors.Is(err, io.EOF) {
return st, err
}
if escaped {
// Skip the backslash character.
if _, err := s.reader.Discard(1); err != nil {
if len(escaped) > 0 {
// Skip the escaped characters.
if _, err := s.reader.Discard(len(escaped)); err != nil {
return st, fmt.Errorf("parsing string: %w", err)
}
} else {
Expand All @@ -332,10 +343,10 @@
}
return &stateCode{}, nil
}
}

if _, err := s.nextRune(); err != nil {
return st, fmt.Errorf("parsing string: %w", err)
if _, err := s.nextRune(); err != nil {
return st, fmt.Errorf("parsing string: %w", err)

Check warning on line 348 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L348 

Added line #L348 was not covered by tests
}
}
}
}
Expand Down Expand Up @@ -369,9 +380,91 @@
}
}

// processLineCommentOrString processes strings or line comments when they have
// the same start character. e.g. Vim Script.
func (s *CommentScanner) processLineCommentOrString(st *stateLineCommentOrString) (bool, state, error) {
// Discard the string start characters.
if _, err := s.reader.Discard(len(s.config.Strings[st.index].Start)); err != nil {
return false, st, fmt.Errorf("parsing string: %w", err)

Check warning on line 388 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L388 

Added line #L388 was not covered by tests
}

// b is used to build the line comment text.
var b strings.Builder
// Add the opening to the builder since we want it in the output if this is a comment.
b.WriteString(string(s.config.Strings[st.index].Start))
for {
lineEnd, err := s.isLineEnd()
if err != nil {
return false, st, err

Check warning on line 398 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L398 

Added line #L398 was not covered by tests
}

// If we get to the end of the line without the string being closed,
// then it must be a comment. Languages where line comments and strings
// share the same character cannot implement multi-line strings.
if lineEnd {
s.next = &Comment{
Text: b.String(),
Line: s.line,
Multiline: false,
}
return true, &stateCode{}, nil
}

// Handle escaped characters.
escaped, err := s.config.Strings[st.index].EscapeFunc(s, &stateString{
index: st.index,
})
// There may still be characters to process so continue.
if err != nil && !errors.Is(err, io.EOF) {
return false, st, err

Check warning on line 419 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L419 

Added line #L419 was not covered by tests
}
if len(escaped) > 0 {
// Skip the escaped characters.
if _, discardErr := s.reader.Discard(len(escaped)); discardErr != nil {
return false, st, fmt.Errorf("parsing string: %w", discardErr)

Check warning on line 424 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L424 

Added line #L424 was not covered by tests
}

// Write the escaped characters in case this is a comment.
_, err = b.WriteString(string(escaped))
if err != nil {
return false, st, fmt.Errorf("writing runes %q: %w", escaped, err)

Check warning on line 430 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L430 

Added line #L430 was not covered by tests
}
continue
}

// Look for the end of the string.
stringEnd, err := s.peekEqual(s.config.Strings[st.index].End)
if err != nil {
return false, st, fmt.Errorf("parsing string: %w", err)

Check warning on line 438 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L438 

Added line #L438 was not covered by tests
}
if stringEnd {
if _, discardErr := s.reader.Discard(len(s.config.Strings[st.index].End)); discardErr != nil {
return false, st, fmt.Errorf("parsing string: %w", discardErr)

Check warning on line 442 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L442 

Added line #L442 was not covered by tests
}
return false, &stateCode{}, nil
}

rn, err := s.nextRune()
if err != nil {
return false, st, fmt.Errorf("parsing string: %w", err)

Check warning on line 449 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L449 

Added line #L449 was not covered by tests
}

_, err = b.WriteRune(rn)
if err != nil {
return false, st, fmt.Errorf("writing rune %q: %w", rn, err)

Check warning on line 454 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L454 

Added line #L454 was not covered by tests
}
}
}

// processMultilineComment processes multi-line comments and returns the next state.
func (s *CommentScanner) processMultilineComment(st *stateMultilineComment) (state, error) {
// Discard the opening since we don't want to parse it. It could be the same as the closing.
if _, errDiscard := s.reader.Discard(len(s.config.MultilineCommentStart)); errDiscard != nil {
return st, fmt.Errorf("parsing code: %w", errDiscard)

Check warning on line 463 in internal/scanner/scanner.go

View check run for this annotation

Codecov / codecov/patch

internal/scanner/scanner.go#L463 

Added line #L463 was not covered by tests
}

var b strings.Builder
// Add the opening to the builder since we want it in the output.
b.WriteString(string(s.config.MultilineCommentStart))
for {
// Look for the end of the comment.
Expand Down
Loading
Loading