Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'ratchet' mode is very slow (plugin-maven) #1025

Closed
6 tasks
rjmunro opened this issue Dec 8, 2021 · 6 comments
Closed
6 tasks

'ratchet' mode is very slow (plugin-maven) #1025

rjmunro opened this issue Dec 8, 2021 · 6 comments

Comments

@rjmunro
Copy link

rjmunro commented Dec 8, 2021

I have 6830 java files in my maven project (found with git ls-files "*.java" | wc -l) spread across 44 modules. When I have no changes (git diff --name-only origin/develop returns nothing) and I scan in ratchet mode, it takes nearly 10 minutes to do a no-op. When I removed ratchet mode because I wanted to see how many files were still misformatted, it took less than 20 seconds, even though it had to process every java file.

I was expecting it to be a lot slower because it had to actually read and format 6830 java files, rather than just look at git and see nothing has changed.

If you are submitting a bug, please include the following:

  • summary of problem
  • gradle or maven version
  • spotless version
  • operating system and version
  • copy-paste your full Spotless configuration block(s), and a link to a public git repo that reproduces the problem if possible
  • copy-paste the full content of any console errors emitted by gradlew spotless[Apply/Check] --stacktrace

If you're just submitting a feature request or question, no need for the above.

@nedtwigg nedtwigg changed the title 'ratchet' mode is very slow 'ratchet' mode is very slow (plugin-maven) Dec 8, 2021
@nedtwigg
Copy link
Member

nedtwigg commented Dec 8, 2021

That is surprising! Here's the codepath as it exists today, but #935 is gonna change things a bit.

final List<File> files;
if (ratchetFrom.isPresent()) {
files = collectFilesFromGit(formatterFactory, ratchetFrom.get());
} else {
files = collectFilesFromFormatterFactory(formatterFactory);
}
if (filePatterns == null || filePatterns.isEmpty()) {
return files;
}
final String[] includePatterns = this.filePatterns.split(",");
final List<Pattern> compiledIncludePatterns = Arrays.stream(includePatterns)
.map(Pattern::compile)
.collect(Collectors.toList());
final Predicate<File> shouldInclude = file -> compiledIncludePatterns
.stream()
.anyMatch(filePattern -> filePattern.matcher(file.getAbsolutePath())
.matches());
return files
.stream()
.filter(shouldInclude)
.collect(toList());

private List<File> collectFilesFromGit(FormatterFactory formatterFactory, String ratchetFrom) throws MojoExecutionException {
MatchPatterns includePatterns = MatchPatterns.from(
withNormalizedFileSeparators(getIncludes(formatterFactory)));
MatchPatterns excludePatterns = MatchPatterns.from(
withNormalizedFileSeparators(getExcludes(formatterFactory)));
Iterable<String> dirtyFiles;
try {
dirtyFiles = GitRatchetMaven
.instance().getDirtyFiles(baseDir, ratchetFrom);
} catch (IOException e) {
throw new MojoExecutionException("Unable to scan file tree rooted at " + baseDir, e);
}
List<File> result = new ArrayList<>();
for (String file : withNormalizedFileSeparators(dirtyFiles)) {
if (includePatterns.matches(file, true)) {
if (!excludePatterns.matches(file, true)) {
result.add(new File(baseDir.getPath(), file));
}
}
}
return result;
}

Iterable<String> getDirtyFiles(File baseDir, String ratchetFrom) throws IOException {
Repository repository = repositoryFor(baseDir);
ObjectId sha = rootTreeShaOf(baseDir, ratchetFrom);
IndexDiff indexDiff = new IndexDiff(repository, sha, new FileTreeIterator(repository));
indexDiff.diff();
String workTreePath = repository.getWorkTree().getPath();
Path baseDirPath = Paths.get(baseDir.getPath());
Set<String> dirtyPaths = new HashSet<>(indexDiff.getChanged());
dirtyPaths.addAll(indexDiff.getAdded());
dirtyPaths.addAll(indexDiff.getConflicting());
dirtyPaths.addAll(indexDiff.getUntracked());

@nedtwigg
Copy link
Member

I believe this should be improved in plugin-maven 2.17.7. Can you report back @rjmunro?

@rjmunro
Copy link
Author

rjmunro commented Dec 24, 2021

On my project on my Intel Macbook Pro, using version 2.17.7 it still takes about 9 minutes, similar to version 2.17.6. With ratchet turned off it takes 20 seconds with either version, so it's not made a noticeable difference.

@nedtwigg
Copy link
Member

Bummer :(. I'm surprised by this, but the good news is that 2.18.0 adds an up-to-date cache which should help a lot whether ratchet is on or off. (#935)

@rjmunro
Copy link
Author

rjmunro commented Feb 9, 2022

Can this be reopened?

@nedtwigg nedtwigg reopened this Feb 9, 2022
@nedtwigg
Copy link
Member

This should be much improved in our latest plugin-gradle 6.13.0 and plugin-maven 2.30.0 thanks to

I'm going to close this again, but please reopen if it is still issue in the latest releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants