Ignored files are watched by hugo server

[jadephilipoom]

Recently, I ran into an issue where hugo server simply hangs after building the website. After some investigation (see lowRISC/opentitan#12322), it looks like the problem is caused by the Hugo server watching for changes in files that should be ignored.

I was able to reproduce a minimal example of this problem with the latest version. Here's the terminal transcript:

Initial setup; I just downloaded the latest binary locally and set up a new site and an empty index.md:

$ mkdir hugo_test && cd hugo_test
$ curl -sL https://github.com/gohugoio/hugo/releases/download/v0.97.3/hugo_extended_0.97.3_Linux-64bit.tar.gz | tar -xzO hugo > bin/hugo
$ alias hugo=~/hugo_test/bin/hugo
$ chmod +x bin/hugo
$ hugo version
hugo v0.97.3-078053a43d746a26aa3d48cf1ec7122ae78a9bb4+extended linux/amd64 BuildDate=2022-04-18T17:22:19Z VendorInfo=gohugoio
$ hugo new site test
Congratulations! Your new Hugo site is created in /home/jadep/hugo_test/test.
<...full output clipped...>
Visit https://gohugo.io/ for quickstart guide and full documentation.
$ ls
bin  test
$ hugo new test/index.md

Then I created a minimal config.toml with the following contents:

$ vim config.toml
$ cat config.toml 
contentDir = "."
ignoreFiles = ["foo"]
watch = true

So far, so good. I could then run hugo server without issues:

$ hugo server --config config.toml
Start building sites … 
hugo v0.97.3-078053a43d746a26aa3d48cf1ec7122ae78a9bb4+extended linux/amd64 BuildDate=2022-04-18T17:22:19Z VendorInfo=gohugoio
WARN 2022/04/28 12:23:27 found no layout file for "HTML" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2022/04/28 12:23:27 found no layout file for "HTML" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2022/04/28 12:23:27 found no layout file for "HTML" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.

                   | EN  
-------------------+-----
  Pages            |  3  
  Paginator pages  |  0  
  Non-page files   |  2  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Built in 149 ms
Watching for changes in /home/jadep/hugo_test/{bin,public,resources,test}
Watching for config changes in /home/jadep/hugo_test/config.toml
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at //localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

However, then I added a symlink to a very large directory (~/.cache). I'd expect that the symlink should be ignored, because foo is in ignoreFiles in config.toml. However, instead, hugo server just hangs after building the website, consuming a substantial portion of CPU the whole time:

$ ln -s ~/.cache foo
$ hugo server --config config.toml
Start building sites … 
hugo v0.97.3-078053a43d746a26aa3d48cf1ec7122ae78a9bb4+extended linux/amd64 BuildDate=2022-04-18T17:22:19Z VendorInfo=gohugoio
WARN 2022/04/28 12:32:14 found no layout file for "HTML" for kind "home": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2022/04/28 12:32:14 found no layout file for "HTML" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
WARN 2022/04/28 12:32:14 found no layout file for "HTML" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.

                   | EN  
-------------------+-----
  Pages            |  3  
  Paginator pages  |  0  
  Non-page files   | 14  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Built in 324 ms
<no further output, cancelled after 5 minutes>

I believe this behavior has to do with the "watch for changes" routine, because if I pass --watch=false, then the behavior goes away. I am confident it's not an issue with the pattern-recognition in ignoreFiles, because if I symlink foo to a small directory then this problem disappears and I can see that foo reappears and disappears from the output as expected when I change ignoreFiles. I also checked if it was a problem with symlink cycles by creating one under foo, but that also doesn't cause the hanging behavior. My best guess is that it has to do with the size of the link target.

But the exact cause of the hanging behavior is slightly orthogonal here; I simply wouldn't expect Hugo to be watching those files, since they're explicitly excluded in config.toml.

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.82.0-9D960784+extended linux/amd64 BuildDate=2021-03-21T17:28:04Z VendorInfo=gohugoio

Does this issue reproduce with the latest release?

Yes, I've replicated it with the latest binary (0.97.3.0)

[bep]

Hugo's file watching is mostly directory based (we also watch some single config files), and that has to be so for it to scale.

So, currently we drop change events for ignored files, but you're right in that we would also drop watching directories that matches the ignore expression(s).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.