[doc] Bazel artifacts cause Hugo local preview to hang

[jadephilipoom]

I've got master checked out at c6b2b7. When I run util/build_docs.py, everything is fine and the site builds. But when I run with the --preview option, the site builds and then Hugo just hangs, taking up a large percentage of my CPU and never actually locally serving the pages. I don't get any printout beyond the Built in 2336 ms message that the non-preview invocation ends on.

Interestingly, when I do a fresh clone and check out the same commit, the preview functionality works just fine and serves pages from localhost:1313 as expected. But no attempt to clean the local directory (e.g. rm -rf build, or passing --ignoreCache to Hugo) has worked thus far.

In the process of trying to debug this problem, I also found that if I try to checkout a previous commit and run util/build_docs.py, with or without the --preview option, the documentation completely fails to build with a separate error: Hugo complains it can't find a file that exists on master but not in the previous commit (e.g. the recently added baselisk.sh). That error looks like:

Error: Error building site: process: readAndProcessContent: walk: Readdir: decorate: lstat /home/jadep/opentitan/bazelisk.sh: no such file or directory

And it's not just git-tracked files -- renaming local, untracked files and directories instead of checking out previous commits causes the same error as above. Again, the problem persists even if I completely remove build directory and pass --ignoreCache. This may be a totally separate problem, but it seems to me like both have something to do with Hugo not really properly ignoring non-source files (given that the documentation preview works in a fresh clone).

I'd be grateful if someone recognizes this problem and can suggest a fix -- I'm currently just trying to debug it myself, and mostly created this issue to record the process in case someone else runs into the same problem later.

[jadephilipoom]

Okay, I fixed this locally by removing a bunch of files and directories I found in git status --ignored. I don't know which of these worked -- I should have tested it more incrementally! Here's what I removed (a mix of "alphabetical order" and "definitely seems safe" and "maybe this thing is related"):

.mypy_cache/
__pycache__/
pytest_cache/
bazel-master.out 
bazel-opentitan
bazel-out
bazel-results/
bazel-testlogs
build-bin/
util/doxygen/Doxyfile.bak
build/
hw/build-bin/

Then I ran:

util/build_docs.py --preview --hugo-opts --ignoreCache

I'm especially suspicious of util/doxygen/Doxyfile.bak, maybe that had something to do with it. I'll see if I can get to a broken state again and narrow it down, and will comment here if that works!

[jadephilipoom]

Okay, looks like this is related to bazel. Worryingly, I can get back into a broken state by just running a bazel command, for instance:

./bazelisk.sh build //sw/device/silicon_creator/mask_rom

Afterwards, util/build_docs.py --preview will hang as described, until I remove bazel-opentitan, bazel-bin, and bazel-out (that specific set seems to be both necessary and sufficient). Then, util/build_docs.py --preview will succeed (no need for the --ignoreCache option). In concrete terms, this flow is very reproducible for me:

$ util/build_docs.py --preview
<works, serving at localhost:1313>
$ ./bazelisk.sh build //sw/device/silicon_creator/mask_rom
$ util/build_docs.py --preview
<hangs>
$ rm -rf bazel-opentitan bazel-bin bazel-out
$ util/build_docs.py --preview
<works, serving at localhost:1313>

Curious if others can reproduce this? Maybe we need to tell Hugo to ignore these bazel artifacts -- perhaps it's hanging because it's trying to process them somehow.

I'm reopening the issue, since it looks like this is more than a one-time problem and I want to find a better fix than having to manually clear bazel artifacts every time I preview documentation.

CC @drewmacrae @cfrantz

[jadephilipoom]

More debug log:

The bazel-* directories should already be ignored by this rule here:

opentitan/site/docs/config.toml

Line 49 in 689a36a

"/bazel-.*/.*",

Adding specific bazel-bin, bazel-out, and bazel-opentitan rules doesn't seem to work either.

Preventing bazel from creating symlinks does seem to work. If I mkdir some dummy directories for bazel-{bin,out,opentitan}, then Bazel warns me it can't create symlinks there but Hugo builds and previews just fine. Perhaps ignoreFiles doesn't handle symlinks in a reasonable manner...

[drewmacrae]

There's a new directory .bin that isn't ignored yet afaik

[drewmacrae]

I also experienced this, hugo appears to get flaky after a build.

[jadephilipoom]

There's a new directory .bin that isn't ignored yet afaik

Unfortunately, ignoring it doesn't seem to have an effect. 😞

The ignoreFiles rules do seem to get picked up properly; when I remove them, bazel-bin and friends appear under build/docs, and when I add them back these disappear again. My best guess is that there's somewhere where Hugo is enumerating everything from the root of its content directory, including following the symlinks, and not excluding the stuff from ignoreFiles. Maybe when it decides what changes to watch for?

[jadephilipoom]

For testing I've blocked the bazel-opentitan and bazel-out symlinks, and I'm now just experimenting with bazel-bin. If I rename the symlink, I still observe the hanging behavior. If I link it to a smaller subdirectory, then the hanging behavior stops. If I link it to a very large subdirectory (e.g.<default target of bazel-bin>/third_party) then I get the hanging behavior again, which supports the guess that the hanging is due to Hugo erroneously trying to process things it should be ignoring.

[mundaym]

I suspect it won't help but I've sent a PR to update hugo to the latest version in case it is useful: #12326

[jadephilipoom]

I suspect it won't help but I've sent a PR to update hugo to the latest version in case it is useful: #12326

Thanks, I was wondering about that! But unfortunately your suspicion is correct; it doesn't fix the problem.

[drewmacrae]

Is bazel making any bazel-* directories anywhere but the repository root? I scoped the exclusion relatively narrowly for hugo.

[jadephilipoom]

I'm fairly certain by now that:

• the problem is limited to the bazel-{opentitan,bin,out} symlinks in the root (because if I manually unlink them all then the issue goes away)
• the ignoreFiles rule does in fact ignore these links (because changing that does remove them from the build/docs output directory)

There's something deeper going on with Hugo here that I don't fully understand, but I think it's something along the lines of "a step where Hugo lists all the files from the content root, following symlinks, and doesn't ignore the stuff in ignoreFiles". Current suspicion is that this happens when Hugo scans which files it's watching -- if I symlink bazel-bin to some small subdirectory of bazel-bin, the preview works, but I noticed that the "watching" message includes ~/.cache, the directory that the symlink's target is under, even though bazel-bin is theoretically ignored and doesn't occur in the build output.

For some reason I can't seem to disable watching in order to confirm that suspicion, though, even by adding watch = false to site/docs/config.toml. Hugo still insists on watching for changes, which is a bit confusing to me still.

[drewmacrae]

I think the server configuration used by --preview might now watch by default and that I'm able to override it at the commandline. Can you change util/build_docs.py to run as if

./util/build_docs.py --preview --hugo-opts --watch=false

was used to execute it? That command appears to keep it from getting stuck.

[jadephilipoom]

That works, awesome! Thanks @drewmacrae !

It's kind of nice to have the pages automatically update when previewing, but I think it's less painful to restart the server than it is to remove or unlink build artifacts. (I suppose we could have build_docs.py unlink and then re-link the symlinks, but that seems like an unexpected enough move that it could really trip someone up at some point.) I'll make a PR.

[jadephilipoom]

Minimally reproduced the bug and reported it to Hugo as gohugoio/hugo#9838