Add straight-watcher.el

[progfolio]

Reimplementation of the watcher script in elisp.

See: #692

Not quite complete, but I figured I'd share it to show the general direction so far.

[progfolio]

In consideration of #390, #447, #488, #508, #519, and #712:

I think it would be best to move away from using file mtimes to determine modifications.
The mtime approach works well in that find is fast, but it has no way of telling us whether or not the current
content of the package warrants a rebuild. What I'm thinking is that we could store a hash and compare against that.

The question is, what do we hash? We could have the watcher process expand the package's :files, append those files in a temp buffer and hash the contents.
The watcher can store these hashes in a hash table on disk. We can then compare the package's last built hash (constructed as above and stored in straight--build-cache instead of the mtime during the build process) with the latest value the watcher reported.
We only rebuild if the hashes do not match.

Pros:

• replaces find dependency with elisp
• Should have similar performance to the find method on start-up. Of course I'd have to measure this, but we'd only be paying the one-time cost of reading the watcher's data in from disk and one hash lookup + string comparison per package.
• Simplifies the codebase. We would have less points of failure if/when packages are not being built at the appropriate time.

Cons:

• Memory/CPU usage of having the watcher process run. I haven't done any formal measurements, but the cost seems negligible IMO. Especially if we were to measure it against the cost of spurious rebuilds.
  In any case I'd aim to keep that cost minimal.
• Less redundancy to detect changes. I'm not sure how much this will matter in practice, though. I personally only use the watcher now and spare Filesystem watcher fails silently on upgrading system Python #447 (which would be a non-issue with an elisp implementation) have rarely had any problems.

Thoughts?

[raxod502]

Yeah! I think that makes a lot of sense. In fact, if we implement this, we might be able to get rid of the find approach entirely, and force everyone to use the filesystem watcher, because it wouldn't have problematic system dependencies. That would certainly be an improvement in maintainability.

I think hashing the :files recipe makes a lot of sense, and it would have no problem with performance as long as it's done in the watcher subprocess which won't block init. I suppose there could be a race condition where you make a modification and then restart Emacs before the watcher has the chance to do its analysis, but that's no worse than what we have with the current filesystem watcher, and I've never actually run into that case in my few years of usage.

Here's a fun trick I've used in the past to have an external process write hashes like this in a format that's super fast to read: when package xyz gets processed by the watcher process and it computes hash abcdef, write a file xyz-abcdef into the results directory. That way, you can get a listing of all the detected modifications as well as their hashes, from a single directory-files call, and you can copy the relevant data into the build cache before clearing out the directory.

Thanks for taking the lead on this improvement!

[progfolio]

This functionality (and arguably many of the other components of straight.el) would be easier to reason about if it were in its own file/sub-feature. However, loading that file is complicated by the bootstrap/installation process. Haven't found a clean way to do this yet.

[raxod502]

I'll certainly agree with you that straight.el would be more maintainable if it were split into multiple files. In my past work I had an alarming tendency to never split code into multiple files, which I've since learned from.

I think I see what you mean about the bootstrap process. Here's one useful note, though: anything that isn't actually needed during initial bootstrap can be in a separate file no problem, since the single-file download used during bootstrap is only used to install straight.el itself, and then is discarded.

You just need to make sure that any of these separate files used after init/bootstrap are loaded only when needed, and not eagerly. (Which is good for performance anyway.)

Since a filesystem watcher doesn't need to be launched during initial installation, it's fine to put this code in a separate file, as long as we require it only when needed.

[progfolio]

I won't be working on this feature any time soon, but anyone who is interested is welcome to any of the code in this patch.

One potential problem with this approach is the large number of open file descriptors when simulating recursive directory watching. There was a proposed patch to Emacs's library to allow recursive directory watching, but it stalled.