Error on finder thread hangs

[donkirkby]

We had an error where the data folder was inaccessible:

2019-01-29 04:19:08[ERROR]micall.monitor.kive_watcher.find_samples(): Failed while finding samples.
Traceback (most recent call last):
  File "/src/MiCall/micall/monitor/kive_watcher.py", line 79, in find_samples
    wait)
  File "/src/MiCall/micall/monitor/kive_watcher.py", line 91, in scan_samples
    reverse=True)
  File "/usr/lib64/python3.6/pathlib.py", line 1078, in glob
    for p in selector.select_from(self):
  File "/usr/lib64/python3.6/pathlib.py", line 487, in select_from
    if not is_dir(parent_path):
  File "/usr/lib64/python3.6/pathlib.py", line 1326, in is_dir
    return S_ISDIR(self.stat().st_mode)
  File "/usr/lib64/python3.6/pathlib.py", line 1136, in stat
    return self._accessor.stat(self)
  File "/usr/lib64/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
BlockingIOError: [Errno 11] Resource temporarily unavailable: '/data'

However, it was very confusing, because the micall_watcher service was still running and its status looked fine. The only sign that something was wrong was that the "No folders need processing" messages weren't written into the log every hour. That and the fact that the next week's run didn't start processing.
Either the scanner should keep running after an error, with some kind of sleeping strategy, or it should shut down the service and send a more explicit e-mail that it has shut down.

[donkirkby]

The same error occurred last night at 10:42, three hours after it finished processing this week's run.