Config issues with custom Sandbox

[alecgunny]

Full reproducing code can be found here, this issue is concerned with issue 2 on that repo's README. In particular, trying to implement a custom Sandbox subclass causes issues at runtime with the Config looks for the corresponding section and finds it doesn't exist. Using the example above, if the sandbox_type is called "singularity_dev", you might see something like:

DEBUG: Checking if DevGreet(name=Thom, dev=False) is complete
INFO: Informed scheduler that task   DevGreet_False_Thom_80b4044546   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 2788817] Worker Worker(salt=3304246859, workers=1, host=dgx1, username=alec.gunny, pid=2788817) running   DevGreet(name=Thom, dev=False)
ERROR: [pid 2788817] Worker Worker(salt=3304246859, workers=1, host=dgx1, username=alec.gunny, pid=2788817) failed    DevGreet(name=Thom, dev=False)
Traceback (most recent call last):
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 344, in run
    cmd = self.sandbox_inst.cmd(self.create_proxy_cmd())
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/contrib/singularity/sandbox.py", line 173, in cmd
    env = self._get_env()
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/sandbox/base.py", line 241, in _get_env
    for name, value in cfg.items(section):
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 477, in items
    options = self.options(section, prefix=prefix, expand_vars=expand_vars,
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 452, in options
    for option in ConfigParser.options(self, section):
  File "/home/alec.gunny/miniconda3/lib/python3.9/configparser.py", line 675, in options
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'singularity_dev_sandbox_env'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DevGreet_False_Thom_80b4044546   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=3304246859, workers=1, host=dgx1, username=alec.gunny, pid=2788817) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 failed:
    - 1 DevGreet(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Haven't looked too much into how arguments get registered with the Config, but could this potentially be handled by a metaclass for Sandbox that automatically registers its config_defaults with the global Config, maybe via a staticmethod on the class being registered?

[riga]

With 52fe3f9, config sections of sandboxes are not constructed dynamically any longer via <sandbox_type>_sandbox (in your case to singularity_dev_sandbox(|_env|_volumes)).

The leading sandbox_type is now replaced with config_section_prefix which has to be defined per Sandbox class, but can be inherited. The base sandboxes coming with law (Bash, Venv, Docker, Singularity, CMSSW) define this prefix, so that your SingularityDev sandbox will now check the singularity_sandbox(|_env|_volumes) sections which should always exist.

Alternatively you can set the class attribute config_section_prefix to "singularity_dev" and put

class SingularityDevSandbox(law.singularity.SingularitySandbox):
    config_section_prefix = "singularity_dev"
    ...


law.config.update({
    "singularity_dev": {
        "stagein_dir_name": "stagein",
        "stageout_dir_name": "stageout",
        "law_executable": "law",
    },
    "singularity_dev_env": {},
    "singularity_dev_volumes": {},
})

under your class definition to register the minimal set of required options.

[alecgunny]

This is great, thanks @riga !

[alecgunny]

@riga tried running with the latest commit, and regardless of whether I use my custom sandbox or not I'm getting the following error:

Traceback (most recent call last):
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/bin/law", line 9, in <module>
    from law.cli import run
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/__init__.py", line 46, in <module>
    law.logger.setup_logging()
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/logger.py", line 43, in setup_logging
    logger = setup_logger(logger, add_console_handler=False)
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/logger.py", line 107, in setup_logger
    from law.config import Config
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 626, in <module>
    locals()[name] = closure(name)
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 612, in closure
    config = Config.instance()
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 167, in instance
    cls._instance = cls(*args, **kwargs)
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 241, in __init__
    include_configs(self.get_expanded("core", opt))
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 206, in include_configs
    self.include(filename)
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 353, in include
    p = self.__class__(filename, skip_defaults=True, skip_fallbacks=True, skip_env_sync=True,
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 241, in __init__
    include_configs(self.get_expanded("core", opt))
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 444, in get_expanded
    return self.get_default(*args, **kwargs)
  File "/home/alec.gunny/miniconda3/envs/law-repro-DBCx7gBt-py3.9/lib/python3.9/site-packages/law/config.py", line 394, in get_default
    value = self.get(section, option)
  File "/home/alec.gunny/miniconda3/lib/python3.9/configparser.py", line 781, in get
    d = self._unify_values(section, vars)
  File "/home/alec.gunny/miniconda3/lib/python3.9/configparser.py", line 1149, in _unify_values
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'core'

Should I file a new issue for this?

[riga]

@alecgunny Sorry for that, this was a testing issue on my end. This should be fixed now with 500c1a8.

[alecgunny]

Confirmed this is fully working now!