Proposed improvements to the API

[valentyn1boreiko]

I propose these improvements:

• use tqdm in shifthappens/models/base.py by introducing verbose attribute to the class Model and using it respectively in shifthappens/models/base.py
• adjust for an additional files, such as .zip archives in file shifthappens/data/imagenet.py by changing to len(os.listdir(ImageNetValidationData)) >= 1000
• add in requirements.txt the used tqdm package

[zimmerrol]

Mostly looking good, though I'm not completely sure yet about one of the changes.

[zimmerrol]

Instead of storing this as a class variable, we could also create a global flag that controls this behavior for all models at the same time. I'm not sure yet which option I prefer.

[valentyn1boreiko]

Yes, I see your point - I think a global flag would be even better. So far, it seems that we don't have use-cases, when several models are evaluated, but if we will at some point, it would be a bit more clean to do it like you say.

[stes]

Hi both, I think ability to configure these options will also come handy for other tasks. For example, if this task here should be verbosely evaluated, probably all other tasks in the same run should be too.

I made a proposal for adding global config options to the package. In case we go ahead and merge this, @valentyn1boreiko you simply use the new shifthappens.config.verbose option instead of defining this option here.

Let me know what you think (ideally directly on the referenced PR).

[stes]

Hi @valentyn1boreiko, #75 was merged now.

What do you think about the proposed solution, do you think you could leverage it to adapt this PR?

In case you dont find time, @kotekjedi might probably be able to help with this!

[valentyn1boreiko]

Hi, @stes, I added two lines
import shifthappens.config
and
if shifthappens.config.verbose:
in models/base.py

This should be enough, right?

[valentyn1boreiko]

Hi @stes,
I added

available_models_dict = {'resnet18': ResNet18,
'resnet50': ResNet50,
'vgg16': VGG16}

in worst_case.py

I think it would make sense to have a model card somewhere in the global config, such that one could easy choose a model with a parameter to a script, right?

Should I open a separate PR request for this, or could you add it?

[stes]

Hi all ( @valentyn1boreiko @zimmerrol @kotekjedi ) what is the status here? Are the open issue we should discuss before moving forward? I also see the potential of combining this with the changed proposed in #75 .

[kotekjedi]

I am still looking at 75#. I think the issue with global variables is addressed in the proposed config.py, and we only need to add a tqdm wrapper.

[github-actions]

Docstring Coverage Report

Name	Total	Miss	Cover	Cover%
init.py	1	0	1	100%
benchmark.py	6	0	6	100%
config.py	4	0	4	100%
model_comparison.py	5	0	5	100%
utils.py	4	0	4	100%
data/init.py	1	0	1	100%
data/base.py	7	0	7	100%
data/imagenet.py	6	0	6	100%
data/torch.py	4	0	4	100%
models/init.py	1	0	1	100%
models/base.py	7	0	7	100%
models/mixins.py	6	0	6	100%
models/torchvision.py	4	0	4	100%
task_data/init.py	1	0	1	100%
task_data/task_metadata.py	2	0	2	100%
task_data/task_registration.py	2	0	2	100%
tasks/init.py	1	0	1	100%
tasks/base.py	8	0	8	100%
tasks/metrics.py	2	0	2	100%
tasks/mixins.py	6	0	6	100%
tasks/task_result.py	2	0	2	100%
tasks/utils.py	3	0	3	100%
tasks/imagenet_c/init.py	1	1	0	0%
tasks/imagenet_c/imagenet_c.py	20	0	20	100%
tasks/imagenet_r/init.py	1	1	0	0%
tasks/imagenet_r/imagenet_r.py	3	0	3	100%
tasks/raccoons_ood/init.py	1	1	0	0%
tasks/raccoons_ood/raccoons_ood.py	3	0	3	100%
TOTAL	112	3	109	97.3%

RESULT: PASSED (minimum: 0.0%, actual: 97.3%)

[zimmerrol]

@valentyn1boreiko What is the status of this PR? It looks like you (accidentially?) added files that belong to same task to this PR. Do you mind cleaning the PR up such that we can soon merge it?