Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposed improvements to the API #64

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

Proposed improvements to the API #64

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ec36db1
Proposed task in worst_case.py
valentyn1boreiko Jun 27, 2022
d2a0ea6
Proposed task in worst_case.py
valentyn1boreiko Jun 27, 2022
a5547e9
Proposed task in worst_case.py
valentyn1boreiko Jun 27, 2022
7be8daf
Proposed task in worst_case.py
valentyn1boreiko Jun 27, 2022
b6babbd
Proposed task in worst_case.py
valentyn1boreiko Jun 27, 2022
5ecc718
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
3f0ee0c
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
2e2da01
Merge pull request #1 from shift-happens-benchmark/main
valentyn1boreiko Jul 19, 2022
29ed2ae
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
3c8fcbe
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
a7a9673
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
9255b75
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
2167637
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
19217ea
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
8d46e34
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
0b856c1
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
5c32a7a
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
673fec5
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
b8659be
Proposed task in worst_case.py
valentyn1boreiko Jul 19, 2022
ea4678a
Merge branch 'main' into adjustment_to_api
valentyn1boreiko Jul 19, 2022
2efdde8
Latest.
valentyn1boreiko Jul 19, 2022
ed1ec54
Added verbose.
valentyn1boreiko Jul 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ install:

build:
python3 -m pip install --upgrade build
python3 -m build
python3 -m build

upload_test:
python3 -m twine upload --repository testpypi dist/*
Expand Down
1 change: 1 addition & 0 deletions examples/raccoons_OOD.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
The original dataset was collected by Dat Tran for the object detection task
and can be found at https://github.com/datitran/raccoon_dataset.
"""

import dataclasses
import os

Expand Down
3 changes: 2 additions & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
numpy
torch
torchvision
surgeon_pytorch
surgeon_pytorch
tqdm
4 changes: 1 addition & 3 deletions setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,7 @@ license_file = LICENSE
license = Apache 2.0

[options]
package_dir =
= .
py_modules = shifthappens
packages = find:
python_requires = >=3.8

[options.extras_require]
Expand Down
2 changes: 1 addition & 1 deletion shifthappens/data/imagenet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def _check_imagenet_folder():
)

assert (
len(os.listdir(ImageNetValidationData)) == 1000
len(os.listdir(ImageNetValidationData)) >= 1000
), "ImageNetValidationData folder contains less or more folders than ImageNet classes."


Expand Down
10 changes: 9 additions & 1 deletion shifthappens/models/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from typing import Iterator

import numpy as np
from tqdm import tqdm

from shifthappens.data import imagenet as sh_imagenet
from shifthappens.data.base import DataLoader
Expand Down Expand Up @@ -109,6 +110,7 @@ class Model(abc.ABC):

def __init__(self):
self._imagenet_validation_result = None
self.verbose = False
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of storing this as a class variable, we could also create a global flag that controls this behavior for all models at the same time. I'm not sure yet which option I prefer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I see your point - I think a global flag would be even better. So far, it seems that we don't have use-cases, when several models are evaluated, but if we will at some point, it would be a bit more clean to do it like you say.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi both, I think ability to configure these options will also come handy for other tasks. For example, if this task here should be verbosely evaluated, probably all other tasks in the same run should be too.

I made a proposal for adding global config options to the package. In case we go ahead and merge this, @valentyn1boreiko you simply use the new shifthappens.config.verbose option instead of defining this option here.

Let me know what you think (ideally directly on the referenced PR).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @valentyn1boreiko, #75 was merged now.

What do you think about the proposed solution, do you think you could leverage it to adapt this PR?

In case you dont find time, @kotekjedi might probably be able to help with this!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @stes, I added two lines
import shifthappens.config
and
if shifthappens.config.verbose:
in models/base.py

This should be enough, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @stes,
I added

available_models_dict = {'resnet18': ResNet18,
'resnet50': ResNet50,
'vgg16': VGG16}

in worst_case.py

I think it would make sense to have a model card somewhere in the global config, such that one could easy choose a model with a parameter to a script, right?

Should I open a separate PR request for this, or could you add it?


@property
def imagenet_validation_result(self):
Expand Down Expand Up @@ -206,7 +208,13 @@ def _predict_imagenet_val(self):
score_type for score_type in score_types if score_types[score_type]
]
}
for prediction in self._predict(imagenet_val_dataloader, targets):

if self.verbose:
pred_loader = tqdm(self._predict(imagenet_val_dataloader, targets), desc='Predictions', total=int(len(imagenet_val_dataloader._dataset)/imagenet_val_dataloader.max_batch_size))
else:
pred_loader = self._predict(imagenet_val_dataloader, targets)

for prediction in pred_loader:
for prediction_type in predictions_dict:
prediction_score = prediction.__getattribute__(prediction_type)
predictions_dict[prediction_type] = sum(
Expand Down