Add Lost in Translation #97
Conversation
Docstring Coverage Report
RESULT: PASSED (minimum: 0.0%, actual: 97.3%) |
There was a problem hiding this comment.
Hi Leander!
I commented on mistakes that did not allow me to run the code.
About the part where only the torchvision models can be used - I am still looking at this code and trying to understand what causes the problem.
If you need any help - feel free to write me!
| import math | ||
|
|
||
|
|
||
| @sh_benchmark.register_task( |
There was a problem hiding this comment.
Since it is a parent class of tasks here, we don't need to register the task.
Or is this class also a task itself?
| default_batch_size: int = 700 | ||
|
|
||
| resolution: int = parameter( | ||
| default=224, |
There was a problem hiding this comment.
| default=224, | |
| default=224, | |
| options=(244,), |
We need to add the option's value here.
| ) | ||
|
|
||
| rotation_cutoff: int = parameter( | ||
| default=0, |
There was a problem hiding this comment.
| default=0, | |
| default=0, | |
| options=(0,), |
We need to add the option's value here.
| ) | ||
|
|
||
| translation_cutoff: int = parameter( | ||
| default=0, |
There was a problem hiding this comment.
| default=0, | |
| default=0, | |
| options=(0,), |
We need to add the option's value here.
|
|
||
| subdir = 'validation-segmentation' | ||
| gt_dir = os.path.join(imagent_s_folder, params['dir'], subdir) | ||
| self.dataset = i_s.ImageNetSEvalDataset(imagenet_root_path, gt_dir, name_list, transform=tt, |
There was a problem hiding this comment.
I think we need to pass label_path here after the name_list
| ) | ||
| @dataclasses.dataclass | ||
| class LostInTranslationTranslation(LostInTranslationBase): | ||
| def _eval_model(model, model_name, batch_size_model, data, eval_device): |
There was a problem hiding this comment.
| def _eval_model(model, model_name, batch_size_model, data, eval_device): | |
| def _eval_model(self, model, model_name, batch_size_model, data, eval_device): |
| ) | ||
| @dataclasses.dataclass | ||
| class LostInTranslationRotation(LostInTranslationBase): | ||
| def _eval_model(model, model_name, batch_size_model, data, eval_device): |
There was a problem hiding this comment.
| def _eval_model(model, model_name, batch_size_model, data, eval_device): | |
| def _eval_model(self, model, model_name, batch_size_model, data, eval_device): |
Very drafty yet, it's more work than expected to slim down the code and integrate it into the framework. For now only works with pytorch models.
Task Description
Add two datasets showing robustness with regards to translation and rotation, while guaranteeing that both the object is always fully visible and we never leave the image. [Paper]((https://openreview.net/forum?id=L8awC6wq_Fs) published at the Shift-Happens workshop.
Dataset Creation
We only depend on previously released datasets (imagenet-s, which in turn depends on new imagenet-labels).
Evaluation Metrics
Accuracy, base and worst-case
Expected Insights/Relevance
the accuracy decreases significantly for these datasets, even for state of the art models.
Access
Only depends on freely available datasets.
Data
Imagenet-s
License
No data provided, but the data is without a license (after correspondence with the author of imagenet-s).