Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add test time augmentation base model. #538

Merged
merged 28 commits into from
Oct 10, 2022

Conversation

HAOCHENYE
Copy link
Collaborator

@HAOCHENYE HAOCHENYE commented Sep 16, 2022

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Add base model for test time augmentation.

Modification

If we want to use TTA model based on MMEngine, we should implement a model inherited from BaseTTAModel like this:

class CustomTTAModel(BaseTTAModel):
    def merge_preds(self, data_samples_list):
        """ merge data_sample_list to one. """
        ...

and then, TestTimeAug transform in mmcv should be used in test_pipeline:

file_client_args = dict(backend='disk')
tta_pipeline = [
        dict(type='LoadImageFromFile', file_client_args=file_client_args),
        dict(type='TestTimeAug',
             transforms=[
                 [dict(type='Resize', scale=(1333, 800), keep_ratio=True),
                  dict(type='Resize', scale=(1333, 800), keep_ratio=True)],
                 [dict(type='RandomFlip', prob=1.),
                  dict(type='RandomFlip', prob=0.)],
                 [dict(type='LoadAnnotations', with_bbox=True)],
                 [dict(type='PackDetInputs',
                       meta_keys=('img_id', 'img_path', 'ori_shape',
                                  'img_shape', 'scale_factor', 'flip',
                                  'flip_direction'))]])
    ]

TestTimeAug and CustomTTAModel are a combo, and should be used together.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@HAOCHENYE HAOCHENYE changed the base branch from HAOCHENYE/tta to main September 16, 2022 07:09
@codecov
Copy link

codecov bot commented Sep 16, 2022

Codecov Report

Base: 77.61% // Head: 77.67% // Increases project coverage by +0.05% 🎉

Coverage data is based on head (247a656) compared to base (5d7527d).
Patch coverage: 76.74% of modified lines in pull request are covered.

❗ Current head 247a656 differs from pull request most recent head a738cce. Consider uploading reports for the commit a738cce to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #538      +/-   ##
==========================================
+ Coverage   77.61%   77.67%   +0.05%     
==========================================
  Files         116      117       +1     
  Lines        8641     8677      +36     
  Branches     1785     1793       +8     
==========================================
+ Hits         6707     6740      +33     
- Misses       1632     1633       +1     
- Partials      302      304       +2     
Flag Coverage Δ
unittests 77.67% <76.74%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmengine/model/wrappers/distributed.py 51.51% <0.00%> (+2.94%) ⬆️
mmengine/runner/checkpoint.py 44.00% <50.00%> (ø)
mmengine/model/wrappers/test_time_aug.py 81.08% <81.08%> (ø)
mmengine/model/__init__.py 75.00% <100.00%> (ø)
mmengine/model/wrappers/__init__.py 70.00% <100.00%> (+3.33%) ⬆️
mmengine/model/base_model/base_model.py 86.11% <0.00%> (+2.77%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@HAOCHENYE HAOCHENYE changed the title [WIP] Add TTA BaseModel [Feature] Add test time augmentation base model. Sep 16, 2022
@HAOCHENYE HAOCHENYE changed the title [Feature] Add test time augmentation base model. [WIP] Add test time augmentation base model. Sep 16, 2022
@ZwwWayne ZwwWayne added this to the 0.2.0 milestone Sep 17, 2022
@ZwwWayne ZwwWayne added the P0 Highest priority label Sep 17, 2022
mmengine/model/__init__.py Outdated Show resolved Hide resolved
mmengine/model/wrappers/test_time_aug.py Outdated Show resolved Hide resolved
mmengine/model/wrappers/test_time_aug.py Outdated Show resolved Hide resolved
mmengine/model/wrappers/test_time_aug.py Outdated Show resolved Hide resolved
mmengine/model/wrappers/test_time_aug.py Outdated Show resolved Hide resolved
mmengine/model/wrappers/test_time_aug.py Outdated Show resolved Hide resolved
@HAOCHENYE HAOCHENYE changed the base branch from main to dev September 23, 2022 07:12
@zhouzaida zhouzaida changed the base branch from dev to main September 28, 2022 02:51
RangiLyu
RangiLyu previously approved these changes Oct 9, 2022
RangiLyu
RangiLyu previously approved these changes Oct 9, 2022
MeowZheng
MeowZheng previously approved these changes Oct 9, 2022
@HAOCHENYE HAOCHENYE dismissed stale reviews from MeowZheng and RangiLyu via c35dc3b October 9, 2022 08:18
@ZwwWayne ZwwWayne merged commit 6245621 into open-mmlab:main Oct 10, 2022
C1rN09 pushed a commit to C1rN09/mmengine that referenced this pull request Nov 1, 2022
* First commit

* add BaseTestTimeAugModel

* Add unit test

* move loop logic to test_step

* fix ddp test

* rename model to module

* optim __init__

* Fix as comment

* Fix as comment

* make val_step should not be called

* make tta do not inherit base model

* Fix unit test

* Enhance docstring

* Fix as comment

* Fix as comment

* minor refine

* minor refine

* minor refine

* fix unit test

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* fix unit test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0 Highest priority
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants