[Feature] Support ViT-Adapter

[okotaku]

Motivation

paper: https://arxiv.org/abs/2205.08534
code: https://github.com/czczup/ViT-Adapter
issue: open-mmlab/mmdetection#9044

Related PR

open-mmlab/mmdetection#9354
open-mmlab/mmcv#2451
open-mmlab/mmcv#2452

Remaining task

• Reproduce mAP.
• Allow value_proj_ratio to be set. [Enhancement] Support value_proj_ratio in MultiScaleDeformableAttention mmcv#2452
• add large arch
• wait for mmcv PR merged
• beit adapter

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
• CLA has been signed and all committers have signed the CLA in this PR.

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: +87.08 🎉

Comparison is base (b8b31e9) 0.02% compared to head (f22e021) 87.11%.

❗ Current head f22e021 differs from pull request most recent head c92beda. Consider uploading reports for the commit c92beda to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##           dev-1.x    #1209       +/-   ##
============================================
+ Coverage     0.02%   87.11%   +87.08%     
============================================
  Files          121      170       +49     
  Lines         8217    14356     +6139     
  Branches      1368     2267      +899     
============================================
+ Hits             2    12506    +12504     
+ Misses        8215     1449     -6766     
- Partials         0      401      +401     

Flag	Coverage Δ
unittests	87.11% <ø> (+87.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/datasets/transforms/compose.py
mmcls/models/backbones/replknet.py	93.00% <0.00%> (ø)
mmcls/datasets/inshop.py	100.00% <0.00%> (ø)
mmcls/models/retrievers/__init__.py	100.00% <0.00%> (ø)
mmcls/engine/optimizers/adan_t.py	10.60% <0.00%> (ø)
mmcls/engine/hooks/switch_recipe_hook.py	88.46% <0.00%> (ø)
mmcls/models/backbones/tinyvit.py	91.20% <0.00%> (ø)
mmcls/models/heads/vig_head.py	42.30% <0.00%> (ø)
mmcls/models/backbones/mobileone.py	94.47% <0.00%> (ø)
mmcls/utils/analyze.py	100.00% <0.00%> (ø)
... and 161 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[okotaku]

There are several points of concern, including

・Duplication of ViTAdapter and BEiTAdapter
・Implementation of is_cls_token for BEiTAttention

Please review.

[Ezra-Yu]

There are several points of concern, including

・Duplication of ViTAdapter and BEiTAdapter ・Implementation of is_cls_token for BEiTAttention

Please review.

Get it, we will review it soon. We will try to merge it into the new version.

[Ezra-Yu]

Implementation of is_cls_token for BEiTAttention

is the is_cls_token the same as with_cls_token? but the num_extra_tokens seems to be 1 in your code and to be 3 in here.

[Ezra-Yu]

LGTM.