Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhance] Support flat param decay mult and fix bias decay mult of depth-wise conv. #771

Merged
merged 6 commits into from
Dec 16, 2022
Merged

[Enhance] Support flat param decay mult and fix bias decay mult of depth-wise conv. #771

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bdbfbde
[Fix] Fix bias decay mult of depth-wise conv.
RangiLyu Nov 28, 2022
0ff2caa
support flatten param weight decay multiplier
RangiLyu Nov 30, 2022
73b50f4
add unit tests
RangiLyu Dec 8, 2022
c0e6c49
REMOVE TODO
RangiLyu Dec 9, 2022
47c4bfa
update doc
RangiLyu Dec 15, 2022
7fabb40
Merge remote-tracking branch 'origin/main' into fix_dw_bias_mult
RangiLyu Dec 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions mmengine/optim/optimizer/default_constructor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,15 +239,15 @@ def add_params(self,
if is_norm:
param_group[
'weight_decay'] = self.base_wd * norm_decay_mult
# depth-wise conv
elif is_dwconv:
param_group[
'weight_decay'] = self.base_wd * dwconv_decay_mult
# bias lr and decay
elif name == 'bias' and not is_dcn_module:
# TODO: current bias_decay_mult will have affect on DCN
RangiLyu marked this conversation as resolved.
Show resolved Hide resolved
param_group[
'weight_decay'] = self.base_wd * bias_decay_mult
# depth-wise conv
elif is_dwconv:
param_group[
'weight_decay'] = self.base_wd * dwconv_decay_mult
params.append(param_group)
for key, value in param_group.items():
if key == 'params':
Expand Down
3 changes: 1 addition & 2 deletions tests/test_optim/test_optimizer/test_optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,8 +172,7 @@ def _check_sgd_optimizer(self,
# sub.conv1.bias
sub_conv1_bias = param_groups[8]
assert sub_conv1_bias['lr'] == self.base_lr * bias_lr_mult
assert sub_conv1_bias[
'weight_decay'] == self.base_wd * dwconv_decay_mult
assert sub_conv1_bias['weight_decay'] == self.base_wd * bias_decay_mult
# sub.gn.weight
sub_gn_weight = param_groups[9]
assert sub_gn_weight['lr'] == self.base_lr
Expand Down