Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Idea) feature: update configuration for turn-based #241

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from
Open

(Idea) feature: update configuration for turn-based #241

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
439be2b
feature: update configuration for turn-based
YuriCat Jan 22, 2022
1c38ca6
fix: use zero_sum_averaging
YuriCat Jan 22, 2022
0c66a5c
feature: 2p zero-sum setting should be False in default config
YuriCat Jan 22, 2022
203c63b
fix: create dual-player batch when zero_sum_averaging is True
YuriCat Jan 22, 2022
27fff3c
Merge develop
YuriCat Feb 1, 2022
c246d3b
fix: debug comment
YuriCat Feb 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ env_args:
#env: 'handyrl.envs.parallel_tictactoe' # specify by path

train_args:
turn_based_training: True
turn_based_training: False # for turn-based games
zero_sum_averaging: False # for 2p zero-sum games
observation: False
gamma: 0.8
forward_steps: 16
Expand Down
10 changes: 7 additions & 3 deletions handyrl/train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def replace_none(a, b):
moments_ = sum([pickle.loads(bz2.decompress(ms)) for ms in ep['moment']], [])
moments = moments_[ep['start'] - ep['base']:ep['end'] - ep['base']]
players = list(moments[0]['observation'].keys())
if not args['turn_based_training']: # solo training
if not (args['turn_based_training'] or args['zero_sum_averaging']): # solo training
players = [random.choice(players)]

obs_zeros = map_r(moments[0]['observation'][moments[0]['turn'][0]], lambda o: np.zeros_like(o)) # template for padding
Expand Down Expand Up @@ -165,7 +165,10 @@ def forward_prediction(model, hidden, batch, args):
o = o.view(*batch['turn_mask'].size()[:2], -1, o.size(-1))
if k == 'policy':
# gather turn player's policies
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment and line 170 comment is duplicated.

outputs[k] = o.mul(batch['turn_mask']).sum(2, keepdim=True) - batch['action_mask']
outputs[k] = o.mul(batch['turn_mask'])
if args['turn_based_training']:
outputs[k] = outputs[k].sum(2, keepdim=True) # gather turn player's policies
outputs[k] = outputs[k] - batch['action_mask']
else:
# mask valid target values and cumulative rewards
outputs[k] = o.mul(batch['observation_mask'])
Expand Down Expand Up @@ -221,7 +224,8 @@ def compute_loss(batch, model, hidden, args):

if 'value' in outputs_nograd:
values_nograd = outputs_nograd['value']
if args['turn_based_training'] and values_nograd.size(2) == 2: # two player zerosum game
if args['zero_sum_averaging']: # two player zerosum game
assert values_nograd.size(2) == 2
values_nograd_opponent = -torch.stack([values_nograd[:, :, 1], values_nograd[:, :, 0]], dim=2)
values_nograd = (values_nograd + values_nograd_opponent) / (batch['observation_mask'].sum(dim=2, keepdim=True) + 1e-8)
outputs_nograd['value'] = values_nograd * emasks + batch['outcome'] * (1 - emasks)
Expand Down