-
Notifications
You must be signed in to change notification settings - Fork 615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a native dataloader to RN50 PyTorch example #4807
Conversation
!build |
CI MESSAGE: [8078512]: BUILD STARTED |
CI MESSAGE: [8078512]: BUILD PASSED |
- extends DALI RN50 PyOtrch example based on https://github.com/NVIDIA/apex/tree/master/examples/imagenet by adding a native dataloader and easy ability to switch between DALI on and DALI off Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
def __iter__(self): | ||
return self | ||
|
||
def __next__(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this on top of the original APEX example to make it more aligned with DALI iterator.
Btw. Do we run the baseline anyhow in CI? |
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Added a basic run (to check if the script works). |
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
!build |
CI MESSAGE: [8089640]: BUILD STARTED |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few small nitpicks, I usually don't do that, but there are formatting differences between neighbouring lines of code.
pipe.build() | ||
train_loader = DALIClassificationIterator(pipe, reader_name="Reader", | ||
last_batch_policy=LastBatchPolicy.PARTIAL, | ||
auto_reset =True) | ||
|
||
pipe = create_dali_pipeline(batch_size=args.batch_size, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick, my eyes hurt due to overwriting of the pipeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
train_loader = torch.utils.data.DataLoader( | ||
train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None), | ||
num_workers=args.workers, pin_memory=True, sampler=train_sampler, collate_fn=collate_fn) | ||
|
||
val_loader = torch.utils.data.DataLoader( | ||
val_dataset, | ||
batch_size=args.batch_size, shuffle=False, | ||
num_workers=args.workers, pin_memory=True, | ||
sampler=val_sampler, | ||
collate_fn=collate_fn) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you format it in the same way?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
train_dataset = datasets.ImageFolder( | ||
traindir, | ||
transforms.Compose([ | ||
transforms.RandomResizedCrop(crop_size), | ||
transforms.RandomHorizontalFlip(), | ||
])) | ||
val_dataset = datasets.ImageFolder(valdir, transforms.Compose([ | ||
transforms.Resize(val_size), | ||
transforms.CenterCrop(crop_size), | ||
])) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you format it in the same way?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
parser.add_argument('--disable_dali', default=False, action='store_true', | ||
help='If use DALI at all') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
parser.add_argument('--disable_dali', default=False, action='store_true', | |
help='If use DALI at all') | |
parser.add_argument('--disable_dali', default=False, action='store_true', | |
help='Disable DALI data loader and use native PyTorch one instead.') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
CI MESSAGE: [8089640]: BUILD FAILED |
CI MESSAGE: [8089640]: BUILD PASSED |
https://github.com/NVIDIA/apex/tree/master/examples/imagenet
by adding a native dataloader and easy ability to switch between DALI on and DALI off
Category:
Other (e.g. Documentation, Tests, Configuration)
Description:
https://github.com/NVIDIA/apex/tree/master/examples/imagenet
by adding a native dataloader and easy ability to switch between DALI on and DALI off
Additional information:
Affected modules and functionalities:
Key points relevant for the review:
Tests:
Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: N/A