[GraphBolt] Add offline sampling support #7679
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
To trigger regression tests:
|
mfbalin
reviewed
Aug 10, 2024
| minibatch.seeds.cpu(), | ||
| minibatch.input_nodes.cpu(), | ||
| minibatch.labels.cpu(), | ||
| [block.cpu() for block in minibatch.blocks], |
There was a problem hiding this comment.
This will only work for DGL. I would recommend changing your code so that it works with any GNN framework. Otherwise, your methods should be called DGLMinibatchProvider or something similar.
There was a problem hiding this comment.
For now this is just for alignment with DiskGNN, as if I directly save all attributes in a minibatch, the size will be large and loading the minibatches is time consuming.
There was a problem hiding this comment.
You don't have to save all attributes in minibatch, you can only save minibatch.sampled_subgraphs as it is the counterpart to DGL's blocks.
mfbalin
reviewed
Aug 10, 2024
| if args.cpu_cache_size_in_gigabytes > 0 and isinstance( | ||
| features[("node", None, "feat")], gb.DiskBasedFeature | ||
| ): | ||
| features[("node", None, "feat")] = gb.CPUCachedFeature( |
There was a problem hiding this comment.
Suggested change
| features[("node", None, "feat")] = gb.CPUCachedFeature( | |
| features[("node", None, "feat")] = features[("node", None, "feat")].read_into_memory() | |
| features[("node", None, "feat")] = gb.CPUCachedFeature( |
There was a problem hiding this comment.
You can have this line to debug, that way you won't be affected by any potential bugs inside DiskBasedFeature, it might not be handling exceptions correctly.
There was a problem hiding this comment.
That way, the error will be inside TorchBasedFeature if the rest of your code has a bug.
|
I sill encounter similar issues with the updated master. It indicates a worker thread does not exit normally. And this error disappears when I read the whole feature into main memory. Here is the error message: |
I couldn't reproduce the issue on my local machine. Trying on another machine. |
|
@Liu-rj Can you provide more information on the machine you are getting this error on? |
|
The machine is g5.8xlarge with 32 cores, 128 RAM, one 24GB A10G GPU. And the error is got on EBS io2 SSD. The same instruction runs normally on instance NVMe storage. And other interactions adjusting CPU and GPU cache can also run normally on EBS io2 SSD. I don't know if it's the io_uring issue (maybe related to hardware) or bugs in the code. |
Since I can't reproduce the issue "yet", I might ask you to run the code with a modification to see if it will fix it. I am currently using thread sanitizer to see if it will catch anything. |
mfbalin
reviewed
Aug 14, 2024
|
|
||
|
|
||
| def main(): | ||
| start = time.time() |
There was a problem hiding this comment.
Suggested change
| start = time.time() | |
| torch.ops.graphbolt.set_num_io_uring_threads(4) | |
| start = time.time() |
#7698 fixes the issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sample minibatches in advance and use
MinibatchLoaderto load during online training.Checklist
Please feel free to remove inapplicable items for your PR.
Changes