Fix local recordio reader #3479
Conversation
helinwang
force-pushed
the
recordio
branch
2 times, most recently
from
3914045
to
785db4e
Compare
| path = paths | ||
| else: | ||
| path = ",".join(paths) | ||
| f = rec.reader(path) |
There was a problem hiding this comment.
When training on distributed environment, reader should fetch only a part of all the data.
There was a problem hiding this comment.
@typhoonzero The files will be read file by file. If we need to fetch only part of the data, maybe we can split them to different files. cloud_reader is much better for the distributed training environment.
There was a problem hiding this comment.
cloud_reader is using master to dispatch tasks, I think we still need a version with out master, or master can dispatch files rather than recordio. This is in case that there are still many cases using Mapreduce output directly as training inputs.
Anyway, will discuss in another issue.
There was a problem hiding this comment.
I see, thanks. The recordio file will be read entirely to parse the index before reading the first item, so in that case we need to shard the files.
| @@ -57,7 +57,7 @@ def reader(): | |||
| return reader | |||
|
|
|||
|
|
|||
| def recordio_local(paths, buf_size=100): | |||
| def recordio(paths, buf_size=100): | |||
There was a problem hiding this comment.
Maybe rename to recordio_reader to indicate it's a reader?
There was a problem hiding this comment.
@typhoonzero It's used as paddle.v2.reader.creator.recordio, probably reader is indicated in the import path.
No description provided.