data reader for mnist #1325
Closed
data reader for mnist #1325
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luotao1
reviewed
Feb 13, 2017
|
|
||
|
|
||
| class DataReader(object): | ||
| def __init__(self, data, labels, batch_size, is_shuffle=False): |
There was a problem hiding this comment.
如果有多个data或labels为空的情况，这个接口可以复用么
There was a problem hiding this comment.
这个只是针对MNIST，不是通用的，其他的任务需要重新写。
Closed
helinwang
reviewed
Feb 14, 2017
| num_magic, n, num_row, num_col = struct.unpack(">IIII", f.read(16)) | ||
| images = np.fromfile(f, 'ubyte', count=n * num_row * num_col).\ | ||
| reshape(n, num_row * num_col).astype('float32') | ||
| images = images / 255.0 * 2.0 - 1.0 |
There was a problem hiding this comment.
好奇images = images / 255.0 * 2.0 - 1.0这样把均值往0.0拉近一些，会比images = images / 255.0大概好多少？（比如说是98.55％ -> 98.57％或者98.5%->98.9%），非常大概的估计就好。
There was a problem hiding this comment.
images = images / 255.0 * 2.0 - 1.0 -> 是归到[-1, 1]
images = images / 255.0 ->[0, 1] 两者结果得做实验对比吧，感觉相差可能不会太大。
这里是继续采用了原始mnist demo的处理方式。
There was a problem hiding this comment.
这两个取值范围就不一样吧，一个是[-1, 1]，一个是[0, 1]。
|
|
||
| def create_datasets(dir='./data/raw_data/'): | ||
| ''' | ||
| 数据download 和 load可以依据https://github.com/PaddlePaddle/Paddle/pull/872来简化 |
There was a problem hiding this comment.
没有看到download的函数，感觉要是能自动download会方便用户使用一些。
There was a problem hiding this comment.
#872 这个PR merge了之后会自动下载数据，这里就没有写。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
大数据不一定都能load到内存，可能需要设计其他的缓存机制。