ndarray: more Base-like APIs #303

iblislin · 2017-10-31T12:58:28Z

remapping
- mean
- sum
- maximum
- minimum
- permutedims
- prod
also fix the axis value mapping
mean(arr, axis=0) is not Julian

- also fix the axis value mapping - `mean(arr, axis=0)` is not Julian

codecov-io · 2017-10-31T20:06:56Z

Codecov Report

Merging #303 into master will decrease coverage by 1.02%.
The diff coverage is 72.72%.

@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
- Coverage    70.3%   69.27%   -1.03%     
==========================================
  Files          25       25              
  Lines        1926     1894      -32     
==========================================
- Hits         1354     1312      -42     
- Misses        572      582      +10

Impacted Files	Coverage Δ
src/util.jl	`32.95% <100%> (-2.61%)`	⬇️
src/ndarray.jl	`83.18% <66.66%> (-1.51%)`	⬇️
src/compat.jl	`0% <0%> (-50%)`	⬇️
src/base.jl	`30.76% <0%> (-3.28%)`	⬇️
src/executor.jl	`86.9% <0%> (-1.74%)`	⬇️
src/symbolic-node.jl	`71.08% <0%> (-1.19%)`	⬇️
src/kvstore.jl	`75% <0%> (-0.87%)`	⬇️
src/model.jl	`70.27% <0%> (-0.78%)`	⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41f8f55...a267493. Read the comment docs.

pluskid · 2017-10-31T20:47:59Z

min, max, etc?

- remove `mx.max`, `mx.min`, `mx.max_axis` and `mx.min_axis`

iblislin · 2017-11-01T02:02:02Z

Done for min and max. Anything else I missed?

pluskid · 2017-11-01T16:05:12Z

I'm searching the doc for the keyword 'dim' or 'axis', it seems there are a lot of operators to consider (e.g. prod). I'm wondering if there is better way to handle this. E.g. detecting in the meta data from libmxnet whether that argument is called dim / axis?

iblislin · 2017-11-01T17:08:54Z

I'm wondering if there is better way to handle this. E.g. detecting in the meta data from libmxnet whether that argument is called dim / axis?

I guess that docstring is the only set we can search. doesnt see any API about the function signature or similar.

I still prefer to do it manually.
I think a better way will be:

add a utility script which does some text seaching/parsing for function signature
run it in CI (maybe) and it pops up some warning
then, (if someone find these warning message) open an issue for remapping the functions that bug by this script.
one of the purpose of manually remapping function is add test cases for them, in order to prevent unaware breaking changes from upsream.

for discovering non-Julian APIs

iblislin · 2017-11-04T07:42:40Z

@pluskid please checkout 8c5b6cf

julia> mx._sig_checker()                        
WARNING: BatchNorm(data, gamma, beta, moving_mean, moving_var, eps, momentum, fix_gamma, use_global_stats, output_mean_var, axis, cudnn_off)
WARNING: CuDNNBatchNorm(data, eps, momentum, fix_gamma, use_global_stats, output_mean_var, axis, cudnn_off)
WARNING: Deconvolution(data, weight, bias, kernel, stride, dilate, pad, adj, target_shape, num_filter, num_group, workspace, no_bias, cudnn_tune, c
udnn_off, layout)                                                      
WARNING: GridGenerator(data, transform_type, target_shape)
WARNING: SliceChannel(data, num_outputs, axis, squeeze_axis)
WARNING: Softmax(data, grad_scale, ignore_label, multi_output, use_ignore, preserve_shape, normalization, out_grad, smooth_alpha)
WARNING: SoftmaxOutput(data, label, grad_scale, ignore_label, multi_output, use_ignore, preserve_shape, normalization, out_grad, smooth_alpha)
WARNING: SpatialTransformer(data, loc, target_shape, transform_type, sampler_type)
WARNING: _backward_slice_axis()                      
WARNING: _ones(shape, ctx, dtype)                                          
WARNING: _random_exponential(lam, shape, ctx, dtype)      
WARNING: _random_gamma(alpha, beta, shape, ctx, dtype)
WARNING: _random_generalized_negative_binomial(mu, alpha, shape, ctx, dtype)
WARNING: _random_negative_binomial(k, p, shape, ctx, dtype)
WARNING: _random_normal(loc, scale, shape, ctx, dtype)
WARNING: _random_poisson(lam, shape, ctx, dtype)
WARNING: _random_uniform(low, high, shape, ctx, dtype)
WARNING: _sample_exponential(lam, shape, dtype) 
WARNING: _sample_gamma(alpha, shape, dtype, beta)
WARNING: _sample_generalized_negative_binomial(mu, shape, dtype, alpha)
WARNING: _sample_multinomial(data, shape, get_prob, dtype)
WARNING: _sample_negative_binomial(k, shape, dtype, p)
WARNING: _sample_normal(mu, shape, dtype, sigma)
WARNING: _sample_poisson(lam, shape, dtype)
WARNING: _sample_uniform(low, shape, dtype, high)
WARNING: _sparse_sum(data, axis, keepdims, exclude)
WARNING: _square_sum(data, axis, keepdims, exclude)
WARNING: _zeros(shape, ctx, dtype)
WARNING: argmax(data, axis, keepdims)
WARNING: argmin(data, axis, keepdims)
WARNING: argsort(data, axis, is_ascend)
WARNING: broadcast_axes(data, axis, size)
WARNING: broadcast_axis(data, axis, size)
WARNING: broadcast_to(data, shape)
WARNING: expand_dims(data, axis)
WARNING: flip(data, axis)
WARNING: log_softmax(data, axis)
WARNING: nanprod(data, axis, keepdims, exclude)
WARNING: nansum(data, axis, keepdims, exclude)
WARNING: normal(loc, scale, shape, ctx, dtype)
WARNING: pick(data, index, axis, keepdims)
WARNING: random_exponential(lam, shape, ctx, dtype)
WARNING: random_gamma(alpha, beta, shape, ctx, dtype)
WARNING: random_generalized_negative_binomial(mu, alpha, shape, ctx, dtype)
WARNING: random_negative_binomial(k, p, shape, ctx, dtype)
WARNING: random_normal(loc, scale, shape, ctx, dtype)
WARNING: random_poisson(lam, shape, ctx, dtype)
WARNING: random_uniform(low, high, shape, ctx, dtype)
WARNING: repeat(data, repeats, axis)
WARNING: reshape_like(lhs, rhs)
WARNING: reverse(data, axis)
WARNING: sample_exponential(lam, shape, dtype)
WARNING: sample_gamma(alpha, shape, dtype, beta)
WARNING: sample_generalized_negative_binomial(mu, shape, dtype, alpha)
WARNING: sample_multinomial(data, shape, get_prob, dtype)
WARNING: sample_negative_binomial(k, shape, dtype, p)
WARNING: sample_normal(mu, shape, dtype, sigma)
WARNING: sample_poisson(lam, shape, dtype)
WARNING: sample_uniform(low, shape, dtype, high)
WARNING: scatter_nd(data, indices, shape)
WARNING: slice_axis(data, axis, begin, end)
WARNING: softmax(data, axis)
WARNING: sort(data, axis, is_ascend)
WARNING: split(data, num_outputs, axis, squeeze_axis)
WARNING: stack(data, axis, num_args)
WARNING: sum_axis(data, axis, keepdims, exclude)
WARNING: swapaxes(data, dim1, dim2)
WARNING: take(a, indices, axis, mode)
WARNING: topk(data, axis, k, ret_typ, is_ascend)
WARNING: uniform(low, high, shape, ctx, dtype)

iblislin · 2017-11-04T07:47:25Z

I can keep refining those APIs, but at this moment, I want to make a new release first. So I won't add more remap to this PR.

pluskid · 2017-11-06T19:35:01Z

src/util.jl

+end
+
+"""
+libmxnet operators signature checker.


Can you add more desc to the docstring?

pluskid · 2017-11-06T19:36:19Z

Thanks! Merged. But can you add a more detailed docstring for _sig_checker?

iblislin · 2017-11-07T04:55:38Z

sure.

iblislin · 2017-11-07T05:14:02Z

It's #305

ndarray: make API of sum and mean be Base-like

4d8dfc4

- also fix the axis value mapping - `mean(arr, axis=0)` is not Julian

iblislin force-pushed the remap-mean-sum branch from 24cb30c to 4d8dfc4 Compare October 31, 2017 16:28

iblislin added 2 commits November 1, 2017 09:46

ndarray: Base-like maximum and minimum

ac9a5b9

- remove `mx.max`, `mx.min`, `mx.max_axis` and `mx.min_axis`

ndarray: simple doc while remapping with @_remap

87b7714

iblislin mentioned this pull request Nov 1, 2017

Cut a release for mxnet v0.12.0 #304

Merged

5 tasks

ndarray: more test cases for dim as tuple

6f57a29

iblislin changed the title ~~ndarray: make API of sum and mean be Base-like~~ ndarray: more Base-like APIs Nov 1, 2017

iblislin added 3 commits November 1, 2017 15:35

ndarray: remap dot, the elegent way

6f1f19c

ndarray: remap transpose and add permutedims

daebf13

ndarray: docs of _remap

7e19bec

iblislin force-pushed the remap-mean-sum branch from f62d2d4 to 7e19bec Compare November 1, 2017 08:12

iblislin added 2 commits November 2, 2017 01:33

ndarray: remap prod

bcf695f

util: add _sig_checker

8c5b6cf

for discovering non-Julian APIs

travis: add _sig_checker after testing

a267493

pluskid reviewed Nov 6, 2017

View reviewed changes

src/util.jl

end

"""

libmxnet operators signature checker.

Copy link

Member

pluskid Nov 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add more desc to the docstring?

pluskid merged commit 4f182ee into dmlc:master Nov 6, 2017

iblislin deleted the remap-mean-sum branch November 7, 2017 01:02

iblislin added this to the v0.3.0 milestone Nov 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ndarray: more Base-like APIs #303

ndarray: more Base-like APIs #303

iblislin commented Oct 31, 2017 •

edited

Loading

codecov-io commented Oct 31, 2017 •

edited

Loading

pluskid commented Oct 31, 2017

iblislin commented Nov 1, 2017

pluskid commented Nov 1, 2017

iblislin commented Nov 1, 2017

iblislin commented Nov 4, 2017

iblislin commented Nov 4, 2017

pluskid Nov 6, 2017

pluskid commented Nov 6, 2017

iblislin commented Nov 7, 2017

iblislin commented Nov 7, 2017

ndarray: more Base-like APIs #303

ndarray: more Base-like APIs #303

Conversation

iblislin commented Oct 31, 2017 • edited Loading

codecov-io commented Oct 31, 2017 • edited Loading

Codecov Report

pluskid commented Oct 31, 2017

iblislin commented Nov 1, 2017

pluskid commented Nov 1, 2017

iblislin commented Nov 1, 2017

iblislin commented Nov 4, 2017

iblislin commented Nov 4, 2017

pluskid Nov 6, 2017

Choose a reason for hiding this comment

pluskid commented Nov 6, 2017

iblislin commented Nov 7, 2017

iblislin commented Nov 7, 2017

iblislin commented Oct 31, 2017 •

edited

Loading

codecov-io commented Oct 31, 2017 •

edited

Loading