Complete unittest for trainer_config_helpers. #108

reyoung · 2016-09-23T08:25:55Z

No description provided.

reyoung

This PR is under developing. Added just for early discuss how to unittest config file generator.

reyoung · 2016-09-23T08:28:07Z

python/paddle/trainer_config_helpers/tests/configs/check.md5

@@ -0,0 +1,3 @@
+9c038249ec8ff719753a746cdb04c026  layer_activations.protostr


@emailweixu I just generate proto string and check the md5sum same.

I read the proto string manually to check proto string correctly. Is this way ok?

* Refine img_conv_op * Add conv_shift layer * Add tensor layer, refine selective fc * Try fix some styles * simple rnn layers * Add image layers * util_layers * Use collection.Sequence instead of list|tuple * BugFix for print_layer * Unitest for poolings. * Add test_grumemory_layer * add last_first_seq to unittest * Add test expand layer, test ntm layers * add *args for outputs * Add trans layer * Add test for hsigmoid * also add interface to set param attr * sampling_id, linear_comb * ctc_layer, crf_layer * cost_layers * beam_search

reyoung · 2016-09-27T10:14:58Z

layers.py unittest complete.

luotao1 · 2016-09-28T03:33:44Z

python/paddle/trainer_config_helpers/layers.py

    """
-    TODO(yuyang18): Add docs
+    Simple recurrent unit layer. It is just a fully connect layer through both
+    time and neural network.


The note can copy formula from recurrent_layer.

luotao1 · 2016-09-28T03:38:42Z

python/paddle/trainer_config_helpers/layers.py

@@ -2802,18 +2939,20 @@ def tensor_layer(input, size, act=None, name=None,

    .. code-block:: python

-       tensor = tensor_layer(input=[layer1, layer2])
+       tensor = tensor_layer(x1=layer1, x2=layer2, size=1000)


the input of tensor_layer is x1 and x2, conv_shift_layer is a and b, and the dotmul_operator is x and y, can make them unifined?

luotao1 · 2016-09-28T03:49:56Z

python/paddle/trainer_config_helpers/tests/configs/beam_search.py

+                       bos_id=0,
+                       eos_id=1,
+                       beam_size=5,
+                       result_file="./generated_sequences.txt")


there has unitest (sample_trainer_rnn_gen.conf) for beam search, and the api is refined in pull#122

Conflicts: python/paddle/trainer_config_helpers/layers.py

emailweixu · 2016-09-26T20:54:55Z

python/paddle/trainer_config_helpers/layers.py

@@ -897,6 +926,12 @@ def lstmemory(input, name=None, reverse=False, act=None,
    assert gate_act.support_hppl
    assert state_act.support_hppl
    assert act.support_hppl
+    assert input.size is not None and input.size % 4 == 0
+    if size is not None:
+        logger.warning("NOTE: The lstmemory layer[%s]'s size is set by previous"


should be logger.fatal

emailweixu · 2016-09-26T20:55:29Z

python/paddle/trainer_config_helpers/layers.py

@@ -999,6 +1034,11 @@ def grumemory(input, name=None, reverse=False, act=None,

    assert act.support_hppl
    assert gate_act.support_hppl
+    assert input.size is not None and input.size % 3 == 0
+    if size is not None:
+        logger.warning("NOTE: the gru memory layer's size is set by previous "


logger.fatal

emailweixu · 2016-09-28T06:01:23Z

python/paddle/trainer_config_helpers/layers.py


    Layer(
        name=name,
        type=LayerType.PRINT_LAYER,
        inputs=[l.name for l in input],
    )
-    LayerOutput(name, LayerType.PRINT_LAYER, input)
+    return LayerOutput(name, LayerType.PRINT_LAYER, input)


print_layer should not return anything

* Fix a unittest bug before. * dotmul inputs' sizes should be equal * add equation to recurrent_layer * unify x,y x1,x2 => a,b * log fatal when lstm/gru set wrong size

Fix incorrect softlink

* fix installation docs

* fix ins no * fix FillOnestep args * fix infer && add infer table cap * fix infer

* Optimizing the zero key problem in the push phase * Optimize CUDA thread parallelism in MergeGrad phase * Optimize CUDA thread parallelism in MergeGrad phase * Performance optimization, segment gradient merging * Performance optimization, segment gradient merging * Optimize pullsparse and increase keys aggregation * sync gpugraph to gpugraph_v2 (#86) * change load node and edge from local to cpu (#83) * change load node and edge * remove useless code Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * extract pull sparse as single stage(#85) Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> * [GPUGraph] graph sample v2 (#87) * change load node and edge from local to cpu (#83) * change load node and edge * remove useless code Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * extract pull sparse as single stage(#85) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * support ssdsparsetable;test=develop (#81) * graph sample v2 * remove log Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: danleifeng <52735331+danleifeng@users.noreply.github.com> * Release cpu graph * uniq nodeid (#89) * compatible whole HBM mode (#91) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * Gpugraph v2 (#93) * compatible whole HBM mode * unify flag for graph emd storage mode and graph struct storage mode * format Co-authored-by: yangjunchao <yangjunchao@baidu.com> * split generate batch into multi stage (#92) * split generate batch into multi stage * fix conflict Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * [GpuGraph] Uniq feature (#95) * uniq feature * uniq feature * uniq feature * [GpuGraph] global startid (#98) * uniq feature * uniq feature * uniq feature * global startid * load node edge seperately and release graph (#99) * load node edge seperately and release graph * load node edge seperately and release graph Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * v2 infer (#102) * optimize begin pass and end pass (#106) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * fix ins no (#104) * [GPUGraph] fix FillOneStep args (#107) * fix ins no * fix FillOnestep args * fix bug for whole hbm mode (#110) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * [GPUGraph] fix infer && add infer_table_cap (#108) * fix ins no * fix FillOnestep args * fix infer && add infer table cap * fix infer * 【PSCORE】perform ssd sparse table (#111) * perform ssd sparsetable;test=develop Conflicts: paddle/fluid/framework/fleet/ps_gpu_wrapper.cc * perform ssd sparsetable;test=develop * remove debug code; * remove debug code; * add jemalloc cmake;test=develop * fix wrapper;test=develop * fix sample core (#114) * [GpuGraph] optimize shuffle batch (#115) * fix sample core * optimize shuffle batch * release gpu mem when sample end (#116) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * fix class not found err (PaddlePaddle#118) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * optimize sample (PaddlePaddle#117) * optimize sample * optimize sample Co-authored-by: yangjunchao <yangjunchao@baidu.com> * fix clear gpu mem (PaddlePaddle#119) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * fix sample core (PaddlePaddle#121) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * add ssd cache (PaddlePaddle#123) * add ssd cache;test=develop * add ssd cache;test=develop * add ssd cache;test=develop * add multi epoch train & fix train table change ins & save infer embeding (PaddlePaddle#129) * add multi epoch train & fix train table change ins & save infer embedding * change epoch finish judge * change epoch finish change Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * Add debug log (PaddlePaddle#131) * Add debug log * Add debug log Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0008.yq01.baidu.com> * optimize mem in uniq slot feature (PaddlePaddle#130) * [GpuGraph] cherry pick var slot feature && fix load multi path node (PaddlePaddle#136) * optimize mem in uniq slot feature * cherry-pick var slot_feature Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com> * [GpuGraph] fix kernel overflow (PaddlePaddle#138) * optimize mem in uniq slot feature * cherry-pick var slot_feature * fix kernel overflow && add max feature num flag Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com> * fix ssd cache;test=develop (PaddlePaddle#139) * slot feature secondary storage (PaddlePaddle#140) * slot feature secondary storage * slot feature secondary storage Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0008.yq01.baidu.com> Co-authored-by: xuewujiao <105861147+xuewujiao@users.noreply.github.com> Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: Thunderbrook <52529258+Thunderbrook@users.noreply.github.com> Co-authored-by: danleifeng <52735331+danleifeng@users.noreply.github.com> Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com>

* Optimizing the zero key problem in the push phase * Optimize CUDA thread parallelism in MergeGrad phase * Optimize CUDA thread parallelism in MergeGrad phase * Performance optimization, segment gradient merging * Performance optimization, segment gradient merging * Optimize pullsparse and increase keys aggregation * sync gpugraph to gpugraph_v2 (PaddlePaddle#86) * change load node and edge from local to cpu (PaddlePaddle#83) * change load node and edge * remove useless code Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * extract pull sparse as single stage(PaddlePaddle#85) Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> * [GPUGraph] graph sample v2 (PaddlePaddle#87) * change load node and edge from local to cpu (PaddlePaddle#83) * change load node and edge * remove useless code Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * extract pull sparse as single stage(PaddlePaddle#85) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * support ssdsparsetable;test=develop (PaddlePaddle#81) * graph sample v2 * remove log Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: danleifeng <52735331+danleifeng@users.noreply.github.com> * Release cpu graph * uniq nodeid (PaddlePaddle#89) * compatible whole HBM mode (PaddlePaddle#91) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * Gpugraph v2 (PaddlePaddle#93) * compatible whole HBM mode * unify flag for graph emd storage mode and graph struct storage mode * format Co-authored-by: yangjunchao <yangjunchao@baidu.com> * split generate batch into multi stage (PaddlePaddle#92) * split generate batch into multi stage * fix conflict Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * [GpuGraph] Uniq feature (PaddlePaddle#95) * uniq feature * uniq feature * uniq feature * [GpuGraph] global startid (PaddlePaddle#98) * uniq feature * uniq feature * uniq feature * global startid * load node edge seperately and release graph (PaddlePaddle#99) * load node edge seperately and release graph * load node edge seperately and release graph Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * v2 infer (PaddlePaddle#102) * optimize begin pass and end pass (PaddlePaddle#106) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * fix ins no (PaddlePaddle#104) * [GPUGraph] fix FillOneStep args (PaddlePaddle#107) * fix ins no * fix FillOnestep args * fix bug for whole hbm mode (PaddlePaddle#110) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * [GPUGraph] fix infer && add infer_table_cap (PaddlePaddle#108) * fix ins no * fix FillOnestep args * fix infer && add infer table cap * fix infer * 【PSCORE】perform ssd sparse table (PaddlePaddle#111) * perform ssd sparsetable;test=develop Conflicts: paddle/fluid/framework/fleet/ps_gpu_wrapper.cc * perform ssd sparsetable;test=develop * remove debug code; * remove debug code; * add jemalloc cmake;test=develop * fix wrapper;test=develop * fix sample core (PaddlePaddle#114) * [GpuGraph] optimize shuffle batch (PaddlePaddle#115) * fix sample core * optimize shuffle batch * release gpu mem when sample end (PaddlePaddle#116) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * fix class not found err (PaddlePaddle#118) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * optimize sample (PaddlePaddle#117) * optimize sample * optimize sample Co-authored-by: yangjunchao <yangjunchao@baidu.com> * fix clear gpu mem (PaddlePaddle#119) Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * fix sample core (PaddlePaddle#121) Co-authored-by: yangjunchao <yangjunchao@baidu.com> * add ssd cache (PaddlePaddle#123) * add ssd cache;test=develop * add ssd cache;test=develop * add ssd cache;test=develop * add multi epoch train & fix train table change ins & save infer embeding (PaddlePaddle#129) * add multi epoch train & fix train table change ins & save infer embedding * change epoch finish judge * change epoch finish change Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> * Add debug log (PaddlePaddle#131) * Add debug log * Add debug log Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0008.yq01.baidu.com> * optimize mem in uniq slot feature (PaddlePaddle#130) * [GpuGraph] cherry pick var slot feature && fix load multi path node (PaddlePaddle#136) * optimize mem in uniq slot feature * cherry-pick var slot_feature Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com> * [GpuGraph] fix kernel overflow (PaddlePaddle#138) * optimize mem in uniq slot feature * cherry-pick var slot_feature * fix kernel overflow && add max feature num flag Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com> * fix ssd cache;test=develop (PaddlePaddle#139) * slot feature secondary storage (PaddlePaddle#140) * slot feature secondary storage * slot feature secondary storage Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0008.yq01.baidu.com> Co-authored-by: xuewujiao <105861147+xuewujiao@users.noreply.github.com> Co-authored-by: miaoli06 <106585574+miaoli06@users.noreply.github.com> Co-authored-by: root <root@yq01-inf-hic-k8s-a100-ab2-0009.yq01.baidu.com> Co-authored-by: chao9527 <33347532+chao9527@users.noreply.github.com> Co-authored-by: yangjunchao <yangjunchao@baidu.com> Co-authored-by: Thunderbrook <52529258+Thunderbrook@users.noreply.github.com> Co-authored-by: danleifeng <52735331+danleifeng@users.noreply.github.com> Co-authored-by: huwei02 <53012141+huwei02@users.noreply.github.com>

* trie by custname, aigc-model-42 * one stage topk, aigc-model-74 * copy from cpu paddle tensor & infer run release GIL, aigc-model-79 * reduced batch topk, aigc-model-85 --------- Co-authored-by: wanglipeng <wanglipeng@baidu.com>

reyoung commented Sep 23, 2016

View reviewed changes

reyoung force-pushed the unittest_for_helpers branch 6 times, most recently from 7bbfb9e to fbbd7a0 Compare September 27, 2016 08:54

reyoung force-pushed the unittest_for_helpers branch from fbbd7a0 to 6343d4a Compare September 27, 2016 10:12

reyoung assigned luotao1, qingqing01 and emailweixu Sep 27, 2016

luotao1 reviewed Sep 28, 2016

View reviewed changes

Merge branch 'master' into unittest_for_helpers

96df7e5

Conflicts: python/paddle/trainer_config_helpers/layers.py

emailweixu reviewed Sep 28, 2016

View reviewed changes

Follow comments

584c0fc

* Fix a unittest bug before. * dotmul inputs' sizes should be equal * add equation to recurrent_layer * unify x,y x1,x2 => a,b * log fatal when lstm/gru set wrong size

reyoung force-pushed the unittest_for_helpers branch from 86f095a to 584c0fc Compare September 28, 2016 06:27

luotao1 approved these changes Sep 28, 2016

View reviewed changes

reyoung merged commit d130d18 into PaddlePaddle:master Sep 28, 2016

reyoung deleted the unittest_for_helpers branch September 28, 2016 06:37

zhhsplendid pushed a commit to zhhsplendid/Paddle that referenced this pull request Sep 25, 2019

Merge pull request PaddlePaddle#108 from tink2123/fix_softlink

8a8202a

Fix incorrect softlink

thisjiang pushed a commit to thisjiang/Paddle that referenced this pull request Oct 28, 2021

fix auto separate bug (PaddlePaddle#108)

c062900

wangxicoding pushed a commit to wangxicoding/Paddle that referenced this pull request Dec 9, 2021

fix installation docs (PaddlePaddle#108)

8d54a20

* fix installation docs

danleifeng pushed a commit to danleifeng/Paddle that referenced this pull request Sep 14, 2022

[GPUGraph] fix infer && add infer_table_cap (PaddlePaddle#108)

1462c54

* fix ins no * fix FillOnestep args * fix infer && add infer table cap * fix infer

AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this pull request Sep 19, 2022

Fix some bugs. (PaddlePaddle#108)

d360801

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complete unittest for trainer_config_helpers. #108

Complete unittest for trainer_config_helpers. #108

reyoung commented Sep 23, 2016

reyoung left a comment

reyoung Sep 23, 2016

reyoung commented Sep 27, 2016

luotao1 Sep 28, 2016

luotao1 Sep 28, 2016

luotao1 Sep 28, 2016

emailweixu Sep 26, 2016

emailweixu Sep 26, 2016

emailweixu Sep 28, 2016

		@@ -0,0 +1,3 @@
		9c038249ec8ff719753a746cdb04c026 layer_activations.protostr

Complete unittest for trainer_config_helpers. #108

Complete unittest for trainer_config_helpers. #108

Conversation

reyoung commented Sep 23, 2016

reyoung left a comment

Choose a reason for hiding this comment

reyoung Sep 23, 2016

Choose a reason for hiding this comment

reyoung commented Sep 27, 2016

luotao1 Sep 28, 2016

Choose a reason for hiding this comment

luotao1 Sep 28, 2016

Choose a reason for hiding this comment

luotao1 Sep 28, 2016

Choose a reason for hiding this comment

emailweixu Sep 26, 2016

Choose a reason for hiding this comment

emailweixu Sep 26, 2016

Choose a reason for hiding this comment

emailweixu Sep 28, 2016

Choose a reason for hiding this comment