[CI][Caffe Frontend] add caffe environment #6023

fernchen · 2020-07-09T06:07:01Z

Recently，we try to introduce Caffe frontend to TVM, please to ref to #6000 , but original CI environment dose not support for Caffe. In order to pass CI test, we add Caffe environment including Caffe dependent packages and pycaffe for ssd-version, it comes from https://github.com/weiliu89/caffe

@zhiics @tqchen @FrozenGene
Could you please have a look on this PR? Thanks!

fernchen · 2020-07-10T01:51:37Z

May you help me review this pr? @siju-samuel @Huyuwei @hlu1

docker/install/ubuntu_install_caffe.sh

fernchen · 2020-07-15T05:52:05Z

Based on disscution above, there are two kinds of ways to support caffe frontend in TVM:

We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;
We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.

We are very welcome to participate in the discussion！https://discuss.tvm.ai/t/introduce-new-frontend-for-caffe/6918

@tqchen @siju-samuel @zhiics @Huyuwei @hlu1 @kazum @srkreddy1238

FrozenGene · 2020-07-17T08:04:58Z

Gental ping @tqchen @siju-samuel

siju-samuel · 2020-07-17T12:41:34Z

We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;

We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.

I vote for second option. BVLC caffe models will have more users compared to forked versions.
Another concern is lot of dependancies are getting added for running the ci with custom distribution, i think bvlc have support for precompiled caffe installation.

fernchen · 2020-07-18T02:16:41Z

We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;

We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.

I vote for second option. BVLC caffe models will have more users compared to forked versions.
Another concern is lot of dependancies are getting added for running the ci with custom distribution, i think bvlc have support for precompiled caffe installation.

Yes, bvlc distribution has supported for precompiled caffe installation, and It's really convenient to use.
If other people agree, we can support the official version first, and then gradually support other distribution. We can even maintain our own Caffe branch to support other layers, of course it is another topic. @tqchen

fernchen · 2020-07-20T13:10:06Z

Hi @FrozenGene @siju-samuel @tqchen :
I have changed the caffe deps into BVLC distribuition, now we support the official caffe version for TVM firstly.

May you please review them? If there is no problem, I hope to merge as soon as possible in order to pass the CI test of caff front-end codes.

docker/install/ubuntu_install_caffe.sh

tqchen

Some final comments

tqchen · 2020-07-27T22:36:35Z

docker/Dockerfile.ci_gpu

@@ -83,6 +83,9 @@ RUN bash /install/ubuntu_install_dgl.sh
 COPY install/ubuntu_install_vulkan.sh /install/ubuntu_install_vulkan.sh
 RUN bash /install/ubuntu_install_vulkan.sh

+COPY install/ubuntu_install_caffe.sh /install/ubuntu_install_caffe.sh


Can we just install the caffe on the CPU and run ci tests through the CPU side? This would reduce our pressure in terms of depending on the GPU resources

We can add the tests to here https://github.com/apache/incubator-tvm/blob/master/tests/scripts/task_python_frontend_cpu.sh

Good, if we just install the caffe on the CPU, it will reduce a lot of work.

tqchen · 2020-07-27T22:37:11Z

docker/install/ubuntu_install_caffe.sh

+
+apt install caffe-cpu -y
+
+pip install --upgrade scikit-image


Need pip3 instead of pip to introduce it to the python3 env.

I have updated the codes, please review them, thanks.

…r precompiled caffe.

tqchen · 2020-07-28T15:58:57Z

Thanks @fernchen This PR is now merged. Note that we still need to update the ci-cpu image, will rely to this thread once that is done.

tqchen · 2020-07-29T15:56:01Z

ci-cpu is now updated to the latest state

* [CI][Caffe Frontend] add caffe environment * [CI][Caffe Frontend] change the caffe deps into BVLC distribution. * [CI][Caffe Fronted] simplify configuration while installing tzdata for precompiled caffe. * [CI][Caffe Frontend] add more information about tzdata. * [CI][CaffeFrontend]remove the ci for gpu env and change to pip3 env Co-authored-by: fernchen <zifeng.cf@alibaba-inc.com>

mshr-h · 2021-09-16T08:39:32Z

Since BVLC distribution support has successfully completed, it would be great if we could replace BVLC distribution with SSD distribution.

Anyone working on that?
If no, I'll try to do that.

FrozenGene · 2021-09-16T09:30:14Z

Since BVLC distribution support has successfully completed, it would be great if we could replace BVLC distribution with SSD distribution.

Anyone working on that?
If no, I'll try to do that.

Please go ahead. And you could tag @fernchen when you pull request.

FrozenGene reviewed Jul 10, 2020

View reviewed changes

docker/install/ubuntu_install_caffe.sh Outdated Show resolved Hide resolved

fernchen force-pushed the ci_caffe branch from 98bd8a0 to f4d1b0b Compare July 20, 2020 13:02

tqchen requested changes Jul 21, 2020

View reviewed changes

docker/install/ubuntu_install_caffe.sh Show resolved Hide resolved

fernchen force-pushed the ci_caffe branch from f4d1b0b to 383db6e Compare July 21, 2020 02:02

fernchen requested review from tqchen and FrozenGene July 21, 2020 11:14

tqchen requested changes Jul 27, 2020

View reviewed changes

fernchen requested a review from tqchen July 28, 2020 04:28

fernchen added 5 commits July 28, 2020 13:39

[CI][Caffe Frontend] add caffe environment

b5bbbe4

[CI][Caffe Frontend] change the caffe deps into BVLC distribution.

dee15c4

[CI][Caffe Fronted] simplify configuration while installing tzdata fo…

6691e77

…r precompiled caffe.

[CI][Caffe Frontend] add more information about tzdata.

a1e78f4

[CI][CaffeFrontend]remove the ci for gpu env and change to pip3 env

f03a832

fernchen force-pushed the ci_caffe branch from 18dd31e to f03a832 Compare July 28, 2020 05:40

tqchen approved these changes Jul 28, 2020

View reviewed changes

tqchen merged commit d35a149 into apache:master Jul 28, 2020

fernchen mentioned this pull request Aug 6, 2020

[Caffe Frontend] introduce caffe frontend for tvm #6206

Merged

42 tasks

mshr-h mentioned this pull request Sep 21, 2021

[CI][Caffe Frontend] Change the caffe deps into SSD distribution #9060

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI][Caffe Frontend] add caffe environment #6023

[CI][Caffe Frontend] add caffe environment #6023

fernchen commented Jul 9, 2020

fernchen commented Jul 10, 2020

fernchen commented Jul 15, 2020

FrozenGene commented Jul 17, 2020

siju-samuel commented Jul 17, 2020

fernchen commented Jul 18, 2020

fernchen commented Jul 20, 2020

tqchen left a comment

tqchen Jul 27, 2020

tqchen Jul 27, 2020

fernchen Jul 28, 2020

tqchen Jul 27, 2020

fernchen Jul 28, 2020

tqchen commented Jul 28, 2020

tqchen commented Jul 29, 2020

mshr-h commented Sep 16, 2021

FrozenGene commented Sep 16, 2021

[CI][Caffe Frontend] add caffe environment #6023

[CI][Caffe Frontend] add caffe environment #6023

Conversation

fernchen commented Jul 9, 2020

fernchen commented Jul 10, 2020

fernchen commented Jul 15, 2020

FrozenGene commented Jul 17, 2020

siju-samuel commented Jul 17, 2020

fernchen commented Jul 18, 2020

fernchen commented Jul 20, 2020

tqchen left a comment

Choose a reason for hiding this comment

tqchen Jul 27, 2020

Choose a reason for hiding this comment

tqchen Jul 27, 2020

Choose a reason for hiding this comment

fernchen Jul 28, 2020

Choose a reason for hiding this comment

tqchen Jul 27, 2020

Choose a reason for hiding this comment

fernchen Jul 28, 2020

Choose a reason for hiding this comment

tqchen commented Jul 28, 2020

tqchen commented Jul 29, 2020

mshr-h commented Sep 16, 2021

FrozenGene commented Sep 16, 2021