Add paddle hub #31873

lyuwenyu · 2021-03-26T02:41:24Z

PR types

New features

PR changes

Others

Describe

Usage:

import paddle.hub as hub

if __name__ == '__main__':

    models = hub.list('lyuwenyu/PaddleClas:hub_L', force_reload=False)
    print(len(models))
    
    # print(hub.list('/Users/lvwenyu01/.cache/paddle/hub/lyuwenyu_PaddleClas_hub_L', source='local'))
    doc = hub.help('lyuwenyu/PaddleClas:hub_L', model='ResNet18')
    print(doc)

    model = hub.load('lyuwenyu/PaddleClas:hub_L', model='MobileNetV3_large_x0_35', source='github', pretrained=True)
    # print(model)
    # print(type(model))

paddle-bot-old · 2021-03-26T02:41:27Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

LielinJiang · 2021-03-31T13:13:34Z

python/paddle/hapi/hub.py

+        ```python
+        import paddle
+
+        paddle.hub.help('lyuwenyu/PaddleClas:hub_L', source='github', force_reload=True)


using official repo will be better

sounds good. It's testing code, beacuse PaddleCls dont have hub.conf by now.

LielinJiang · 2021-03-31T13:13:53Z

Need a simple unittest

qingqing01 · 2021-04-02T05:51:49Z

python/paddle/hapi/hub.py

+    Load model
+
+    Args:
+        repo_dir(str): github or local path


as offline discussion, also can add gitee

ok, will push it after testing offline

ZeyuChen

LGTM

nepeplwu · 2021-04-22T12:06:22Z

python/paddle/hapi/hub.py

+            repo_dir, force_reload, True, source=source)
+
+    sys.path.insert(0, repo_dir)
+    hub_module = __import__(MODULE_HUBCONF.split('.')[0])


It seems that sys.modules['hubconf'] will be used by default on the second load, which will causes unexpected behavior.

You are right. I should pop it after __import__ module to keep sys.module clean

sys.modules.pop('hubconf')

ps. using importlib seems beeter, but it‘s incompatible in py2 and py3 😢

nepeplwu · 2021-04-22T12:30:50Z

python/paddle/hapi/hub.py

+    # Setup hub_dir to save downloaded files
+    hub_dir = HUB_DIR
+    if not os.path.exists(hub_dir):
+        os.makedirs(hub_dir)


It seems that multi-process calls may cause the problem of repeated directory creation and raise an exception.

suggest：

os.makedirs(hub_dir, exist_ok=True)

nepeplwu · 2021-04-22T12:37:23Z

python/paddle/hapi/hub.py

+    func = getattr(m, name, None)
+
+    if func is None or not callable(func):
+        raise RuntimeError('Canot find callable {} in hubconf'.format(name))


Canot -> Cannot

jzhang533 · 2021-04-22T14:42:55Z

python/paddle/hapi/hub.py

+    Args:
+        repo_dir(str): github or local path
+            github path (str): a str with format "repo_owner/repo_name[:tag_name]" with an optional
+                tag/branch. The default branch is `master` if not specified.


default是main还是master
MASTER_BRANCH定义是main

应该是main, 我修改下

jzhang533 · 2021-04-22T14:44:04Z

python/paddle/hapi/hub.py

+        ```python
+        import paddle
+
+        paddle.hub.help('lyuwenyu/paddlehub_demo:main', source='github', force_reload=False)


示例的api不对

Got it ~😢

jzhang533 · 2021-04-22T14:45:49Z

python/paddle/hapi/hub.py

+    Args:
+        repo_dir(str): github or local path
+            github path (str): a str with format "repo_owner/repo_name[:tag_name]" with an optional
+                tag/branch. The default branch is `master` if not specified.


jzhang533 · 2021-04-22T14:46:29Z

python/paddle/hapi/hub.py

+    Args:
+        repo_dir(str): github or local path
+            github path (str): a str with format "repo_owner/repo_name[:tag_name]" with an optional
+                tag/branch. The default branch is `master` if not specified.


jzhang533 · 2021-04-22T14:46:51Z

python/paddle/hapi/hub.py

+            github path (str): a str with format "repo_owner/repo_name[:tag_name]" with an optional
+                tag/branch. The default branch is `master` if not specified.
+            local path (str): local repo path
+        mdoel (str): model name


XiaoguangHu01 · 2021-04-22T14:51:54Z

python/paddle/tests/test_hapi_hub.py

+        np.testing.assert_equal(out.shape, [1, 8, 50, 50])
+
+        model = hub.load(
+            self.github_repo, model='MM', source='github', force_reload=True)


是否能检查下load之后的结果是否正确？
这几个参数是否发挥作用，这个用例并不能很好的测试出来。

已修改, 增加固定输入和pretrained参数并测试输出的正确性

这几个参数是否发挥作用，是的一个用例不能很好的反馈出来但是结合上下文多个tests按顺序是隐式的 1. hub.load source local和remote模型使用的地址不一致，2. 首次使用remote时候本地是没有缓存的当为False时候不会download数据测试能过说明前边为True时候起作用了，3. 不符合要求的参数在异常里测试的

XiaoguangHu01 · 2021-04-22T14:52:39Z

python/paddle/tests/test_hapi_hub.py

+        docs = hub.help(
+            self.github_repo, model='MM', source='github', force_reload=False)
+
+        print(docs)


检查下docs的值是否正确？

嗯嗯可以

XiaoguangHu01 · 2021-04-22T14:53:30Z

python/paddle/tests/test_hapi_hub.py

+            source='github',
+            force_reload=False, )
+
+        print(models)


检查下list的执行结果是否符合预期？

嗯嗯可以

jzhang533 · 2021-04-23T09:42:33Z

python/paddle/hapi/hub.py

@@ -19,7 +19,7 @@
 import zipfile
 from paddle.utils.download import get_path_from_url

-MASTER_BRANCH = 'main'
+MAIN_BRANCH = 'main'


变量名称是不是应该叫DEFAULT_BRANCH
另外，是不是github的时候default是main, gitee的时候default是master，比较合理。

jzhang533

lgtm

saxon-zh

LGTM

LielinJiang and others added 9 commits March 9, 2021 08:41

add hub list function

6ef62e0

add help load func

a71dff5

add load and help funcs.

64570eb

add local repo support for list/help/load func

af43bfd

add local repo support for list/help/load func

00a9786

add _check_dependencies hubconf

1b613cb

update hub docs, test=develop

884bbf9

update hub docs, test=develop

2c52a28

update docs

f9dd61d

lyuwenyu changed the title ~~Hub l~~ add paddle Hub Mar 26, 2021

lyuwenyu closed this Mar 26, 2021

lyuwenyu reopened this Mar 26, 2021

lyuwenyu closed this Mar 26, 2021

lyuwenyu reopened this Mar 26, 2021

upadte docs

c8d2ae3

lyuwenyu changed the title ~~add paddle Hub~~ Add paddle hub Mar 31, 2021

Merge remote-tracking branch 'upstream/develop' into hub_L

8acff14

qingqing01 requested review from LielinJiang, qingqing01 and heavengate March 31, 2021 12:55

LielinJiang reviewed Mar 31, 2021

View reviewed changes

qingqing01 reviewed Apr 2, 2021

View reviewed changes

lyuwenyu added 6 commits April 6, 2021 15:35

init unittest for hapi/hub

2237120

add hub unittest, test=develop

8185cc4

support source gitee

c4347b2

update hapi hub test

aa0eac9

for ci test

dfa6d03

add hub test

ec5183d

ZeyuChen previously approved these changes Apr 22, 2021

View reviewed changes

nepeplwu reviewed Apr 22, 2021

View reviewed changes

jzhang533 reviewed Apr 22, 2021

View reviewed changes

XiaoguangHu01 reviewed Apr 22, 2021

View reviewed changes

fix docs, bugs of import, and more unittest

ae0bb17

lyuwenyu dismissed stale reviews from ZeyuChen, qingqing01, and XieYunshen via ae0bb17 April 23, 2021 02:46

update

59e57e3

PaddlePaddle locked and limited conversation to collaborators Apr 23, 2021

PaddlePaddle unlocked this conversation Apr 23, 2021

PaddlePaddle locked and limited conversation to collaborators Apr 23, 2021

PaddlePaddle unlocked this conversation Apr 23, 2021

lyuwenyu added 3 commits April 23, 2021 15:20

update

204e047

update

56e1042

Merge remote-tracking branch 'upstream/develop' into hub_L

aea875a

XieYunshen previously approved these changes Apr 23, 2021

View reviewed changes

jzhang533 reviewed Apr 23, 2021

View reviewed changes

update

67d17ca

lyuwenyu dismissed XieYunshen’s stale review via 67d17ca April 23, 2021 13:02

update

cad8c80

XieYunshen previously approved these changes Apr 25, 2021

View reviewed changes

update default branch

7408e84

lyuwenyu dismissed XieYunshen’s stale review via 7408e84 April 25, 2021 02:43

update, remove default branch var

7829730

jzhang533 approved these changes Apr 25, 2021

View reviewed changes

saxon-zh approved these changes Apr 25, 2021

View reviewed changes

XieYunshen approved these changes Apr 25, 2021

View reviewed changes

qingqing01 merged commit 4e460d7 into PaddlePaddle:develop Apr 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add paddle hub #31873

Add paddle hub #31873

lyuwenyu commented Mar 26, 2021

paddle-bot-old bot commented Mar 26, 2021

LielinJiang Mar 31, 2021

lyuwenyu Apr 2, 2021

LielinJiang commented Mar 31, 2021

qingqing01 Apr 2, 2021

lyuwenyu Apr 2, 2021

ZeyuChen left a comment

nepeplwu Apr 22, 2021

lyuwenyu Apr 22, 2021 •

edited

Loading

nepeplwu Apr 22, 2021

lyuwenyu Apr 22, 2021

nepeplwu Apr 22, 2021

lyuwenyu Apr 22, 2021

jzhang533 Apr 22, 2021

lyuwenyu Apr 22, 2021

jzhang533 Apr 22, 2021

lyuwenyu Apr 22, 2021

jzhang533 Apr 22, 2021

jzhang533 Apr 22, 2021

jzhang533 Apr 22, 2021

XiaoguangHu01 Apr 22, 2021

lyuwenyu Apr 23, 2021 •

edited

Loading

XiaoguangHu01 Apr 22, 2021

lyuwenyu Apr 22, 2021

XiaoguangHu01 Apr 22, 2021

lyuwenyu Apr 22, 2021

jzhang533 Apr 23, 2021

lyuwenyu Apr 25, 2021

jzhang533 left a comment

saxon-zh left a comment

Add paddle hub #31873

Add paddle hub #31873

Conversation

lyuwenyu commented Mar 26, 2021

PR types

PR changes

Describe

paddle-bot-old bot commented Mar 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LielinJiang commented Mar 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZeyuChen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyuwenyu Apr 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyuwenyu Apr 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

saxon-zh left a comment

Choose a reason for hiding this comment

lyuwenyu Apr 22, 2021 •

edited

Loading

lyuwenyu Apr 23, 2021 •

edited

Loading