Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can NCS2 use multiple processes? Or can an NCS2 run multiple models simultaneously? #2

Open
ZhaoMonica opened this issue May 23, 2019 · 17 comments
Open

Can NCS2 use multiple processes? Or can an NCS2 run multiple models simultaneously? #2

ZhaoMonica opened this issue May 23, 2019 · 17 comments
Labels
question Further information is requested

Comments

@ZhaoMonica
Copy link

Hello, may I ask "_worker = threading. Thread (target = async_infer_worker, args= (exec_nets [f], request_number, image_queue, out_blob)" in your "async_api_multi-processes_multi-requests_multi-ncs.py" file?

_ worker. start ()

Infer_threads. append (_worker)

Can "preprocess_process. join ()" be changed to process run? I tried unsuccessfully myself. I only have one NCS2. I want to run two models, one Yolo and one SSD model at the same time. Does NCS2 support it?
@decemberpei
Copy link
Owner

decemberpei commented May 23, 2019 •
edited
Loading

am sorry your post is badly formatted i can't catch your meaning. For your last question, The answer is Yes. you can run two models simutaneously with only one NCS2, with lower performance result of course.

@ZhaoMonica
Copy link
Author

from openvino.inference_engine import IENetwork, IEPlugin
import cv2
import multiprocessing as mp

def main(net, plugin):

input_blob = next(iter(net.inputs))
out_blob = next(iter(net.outputs))

exec_net = plugin.load(network=net)
frame = cv2.imread("/home/sdu/my_test/1557038029.0015957.jpeg")
n, c, h, w = net.inputs[input_blob].shape

image = cv2.resize(frame, (w, h))
image = image.transpose((2, 0, 1))
image = image.reshape((n, c, h, w))

res = exec_net.infer({input_blob: image})
output_blob = res
print(output_blob[out_blob].shape)

#####################################################
def run():
net1 = IENetwork(
model='/home/sdu/my_test/model_data/yolov2/FP16_new/yolo_test.xml',
weights='/home/sdu/my_test/model_data/yolov2/FP16_new/yolov_test.bin')
net2 = IENetwork(model='/home/sdu/my_test/model_data/ssd/ssd/deploy.xml',
weights='/home/sdu/my_test/model_data/ssd/ssd/deploy.bin')

plugin = IEPlugin(device="MYRIAD")
nets = [net1, net2]
processes = []
for net in nets:
    processes.append(mp.Process(target=main, args=(net,plugin)))

[setattr(process, "daemon", True) for process in processes]  # process.daemon = True
[process.start() for process in processes]
[process.join() for process in processes]
del net
del plugin

run()

@ZhaoMonica
Copy link
Author

I use the above code and run two models at the same time to report the following errors:

Process Process-1:

Traceback (most recent call last):

File "/usr/lib/python 3.5/multiprocessing/process.py", line 249, in_bootstrap

Self.run ()

File "/usr/lib/python 3.5/multiprocessing/process.py", line 93, in run

Self. _target (* self. _args,** self. _kwargs)

File "mulit_nets_test_01.py", line 12, in main

Exec_net = plugin. load (network = net)

File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load

File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load

Runtime Error: Can not init USB device: NC_DEVICE_NOT_FOUND

E: [ncAPI] [719106] ncDeviceOpen: 870 failed to find device

Process Process-2:

Traceback (most recent call last):

File "/usr/lib/python 3.5/multiprocessing/process.py", line 249, in_bootstrap

Self.run ()

File "/usr/lib/python 3.5/multiprocessing/process.py", line 93, in run

Self. _target (* self. _args,** self. _kwargs)

File "mulit_nets_test_01.py", line 12, in main

Exec_net = plugin. load (network = net)

File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load

File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load

Runtime Error: Can not init USB device: NC_DEVICE_NOT_FOUND

In fact, I can be sure that my NCS2 is plugged in and there is no problem. If I use "processes. append" (threading. Thread (target = main, args = (net, plugin)) to get results, does this mean that NCS2 cannot run two models in parallel?

@ZhaoMonica
Copy link
Author

processes.append(threading.Thread(target=main, args=(net,plugin )))

@decemberpei
Copy link
Owner

if you are sure the issue is not caused by your NCS2, suggest you try use threads instead of processes ( since different processed have different memory spaces, "passing" IEPlugin objects between processes may cause unexpected behavior like what you've ran into).

if you really want multiple processes, you can init IEPlugin instances in child processes. this should also work.

@ZhaoMonica
Copy link
Author

Hello, I don't quite understand what you mean by "Initializing IEPlugin in a child process". Can you show it in my code above? Thank you very much.

@ZhaoMonica
Copy link
Author

[<Process(Process-1, initial)>, <Process(Process-2, initial)>]
E: [ncAPI] [ 175625] ncDeviceOpen:870 failed to find device

Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "mulit_nets_test_01.py", line 12, in main
exec_net = plugin.load(network=net)
File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load
File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load
RuntimeError: Can not init USB device: NC_DEVICE_NOT_FOUND
(1, 1, 100, 7)

I put the initialization of IEPlugin in the main () function of the code above, which shows the results. Only one model has been loaded and output. Do you know why?

@decemberpei
Copy link
Owner

decemberpei commented May 23, 2019 via email

@decemberpei decemberpei added the question Further information is requested label May 24, 2019
@ZhaoMonica
Copy link
Author

Hello, can I understand that a NCS2 can only perform "plugin = IEPlugin" (device = "MYRIAD") once, and if it is a process, it will need several NCS2 to initialize IEPlugin once for each process.

@decemberpei
Copy link
Owner

Hello, can I understand that a NCS2 can only perform "plugin = IEPlugin" (device = "MYRIAD") once, and if it is a process, it will need several NCS2 to initialize IEPlugin once for each process.

I haven't tried but I don't agree,

  1. If in your OS (say, Linux/Windows), there are several separate apps which are trying to use the only one NCS2, probably they don't know the existance of each other, and will init their own IEPlugin instance. it would be very confusing if one App succeed to init IEPlugin while others fail to do so.
  2. from OS point of view, "processes" are just different Apps that run in parallel, when you pass objects such as IEPlugin instances to other processes, the object get "cloned", the cloned object is by no mean the original one and may cause errors when you try to invoke it. hence If I were you i won't pass IEPlugin instances between processes.

@ZhaoMonica
Copy link
Author

I have some questions about your first point: if there are several separate applications trying to use the same NCS2, does NCS2 support separate access for several separate applications?

@ZhaoMonica
Copy link
Author

I use process simulation to simulate several separate applications to access NCS2 at the same time. In fact, when I insert an NCS2, only one application is working properly. Others will "fail to find NCS2". When I insert two NCS2, two programs will run normally. Does this mean that NCS2 cannot be processed or accessed simultaneously?

@decemberpei
Copy link
Owner

decemberpei commented May 24, 2019 via email

@decemberpei
Copy link
Owner

decemberpei commented May 24, 2019 via email •
edited
Loading

@ZhaoMonica
Copy link
Author

Hi, how can I avoid having only one thread running if I have an NCS2 and start four threads, each of which is an infinite loop?I want all four threads to execute in rotation, not just one running and all the others blocking.

@decemberpei
Copy link
Owner

decemberpei commented Jun 3, 2019 •
edited
Loading

Hi, how can I avoid having only one thread running if I have an NCS2 and start four threads, each of which is an infinite loop?I want all four threads to execute in rotation, not just one running and all the others blocking.

sorry your question has nothing to do with NCS2 nor this repo, Please google it yourself.

@decemberpei
Copy link
Owner

just found some tips in official doc:
http://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_MYRIAD.html#MYRIAD_DEVICE_ALLOC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@decemberpei @ZhaoMonica