RunAsync C/CXX API #16613

RandySheriffH · 2023-07-06T18:42:12Z

Implement RunAsync API - the session will run in a thread of intra-op thread pool.

pranavsharma

How will users turn on this feature? By configuring num inter op threads? Should we've a session option to turn on async runs? Not all sessions require it and for those that don't creating an inter-op threadpool is waste of resources.
Reminder to add documentation.

include/onnxruntime/core/session/onnxruntime_c_api.h

onnxruntime/core/session/inference_session.cc

RandySheriffH · 2023-07-06T23:59:39Z

How will users turn on this feature? By configuring num inter op threads? Should we've a session option to turn on async runs? Not all sessions require it and for those that don't creating an inter-op threadpool is waste of resources. Reminder to add documentation.

Let's stick with intra op thread pool to avoid all those.

include/onnxruntime/core/session/onnxruntime_c_api.h

include/onnxruntime/core/session/onnxruntime_cxx_inline.h

include/onnxruntime/core/session/onnxruntime_cxx_api.h

onnxruntime/core/session/inference_session.cc

onnxruntime/test/shared_lib/test_inference.cc

include/onnxruntime/core/session/onnxruntime_c_api.h

include/onnxruntime/core/session/onnxruntime_cxx_inline.h

include/onnxruntime/core/session/onnxruntime_cxx_api.h

include/onnxruntime/core/session/onnxruntime_cxx_inline.h

Add ut --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>

include/onnxruntime/core/session/onnxruntime_cxx_inline.h

pranavsharma · 2023-07-11T19:53:44Z

Also, a reminder that the documentation should call out that the callback is going to be executed in one of the intra op threadpool threads.

yuslepukhin · 2023-07-11T21:27:41Z

Is there a design document for this? In my book running async requires a huge number of changes.
Also what is the purpose of running async? There is no IO, we are CPU bound. What would a callback do? Set a condition var? What is the difference between sync call return and wait on cond var?
I fail to understand what the async mean.

RandySheriffH · 2023-07-11T23:23:18Z

Also, a reminder that the documentation should call out that the callback is going to be executed in one of the intra op threadpool threads.

Would this be good enough?

onnxruntime/include/onnxruntime/core/session/onnxruntime_c_api.h

Line 4332 in 2baa70a

    
           /** \brief Run the model asynchronously in a thread owned by intra op thread pool

fs-eire · 2023-07-11T23:47:47Z

Is there a design document for this? In my book running async requires a huge number of changes. Also what is the purpose of running async? There is no IO, we are CPU bound. What would a callback do? Set a condition var? What is the difference between sync call return and wait on cond var? I fail to understand what the async mean.

I have to agree with you... My understanding is that, this PR is just a change to introduce the async API, which does not have a real "async" implementation. The major value of this PR is for reviewing the interface. The implementation does not work in ort-web anyway.

RandySheriffH · 2023-07-11T23:58:57Z

Is there a design document for this? In my book running async requires a huge number of changes. Also what is the purpose of running async? There is no IO, we are CPU bound. What would a callback do? Set a condition var? What is the difference between sync call return and wait on cond var? I fail to understand what the async mean.

I have to agree with you... My understanding is that, this PR is just a change to introduce the async API, which does not have a real "async" implementation. The major value of this PR is for reviewing the interface. The implementation does not work in ort-web anyway.

No, it does not target ort-web, at least not for now. The target of the PR is to make RunAsync work for desktop and mobile for 1.16, specifically, run session in separate thread and invoke callback. For web cases, the API might migrate with a different behavior in the future.

include/onnxruntime/core/session/onnxruntime_c_api.h

include/onnxruntime/core/session/onnxruntime_cxx_api.h

include/onnxruntime/core/session/onnxruntime_c_api.h

include/onnxruntime/core/session/onnxruntime_cxx_api.h

onnxruntime/core/session/inference_session.h

onnxruntime/core/session/inference_session.cc

onnxruntime/core/session/onnxruntime_c_api.cc

pranavsharma

LGTM 👍

include/onnxruntime/core/session/onnxruntime_c_api.h

onnxruntime/core/session/inference_session.cc

RandyShuai added 2 commits July 6, 2023 11:36

implement RunAsync c api

a872264

implement RunAsync cxx api

28ed47e

pranavsharma reviewed Jul 6, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

add ut

cb7d9ee

RandySheriffH changed the title ~~RunAsync C API~~ RunAsync C/CXX API Jul 6, 2023

handle exception

4db5c59

RandyShuai added 2 commits July 6, 2023 18:19

poll for ret

0cf25df

rename var

f9df857

RandySheriffH marked this pull request as ready for review July 7, 2023 17:23

RandySheriffH commented Jul 7, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

fs-eire reviewed Jul 7, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_cxx_inline.h Outdated Show resolved Hide resolved

fix warning

4283709

fs-eire reviewed Jul 7, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_cxx_api.h Outdated Show resolved Hide resolved

RandyShuai added 3 commits July 7, 2023 18:02

rewrapper cxx interface

8869dcd

fix format

0ac824e

minor fix“

a499dbf

pranavsharma reviewed Jul 10, 2023

View reviewed changes

RandyShuai and others added 6 commits July 10, 2023 13:34

add ut

b18aa26

Add ut (#16647)

43d01b3

Add ut --------- Co-authored-by: Randy Shuai <rashuai@microsoft.com>

use inline vector internally

3688327

format code

a736acb

resolve conflict

0615511

format code

a478081

fs-eire reviewed Jul 10, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_cxx_inline.h Outdated Show resolved Hide resolved

RandyShuai added 3 commits July 11, 2023 14:48

add output value to RunAsync

95c9fdd

fix build

3c9e80e

format code

2baa70a

format

813b0d2

pranavsharma reviewed Jul 12, 2023

View reviewed changes

fix comments

6c67e44

pranavsharma previously approved these changes Jul 12, 2023

View reviewed changes

skottmckay reviewed Jul 13, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

skottmckay reviewed Jul 13, 2023

View reviewed changes

fix comments

bc607f2

RandySheriffH dismissed pranavsharma’s stale review via bc607f2 July 13, 2023 02:28

keep cxx styling

ce1f898

skottmckay reviewed Jul 14, 2023

View reviewed changes

onnxruntime/core/session/onnxruntime_c_api.cc Outdated Show resolved Hide resolved

RandyShuai added 5 commits July 13, 2023 17:29

remove extra space

f92c937

Merge branch 'main' into rashuai/RunAsync

092d23f

keep const inside span

bb73ab1

keep const inside span

c64eac4

fix typo

8206220

pranavsharma reviewed Jul 14, 2023

View reviewed changes

skottmckay reviewed Jul 14, 2023

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Show resolved Hide resolved

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

RandyShuai added 2 commits July 14, 2023 18:00

move exception handling out of catch

f3cfd91

tune callback in exception

11c09b0

skottmckay approved these changes Jul 16, 2023

View reviewed changes

RandySheriffH merged commit e1ca8ee into main Jul 16, 2023

RandySheriffH deleted the rashuai/RunAsync branch July 16, 2023 23:51

MaxMood96 mentioned this pull request May 23, 2023

[Snyk] Security upgrade webpack from 5.36.2 to 5.76.0 MaxMood96/onnxruntime#411

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RunAsync C/CXX API #16613

RunAsync C/CXX API #16613

RandySheriffH commented Jul 6, 2023 •

edited

Loading

pranavsharma left a comment

RandySheriffH commented Jul 6, 2023 •

edited

Loading

pranavsharma commented Jul 11, 2023

yuslepukhin commented Jul 11, 2023

RandySheriffH commented Jul 11, 2023

fs-eire commented Jul 11, 2023 •

edited

Loading

RandySheriffH commented Jul 11, 2023 •

edited

Loading

pranavsharma left a comment

RunAsync C/CXX API #16613

RunAsync C/CXX API #16613

Conversation

RandySheriffH commented Jul 6, 2023 • edited Loading

pranavsharma left a comment

Choose a reason for hiding this comment

RandySheriffH commented Jul 6, 2023 • edited Loading

pranavsharma commented Jul 11, 2023

yuslepukhin commented Jul 11, 2023

RandySheriffH commented Jul 11, 2023

fs-eire commented Jul 11, 2023 • edited Loading

RandySheriffH commented Jul 11, 2023 • edited Loading

pranavsharma left a comment

Choose a reason for hiding this comment

RandySheriffH commented Jul 6, 2023 •

edited

Loading

RandySheriffH commented Jul 6, 2023 •

edited

Loading

fs-eire commented Jul 11, 2023 •

edited

Loading

RandySheriffH commented Jul 11, 2023 •

edited

Loading