Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
.. Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed
with this work for additional information regarding copyright
ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the "License"); you may not use this file
except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.

.. include:: ../../../common.defs

.. default-domain:: c

TSContScheduleEveryOnEntirePool
*******************************

Synopsis
========

.. code-block:: cpp

#include <ts/ts.h>

.. function:: std::vector<TSAction> TSContScheduleEveryOnEntirePool(TSCont contp, TSHRTime every, TSThreadPool tp)

Description
===========

Schedules :arg:`contp` to run :arg:`every` milliseconds, on all threads that
belongs to :arg:`tp`. The :arg:`every` is an approximation, meaning it will be at least
:arg:`every` milliseconds but possibly more. Resolutions finer than roughly 5 milliseconds will
not be effective. Note that :arg:`contp` is required to NOT have a mutex, since the continuation
is scheduled on multiple threads. This means the continuation must handle synchronization itself.

The return value can be used to cancel the scheduled event via :func:`TSActionCancel`. This
is effective until the continuation :arg:`contp` is being dispatched. However, if it is
scheduled on another thread this can be problematic to be correctly timed. The return value
can be checked with :func:`TSActionDone` to see if the continuation ran before the return.

.. note:: Due to scheduling multiple events, the return value is changed to :type:`std::vector<TSAction>`, as compared to :type:`TSAction` of the other `TSContSchedule` APIs.

Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

See Also
========

:doc:`TSContScheduleOnPool.en`
:doc:`TSContScheduleOnThread.en`
:doc:`TSContScheduleOnEntirePool.en`
:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleEveryOnThread.en`
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Synopsis

#include <ts/ts.h>

.. function:: TSAction TSContScheduleEveryOnPool(TSCont contp, TSHRTime every)
.. function:: TSAction TSContScheduleEveryOnPool(TSCont contp, TSHRTime every, TSThreadPool tp)

Description
===========
Expand All @@ -42,17 +42,19 @@ effective. Note that :arg:`contp` is required to have a mutex, which is provided
The return value can be used to cancel the scheduled event via :func:`TSActionCancel`. This
is effective until the continuation :arg:`contp` is being dispatched. However, if it is
scheduled on another thread this can be problematic to be correctly timed. The return value
can be checked with :func:`TSActionDone` to see if the continuation ran before the return,
which is possible if :arg:`timeout` is `0`.
can be checked with :func:`TSActionDone` to see if the continuation ran before the return.

If :arg:`contp` has no thread affinity set, the thread it is now scheduled on will be set
as its thread affinity thread.

Note that the TSContSchedule() family of API shall only be called from an ATS EThread.
Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

See Also
========

:doc:`TSContScheduleOnPool.en`
:doc:`TSContScheduleOnThread.en`
:doc:`TSContScheduleOnEntirePool.en`
:doc:`TSContScheduleEveryOnThread.en`
:doc:`TSContScheduleEveryOnEntirePool.en`
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
.. Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed
with this work for additional information regarding copyright
ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the "License"); you may not use this file
except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.

.. include:: ../../../common.defs

.. default-domain:: c

TSContScheduleEveryOnThread
***************************

Synopsis
========

.. code-block:: cpp

#include <ts/ts.h>

.. function:: TSAction TSContScheduleEveryOnThread(TSCont contp, TSHRTime every, TSEventThread ethread)

Description
===========

Schedules :arg:`contp` to run :arg:`every` milliseconds, on the thread specified by
:arg:`ethread`. The :arg:`every` is an approximation, meaning it will be at least :arg:`every`
milliseconds but possibly more. Resolutions finer than roughly 5 milliseconds will not be
effective. Note that :arg:`contp` is required to have a mutex, which is provided to
:func:`TSContCreate`.

The return value can be used to cancel the scheduled event via :func:`TSActionCancel`. This
is effective until the continuation :arg:`contp` is being dispatched. However, if it is
scheduled on another thread this can be problematic to be correctly timed. The return value
can be checked with :func:`TSActionDone` to see if the continuation ran before the return.

If :arg:`contp` has no thread affinity set, the thread it is now scheduled on will be set
as its thread affinity thread.

Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

See Also
========

:doc:`TSContScheduleOnPool.en`
:doc:`TSContScheduleOnThread.en`
:doc:`TSContScheduleOnEntirePool.en`
:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleEveryOnEntirePool.en`
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
.. Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed
with this work for additional information regarding copyright
ownership. The ASF licenses this file to you under the Apache
License, Version 2.0 (the "License"); you may not use this file
except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.

.. include:: ../../../common.defs

.. default-domain:: c

TSContScheduleOnEntirePool
**************************

Synopsis
========

.. code-block:: cpp

#include <ts/ts.h>

.. function:: std::vector<TSAction> TSContScheduleOnEntirePool(TSCont contp, TSHRTime timeout, TSThreadPool tp)
Copy link
Contributor

@zwoop zwoop Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeout seems wrong name here. It's a delay, right ? I got confused here because the "every" version also uses both timeout and delay "randomly"?

Are these API prototypes the same as what was sent to the mailing list? If not, maybe an update there as well ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just following the naming convention for the other TSContSchedule APIs. I'll update the email in the mailing list to show the changes of this PR.

The timeout is kind of like a delay, but it could also be a negative number for negative queue events.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we got rid of the negative queue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty sure it's still here?

process_queue(&NegativeQueue, &ev_count, &nq_count);


Description
===========

Schedules :arg:`contp` to run :arg:`timeout` milliseconds in the future, on all threads that
belongs to :arg:`tp`. The :arg:`timeout` is an approximation, meaning it will be at least
:arg:`timeout` milliseconds but possibly more. Resolutions finer than roughly 5 milliseconds will
not be effective. Note that :arg:`contp` is required to NOT have a mutex, since the continuation
is scheduled on multiple threads. This means the continuation must handle synchronization itself.

The return value can be used to cancel the scheduled event via :func:`TSActionCancel`. This
is effective until the continuation :arg:`contp` is being dispatched. However, if it is
scheduled on another thread this can be problematic to be correctly timed. The return value
can be checked with :func:`TSActionDone` to see if the continuation ran before the return,
which is possible if :arg:`timeout` is `0`.

.. note:: Due to scheduling multiple events, the return value is changed to :type:`std::vector<TSAction>`, as compared to :type:`TSAction` of the other `TSContSchedule` APIs.

Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

See Also
========

:doc:`TSContScheduleOnPool.en`
:doc:`TSContScheduleOnThread.en`
:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleEveryOnThread.en`
:doc:`TSContScheduleEveryOnEntirePool.en`
47 changes: 25 additions & 22 deletions doc/developer-guide/api/functions/TSContScheduleOnPool.en.rst
Original file line number Diff line number Diff line change
Expand Up @@ -68,28 +68,28 @@ another thread this can be problematic to be correctly timed. The return value c
If :arg:`contp` has no thread affinity set, the thread it is now scheduled on will be set
as its thread affinity thread.

Note that the TSContSchedule() family of API shall only be called from an ATS EThread.
Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

Note that some times if the TASK threads are not ready when you schedule a "contp" on the ``TS_THREAD_POOL_TASK`` then
the "contp" could end up being executed in a ``ET_NET`` thread instead, more likely this is not what you want.
Note that some times if the TASK threads are not ready when you schedule a :arg:`contp` on the ``TS_THREAD_POOL_TASK`` then
the :arg:`contp` could end up being executed in a ``ET_NET`` thread instead, more likely this is not what you want.
To avoid this you can use the ``TS_LIFECYCLE_TASK_THREADS_READY_HOOK`` to make sure that the task threads
have been started when you schedule a "contp". You can refer to :func:`TSLifecycleHookAdd` for more details.
have been started when you schedule a :arg:`contp`. You can refer to :func:`TSLifecycleHookAdd` for more details.

Example Scenarios
=================

Scenario 1 (no thread affinity info, different types of threads)
----------------------------------------------------------------

When thread affinity is not set, a plugin calls the API on thread "A" (which is an "ET_TASK" type), and
wants to schedule on an "ET_NET" type thread provided in "tp", the system would see there is no thread
When thread affinity is not set, a plugin calls the API on thread "A" (which is an ``ET_TASK`` type), and
wants to schedule on an ``ET_NET`` type thread provided in :arg:`tp`, the system would see there is no thread
affinity information stored in "contp."

In this situation, system sees there is no thread affinity information stored in "contp". It then
checks whether the type of thread "A" is the same as provided in "tp", and sees that "A" is "ET_TASK",
but "tp" says "ET_NET". So "contp" gets scheduled on the next available "ET_NET" thread provided by a
round robin list, which we will call thread "B". Since "contp" doesn't have thread affinity information,
In this situation, system sees there is no thread affinity information stored in :arg:`contp`. It then
checks whether the type of thread "A" is the same as provided in :arg:`tp`, and sees that "A" is ``ET_TASK``,
but :arg:`tp` says ``ET_NET``. So :arg:`contp` gets scheduled on the next available ``ET_NET`` thread provided by a
round robin list, which we will call thread "B". Since :arg:`contp` doesn't have thread affinity information,
thread "B" will be assigned as the affinity thread for it automatically.

The reason for doing this is most of the time people want to schedule the same things on the same type
Expand All @@ -99,11 +99,11 @@ thread.
Scenario 2 (no thread affinity info, same types of threads)
-----------------------------------------------------------

Slight variation of scenario 1, instead of scheduling on a "ET_NET" thread, the plugin wants to schedule
on a "ET_TASK" thread (i.e. "tp" contains "ET_TASK" now), all other conditions stays the same.
Slight variation of scenario 1, instead of scheduling on a ``ET_NET`` thread, the plugin wants to schedule
on a ``ET_TASK`` thread (i.e. :arg:`tp` contains ``ET_TASK`` now), all other conditions stays the same.

This time since the type of the desired thread for scheduling and thread "A" are the same, the system
schedules "contp" on thread "A", and assigns thread "A" as the affinity thread for "contp".
schedules :arg:`contp` on thread "A", and assigns thread "A" as the affinity thread for :arg:`contp`.

The reason behind this choice is that we are trying to keep things simple such that lock contention
problems happens less. And for the most part, there is no point of scheduling the same thing on several
Expand All @@ -114,14 +114,14 @@ serialized since its on the same continuation).
Scenario 3 (has thread affinity info, different types of threads)
-----------------------------------------------------------------

Slight variation of scenario 1, thread affinity is set for continuation "contp" to thread "A", all other
Slight variation of scenario 1, thread affinity is set for continuation :arg:`contp` to thread "A", all other
conditions stays the same.

In this situation, the system sees that the "tp" has "ET_NET", but the type of thread "A" is "ET_TASK".
So even though "contp" has an affinity thread, the system will not use that information since the type is
different, instead it schedules "contp" on the next available "ET_NET" thread provided by a round robin
In this situation, the system sees that the :arg:`tp` has ``ET_NET``, but the type of thread "A" is ``ET_TASK``.
So even though :arg:`contp` has an affinity thread, the system will not use that information since the type is
different, instead it schedules :arg:`contp` on the next available ``ET_NET`` thread provided by a round robin
list, which we will call thread "B". The difference with scenario 1 is that since thread "A" is set to
be the affinity thread for "contp" already, the system will NOT overwrite that information with thread "B".
be the affinity thread for :arg:`contp` already, the system will NOT overwrite that information with thread "B".

Most of the time, a continuation will be scheduled on one type of threads, and rarely gets scheduled on
a different type. But when that happens, we want it to return to the thread it was previously on, so it
Expand All @@ -131,10 +131,10 @@ types and thread pointers.
Scenario 4 (has thread affinity info, same types of threads)
------------------------------------------------------------

Slight variation of scenario 3, the only difference is "tp" now says "ET_TASK".
Slight variation of scenario 3, the only difference is :arg:`tp` now says ``ET_TASK``.

This is the easiest scenario since the type of thread "A" and "tp" are the same, so the system schedules
"contp" on thread "A". And, as discussed, there is really no reason why one may want to schedule
This is the easiest scenario since the type of thread "A" and :arg:`tp` are the same, so the system schedules
:arg:`contp` on thread "A". And, as discussed, there is really no reason why one may want to schedule
the same continuation on two different threads of the same type.

.. note:: In scenario 3 & 4, it doesn't matter which thread the plugin is calling the API from.
Expand All @@ -143,6 +143,9 @@ the same continuation on two different threads of the same type.
See Also
========

:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleOnThread.en`
:doc:`TSContScheduleOnEntirePool.en`
:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleEveryOnThread.en`
:doc:`TSContScheduleEveryOnEntirePool.en`
:doc:`TSLifecycleHookAdd.en`
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,14 @@ another thread this can be problematic to be correctly timed. The return value c
If :arg:`contp` has no thread affinity set, the thread it is now scheduled on will be set
as its thread affinity thread.

Note that the TSContSchedule() family of API shall only be called from an ATS EThread.
Note that the `TSContSchedule` family of API shall only be called from an ATS EThread.
Calling it from raw non-EThreads can result in unpredictable behavior.

See Also
========

:doc:`TSContScheduleOnPool.en`
:doc:`TSContScheduleOnEntirePool.en`
:doc:`TSContScheduleEveryOnPool.en`
:doc:`TSContScheduleEveryOnThread.en`
:doc:`TSContScheduleEveryOnEntirePool.en`
3 changes: 3 additions & 0 deletions include/iocore/eventsystem/EventProcessor.h
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,9 @@ class EventProcessor : public Processor
Event *schedule_every(Continuation *c, ink_hrtime aperiod, EventType event_type = ET_CALL, int callback_event = EVENT_INTERVAL,
void *cookie = nullptr);

std::vector<TSAction> schedule_entire(Continuation *c, ink_hrtime atimeout, ink_hrtime aperiod, EventType event_type = ET_CALL,
int callback_event = EVENT_IMMEDIATE, void *cookie = nullptr);

////////////////////////////////////////////
// reschedule an already scheduled event. //
// may be called directly or called by //
Expand Down
3 changes: 3 additions & 0 deletions include/ts/ts.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
#endif

#include <type_traits>
#include <vector>

#include "tsutil/DbgCtl.h"
#include "ts/apidefs.h"
Expand Down Expand Up @@ -1283,8 +1284,10 @@ void TSContDataSet(TSCont contp, void *data);
void *TSContDataGet(TSCont contp);
TSAction TSContScheduleOnPool(TSCont contp, TSHRTime timeout, TSThreadPool tp);
TSAction TSContScheduleOnThread(TSCont contp, TSHRTime timeout, TSEventThread ethread);
std::vector<TSAction> TSContScheduleOnEntirePool(TSCont contp, TSHRTime timeout, TSThreadPool tp);
TSAction TSContScheduleEveryOnPool(TSCont contp, TSHRTime every /* millisecs */, TSThreadPool tp);
TSAction TSContScheduleEveryOnThread(TSCont contp, TSHRTime every /* millisecs */, TSEventThread ethread);
std::vector<TSAction> TSContScheduleEveryOnEntirePool(TSCont contp, TSHRTime every /* millisecs */, TSThreadPool tp);
TSReturnCode TSContThreadAffinitySet(TSCont contp, TSEventThread ethread);
TSEventThread TSContThreadAffinityGet(TSCont contp);
void TSContThreadAffinityClear(TSCont contp);
Expand Down
Loading