`doc/intro.html#the-callback-smallprint` needs more detailed/improved explanations

[victorel-petrovich]

I found couldn't understand the topic, probably many others will find it the same. Who does understand it (at least partially), please comment.
After clarifying these, I plan to make a PR to improve it. (Or you can do it).

From earlier in the section, we get that an event is a particular kind of change that happens in Notepad++ (things like the active document changing, a file being opened or saved; a character being added, a save point being reached, the document being made dirty, etc.).
Say the focus changes from one tab (file) to another one.
When it happens, I get that the handler function (registered with notification.callback()) is called, or in other words: a notification from the event received, correct?
And the handler starts working.

Due to the nature of Scintilla events, they are by default processed internally slightly differently to Notepad++ events.

Notepad++ events are always processed synchronously, i.e. your event handler finishes before Python Script lets Notepad++ continue.

your event handler finishes before what exactly? ... lets Notepad++ continue with what?
Would it be that Notepad++ waits for hanldler to finish before responding to the user for another event/action? Sort of freezes until then?

Scintilla events are placed in a queue, and your event handlers are called as the queue is asynchronously processed - i.e. the event completes before your event handler is complete (or potentially before your event handler is even called).

What means for an event to "complete" in that sentence? When/how the event "completes"?
Is there a deadline for (a notification from) an event to be processed, so that if deadline passed than the event is considered "gone/past", and the event handler misses it ?

The only visible difference is that if you have a lot of callbacks registered, or your callbacks perform a lot of work, you might receive the event some time after it has actually occurred.

But isn't that always expected? I mean, event happens, then it is noticed. Cause and effect, laws of the universe :) .
Or is it that pythonscript can lag behind that queue with event notifications, which fills up a lot faster than pythonscript can manage. And therefore: the event handler function might receive the notification from event at a noticeably later time than when event (action from user) actually happened?

More questions on the smaller-print there about "As of version 1.0, you can use Editor.callbackSync() [...]" , but they are dependent on clarifying the above first.

[Ekopalypse]

I found it hard to understand, probably many others will find it the same. Who understands the topic, please comment. After clarifying these, I plan to make a PR to improve it. (Or you can do it).

From earlier in the section, we get that an event is a particular kind of change that happens in Notepad++ (things like the active document changing, a file being opened or saved etc.). Say the focus changes from one tab (file) to another one. When it happens, I get that the handler function (registered with notification.callback()) is called, or in other words: a notification from the event received, correct? And the handler starts working.

Yes

Due to the nature of Scintilla events, they are by default processed internally slightly differently to Notepad++ events.
...

As mentioned earlier, Notepad notifications are always synchronous, while for Scintilla you can register either synchronous or asynchronous notifications.
Let's consider the following

import time

def on_buffer_activated(args):
   time.sleep(5)

def on_update_ui(args):
    print(time.time())
    time.sleep(5)
    print(time.time())
    


notepad.callback(on_buffer_activated, [NOTIFICATION.BUFFERACTIVATED])
editor.callback(on_update_ui, [SCINTILLANOTIFICATION.UPDATEUI])

Whenever a buffer is activated, either by switching tabs or loading files etc....
on_buffer_activated is called and you will notice that it hangs for 5 seconds because of the blocking time.sleep call. This is because the callbacks are called synchronously, meaning the caller waits until the function is done before taking the next step. The on_update_ui callback, on the other hand, is asynchronous and does not block the user interface, but you will notice that there is a 5-second delay on each new notification because the events are queued and processed one after the other in there own threads. Does this make sense to you?

[victorel-petrovich]

@Ekopalypse , thanks a lot for the explanation and concrete example!

I found it easier to understand by experimenting separately with notepad vs editor events, here are the 2 scripts and my observations (which I think just rephrase some of what you wrote).

Script 1

console.clear()
import time

starttime=time.time()

def on_buffer_activated(args):
    print("on_buffer_activated")
    print((time.time()-starttime)//1) , 
    print("   ") , 
    time.sleep(4)
    print((time.time()-starttime)//1)

notepad.callback(on_buffer_activated, [NOTIFICATION.BUFFERACTIVATED])

time.sleep(20)

notepad.clearCallbacks()

print("\nExperiment is over.")

During the run, I attempted to switch tabs, select text, click menues etc.
N++ did not respond to any user action until it finishes running the handler for a particular event.

Script 2

console.clear()
import time

starttime=time.time()

def on_update_ui(args):
    print("on_update_ui")
    print((time.time()-starttime)//1) , 
    print("   ") , 
    time.sleep(4)
    print((time.time()-starttime)//1)
    
editor.callback(on_update_ui, [SCINTILLANOTIFICATION.UPDATEUI])

time.sleep(20)

editor.clearCallbacks()

print("\nExperiment is over.")

Again tried many actions (clicks, tab switches etc ). Saw that the editor responds no problem, while the handler also processing events at his own pace, lagging.

In fact, if you tried very many clicks etc during the run, after script finishes, thus callback unregistered, the handler was STILL processing some past events for a while.
Which means clearCallbacks(...) functions only disable the handler for NEW events. Which tracks with there being a queue of notifications.

You mentioned threads: what I get is that, for asynchronous queue, in order to process new events while the handler works, N++ uses other threads, to work concurently.

I'll add these insights to the PR I made for the doc.

[Ekopalypse]

@victorel-petrovich

You mentioned threads: what I get is that, for asynchronous queue, in order to process new events while the handler works, N++ uses other threads, to work concurently.

Strictly speaking, it is the Pythonscript plugin that forces different threads to be started to handle the asynchronous notifications. Npp doesn't know about their existence from the application's point of view, which also means that you have to be careful when changing something that is accessible from different threads.

[victorel-petrovich]

@Ekopalypse
I meant, N++ must be using other threads (not managed by Pythonscript) in order to respond to user actions while Pythonscript is currently busy handling one particular Scintilla event (notification) in the queue.

I'm not sure if it is relevant, but is every created queue specific to the a particular handler function, so that it will house all the notifications specified in editor.callback(...,[...]) ? Or is there a single queue common for all handlers and all event types ?

Notifications on the queue are processed in strict order, non overlapping in time, so I don't know why it's important that different threads are issued for different notifications listed on the queue. (I thought threads are there to make concurrency possible).

you have to be careful when changing something that is accessible from different threads.

Is this about changing some variables (state) that are sort-of "global", persistent from processing one notification to the next one?

Does this have to do with args that can be used in the handler function, so that some of them (for some events) are "writable" ? Or are you talking about something else entirely?
Maybe you could give a little example or reference.

[Ekopalypse]

I'm not sure if it is relevant, but is every created queue specific to the a particular handler ...

From my understanding, there is a queue for synchronous and one for asynchronous notifications and a handler for each.

From a high level point of view this is what happens when registering

a sync notifications
-> Npp calls the beNotified callback of PythonScript(PS) and waits until it completes

in case of an async scenario,
-> Npp calls the beNotified callback and PS queues the notification for later processing.
Informs its thread handling these notifications that an event has occurred and returns.
Npp does other things.
The asynchronous callback handler does its tasks.

From a script point of view

x = 1  # code outside of my_async_callback runs in the main PS thread

def my_sync_callback(args):
	# runs in the main PS thread
	# interacting with e.g. x is safe

def my_async_callback(args):
	# this part runs in the thread which handles ALL async callbacks
	# one after the other
	# changing stuff which has been created in the main thread from here is dangerous
	# with Python and e.g. the ctypes module you can do almost everything to confuse not only Npp.

editor.callbackSync(my_sync_callback, [whatever scinitilla notifcation])
editor.callback(my_async_callback, [whatever scinitilla notifcation])

Is this about changing some variables ...

Yes, if you change something from two different threads, it can lead to unexpected behavior.

[victorel-petrovich]

The high level point of view is good!
I assume "beNotified" is the mechanism via which PS is registered to be notified, thus called, by N++ when events of interest to PS happen in N++. (That's enough for me to know)

So, you're saying there is a single thread handling the queue of asynchronous notifications, not a new thread for every new notification in that queue (as Ithink you were writing in previous replies) ?
Results of script 2 above confirm the single thread version. I was creating clicking events there fast, so , if it were a new thread for every notification, I think we'd see same or almost same start and end time printed below:

on_update_ui
1.0     5.0
on_update_ui
5.0     9.0
on_update_ui
9.0     13.0
on_update_ui
13.0     17.0
on_update_ui
17.0     
Experiment is over.
21.0
on_update_ui
21.0     25.0
on_update_ui
25.0     29.0

What I understood from your explanations is that the danger with asynchronous callbacks is due to N++ not waiting: both N++ and the PS event handler work concurently, from different threads, and they may both access some same variables .

I read today in a N++ community thread (again threads :D ; all of them having access to my same memory state... )
that the asynchronous callback function could modify , for example, some Scintilla variables (ex, set selection range or something like that) with which N++ itself might be working at that time, concurently.

Maybe that's what you meant (or at least some of it !).

[Ekopalypse]

I assume "beNotified" is the mechanism via which PS is registered to be notified, thus called, by N++ when events of interest to PS happen in N++. (That's enough for me to know)

Yes, each plugin must export a set of fixed callbacks, defined by the plugin interface, to be used by Npp.

So, you're saying there is a single thread handling the queue of asynchronous notifications

Yes.

Results of script 2 above confirm the single thread version.

I think it's very good that you don't take it for granted, but test yourself to see if something fits.

Maybe that's what you meant (or at least some of it !).

Yes ... but not only limited to Scintilla interaction. One day you might get the idea to intercept messages sent from Windows to Npp, or you might create a script that itself manipulates a database used by other programs and/or users, and then you should pay attention to which part of your script changes what in memory.

[victorel-petrovich]

Got it! Thank you.
Hope all this will be instructive to other readers here too.

I'll include a short note on threads and potential dangers in the PR I've been writing.

By the way, do you have an idea why my PR here to update the doc has been ignored so far (5 days ) ? How long it's normal to wait?
I expect that if there was another reason, the maintainters will comment on my PR.

[Ekopalypse]

By the way, do you have an idea why my PR here to update the doc has been ignored so far (5 days ) ? How long it's normal to wait?

I don't know, maybe @chcg is on vacation or taking time off from PS.
Unfortunately, reviewing the modified documentation is also very time consuming.
Please don't get me wrong, I think your dedication is good, but unfortunately reality dictates how much time you can invest in such hobby projects. If I find some time soon, I can take a look at the changes, but I can't and won't promise anything.

[victorel-petrovich]

@Ekopalypse
thanks for the reply. I see; ok, no pressure on anyone. I won't delete that PR.

Sometimes, someone appears "out of the blue" (like me here ;) ) and is inspired to contribute, and-- who knows for how long he/she will hang around? In Romanian we have a proverb like "beat the iron while it's hot" ...

[Ekopalypse]

Yes, we have the same/similar saying in Germany: "You have to strike while the iron is hot" :-)