Enhancement: Proper touch input support

[ericoporto]

AGS assumes that the player uses a mouse to play the game. In today times, there is a myriad of devices that support touch input that are used quite frequently, famously our mobile ports should be able to run in such devices. As developers mature they would need to be able to cater specifically for the constraints of such devices. The assumption of a mouse that can click can make some control schemes challenging to support in touch first devices.

• I propose we implement either touch specific events or pointer events (mixed touch and mouse) that could be used in the script API in multitouch control schemes.

  • since there is only on_mouse_click but not on_mouse_down, on_mouse_move and on_mouse_release, it may be better to use the pointer concept and create on_pointer_start, on_pointer_move, on_pointer_end to cover mouse and touch on the same elements.
  • html also has an additional on_touch_cancel, need to see if this is needed/possible with sdl2.
  • AGS currently has a limit of 5 events in the queue, it may be needed to allow a bigger queue or some touched points may be missed in a frame.

• Additionally, I would like to propose additional event bindings for the Button GUI Control, so it has additionally a touch down and touch release event (or pointer down, pointer release), and also a property that can be checked (IsTouched / IsPointed)

  • We can postpone doing the GUI button events

These are drawn from my observations of other engines, that I wrote about it here: https://ericonotes.blogspot.com/2020/11/a-quick-look-at-touch-handling-apis-in.html

Such API additions would allow for multitouch in GUIs, which can be used to better handle mobile device usage and respond more quickly in the interface (hitting two fingers at screen today will result in a finger being dismissed), and also this allows support for on screen joysticks.

Forum topic

• https://www.adventuregamestudio.co.uk/forums/index.php?topic=59710.0

On Screen Joystick example

Using AGS GUIs, it should be possible to construct something like below. Right now, it's not possible since using two or more fingers at the same time is not possible.

Note

https://youtu.be/B_IqYy4T_AA?si=xOASxzLCrI0F1lV8&t=916

In this talk, the Broken Sword dev talks about the recent mobile port and it's interesting how their interface got adapted.

[ivan-mogilko]

Just for a quick reference on SDL2 touch control support:

SDL2 touch-related events:
https://wiki.libsdl.org/SDL_TouchFingerEvent
https://wiki.libsdl.org/SDL_MultiGestureEvent
https://wiki.libsdl.org/SDL_DollarGestureEvent

---

There is a couple of hints that control SDL2 behavior when it comes to simulating touch and mouse through each other. Some hints are missing in the SDL2 wiki, but their brief description may be found in the source code.
(https://github.com/libsdl-org/SDL/blob/main/include/SDL_hints.h):

• SDL_HINT_MOUSE_TOUCH_EVENTS

A variable controlling whether mouse events should generate synthetic touch events

https://github.com/libsdl-org/SDL/blob/1fc7f681187f80ccd6b9625214b47db665cd9aaf/include/SDL_hints.h#L1146-L1150

• SDL_HINT_TOUCH_MOUSE_EVENTS

A variable controlling whether touch events should generate synthetic mouse events

https://github.com/libsdl-org/SDL/blob/1fc7f681187f80ccd6b9625214b47db665cd9aaf/include/SDL_hints.h#L1514-L1522

I guess SDL_HINT_TOUCH_MOUSE_EVENTS is the one that matters more for the mobile devices. If AGS has its own proper touch API in script, then this hint should likely be disabled for games with such support, and enabled for games without such support.

When receiving mouse events you can distinguish real mouse from emulated using "event.button.which" parameter, which means mouse ID. For emulated mouse ID = SDL_TOUCH_MOUSEID.

---

The SDL2's implementation of synthetic mouse events may be found in the following code (in latest version):
https://github.com/libsdl-org/SDL/blob/main/src/events/SDL_touch.c
under SYNTHESIZE_TOUCH_TO_MOUSE macro test.
e.g. as of recent commit: https://github.com/libsdl-org/SDL/blob/970344719a958460aadd73342a2b0524981a59f4/src/events/SDL_touch.c#L266

[ericoporto]

Stranga pinged me again about this today, I mentioned I believe this is something for a 3.6.1 release.

[ericoporto]

So in AGS currently the mouse can imply in on_mouse_click only when nothing is blocking, and when something is blocking the mouse goes through the skipping stuff. This is because on_mouse_click (and any event in AGS!) can't run when something is blocking (similar to repeatedly_execute and not repeatedly_execute_always), this means that whatever is the api added, we also need to add touch as a way to skip things in all things that can be skipped (video, cutscene, speech, ...), and similar to mouse click, a touch anywhere on screen will cause it to skip.

[ericoporto]

Trying to come up with a minimalistic approach

builtin struct Pointer {
  /// Number of pointers, this is a fixed amount
  readonly import static attribute int Count;      // $AUTOCOMPLETESTATICONLY$
  /// Takes pointer ID and returns where the pointer is in game screen, (-1,-1) if invalid
  readonly import static attribute Point* Position[];      // $AUTOCOMPLETESTATICONLY$
  /// Takes pointer ID and returns true if the pointer is pressed, the finger is on screen or left mouse button is down
  readonly import static attribute bool IsDown[];       // $AUTOCOMPLETESTATICONLY$
};

Here's how it works

• Reuse the unified finger tracking from the touch to mouse emulation (here)
• Pointer 0 is the mouse and other indexes maps to a finger in order they actively touched the screen. The maximum amount of fingers is limited by MAX_FINGERS internally.
• The state of the Pointer struct doesn't change in the frame, it will be like it gets a picture of all this at frame start, this includes being immutable to Mouse.Update() calls.

Here is it's initial version: https://github.com/ericoporto/ags/tree/experimental-pointer-api

[ivan-mogilko]

I'm concerned about the bare "Pointer" name, this term has many uses in programming. Are there other alternatives to this? If not, perhaps adding something to it may clarify the purpose. A quick example: "PointerDevice".

[ericoporto]

I agree it's a terrible name, could also go with "TouchPoint", "Interaction" or "TouchInput".

I would like to somehow have the mouse input be one of the things there just to make it easier to iterate through testing Game Script code in the AGS Editor.

I also thought about the API being like, say in System.TouchPoint[] and then returning an object that is just data, like Point is, so holding a reference to it isn't a problem. I think I would need to create my own ScriptUserObject like Point - something like ScriptStructHelpers::CreateTouchPoint(int x, int y, int id, bool down), but need to really think through - it looks a bit problematic if I need to add something in it.

[ericoporto]

My first test of the thing

https://ericoporto.github.io/public_html/382d947/

• my iPhone: only two fingers at maximum appears to work
• my Android phone: can throw all five fingers at the screen, it looks like the fingers "roll" from 1 to 10.

It also looks like my screen position calculation is completely wrong.

[ericoporto]

I renamed to touch points (I haven't renamed the files too yet, but will eventually)

managed struct TouchPoint {
	int ID, X, Y;
	bool IsDown;
};

builtin struct Touch {
  /// Number of pointers, this is a fixed amount
  readonly import static attribute int TouchPointCount;      // $AUTOCOMPLETESTATICONLY$
  /// Takes pointer ID and returns where the pointer is in game screen, (-1,-1) if invalid
  readonly import static attribute TouchPoint* TouchPoint[];      // $AUTOCOMPLETESTATICONLY$
};

I still can't figure the screen position calculation. Trying to pickup things from mousew32, because I would want the same position one gets from mousex and mousey globals. Edit: something like ericoporto@20c6bb7

Edit: testing in a few devices and it almost works, except it's not being clamped to the game borders. Edit: ericoporto@7c1febc fixed it!

[ericoporto]

Hey, I would like to try to add multi-touch support to GUIs... but... I can't figure even how they are clicked at all.

https://github.com/adventuregamestudio/ags-template-source/blob/c9f8c8ebde194ad026d095a7e6a30ef6fdd10fe5/Empty%20Game/GlobalScript.asc#L69-L85

There isn't anything for GUIs in Global Script, does it happen through some internals?

I know buttons can be held when clicking, and the event that we normally use happens on release. With multi-touch, I would like to keep that working in multi-touch, and also to add some event it triggers continually while the button is held.

Ah, I think I need to implement some variance of this for multi-touch

ags/Engine/main/game_run.cpp

Line 351 in 466acb5

static void check_mouse_controls(const int was_mouse_on_iface)

[ivan-mogilko]

Right, GUIs are not handled in script at all, this is done purely by polling GUI state: see GUIMain::Poll, where it decides which controls are under mouse, and which events to send (mouse over, pushed, unpushed etc).

EDIT: I did not think about this earlier, but I'd assume that the touches may be seen as extra "mouse" devices. So instead of checking a single mouse device, engine should have a list of devices (or rather list of specific device state, including coordinates and "button" state), and check all of them in a loop.
Similarly, any global variables that mention mouse state, such as "was_mouse_on_iface" should be rewritten to support multiple devices.
IDK how the game should react if there are multiple touches on buttons though. Maybe that should not be supported, and only the first or the last touch should result in a control activation?...

EDIT2: There's another thing, mouse event such as on_mouse_click currently only includes button as an argument, but that's wrong, it should also include at least position saved at the time when event was registered. That's important to have on its own (as mouse may be moving between updates), but even more important with multiple "devices" that may be pressed during same update in different positions.

In other words, it should be:

on_mouse_click(MouseButton button, int mx, int my)
on_touch_down(? touch_id, int tx, int ty)

or similar.

[ericoporto]

Uhm... I made my TouchPoint like this

managed struct TouchPoint {
	int ID, X, Y;
	bool IsDown;
};

Perhaps if the bool IsDown is instead an enum MouseButton ButtonDown that is always either none or left click for fingers but can take other buttons it could work in a generical way.

But yeah, the multiple mouse devices approach, even if only internally to ags could work well. Game maker works in this manner.

Because of the nature of AGS being a Point-and-Click engine the mouse is integral to it's behavior and affects a lot of things. We do a lot of global assumptions around having a single mouse device.

IDK how the game should react if there are multiple touches on buttons though. Maybe that should not be supported, and only the first or the last touch should result in a control activation?...

Yeah, I think so, the first touch in the gui control marks it as being pressed down and only once the last touch point on top of it released the gui control is released.

see GUIMain::Poll, where it decides which controls are under mouse, and which events to send (mouse over, pushed, unpushed etc).

AH, that is the place. In game_run.cpp we call it passing the global mousex and mousey positions. For now I think my approach is to still do that and additionally call it repeatedly with a for that would go through all the fingers that are down.

I think when polling it may not be necessary to tell the ID of the finger, but instead tell it's in the same frame? Meaning, that presses in the same frame could only affect the control state once. I still need to think a little more on this.

---

Edit: it looks like ID of the finger would only be useful to skip processing of the finger that hasn't moved.

Also looking more at the code it looks like the highlighting (HighlightCtrl) would still be kept at one control only. There is a focus (FocusCtrl) thing, but it looks like it's not actually implemented currently.

[ericoporto]

Uhm, there is a behavior that makes sense for Mouse but doesn't make sense for Touch, that kinda tells me we want to have different Polling for this. It's this click, hold that locks the button in down state.

Now in a touch environment, this doesn't happen, the button gets released as soon as the finger is not on top of it, but with mouses this is not expected. I think this signals that we would have two Polling, one for mouse and other for touch.

Going back to the script api, cirrus-ci.com/build/5713709612400640 | ericoporto@ags/experimental-touch-api

[ivan-mogilko]

Now in a touch environment, this doesn't happen, the button gets released as soon as the finger is not on top of it, but with mouses this is not expected. I think this signals that we would have two Polling, one for mouse and other for touch.

Different behavior is better done with either a flag that tells how the device should act, or virtual function(s) overridden in a device class. Having multiple polling loops will complicate code organization (and potentially there may be other differences found in future).

[ivan-mogilko]

Had to refresh my memories on this topic, so re-read everything in this ticket, and also in the linked blog post and related docs from few other engines (Unity and Mozilla).

I come at this setting following questions:

1. Which concepts or objects do we need to have represented in touch API?
2. What should be the relation between mouse support and touch API?
3. How much can be supported when using SDL?

After reading Unity docs about their Touch struct, I guess I understand this concept of a Touch as a gesture performed by a finger which lifetime spans from the moment of finger's touch first registered, and to the moment when it is unregistered, which (from what I understood) comes next frame after being "unpressed".
This "gesture object" has a unique "touch id", and a state which goes like "begin" (just pressed), "end" (just unpressed), "moved", "stayed" and "cancelled".

Since it's a continuous object that may exist for multiple game frame, there's an option of actually returning a persistent object in script. I can't tell whether Unity of other engines do that, but it's an alternative to creating TouchPoint each time user wants to check same touch instance for a duration of multiple frames.
If done so, the struct also would have to be "detached" in a similar style to Overlays and Joystick. The stage property could serve as a lifetime end check, as we know that the same touch instance won't resume after "end" stage.

Then, the most modern Unity Input API has a distinction between fingers and touches:
https://docs.unity3d.com/Packages/com.unity.inputsystem@1.13/manual/Touch.html
In simple words, the Finger there is a fixed Nth pointer which has always same fixed ID, and can be active (touching) or not active.
While the Touch is a current gesture of a Finger.
There's an interesting reason (besides other things) why these two objects have to be kept apart. Some touches may be too short to stay until the next game or script update. If we have finger and touch merged in a single object, then user will never know about that short touch. If they are separate, then we may have a fixed-sized list of fingers, and a dynamic list of touches, where some of these touches might have already ended, but will stay for 1 update frame to let user learn about them in script (like a history of everything happened between frames).

In the end, it seems, we have 3 potential concepts:

1. A Finger, which is something that may create touches. Finger is a fixed object that exists so long as a touch device is connected to the system. The list of Fingers is always fixed-sized, and accessing Finger of same index will always give same exact object.
2. A Touch as a continuous gesture, which is a instance of a Finger touching the screen. It has to have a unique id within all active touches, but these ids may be reused after an active touch is completed. It has current stage that tells its state, and may optionally have a history of all its changes. The list of Touches is dynamic, varying in length, and contains only the gestures that were active within the last frame update (or have just ended).
3. A Touch as a event, which describes a single change in touch, identified by a touch id (unique among all active touches). The list of touch event is a history of changes since the touching started.

We don't have to implement all of these of course.

---

About mixing a mouse device in. Previously there was a suggestion in this thread to replace "IsDown" property with "MouseButton", but that would prevent having stage property in TouchPoint. Another option that comes to mind is to use mouse button ids as "touch ids". That would reserve few first unique ids for the mouse buttons. But also there could be just a separate MouseButton property in a TouchPoint, which would be None for non-mouse "touches".
BTW I've been wondering if it's possible to support multiple mice connected to a system. SDL2 seems to potentially support that, since its mouse events have "device id" parameter. A quick search shows that Unity does not support multiple mouse devices. Idk about other game engines. I guess that's not a popular demand. But for the reference, if we'd ever like to support multiple mice, then having buttons as reserved touch ids won't do as well, in which case it seems that only separate property is suitable.

---

To summarize, we need user to be able to achieve at least two things:

1. Track current state of a pointer and/or current pointer's gesture (each of them).
2. Know the history of changes of a touch state even if multiple changes occured within a single frame (even if it's a pointer not touching anymore).

Now, the history may be recorded by a user in script, if we support pointer down/up/move callbacks. So having it actually present in the Touch struct is optional (we may live without it for starters).
I also suppose that we do not need to split pointer and gesture into 2 separate structs. We might later if that would be necessary, but I am not educated in touch controls enough to tell whether that would be useful beforehand.

Which brings us to the remaining question of a touch state. Supposing we merge "pointer" and "gesture" and have a struct that has a meaning of "pointer touching state".
There's still a question of whether this should be a persistent "pointer" struct, in a fixed-sized list of pointer states, which always exist, or a "pointer's gesture" struct, which lifetime is only while the pointer is active, and then these structs are contained in a dynamically resized list.
In the first case, their indexes will be fixed, and the user will be able to keep checking "finger" states using the same sequential index.
In the second case, user will have to iterate the list, and check "finger id" in each struct in order to match them with the previously done records.

In any case such "TouchPoint" struct should contain:

• finger/pointer id;
• touch stage: begin, end, moved, stayed, cancelled (maybe not all of those if SDL does not let us learn some);
• position on screen;
• optionally, mouse button corresponding to this touch, or None.

In both cases we may decide to not allocate this managed object each time a user requested it, but allocate it when it's first asked.
In case the struct corresponds not to a fixed pointer but to a single gesture, then we also must detach this managed object from the engine whenever it becomes obsolete, just like we do with Overlays and disconnected Joysticks. I suppose that detaching will be done by setting invalid "finger id" and "stage" properties.

[ericoporto]

The touch ID is different between platforms when looking directly from SDL, because it is transparent to the platforms, there is a layer in ags that abstracts this currently. (Unique, but either always increments or uses the first available)

[ivan-mogilko]

The touch ID is different between platforms when looking directly from SDL, because it is transparent to the platforms, there is a layer in ags that abstracts this currently. (Unique, but either always increments or uses the first available)

It makes sense to be platform agnostic. But I am also concerned about proper understanding of what is the meaning of these IDs.

I looked into which IDs does SDL provides, there's something that confused me at first. It has "touchID" and "fingerID" (both SDL2 and SDL3). It appears that "touchID" is not really a "single touch id", but a "touch device id" (like a touchscreen, I suppose). While "fingerID" is actually not the index of Nth finger touching, but an arbitrary id of a "touch action":

https://wiki.libsdl.org/SDL3/SDL_TouchID

This ID is valid for the time the device is connected to the system, and is never reused for the lifetime of the application.

https://wiki.libsdl.org/SDL3/SDL_FingerID

This ID is valid for the time the finger (stylus, etc) is touching and will be unique for all fingers currently in contact, so this ID tracks the lifetime of a single continuous touch. This value may represent an index, a pointer, or some other unique ID, depending on the platform.

So it sais that it may or not match the index of a finger, so it cannot be relied on as something sequential.

[ivan-mogilko]

Alright, so, revisiting the existing engine code again, where it converts from SDL finger id to our finger id... Our ags "finger id" is basically Nth "finger" pressing, it's 0-based and sequential; and if a "finger" in the middle of a sequence was released, then there are gaps that are filled by newly pressed fingers.

The same finger id could be used in the "touch point" struct, be an index in array of touch points, and passed as an argument into "pointer" callbacks, binding them all together.

I suppose that this lets the TouchPoint work as a "touching finger state", telling what the Nth finger is doing until its released.
If we need more sophisticated record for the gesture history in the future, perhaps we could add more structs, like TouchState or GestureState, and so forth, connected to a TouchPoint through the finger id.

Then, I'd suggest to modify the API in the draft PR to something closer to examples shown in this issue thread (in past comments above), where the Touch struct has static indexed property instead of returning dynamic array. And also have TouchPoint struct use properties instead of bare fields, as that will make it easier to maintain and expand on need.

struct Touch {
  /// Number of pointers, may increase as more of them register in game, but never decreases
  readonly import static attribute int TouchPointCount;      // $AUTOCOMPLETESTATICONLY$
  /// Takes pointer ID and returns the pointer state
  readonly import static attribute TouchPoint* TouchPoint[];      // $AUTOCOMPLETESTATICONLY$
}

But since there may be gaps, the good question is what does TouchPointCount returns and what happens when user requests a non-touching TouchPointer in the "gap".
Saying that it's "not down" may not be enough, because we might want to let user detect when a previously touching pointer got unpressed, at least for 1 final frame.
Either such element could be returned as null, or if we use the "touch phase" concept, then there could be another phase meaning "not active" (not doing any touches last frame). Nulls are annoying since users tend to forget to check for them, and the existing array makes an impression of something one can iterate safely up to Count at all times. So probably it should be a property telling that?

Then the minimal TouchPoint could be like:

managed struct TouchPoint {
    import readonly attribute int PointerID; // or just ID, since it's own id
    import readonly attribute bool IsActive; // ???
    import readonly attribute bool IsDown;
    import readonly attribute int X;
    import readonly attribute int Y;
};

I'm bit conflicted on whether we need "touch phase" thing as an enum, or whether it will be possible or useful.
It may be convenient to quickly tell whether the pointer moved, without testing and comparing position. It may be also convenient to know that the pointer is just unpressed.
However, in our system, supposing one finger may be released and then quickly pressed again during same frame, makes it impossible to distinguish these intermediate states without processing pointer callbacks at the same time. So user will have to do this work anyway... unless we present a more sophisticated api with recorded gesture history.

EDIT: hmm, in such case I am not certain about mentioned IsActive property too. Maybe better to leave it out for the time being.

---

The accompanying callbacks should be like:

void on_pointer_start(int pointer_id, int x, int y);
void on_pointer_move(int pointer_id, int x, int y);
void on_pointer_end(int pointer_id, int x, int y);

---

What if want to have the mouse processed by the same system?
I suppose that in such case we will need to record each separately pressed mouse button as another "touching pointer", and add MouseButton MouseButton property to the TouchPoint struct.