Add support for modifier keys to input mappings

[Cervator]

At present all our input mappings are single key only - no ALT-C or anything of the sort. We need to change that!

The actual change popup for when you're modifying a mapping is pretty nice already, but it locks in the first key you enter. Either we need to add logic to accept but not finish if the player clicks a modifier key (CTRL, ALT, SHIFT) or maybe add checkboxes so the modifier keys can be included.

Support should also be added so defaults can be set that include modifier keys. See InventoryButton for an example

Slightly related: #2552, #2384, #2145, #2125

[ThompsonTyler]

Hey, currently planning on sending in an application for GSOC and wanted to see if I could address an issue along with my introduction to the forums. This issue seems interesting enough and after looking into the code and with some interaction so far seems to point towards the code currently being setup based on the only possibility of 1 key attached to an input and seems that the way to implement this would be to overhaul the input to allow for monitoring on 2 keys (or setting up the binding with 2 inputs), making sure to not break any of the implementation based on 1 input.

Any ideas before I try to see what I can do?

Currently have a checkbox with the bind ui added (To allow for just control for crouch to still work) and when this is selected, to detect if a modifier key and which one is pressed, storing it in the UIInputBind at the moment.

[Cervator]

Sounds like you've both picked a good issue to try and already have a nice grasp of the related code! I don't have a lot of feedback or ideas above what's already in the issues, although I'm always up for a bit of brainstorming on IRC, as are others. Depends on who is around, time zones and all. You could also start a PR for further discussion there, just mark it as not for merging yet until happy with it :-)

@rzats probably would have some ideas since he'd done a lot of UI stuff

[ThompsonTyler]

Did a lot more research into the code today and at least at my level of programming and knowledge of the code base, I can't really think of a way to implement this with out completely re-hauling the inputs as currently they are fairly hard-coded to key presses(so can't trick it into 2 key presses) and all the bindings and what not being very reliant on those single key presses. Basically to my knowledge the entire system is very very dependent on an input being 1 key press. I am able to have the binding UI allow for modifier + non-modifier, but can't go any further into the structure with it. And had some implementation with an altered input that would allow for binds, but ran out of luck when trying to actually detect an input or apply the altered input to a binding(that then couldn't be checked anyways, due to the problem with 1 key only inputs that I addressed)

This could definitely be done, but at least in my knowledge would definitely not be seen as "bite-size".

I might still be interested in trying to solve this problem, but I was wondering if anyone else can think of any other ideas that wouldn't require an entire input re-haul. I was going to stop by the irc sometime in the future and see if any kind of brainstorming could result towards anything.
Sorry about the kind of just open message, but don't know the community much at all yet to know of who to even look towards about conversing about this topic.

[Cervator]

Thank you for the research @ThompsonTyler - I've removed the Bite-sized label as it does sound like this would take a fair bit more work. Much appreciated :-)

This communication is fine! As is IRC. Yeah we'll find something there. Pop on any time you can and see when somebody is around, I'm about to head off for the day myself now.

[ThompsonTyler]

Just wanted to send an update, I think I have figured out at least one method of approach to implementing this, going to try about implementing it and test to see if it results in breaking everything or not to see if it is even a viable approach(The classes that are touched seems to build up very quickly). Meaning a lot of testing afterwards will probably be required.

I did have one question though, will look for you/rzats in IRC in the meantime, but how would modifier keys be implemented interface wise? As in if control/shift/alt are modifier keys, would using just that key be a "viable" bind? (Current crouch default bind is just control, fast move default bind of just shift) or would each bind require to be (if looking at keys) either a non-modifier key or a non-modifier key + modifier(or possible multiple modifiers).

And if you had something bound to w, and something to control + w, if control + w is pressed should both w and control + w trigger or just the control + w binding?

[ThompsonTyler]

Well, I think I have at least currently been beaten by the modifier keys. But, I will add my new-found knowledge about a good portion of the Input system as found to allow further documentation for anyone who choses to follow up on this issue. Initial thoughts is that implementing the modifier keys would require a large amount of code introduction due to everything being very based upon 1 input(be it a key, a mouse value, a controller value, etc). So, to add checking to multiple keys at once would probably require a large change to the entire input structure. If someone can think of any other way of doing this, I'm all ears and would be glad to talk about it. (And let you know if I already tried that implementation, I tried a few different methods and all seemed to end to dead-ends regarding a quicker fix vs a full re-implementation)

With out further ado, the documentation I have come up with regarding inputs pertaining mostly upon keyboard strokes :)

Will be adding to this as I have time.

Files related to Input

Files related to binding:

• UIInputBind.java - The class for the logic for setting a bind in the input settings UI, prior to "ok"-ing that binding. Important for grabbing the actual strokes being made to set the binding.
• ChangeBindingPopup.java - Class for the UI for setting a bind in input settings. Will be important if you want to change the UI to allow a modifier selection or take an approach along these lines.
• changeBindingPopup.ui - The UI file for the binding change. If you change ChangeBindingPopup.java and want a UI change you will probably need to edit this file as well.
• BindsConfig.java - Data structure for the "bindings" storing a map of URI to Input. Allows for checking the bindings available in either direction(URIs connected to an input, or all inputs connected to an URI). This file also handles interpretting defaults and binds setup in external files. For example:
  ChatButton.java

@RegisterBindButton(id = "chat", description = "${engine:menu#binding-chat}", category = "general")

@DefaultBinding(type = InputType.KEY, id = Keyboard.KeyId.T)

Files related to input keys:

• Input.java - Overall interface for inputs, all inputs(keys, mouse, controllers, etc) are all implementations of an input.
• InputType.java - Enumeration for the types of an input in order to retrieve instances of those inputs. Allows retrieval of inputs through the id or the string name representing the input based upon the type.
• Keyboard.java - Class that actually implements the Key type for Input. Sets up the functions declared in the Input interface and provides the constants that are referenced across other classes.
• InputSystem.java - The actual system that processes input events across all input types and checks them based on the stored bindings and directs them towards BindableButtons. Allows for access to finding and setting the inputs attached to a button utilizing the URI connected to the BindableButtonImpl
• BindableButtonImpl.java - Implementation of BindableButton. A representation of a button that allows for many different inputs to be tied to the same button and be solely based upon "input" instead of where it came from (mouse, controller, keyboard)

My attempts at solving modifier keys:

• Trying to setup an altered "Input" class that would allow for 2 Inputs to be tracked, failed due to inability to discern needing both inputs and just 2 different inputs bound to an output.
• Having the inputSystem "track" key events for modifier keys and altering the passed id's of the passed inputs (if control is held, add 500 to the input, 1000 if shift, etc). This worked to be able to detect keystrokes of multiple keys, but couldn't be matched to a binding, because bindings are setup with Input's in mind, not the sole integer value for the Id that inputSystem utilizes for keys.
• Implementing a new UI for the changing binding UI that allow for chosing if a modifier was added or not, this would either allow making all new bindings have a modifier attached visually or only popup on actually using the key(actually pressing alt + w), but this also only worked visually. Could not find a way to pass the modifier aspect on any further.
• Having binds be attached to lists of inputs instead of just inputs, in order to "satisfy" a bind, all keys in the list must be satisfied and not just one input attached to the bind. This had problems with tracking all the keys(to see what was currently pressed) needing a large amount of data storage, becoming very messy very quickly, and also no ability to disregard a binding attached to say W if control + W were pressed (which only should trigger control + w binding, not control + w and w bindings)
• Altering the Id's of an Input to allow a temporary bypass around the middle Input structure to connect bindings on ints to the binding changing UI. This failed due to many systems related to input utilizing the constants and lookups on ids/key values (which are now set incorrectly and causes many problems to arise, or just straightup not being able to pass the data through a pipeline that doesn't allow it to pass through)

[Cervator]

Thank you for the extensive research @ThompsonTyler :-)

Sometimes these items turn out to not be as easy as we would've liked. Sorry about sending you down a rabbit hole that turned out to lead half-way through the planet!

I wonder how other LWJGL-based games handle it code-wise. Maybe LibGDX could provide some inspiration as well.

[OvermindDL1]

I wonder how other LWJGL-based games handle it code-wise. Maybe LibGDX could provide some inspiration as well.

What I did in my couple little LWJGL projects is the same as I do in C++, I make an inputmapper. Specifically the style that I go through is I make a tree, kind of like a file-system, and 'mount' things in it. So I make a node that is for a keyboard for example and set it at "/dev/keyboards/0" and I create a link from an event on it to a 'name' for the game to use, so the game may want something like "/players/0/inputs/open_inventory" and I register a link between those with (let me go grab the code, sec... This is in C++ but it was basically identical in Java:

// Grab a keyboard (can also enumerate them or whatever, great to do for keymap menu's)
// The '0' is the index and can be fairly random depending on how the OS enumerates the keyboards
// You can also access them via UUID, which is unique(ish, some keyboard makers are stupid) per keyboard
Keyboard *keyboard = system.getNode<Keyboard>("/dev/keyboards/0");

// When player presses 'w' it sends the forward event at that path, note this is not some kind of
// globally thrown event, at runtime this is as fast as possible, you have to subscribe specifically as below
system.createEventMapping("/dev/keyboards/0/keychange/w", "/players/0/inputs/forward");

// Subscribe to this event, this is done in my PlayerController component systems
system.subscribeEvent("/players/0/inputs/forward", bind(&this, onForward));

// The callback can be like this as an example
void onForward(Atom64 _eventName, VariantMap &event) {
  float speed = 0.0; // default do nothing if no understandable input
  // A keyboard will always return here a 0 or 1, a gamepad would be a range from 0.0 to 1.0 when you map to a half-axis
  if(Variant *range = event.at(INPUT_VALUE)) speed = range.getFloat(1.0); // 1.0 is the default if the variant cannot become a float
  this.setForwardVelocity(speed * this.forwardSpeedPerSecond);
}

// And event mapping can also mutate/change the event as it is passed through, to make it more useful
// for the receiver to handle for example:
system.createEventMapping("/dev/keyboards/0/keychange/w", "/players/0/inputs/forward", bind(&this, mapKeyboardEventToPlayerInputEvent));

// For combo-keys can register those as normal, just to a bouncer node
// Register the node first:
system.setNode("/players/0/inputs/open_inventory", new InputPropogateOnlyWhenAll(2));
// Pass events to it
system.createEventMapping("/dev/keyboards/0/keychange/shift", "/players/0/inputs/open_inventory/listener");
system.createEventMapping("/dev/keyboards/0/keychange/tab", "/players/0/inputs/open_inventory/listener");
// And listen for the final event:
system.subscribeEvent("/players/0/inputs/open_inventory", bind(&this, onOpenInventory));

I've been using this style for over ten years, not only for input handling but also level object registrations, data registrations, even querying monitor information and more. Getting a node from the system is not 'fast' (multiple map lookups), but you can hold on to and cache that node for future usage, plus most links are made at start-up time anyway, and dynamic should be holding a node pointer regardless so it remains fast. A node is just any class that implements a few specific functions from an abstract base (I like to keep class's very flat) so it is pretty easy to make specific nodes, all tied in with the Variant's that are pretty ubiquitous in my setups (easy to tie into scripts, enumerate possible values, etc, it makes it really easy to introspect anything, since this is not java where you can just reflect in ;-)).

Just an idea. :-)

[ThompsonTyler]

Just wanted to give anyone reading this an update, I have gotten modifier keys working in a very basic function(only one alt key, and haven't done much regarding files/parsing config/etc), Will be working on extending it in the near future and hope for this to be done in the next few days :)

[oniatus]

The Input and Binds system was overhauled in #2948 and (hopefully) it should be more easy to implement modifiers in the new system. will close #2859 for now but whoever wants to continue on this feature can have a look at the implementation and start based on the ideas from @ThompsonTyler who really put some thoughts into it.
Another idea for a continuation would be to keep it a bit more flexible and map multiple keys to one input instead of restricting it to the most common modifiers.

[skaldarnar]

Related to MovingBlocks/TeraNUI#37