Keyboard layout for shortcuts

[RastislavKish]

What is the problem or limitation you are having?

I was not sure whether to classify this as a feature request or a bug, I think it depends on the viewpoint.

Basically, right now, if I assign a hotkey for a command, say Key.MOD_1+"1", what I actually need to press to get it triggered is Ctrl+Shift+1, because the Slovak keyboard layout I'm using has the character "+" where the English keyboard has 1, and Shift+"+" produces 1 on our keyboard.

I would guess that Key.MOD_1+Key.EXCLAMATION yields a similar behavior on the English layout, i.e. you need to press Ctrl+Shift+1. Which sort of makes sense in this particular case, but normally, I would rather define this like Key.MOD_1+Key.SHIFT+"1" (it seems there is actually no value for 1 on the Key enum, I do see Key.NUMPAD_1 there but the number row is only represented by the special symbols, so this is what works instead).

Does Toga have a guidelined way of approaching this?

Describe the solution you'd like

For me personally, it makes more sense to define shortcuts through hardware keys, because various countries have very differing stickers on their keyboards, but actually very similar hardware layouts, so when I'm designing a keyboard-centric app, I can select shortcuts based on what's physically most convenient to press and makes most sense, so when my users install my app, most of them will have the fine-tuned logical control system I designed for them out of the box.

While, if I used localized shortcuts, they could be giving perfect sense on my keyboard, but be absolutely out of reah for many other countries, or not possible to press at all (like Ctrl+Ľ or Ctrl+Ô on an English keyboard).

Describe alternatives you've considered

_

Additional context

Environment

• Operating system: Ubuntu Mate 22.04 64-bit
• Toga: 0.4.3.dev275+g5badd9083

[freakboy3742]

Agreed this is a difficult one - it falls in the unfortunate gap between bug (because there are incorrect/misleading/unreachable key combinations), and feature (because fixing it requires a localisation framework of some kind).

Let me say from the outset that I definitely want to fix this. The lack of support for non-US keyboards is entirely a function of my own cultural ignorance, not an intentional design choice. I want Toga to have good support for internationalization, localisation and accessibility (i18n, l10n and a11y) out of the box. However, the tyranny of being a English speaking, non-impaired developer means my practical experience with these features is often lacking; and in an attempt to get something working, i18n, l10 and a11y concerns will often get neglected. The assistance and experience of non-English-native developers like yourself is invaluable when it comes to addressing i18n and l10n issues.

You've mentioned that keyboard shortcuts should be defined through hardware keys. Conceptually, I can see what you're saying, and agree; the complication is that we need to be cross platform, and there isn't a cross-platform representation for hardware keys - and some platforms don't seem be entirely compatible with this concept.

Toga's key definitions essentially are "hardware" representations, although they're defined in text form in an enumeration - toga.Key.EXCLAMATION and toga.Key._1 are different key definitions, defined as "!" and "1" constants internally, even though they're the same physical key on a US keyboard. In the backend, these are then mapped to native platform representations.

The problem we hit is that this mapping is both platform and locale dependent - especially when it comes to "shifted" keys on some platforms.

GTK seems to do this reasonably well - there's a separate GDK key code for Gdk.KEY_exclam and Gdk.KEY_1, and <shift> exists as a separate modifier (one that Toga only adds when the key is an alphabetical key - so A is mapped to <shift>a). We're currently missing mappings for keys like Ľ and Ô that don't exist on a US keyboard, but it should be reasonably straightforward to add these missing entries.

However, on macOS, the mapping is tied to physical keys - so the "1" key is "key 18", and it's up to Toga to map from "key 18 + shift" to toga.Key.EXCLAMATION. As you've noted, this appears to be locale dependent, so it doesn't work non-US keyboards. I don't know if there's a better (or locale dependent) way to handle these key combinations - if the entire keycode is different (so the +/1 key is a different key code on Slovak keyboards), or if there's a different mapping required, or if there's a better API we could be using for defining menu shortcuts.

The situation on Windows is much the same, except even more confusing because the Winforms Key enum has some... very weird design decisions - key names seem somewhat arbitrary in some cases, and I haven't seen any way to represent locale-specific keys like Ľ and Ô.

I'm open to suggestions. As I said at the start - the lack of support for non-US keyboards wasn't an intentional choice, it's purely a function of not having access to non-US keyboard to develop/test with. Any and all assistance on this issue would be most welcome.

[RastislavKish]

Thanks for the quick and detailed response. There is no need to worry, feedback is what community is here for. :)

Also, I wouldn't view keyboard-related stuff as necessarily a pure a11y matter, indeed keyboard control is an important feature for people who can not use the mouse, but also for powerusers who want to use their programs efficiently, Vim with its community is albeit a bit extreme, but great example of the power a good keyboard input can provide. This is the viewpoint I would be primarily considering when designing keyboard shortcuts system specifically, i.e. powerusers, because the aimed benefit is speed and convenience, increased a11y is just a nice benefit on top.

Perhaps it would be good to clarify the terminology a bit, I feel we're using the same terms but actually referring to the opposite ends of the problem:

Toga's key definitions essentially are "hardware" representations, although they're defined in text form in an enumeration - toga.Key.EXCLAMATION and toga.Key._1 are different key definitions, defined as "!" and "1" constants internally, even though they're the same physical key on a US keyboard.

By hardware representation, or hardware key, I imagine a locale-independend identifier of a physical key on a "Generic 105 key PC" keyboard. However, with this definition, the Key enum shouldn't contain EXCLAMATION and _1 at once (I didn't in fact notice the _1 exists because help(toga.Key) doesn't list it likely due to the _ being interpreted as access indicator, thanks for pointing this out).

I would imagine something like the X keycodes, note for sake of convenience, the symbols of the English keyboard layout are often used for labelling the individual keys so they can be referred to in a convenient way, this is used by the win32 libraries, Android keyboard input, WPF, ...

And, what I'm suggesting is that these hardware keycodes would be used for setting shortcuts by the programmer. Because, to give an example, right now, I'm creating a program, where the user is expected to quickly place timemarks of different categories while doing other things by pressing Alt+1,, Alt+2, ... (the number reprresents a category)
But if I use Key.MOD_2+Key._1, what I actually need to press is Alt+Shift+1, what's not the intended behavior, my intention as a programmer was to give the user a shortcut they could press with two fingers without shifting their hands and leaving the writer position too much, now they need 3 fingers and both hands to perform this operation, which is inefficient.

Basically, what I'm trying to say is that keyboard shortcuts are primarily about physiology of hand movements and maximizing their efficiency, therefore, it makes much more sense to define them based on the keyboard geometry (which is mostly generic) rather than layout (which is language dependent).

The only aspect of shortcuts related to the user's language I can think of is memorability, i.e. if the programmer wanted to have a shortcut for say "Go to page" function, they may choose Ctrl+P to make it easier to remember, in such case, a Slovak variant could be Ctrl+S (like strana - page). But this is something that's very rarely done, and even if it is, I believe it should be a matter of localisation stack rather than the GUI one.

These are basically my thoughts on approaching this. I believe it's the method used by most of GUI frameworks I've worked with, albeit some like GTK in particular do have quite confusing systems, but I think most of them should offer a reasonably low-level key representation.

[freakboy3742]

Also, I wouldn't view keyboard-related stuff as necessarily a pure a11y matter, ...

Oh, sure - and accessibility doesn't end with keyboard shortcuts. All I was saying is that Toga tries to do the right thing for accessibility as a general policy. We sometimes (often?) fall short, but we want to do better.

Perhaps it would be good to clarify the terminology a bit, I feel we're using the same terms but actually referring to the opposite ends of the problem:

Toga's key definitions essentially are "hardware" representations, although they're defined in text form in an enumeration - toga.Key.EXCLAMATION and toga.Key._1 are different key definitions, defined as "!" and "1" constants internally, even though they're the same physical key on a US keyboard.

By hardware representation, or hardware key, I imagine a locale-independend identifier of a physical key on a "Generic 105 key PC" keyboard. However, with this definition, the Key enum shouldn't contain EXCLAMATION and _1 at once (I didn't in fact notice the _1 exists because help(toga.Key) doesn't list it likely due to the _ being interpreted as access indicator, thanks for pointing this out).

But that level of representation doesn't help - at least, not without another layer telling you what those physical keys mean. Knowing "the key three from the left on the top row was pressed" isn't a helpful identifier unless I know what that key represents. That's always going to be a symbol of some kind, or something non-printable like "delete", "enter", or a modifier.

And that's before you've gotten into what "three from the top left" even means if you've got a Dvorak-mapped 42 key Corne keyboard :-)

Knowing "the user pressed 1" or "the user pressed !" is a useful representation, and one that is tied to the hardware, because it represents what the user actually typed, independent of whether they needed to press shift or any other modifier to get that key; and if a shift or modifier was involved, it isn't dependent on what the unshifted or unmodified key was. In this representation, the only thing that "shift" is useful for is for differentiating "A" from "a".

These are basically my thoughts on approaching this. I believe it's the method used by most of GUI frameworks I've worked with, albeit some like GTK in particular do have quite confusing systems, but I think most of them should offer a reasonably low-level key representation.

I think you're convolving two different (but equally important) concepts in your analysis.

The first is the key itself. "As a developer, I want to bind a key shortcut to the CTRL-'+' key combination.". This is keyboard dependent, and has a locale component. for a Slovak keyboard, "+" is an unshifted key on the top left of the keyboard; for me, it's a shifted "=" on the top right of the keyboard. As a developer, I don't particularly care where the key is on the keyboard - I just want the "+" symbol to activate it. It's Toga's job to work out where that key is, and how to bind to it, in a locale independent way.

The second is locale-dependent shortcuts - "As a developer, I want the 'go to page' command to be bound to CTRL+p for English, CTRL-s for Slovak, ...". This is an important l10n problem to solve - but it also requires a solution the first, as you can't use "ctrl-s" unless you can reliably describe the "s" key. The shortcut choices could reflect physical keyboard geometry, language, or anything else.

Interestingly, the second problem is a variant on one that Toga already handles - platform-specific shortcuts. For example, does the shortcut to quit the app involve the Q key (for Quit), the E key (for Exit), or the X key (for eXit)? This is the same underlying command, but different platform style guides dictate different shortcuts. We're able to make a split here by defining app commands at the backend level, but that doesn't help users who want to define "system" commands; and it doesn't help the l10n case where "Q" isn't the right shortcut in the current language (I don't know if Slovak apps use "Q/E/X" for quitting apps, but hopefully the point here is clear).

[RastislavKish]

Knowing "the user pressed 1" or "the user pressed !" is a useful representation, and one that is tied to the hardware, because it represents what the user actually typed,

I would say this depends on the application. If the purpose of capturing keys is typing, let's say I would be programming my own TextInput component, then yes, I do need to know the symbol the user wanted to put in the field, because that's what they want to see there.

However, if I'm introducing keyboard shortcuts, my interest is not the symbol on the physical key, but it's location. For example, Vim had mapped the next line command to the J key, because it's on the home finger position, under the strongest finger of right-handed people, therefore easy to press. The symbol is not as much of a deal in this case, it's about the location. Indeed when you can, you select keys with symbols that evoke the functionality in some way to make them rememberable, but the location, number of keys to press, how spread apart they are always gets priority.

As a developer, I don't particularly care where the key is on the keyboard - I just want the "+" symbol to activate it.

For me it's the exact opposite, I don't care what letter is printed on the key I want to use for a shortcut, I just want it's position to be the same as mine, to make sure it will be convenient and comfortable to use. Of course, there will always be physical keyboards that won't satisfy the requirements, there are plenty of them that don't contain some of the modifiers, lack the functional row, don't have special keys like Home, End, but then, such a keyboard is likely not the right tool for powerusing my application and the users who would like to do so likely know that very well, they can use the menus or action icons if they don't have a suitable hardware.

It seems we're talking about two different approaches of choosing shortcuts, a location based and meaning based. Perhaps Toga could implement geometrical shortcuts and textual shortcuts, which would technically be the same, just the textual shortcut would at runtime go through a reverse-translation process to find out what is it's geometrical position in respect to the currently active layout, what needs to be done anyway.

Note the names are what just came on my mind.

[freakboy3742]

Knowing "the user pressed 1" or "the user pressed !" is a useful representation, and one that is tied to the hardware, because it represents what the user actually typed,

I would say this depends on the application.
...
However, if I'm introducing keyboard shortcuts, my interest is not the symbol on the physical key, but it's location.

It may be your interest - but I strongly challenge the assertion that this is the common case. I would argue that the vast majority of app keyboard shortcuts are assigned based on, in order:

1. How can the user remember this?
2. Does the ease with which it can be typed match the frequency with which it needs to be used?

Taking macOS in an English locale as an example - Save is CMD-S, Quit is CMD-Q, Hide is CMD-H, Minimize is CMD-M. "Preferences" might seem like it should be CMD-P, but it's CMD-, because Print is more significant as a "P" command. The position of these keys has no real bearing on anything. The shortcuts have their names because of the command names, followed by the availability of keys.

An app like vim is such an extreme case that it really shouldn't really be used as a primary exemplar here. vim is an app that comes from an era where mice - and even keyboards with arrow keys - effectively didn't exist. "Microsoft Word" is a better exemplar for the types of behavior we're enabling here. It's certainly possible to build vim with Toga... but that's not the use case we should be optimising for.

The other thing to keep in mind is how this will be implemented. How are you going to label the "sixth key from the left on the second row"? That's Y for me, but Z for you. What meaningful name/label can you apply as a constant in code?

More importantly - how do the underlying GUI toolkits represent things? Every single one of them has a constant for "!". None of them have a constant for "the key at the top left of the keyboard".

It seems we're talking about two different approaches of choosing shortcuts, a location based and meaning based.
...
Perhaps Toga could implement geometrical shortcuts and textual shortcuts, which would technically be the same, just the textual shortcut would at runtime go through a reverse-translation process to find out what is it's geometrical position in respect to the currently active layout, what needs to be done anyway.

I agree a translation layer would be possible; however, given that the underlying platform implementations are universally textual/symbol based, it would seem to make a lot more sense to map geometry to symbol than the other way around.

[HalfWhitt]

One possible counterpoint is that while copy is Ctrl/Cmd C, cut and paste are X and V. I don't think those mean anything beyond the fact that they're physically next to C on a US keyboard.

But all these examples may be somewhat beside the point... What exactly do other common apps do across different-language keyboards? @RastislavKish, in most apps, where are, for example, cut, copy, and paste for you? Are they where your X, C, and V are? Are they in the same spot that X, C, and C are on a US keyboard, even though that's Y, X, and C for you? (Or are they in some other spot entirely?) What about Undo — is it modifier + Z, or + \? What about more application-specific shortcuts, if you have a way of telling what the same app uses on a US keyboard? Whatever approach "normal" apps use is probably what Toga should do, at least by default.

Somewhat separately, re: OS-dependent shortcuts for things like quit: we could do something like what Qt does and have a simulated "Quit" keycode that maps to the system default. If I recall, it has those for most common actions like cut, paste, close, etc. I'm not sure how it handles collisions with manually-defined shortcuts, though.

[freakboy3742]

One possible counterpoint is that while copy is Ctrl/Cmd C, cut and paste are X and V. I don't think those mean anything beyond the fact that they're physically next to C on a US keyboard.

Ok - you've got me there. If I was to retcon a rationalisation into my original description, I'd say "C" was picked because of the name; X and V were picked for memory reasons because it's easy to remember they're physically close to C.

Somewhat separately, re: OS-dependent shortcuts for things like quit: we could do something like what Qt does and have a simulated "Quit" keycode that maps to the system default. If I recall, it has those for most common actions like cut, paste, close, etc. I'm not sure how it handles collisions with manually-defined shortcuts, though.

The complication that Qt doesn't handle (AFAIK) is that Windows puts Quit in the File menu, and macOS puts it in the Application menu. However, you're right that system commands could be defined in an abstract "Whatever the key for Cut is" manner. However, that doesn't help us with user-defined commands that need to be locale dependent.

There's essentially 2 issues here - one is a bug, and one is a feature.

The bug is that I can't currently define a keyboard shortcut for CMD-! that is reachable for Slovak (and other non-US keyboard) users on macOS and Windows, because the keyboard handling on those platforms assumes ! is a shifted 1. This requires us to modify the "shift identification" code to be locale dependent. I don't believe GTK isn't affected by this because the Gdk keycode for ! is independent to 1; shifting is only used to differentiate A from a. It also requires us to add key definitions for keys that aren't currently reachable at all by Toga, like all the accent keys on a Slovak keyboard.

The missing feature is that I can't define a localized keyboard shortcut. This is the "Go to Page" being CMD-P in English, but CMD-S for Slovak users. This would also be accompanied by translating the text of the command itself from English to Slovak. Localization is a big task, but one that I definitely want Toga to support.

From there, @RastislavKish's "geometrical layout" is a separate API for specifying keycodes other that localization, mapping from some abstract geometrical representation to Toga keycodes in a locale specific way. I can't say this is a high priority for me personally, but if someone wants to tackle the problem and comes up with a good geometrical representation, then I guess we might as well support it.

[RastislavKish]

It is true I personally usually handle shortcuts symply by listening on window key presses, so I'm not sure how are various backends handling the question of specifically keyboard shortcuts in this regard. And they may actually do some fiddling there, for example Audacity I believe used to have ( and ) keys for selecting audio, the problem was that on Slovak keyboard, these two characters are not beside each other, but they're placed on ) and \ keys, plus you need to press Shift, so doing accurate selections was quite a drag.

The special keys were causing also a lot of issues in programs like Excel, VS Code, the letter approach for picking shortcuts works reasonably well if the app has few functions, but as their amount starts to grow, there's always a need to use a different logic for their placement, these programs either used combinations like Ctrl+Shift+`, which can't even be pressed on Windows with a Slovak layout, because the ` key / character simply doesn't exist there, and, there were combinations like Ctrl+Shift+;, which could be technically done but they usually didn't work likely for similar problems that Toga currently has with Shift modifying the outcoming character.

So, judging by apps, there were different approaches on this topic. And we have even more funny situations, because, Windows (at least on the Slovak keyboard, not sure about the English-one), translates right alt into Ctrl+LeftAlt. It is a system-level translation, meaning there's nothing like AltGr on Linux, the apps will simply see a Ctrl+Alt combination. While at the same time, because many keys on their own cary accented characters (take the number row, as an example, which is ; + Ľ Š Č Ť Ž Ý Á Í É =), we have many special keys assigned for Ctrl+Alt modifiers. For example, Ctrl+Alt+P produces ' on Windows.

Right now, if toga defined one command with a Ctrl+Alt+P shortcut and another command with Ctrl+Alt+', pressing Ctrl+Alt+P would likely trigger either both of them, or, depending on the internal handling, would lead to undefined behavior.

Keyboard layouts are a mess. :)

That's why I don't like to mess with them. But this apparently is a personal preference, so sticking to what the underlying Toga frameworks offer sounds reasonable. Which should be also the starting point for possible implementation of any other approaches. If say GTK understands shortcuts like some modifiers plus a textual character (not physical key), then it indeed makes sense to have this as the base and implement a function, which would take physical key (marked by its position on English layout) + modifiers and return the textual character they produce, this could be used then to assemble shortcuts that are layout independend and Toga wouldn't actually need to introduce any new API except the translation function.

With GDK, this can be done pretty easily, even though the naming and differences between hardware_keycode, keycode, keyval, ... are super-confusing, basically there is a Keymap structure exactly for this purpose, it can be obtained from the GDK display object and provides translate_keyboard_state method which can do the trick. I've done this in one of my projects and it worked really well.

However, I don't know the state of this with Winforms and Cocoa. Those would need further research.

[freakboy3742]

So, judging by apps, there were different approaches on this topic.

Sure - but all these cases would be caught by the ability to define locale-specific key shortcuts. Whether the locale-specific shortcut has been selected for linguistic reasons or because of basic keyboard ergonomics doesn't matter - as long as every key on the keyboard can be mapped on any given locale's keyboard, and there's an ability to define locale-specific key codes, then the problems you describe can be avoided. That's why I've described this as a combination of a bug and a feature.

Keyboard layouts are a mess. :)

No disagreement on that :-)

That's why I don't like to mess with them. But this apparently is a personal preference, so sticking to what the underlying Toga frameworks offer sounds reasonable.

It's not just that it's "reasonable" - it's that it's the only option that is available to us. A Winforms menu item can only be specified in terms of a ShortcutKey. A macOS shortcut can only be specified in terms of Cocoa key values. We can only cook with the ingredients we have, and some platforms simply don't provide the sort of low level key code handling you're describing - at least, not to the best of my knowledge, in a way that is compatible with defining menu shortcuts.

With GDK, this can be done pretty easily, even though the naming and differences between hardware_keycode, keycode, keyval, ... are super-confusing, basically there is a Keymap structure exactly for this purpose, it can be obtained from the GDK display object and provides translate_keyboard_state method which can do the trick. I've done this in one of my projects and it worked really well.

The current GTK implementation is entirely based on keyval; if there's a more appropriate API that we should be using that allows us to continue to define GTK keyboard shortcuts, feel free to suggest a change in a PR.

However, I don't know the state of this with Winforms and Cocoa. Those would need further research.

Sure - if you don't have expertise or access to Windows or macOS, we don't expect you to submit patches for those platforms (unless you're interested, of course!) As long as the GTK-specific changes don't impose API requirements on other platforms for an existing feature, we're happy to accept single-platform fixes.

[corranwebster]

However, on macOS, the mapping is tied to physical keys - so the "1" key is "key 18", and it's up to Toga to map from "key 18 + shift" to toga.Key.EXCLAMATION. As you've noted, this appears to be locale dependent, so it doesn't work non-US keyboards. I don't know if there's a better (or locale dependent) way to handle these key combinations - if the entire keycode is different (so the +/1 key is a different key code on Slovak keyboards), or if there's a different mapping required, or if there's a better API we could be using for defining menu shortcuts.

I fell down this rabbit-hole today (I was writing a widget with key event handling and getting errors when pressing the '§' key on my UK English keyboard). This is expected at the moment because there is no toga.Key.SECTION or similar and the toga_cocoa.keys doesn't have a mapping for "key 10".

What we're doing in the Mac keys right now is when we map from toga to cocoa, we map to character codes which are independent of the physical keys because this is what menu keyboard accelerators expect on cocoa. So for example "Key.ESCAPE" gets mapped to chr(0x001B) (ascii escape) and "Key.F1" gets mapped to chr(0xF704) (a unicode symbol in a "private use area"). Alphabetic characters get mapped to shift plus their lowercase version, but other shifted characters are left alone (eg. Key.HASH is mapped to "#"). I think this is the right way to do things Key.MOD_1 + Key.HASH should map to command plus whatever you need to type to get a "#", and if that doesn't make sense in a different locale, the internationalisation of your app should handle that.

Looking at the reverse direction on the mac (fortunately only used in tests, right now), currently everything is via the physical keyboard key codes, which doesn't internationalise well. However the key event has a number of methods and properties other than the key codes that can help:

• characters seems to give what you'd want for simple text entry (it doesn't work for composed characters or Chinese/Japanese/Korean)
• charactersByApplyingModifiers(NSEventModifierFlagCommand) seems to reliably give the unmodified key that would be bound to a command (on non-Latin systems, macOS seems to mostly use QWERTY layout for command keys). So if I type key code 0 ("a" on an English keyboard) then on French layout this gives me "q", and on Slovak, Ukrainian, Japanese and Hebrew layout is gives me "a").
• If shift is pressed as well, then charactersByApplyingModifiers(NSEventModifierFlagCommand | NSEventModifierFlagShift) and converting alphabetic characters to lower case may be better.
• for some keys which don't really have good character representations, the more stable key codes might be needed, for example to distinguish between things like Key.ESCAPE and Key.CONTROL + "[".

This doesn't solve things on other platforms, but it may help Toga move away from depending heavily on the physical hardware for keyboard interactions.