clx: input: english layout

[dkochmanski]

McCLIM input (interactor, listener) are set to the english layout (even if other layout is set on the X window system), so the user can't insert characters like "łęść". If truetype is loaded these characters are rendered without a problem, just can't insert them.

Up-to-date summary is present on the Bounties wiki page.

[dkochmanski]

This bug has assigned 100$ bounty: https://www.bountysource.com/teams/mcclim/bounties

[ailisp]

Is this include the basic text-field and text-editor panels? After switch to a layout such as pl, characters with right alt doesn't work. I also tried es, in es layout, Spanish symbols typed out directly and symbols with shift can work, such as !"·$%&/()= (these are shift+1 2 ... 0), but symbols with right alt ones don't work. May be a awkward solution is to add right alt+character commands in Libraries/Drei/basic-commands.lisp, as input to any text field/editor in the end call com-self-insert in line 514, basic-commands.lisp to insert the character.

[dkochmanski]

it generally affects everything what takes keyboard input in McCLIM. You can render any character, but you can't type them. I suspect it's a trouble in clx (not a backend, but a library)

[gabriel-laddel]

;;; To switch your keyboard layout between Russian & Eng, run the following and
;;; press ALT-SHIFT.
;;;
;;; "setxkbmap -option grp:switpch,grp:alt_shift_toggle,grp:led:scroll us,ru"
;;;
;;; Eval these expressions, switch to russian and enter it into drei - works.
;;; This is only a proof-of-concept, capital letters won't work, and there
;;; are several languages other than Russian that eg, sbcl supports.

(in-package clim-xcommon)

(loop for (a b c)
      in '((DEFINE-KEYSYM :CYRILLIC_SMALL_LETTERF_SOFT_SIGN 1759) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SOFT_SIGN 1752)
  (DEFINE-KEYSYM :CYRILLIC_CAPITAL_LETTER_SOFT_SIGN 1784) (DEFINE-KEYSYM :CYRILLIC_CAPITAL_LETTER_HARD_SIGN 1791)
  (DEFINE-KEYSYM :CYRILLIC_CAPITAL_LETTER_DZHE 1727) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_DZHE 1711) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_IO 1715)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_IO 1699) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_JE 1720) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_LJE 1721)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_NJE 1722) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_JE 1704) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_LJE 1705)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_NJE 1706) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_YU 1728) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_A 1729)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_BE 1730) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_TSE 1731) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_DE 1732)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_IE 1733) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EF 1734) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_GHE 1735)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_HA 1736) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_I 1737) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EL 1740)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EM 1741) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EN 1742) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_O 1743)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_PE 1744) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_YA 1745) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ER 1746)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ES 1747) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_TE 1748) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_U 1749)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ZHE 1750) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_VE 1751) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_YERU 1753)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ZE 1754) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SHA 1755) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_E 1756)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SHCHA 1757) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_CHE 1758) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_A 1761)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_BE 1762) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_TSE 1763) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_DE 1764)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_IE 1765) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EF 1766) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_GHE 1767)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_HA 1768) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_I 1769) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EL 1772)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EM 1773) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_EN 1774) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_O 1775)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_PE 1776) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_YA 1777) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ER 1778)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ES 1779) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_TE 1780) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_U 1781)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ZHE 1782) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_VE 1783) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_YERU 1785)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_ZE 1786) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SHA 1787) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_E 1788)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SHCHA 1789) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_CHE 1790) (DEFINE-KEYSYM :CYRILLIC_CAPITAL_LETTER_SHORT_I 1770)
  (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_SHORT_I 1738) (DEFINE-KEYSYM :CYRILLIC_CAPITAL_LETTER_KA 1771) (DEFINE-KEYSYM :CYRILLIC_SMALL_LETTER_KA 1739))
      do (setf (gethash c CLIM-XCOMMON::*KEYSYM-NAME-TABLE*) (list b)))

(defun hashtable->list (hashtable)
  (let* ((out)) 
    (maphash (lambda (k v) (push (list k v) out))
         hashtable)
    out))

(loop for (code character) in (MAPCAR (LAMBDA (L) (list (car l) (name-char (symbol-name (CAR (SECOND L))))))
                                      (REMOVE-IF-NOT (LAMBDA (L) (DESTRUCTURING-BIND (A B) L (CL-PPCRE:SCAN "CYRILLIC" (SYMBOL-NAME (CAR B)))))
                                              (HASHTABLE->LIST CLIM-XCOMMON::*KEYSYM-NAME-TABLE*)))
      do (when character ;; we have two codes that don't correspond to anything..
           (setf (gethash code xlib::*keysym->character-map*) (list (list character)))))

[gabriel-laddel]

Oh, an you have to hotpatch XLIB::DISPLAY-KEYBOARD-MAPPING, else new keyboard layouts won't be picked up, in any language.

;;; NOTE, @2016-08-10T01:52:23.738949Z
;;; Fixes https://github.com/robert-strandh/McCLIM/issues/22
(in-package xlib)

(defun display-keyboard-mapping (display)
  (declare (type display display))
  (declare (clx-values (simple-array keysym (display-max-keycode keysyms-per-keycode))))
  (setf (display-keysym-mapping display) (keyboard-mapping display)))

[ailisp]

@gabriel-laddel It works, excellent work!

[gabriel-laddel]

@ailisp Anything else in McCLIM you'd like to see fixed/improved?

[ailisp]

@gabriel-laddel Maybe we will find a way to send a pull issue to clx and automatically add mapping from character representing keywords to unicode characters, then all keyboard layout input may be perfect. a longer term objective is, in fact i'm from China, i would like to add an input method for Chinese, Korean and Japanese. Input method is to show a panel to construct a block word from typing something into a panel. As when use English or Russian, there is no need to use input method so you may not heard of, but if someone want to type hello in Chinese (你好), he must first type "nihao" into the panel and then select correct one from candidate words which have same pronunciation from the panel. So that is much more work than keyboard layout only. I first wonder why, for example, a textbox in GTK+ is able to enable both different kbd layout and input methods. For kbd layout, it directly wrap from x. For input method, it implement a IMEContext mixin based on xim (the official x input method extension) for all text input widgets. So next step I want to add input method feature for McCLIM.

[dkochmanski]

Thanks! I can confirm, that this work for changing keyboard layout (but not keyboard mapping, setxkbmap pl for instance doesn't pick keyboard mapping to issue #\ś on ALT-RIGHT + s).

It's a very good starting point. I'll consult this change with CLX developers after the weekend and hopefully merge this change there.

(NB, as found previously by @gabriel-laddel)
http://stackoverflow.com/questions/35988746/clx-stumpwm-mcclim-keyboard-layout-locked-on-startup

[dkochmanski]

Created a few issues on CLX:
sharplispers/clx#53 (pull request: sharplispers/clx#54)
sharplispers/clx#55 (maybe related: sharplispers/clx#56)

[fiddlerwoaroof]

The change in this commit seems to allow mcclim to detect and handle keymap changes appropriately:

fiddlerwoaroof@a39ec47

[fiddlerwoaroof]

From IRC:

<fiddlerwoaroof> So, I think the solution is a bit more complicated than I
thought. My commit catches and handles the keymap change
event correctly, as far as I can tell (switch from a qwerty
layout like us to an azerty layout like fr + the keys change
appropriately)

<fiddlerwoaroof> With pl, the problem is that some of the characters are
involve iso_level3_shift (I think it's usually called AltGr
or something) and mcclim isn't handling that key correctly.

<fiddlerwoaroof> Also, I've noticed that the combining "dead" keys work in
some situations and don't work in others, I haven't really
figured out what makes the difference yet.

[fiddlerwoaroof]

Also, I've noticed that different gadgets handle input differently. In this example, typing dead keys (e.g. the circumflex accent in the french layout) into the text-editor signals a condition while the interactor handles such characters properly.

(defpackage :fwoar.clim-test-app
  (:use :clim :clim-lisp))

(in-package :fwoar.clim-test-app)

(define-application-frame test-app ()
  ()
  (:menu-bar (("Quit" :command com-quit)))
  (:panes
   (text :text-editor :height 200 :value "Some text")
   (int :interactor :height 400 :width 600))
  (:layouts
   (default (labelling (:label "foo")
              (vertically ()
                text
                int)))))

(define-test-app-command (com-quit :name t) ()
  (frame-exit *application-frame*))

(defun main ()
  (bt:make-thread
   (lambda ()
     (run-frame-top-level (make-instance 'test-app)))))

[fiddlerwoaroof]

Using @gabriel-laddel's keysym definitions and the code in my pr, I can type cyrillic characters

[fiddlerwoaroof]

Ok, using the code in this gist, I've managed to get characters from several different non-us locales to work correctly. This depends on the latest commit in my fork, which needs to be cleaned up a bit before it's ready.

https://gist.github.com/fiddlerwoaroof/5b62a4be6b115c08665d1d5989d3b4e7

[gabriel-laddel]

@dkochmanski What exactly do you need to consider this issue resolved and reward me the $150?

[dkochmanski]

Cleaned up and merged pull request which make McCLIM input obey keyboard layout set with setxkbmap asynchronusly at runtime by the user from another process (i.e terminal).

[fiddlerwoaroof]

The difficult part is that the pull request has to coordinate with CLX because most of the changes necessary involve filling out CLX's keysyms.

[dkochmanski]

We are in luck here, because I'm also CLX contributor and have PR accept permission there too

[gabriel-laddel]

#|CLIM Multilanguage Testbed

Possible portability issue. Does #P"/usr/share/X11/xkb/rules/xorg.xml" exist on all
UNIXen?

Requires cl-ppcre, cl-html-parse|#

(in-package climi)

(defparameter setxkbmap-string nil
  "The MULTILANG program builds up a setxkbmap sh string")

(defparameter multilang-original-layout-string nil)

(defmacro with-getfs (getfs plist &rest body)
  (assert (every 'keywordp getfs))
  `(let* ,(loop for getf in getfs 
 		collect (list (intern (symbol-name getf))
			      (list 'getf plist getf)))
     ,@body))

(defun sformat (control-string &rest format-arguments)
  (apply 'format (append (list nil control-string) format-arguments)))

(defun drop (n l)
  (unless (> n (length l)) (subseq l n (length l))))

(defun walk-tree (fun tree)
  (subst-if t (constantly nil) tree :key fun))

(defun select-node (fun tree)
  (walk-tree (lambda (node) (when (funcall fun node)
                              (return-from select-node node)))
             tree) nil)

(DEFCLASS XKB-CONFIG-ITEM NIL
  ((NAME :ACCESSOR NAME :INITARG :NAME :INITFORM NIL) (DESCRIPTION :ACCESSOR DESCRIPTION :INITARG :DESCRIPTION :INITFORM NIL)
   (SHORT-DESCRIPTION :ACCESSOR SHORT-DESCRIPTION :INITARG :SHORT-DESCRIPTION :INITFORM NIL)
   (LANGUAGE-LIST :ACCESSOR LANGUAGE-LIST :INITARG :LANGUAGE-LIST :INITFORM NIL :DOCUMENTATION "Languages are named by a ISO639ID")
   (COUNTRY-LIST :ACCESSOR COUNTRY-LIST :INITARG :COUNTRY-LIST :INITFORM NIL :DOCUMENTATION "Countries are named by a ISO3166ID")))

(COMMON-LISP:DEFCLASS XKB-LAYOUT (XKB-CONFIG-ITEM) ((VARIANTS :ACCESSOR VARIANTS :INITARG :VARIANTS :INITFORM NIL)))
(COMMON-LISP:DEFCLASS XKB-VARIANT (XKB-CONFIG-ITEM) NIL)
(COMMON-LISP:DEFCLASS XKB-GROUP (XKB-CONFIG-ITEM) ((OPTIONS :ACCESSOR OPTIONS :INITARG :OPTIONS :INITFORM NIL)))
(COMMON-LISP:DEFCLASS XKB-OPTION (XKB-CONFIG-ITEM) ((MULTIPLE-SELECTION :ACCESSOR MULTIPLE-SELECTION :INITARG :MULTIPLE-SELECTION :INITFORM NIL)))

(defmethod print-object ((object xkb-config-item) stream) 
  (with-slots (name description) object
    (format stream "#<~S: ~A, ~A>" (type-of object) name description)))

(defun make-xkb-config-item (config-item-list &key (type 'xkb-config-item) (additional-slot-initializations))
  (labels ((maybe-reduce-language-or-country-list (l) (when l (remove-if-not 'stringp l)))
           (xkb-config-name (l) (second (find :name l :key 'car)))
           (xkb-config-description (l) (second (find :description l :key 'car)))
           (xkb-config-short-description (l) (second (find :shortdescription l :key 'car)))
           (xkb-config-maybe-language-list (l) 
              (maybe-reduce-language-or-country-list
               (second (find-if (lambda (ll) (and (listp ll) (eq :languagelist (car ll)))) l))))
           (xkb-config-maybe-country-list (l) 
              (maybe-reduce-language-or-country-list
               (second (find-if (lambda (ll) (and (listp ll) (eq :countrylist (car ll)))) l)))))
    (let* ((standard-initializations (list type
                                           :NAME (XKB-CONFIG-NAME CONFIG-ITEM-LIST)
                                           :SHORT-DESCRIPTION (XKB-CONFIG-SHORT-DESCRIPTION CONFIG-ITEM-LIST)
                                           :DESCRIPTION (XKB-CONFIG-DESCRIPTION CONFIG-ITEM-LIST)
                                           :LANGUAGE-LIST (XKB-CONFIG-MAYBE-LANGUAGE-LIST CONFIG-ITEM-LIST)
                                           :COUNTRY-LIST (XKB-CONFIG-MAYBE-COUNTRY-LIST CONFIG-ITEM-LIST))))
      (apply 'make-instance (if additional-slot-initializations
                                (append standard-initializations
                                        additional-slot-initializations)
                                standard-initializations)))))

(let* ((raw-parse-info (html-parse:parse-html #P"/usr/share/X11/xkb/rules/xorg.xml")))
  (defparameter xkb-layouts
    (loop with layouts = (rest (SELECT-NODE (lambda (l) (and (listp l)
                                                             (eq :layoutlist (car l))))
                                                      raw-parse-info))
          for layout-l in layouts
          for layout-config-list = (rest (second layout-l))
          for variants = (when (third layout-l)
                           (mapcar (lambda (variant-l) (make-xkb-config-item (rest (second variant-l)) :type 'xkb-variant)) 
                                   (rest (third layout-l))))
          collect (if variants 
                      (make-xkb-config-item layout-config-list :type 'xkb-layout
                                            :additional-slot-initializations (list :variants variants))
                      (make-xkb-config-item layout-config-list :type 'xkb-layout))))
  (defparameter xkb-groups
    (loop with groups = (rest (SELECT-NODE (lambda (l) (and (listp l)
                                                            (eq :optionlist (car l))))
                                                     raw-parse-info))
          for group-block-list in groups
          for allow-multiple-selection? = (string= "true" (third (car group-block-list)))
          for group-config-list = (rest (second group-block-list))
          for xkb-options = (mapcar (lambda (l) (make-xkb-config-item (rest (second l)) 
                                                                      :type 'xkb-option
                                                                      :additional-slot-initializations 
                                                                      (list :multiple-selection allow-multiple-selection?)))
                                    (drop 2 group-block-list))
          collect (make-xkb-config-item group-config-list
                                        :type 'xkb-group
                                        :additional-slot-initializations (list :options xkb-options)))))

(defun xkb-variants () (mappend 'variants xkb-layouts))
(defun xkb-options () (mappend 'options xkb-groups))

(defun keyboard-layout ()
  (loop with keymap = '(("options:" :options) ("layout:" :layout) ("variant:" :variant))
        for (key value) in (mapcar (lambda (s) (remove-if 'emptyp (split #\space s)))
                                   (cl-ppcre::split #\Newline (mm:run-program "setxkbmap -query" :output :string)))
        when (member key (mapcar 'car keymap) :test 'string=)
        appending (let* ((new-key (second (find key keymap :test 'string= :key 'car)))
                         (new-value (case new-key
                                      (:layout (find value xkb-layouts :key 'name :test 'string=))
                                      (:variant (find value (xkb-variants) :key 'name :test 'string=))
                                      (:options (mapcar (lambda (xkb-option-name) (find xkb-option-name (xkb-options) :key 'name :test 'string=))
                                                        (cl-ppcre::split #\, value))))))
                    (list new-key new-value))))

(defun print-xkb-config-for-list-pane (object)
  (with-slots (name description) object
    (sformat "~A, ~A" name description)))

(DEFINE-APPLICATION-FRAME MULTILANG NIL NIL (:MENU-BAR NIL) (:POINTER-DOCUMENTATION T)
  (:PANES (SETXKBMAP-DISPLAY :APPLICATION :DISPLAY-FUNCTION 'RENDER-SETXKBMAP :DISPLAY-TIME :COMMAND-LOOP :SCROLL-BARS :HORIZONTAL)
          (LAYOUT-SELECTION :LIST-PANE :ITEMS XKB-LAYOUTS :MODE :EXCLUSIVE :NAME-KEY 'PRINT-XKB-CONFIG-FOR-LIST-PANE :VALUE-CHANGED-CALLBACK
                            'UPDATE-LAYOUT)
          (VARIANT-SELECTION :LIST-PANE :MODE :EXCLUSIVE :NAME-KEY 'PRINT-XKB-CONFIG-FOR-LIST-PANE :VALUE-CHANGED-CALLBACK
                             'RECALCULATE-SETXKBMAP-STRING)
          (OPTIONS-SELECTION :LIST-PANE :NAME-KEY 'PRINT-XKB-CONFIG-FOR-LIST-PANE :MODE :NONEXCLUSIVE :PREFER-SINGLE-SELECTION NIL :ITEMS
                             (XKB-OPTIONS) :VALUE-CHANGED-CALLBACK 'RECALCULATE-SETXKBMAP-STRING)
          (SWITCH-TO-NEW-KEYMAP :PUSH-BUTTON :LABEL "Enable the requested keymap" :ACTIVATE-CALLBACK 'SWITCH-TO-NEW-KEYMAP)
          (REVERT-KEYMAP :PUSH-BUTTON :LABEL "Revert to original keyboard config" :ACTIVATE-CALLBACK 'REVERT-TO-ORIGINAL-KEYBOARD-CONFIG)
          (CLEAR-OPTIONS-TOGGLE :TOGGLE-BUTTON :LABEL "Clear previous options on keymap switch?" :VALUE T) (EDITOR :TEXT-EDITOR)
          (INTERACTOR :INTERACTOR))
  (:LAYOUTS
   (:DEFAULT
    (VERTICALLY NIL
      (50 SETXKBMAP-DISPLAY)
      (5/6
       (HORIZONTALLY NIL
         (1/2
          (VERTICALLY NIL
            (HORIZONTALLY NIL
              SWITCH-TO-NEW-KEYMAP
              CLEAR-OPTIONS-TOGGLE)
            REVERT-KEYMAP
            (SCROLLING NIL
              EDITOR)))
         (1/4
          (VERTICALLY NIL
            (LABELLING (:LABEL "Layouts")
              (SCROLLING NIL
                LAYOUT-SELECTION))
            (LABELLING (:LABEL "Layout Variants")
              (SCROLLING NIL
                VARIANT-SELECTION))))
         (1/4
          (VERTICALLY NIL
            (LABELLING (:LABEL "Options")
              (SCROLLING NIL
                OPTIONS-SELECTION))))))
      (1/6 INTERACTOR)))))

(DEFUN MULTILANG-FRAME ()
  (OR
   (FIND 'MULTILANG (REMOVE-IF (LAMBDA (O) (TYPEP O 'CLIM-INTERNALS::MENU-FRAME)) (SLOT-VALUE (FIND-FRAME-MANAGER) 'CLIM-INTERNALS::FRAMES)) :KEY
         'CLIM:FRAME-NAME)
   (FIND 'MULTILANG (REMOVE-IF (LAMBDA (O) (TYPEP O 'CLIM-INTERNALS::MENU-FRAME)) (SLOT-VALUE *DEFAULT-FRAME-MANAGER* 'CLIM-INTERNALS::FRAMES)) :KEY
         'CLIM:FRAME-NAME)))

(DEFUN RUN-MULTILANG ()
  (RUN-FRAME-TOP-LEVEL (MAKE-APPLICATION-FRAME 'MULTILANG) :NAME "MULTILANG" :CALLING-FRAME *APPLICATION-FRAME*))

(defmacro with-multilang-list-panes (&rest body)
  `(let* ((layout-selection (find-pane-named *application-frame* 'layout-selection))
          (variant-selection (find-pane-named *application-frame* 'variant-selection))
          (options-selection (find-pane-named *application-frame* 'options-selection)))
     ,@body))

(defmethod run-frame-top-level :before ((frame multilang) &key &allow-other-keys)
  (with-getfs (:layout :variant :options) (keyboard-layout)
    (setf multilang-original-layout-string
          (sformat "setxkbmap -layout ~A ~:[~;-variant ~A~] ~{-option ~A ~}"
                       (awhen layout (name it))
                       (awhen variant (name it))
                       (awhen variant (name it))
                       (awhen options (mapcar 'name it))))
    
    (with-multilang-list-panes 
     (let* ((variant-layout (find mm::variant xkb-layouts
                                  :key 'variants
                                  :test (lambda (variant variant-list) (member variant variant-list))))
            (layout-variants (and variant-layout (variants variant-layout))))
       (setf (clim-extensions::list-pane-items variant-selection) layout-variants
             (gadget-value variant-selection) mm::variant
             (climi::visible-items variant-selection) (length (clim-extensions::list-pane-items variant-selection))
             (gadget-value layout-selection) mm::layout
             (gadget-value options-selection) mm::options))
     (redisplay-frame-pane frame 'variant-selection :force-p t))))

(defun render-setxkbmap (frame pane)
  (princ setxkbmap-string pane))

(defun revert-to-original-keyboard-config (&rest _)
  (uiop:run-program multilang-original-layout-string))

(defun recalculate-setxkbmap-string (&rest _)
  (with-multilang-list-panes    
   (setf setxkbmap-string
         (sformat "setxkbmap -layout ~A ~:[~;-variant ~A~] ~{-option ~A ~}"mm::
                      (awhen (gadget-value layout-selection) (name it))
                      (awhen (gadget-value variant-selection) (name it))
                      (awhen (gadget-value variant-selection) (name it))
                      (awhen (gadget-value options-selection) (mapcar 'name it)))))
  (redisplay-frame-pane *application-frame* 'setxkbmap-display))

(defun switch-to-new-keymap (&rest _)
  (when (gadget-value (find-pane-named *application-frame* 'clear-options-toggle))
    (uiop:run-program "setxkbmap -option"))
  (uiop:run-program setxkbmap-string))

(defun update-layout (gadget new-value)
  (with-multilang-list-panes
   (setf (clim-extensions::list-pane-items variant-selection) (variants new-value)
         (climi::visible-items variant-selection) (length (clim-extensions::list-pane-items variant-selection))
         (gadget-value variant-selection) nil)
   (redisplay-frame-pane *application-frame* 'variant-selection :force-p t)
   (recalculate-setxkbmap-string)))

There are a few misunderstandings in the thread above, and this should help to put everyone on the same page.

If #P"/usr/share/X11/xkb/rules/xorg.xml" does not exist on your machine, you will need to locate it (try "locate xorg.xml") & substitue into the above code snippet.

[MatthewRock]

Hi, I would like to ask about the progress on the bug? I would like to fix it myself if no one else has already tried to do it; otherwise I'll spend my time on other bug.

It also looks like @fiddlerwoaroof has an almost working solution, so I don't want to "steal" the bounty from him. Are you still working on the issue? What about you, @gabriel-laddel ?

[skypher]

So what is the acceptance criterium for this to be solved? Any set of languages or characters?

[dkochmanski]

Hey. The acceptance criterium is:

Technical perspective:

• implement xkb extension in CLX
• integrate if necessary with McCLIM

Expected behavior:

• I can type localized characters just as in xlib clients (with modifiers and such)
• setxkbmap is picked up when changed asynchronously from terminal so next characters are mapped correctly

Solution should support keysyms supported by xlib/xcb (if there is no licensing issue with that such mappings could be generated by hand from xml). First tests will include pl and de layouts, but we want a complete support.

Note that many pieces of this puzzle were tackled by various people, so it will be best to communicate with them. I'm little concerned that this solution has many creators so there may be some dissatisfaction with regard to whom we should give bounty. IMHO @dbjergaard solution is the most complete approach (and his analysis of the situation is a very valuable resource). Quote from the PR 100 in CLX:

Really, I just want something to get the ball rolling so others can pick up and make meaningful contributions instead of re-doing all the research I did into the background of the problem.

Given above and to avoid misunderstandings: @dbjergaard are you willing to coordinate the effort to implement this solution? If so, then I would shed judgement to whom the bounty goes when this issue is solved and integrated to CLX/McCLIM. If you do it yourself that's fine, and if you are OK someone else do it when possible please say so.

It is possible to split a bounty (we will claim it, re-add to McCLIM pool and create N separate issues with bounty parts to be claimed by different people). But this split must be among contributors to the final solution (not to prior incomplete attempts) and it must be consensual.

Previous approaches (most notably by @fiddlerwoaroof and @gabriel-laddel) were fine proof of concept solutions but they are seemingly not pursued twoards a clean pull request nor are as complete and beneficial to the community as XKB implementation for CLX.

CC: @fiddlerwoaroof @gabriel-laddel @quasus

Some related work and pointers:

• clx: input: english layout #35 (this issue)
• Missing keysyms sharplispers/clx#119 (clx issue with missing keysyms)
• http://dbjergaard.github.io/posts/clx_extensions.html (analysis of what needs to be done)
• xkeyboard WIP sharplispers/clx#100
• https://github.com/filonenko-mikhail/clx-xkeyboard
• wip: fix: display-keyboard-mapping: don't cache the result sharplispers/clx#54 (mapping-notify – this is relevant to McCLIM integration - we'd need to handle it)
• Xkeyboard #627 (support for xkb based on clx-keyboard fork)
• https://github.com/quasus/clx-xkeyboard (quasus's fork of clx-keyboard)
• https://en.wikipedia.org/wiki/X_keyboard_extension

[fiddlerwoaroof]

Yeah, I ran out of free time (got a job) but I agree that XKB support is the right way to go.

[skypher]

Sounds interesting, thanks for elaborating. I'll let you know once I decide to tackle it.

[dmb2]

I will help coordinate the solution. We need people contributing however they can since this is such a large issue. This is where the bounty complicates things because a complete solution already has at least three or four authors I can think of.

My development time lately hasn't been dedicated to the xkb extension, but I'm happy to help coordinate efforts and point people to tasks of varying difficulty in getting things going.

[quasus]

Fine with me, how do we proceed?

My point is that clx-xkeyboard almost does the trick but for the lack of events. The fairly simple code I add to McCLIM already fulfills the requirements to support any keyboard layout and to track changes to the keyboard state. (Basically, in some places I replace pre-XKB functions by their XKB counterparts and also cache the keyboard state invalidating the cache on XkbMapNotify events in order to stay up-to-date.) However, I don’t insist on merging it, because the hack I use to track the XkbMapNotify events is not neat and doesn’t deserve to be a part of McCLIM or clx-xkeyboard.

What we need is a serious implementation of XKB events. This is beyond McCLIM and belongs to the realm of CLX. The stumbling block is that the declare-event macro provided by CLX is inadequate for XKB events. The reason is that declare-event assumes a specific type signature of an event, whereas all the XKB events, while being a single event from the X viewpoint, have very different type signatures. I’m working on this at the moment.

[fiddlerwoaroof]

It might be worthwhile to write a code generator that transforms the XCB xml files into common lisp code.

[fiddlerwoaroof]

This would make it a lot easier to keep CLX up to date with various X extensions.

[dmb2]

The best way forward is to port the partial solution from @fiddlerwoaroof to the clx-xkeyboard extension that I’ve started.

It may be that once the extension is in clx McCLIM will get international input for free.

[quasus]

As far as I can tell, @fiddlerwoaroof's idea is to use xkb:mapping-notify whenever a MappingNotify event is detected. Of course, it is a sound idea. The conceptual drawback of the implementation is that it doesn’t use XKB and so it is in principle limited. For instance, it has been noted that Polish 3rd-level characters are not working―and this is not surprising, as the concept of the level only exists in XKB. Also I think it doesn’t work with the third and fourth layouts (groups) e. g. if one sets them with

    setxkbcomp -layout 'us,de,ru,ar'

Thus, a fundamental solution requires XKB.

1. xlib:mapping-notify is defined in the translation.lisp file of clx, so it’s a part of pre-XKB translation system. Instead, we need XKB translations. Fortunately, Mikhail Filonenko took care of that years ago in clx-xkeyboard (including an extensive table of keysyms). This machinery is to be plugged into McCLIM.

2. xlib:mapping-notify updates (or rather, invalidates) the keyboard mapping cache. Unless I’m greatly mistaken the clx-xkeyboard library doesn’t implement any caching.

3. Instead of MappingNotify we must listen to XkbMapNotify events. And this is a problem, because clx-xkeyboard doesn’t implement XKB events, and this is not straightforward.

I have addressed these issues by 1) fixing some places in McCLIM to use XKB translations; 2) implementing XKB keyboard mapping cache for a class of CLIM ports; 3) providing a temporary solution concerning events both on the McCLIM and clx-xkeyboard side. Practically all the McCLIM code with comments is in this file.

[JMC-design]

It seems to me that you have two separate problems and perhaps the bounty should be split.

1. the hard part, have a complete working xkb extension.
2. the now easy part on mcclim, selecting for keyboard change event and responding to the event.

In regards to 2, I don't know how mcclim works(if you depend on the window manager), but in my system the input manager keeps per input per window mappings(been more focused on xinput), so you can do things like have your laptop keyboard enter english in one window, french in an other, while having your plugged in keyboard type russian in the first window and greek in the second. Sure that's a bit a of a contrived example for keyboards but I'm more concerned with graphics applications and per widget input customizations. If mcclim itself is stateless you might want to pass the buck on to programmers, where it's usually tossed.

About 1. Besides xinput this is the most complex extension to x to implement. I think it deserves all the bounty. Now if you follow me to my van in the alley, I'd like to show you an xkb extension I'm selling. ;)
Seriously though, it'll be a while till I've got a complete implementation working but was wondering how pressing of an issue this is. Because if all you guys need is the events I can add those to your implementation if you can't wait a month or two. Though I see your hack on the mcclim side doesn't actually require any of the actual event info. Perhaps define-event-class is something that needs to be looked at, I find it hard sometimes to tell whether you guys are an application or a gui toolkit.

Which brings me to the most important question, to me, of how you guys think things should be reported by xkb, I'm leaning towards outputing structures of type list so that destructuring can be used on nested structures as well as struct accessors. Or does this even matter to you guys? Do you guys need to revisit your implementation of keyboard events, and input in general, that seems to have been based off the original protocol? Xinput is coming soon*, so it might be something you guys need to think about.

*by soon, i mean after xkb because I want your moolah.