term-char-mode no longer usable as a terminal emulator

[clinbr]

With the current 'develop' branch, it seems that all 'top level' bindings are now available in term (char sub-mode) in both insert state and emacs state. This behaviour seems to have been introduced to as a fix for #3927, so this is evidently expected behaviour for some people. I would argue that this bug was a feature--the whole point of term-char-mode is that all keystrokes except for the escape key(s) are sent to the terminal. Any terminal app using M- keybindings is unusable with this behaviour.

I would also argue that #4054 was fixed with a kludge. It was already possible to send C-z with ESC C-z, but the real problem was that C-z was available in term-char-mode.

[darkfeline]

I apologize if you find my fix a kludge. ESC C-z does not send C-z, or I'm not understanding you correctly. ESC exits insert mode, and C-z switches from Evil/normal state to Emacs state.

M- keybindings can be used with the time-honored tradition of sending ESC (bound to C-c C-e) followed by the character in question. This will break kludgy hacks which try to interpret ESC followed by a character within a timeout period differently from ESC followed by a character outside of the timeout period.

Almost all terminals do not support the Meta/Alt key in the first place. M-x is identical to ESC x as far as the terminal is concerned (barring of course the one or two terminals in existence with 8-bit character support, which brings its own host of compatibility problems).

Now, as far as using term-char-mode as a terminal emulator: yes, it's kind of broken, and no, there isn't an easy solution, partly because terminals really, really suck (read Terminals are weird).

the whole point of term-char-mode is that all keystrokes except for the escape key(s) are sent to the terminal

I do agree that this would be nice, but how would you make this integrate cleanly with Spacemacs? I do not want to have to use an inconsistent C-c SPC w l exclusively to switch out of my term buffer, I would just run a separate terminal emulator at that point.

The point of running terms in Emacs is to integrate it with everything else in Emacs. If you need special unergonomic escape sequences in terms, it defeats the purpose.

[clinbr]

Okay, that was too harsh. I said it was a kludge because I think that restoring the documented term-char-mode behaviour is the "obvious" solution. Clearly you disagree. For what it's worth, I do think C-c SPC w l is acceptable, at least from emacs state.

ESC C-z works from emacs state, for me at least. I was using emacs state because it previously gave the expected term-char-mode behaviour. I see that the bug was about insert state; my apologies.

I will have to read your link before making further criticism.

[StreakyCobra]

If the goal of term-char-mode is to directly send raw keystrokes to the terminal, then indeed it's a bug if they are caught by Spacemacs. As long as this mode is not the default one, and must be activated in purpose by the user, then having to do C-c SPC w l or any other long key sequences is not a problem as it's the wanted behaviour.

[clinbr]

If it helps: I fix this locally by reverting 81765e9 . This also fixes #4047.

[StreakyCobra]

The problem is that either:

• We use this commit to make SPC binding available everywhere without having to define it explicitly in each mode, but some mode like helm and term-char-mode get functionalities overridden.
• We don't use this commit, but the we have to explicitly bind SPC in a lot of modes (see Missing SPC binding in several modes #3981).

@justbur @syl20bnr Any thoughts about this and #4047?

[justbur]

@StreakyCobra I guess we just make the override behavior optional since we clearly can't please everyone. The only question then is what's the right default.

[justbur]

By the way the only reason helm was getting functionality overridden is because of the conflicting choice of leader key in #4047

[StreakyCobra]

@justbur Is there an easy way to disable this override per-mode basis?

[clinbr]

Why are C-x, C-c, C-/, and friends being introduced into the term-char-mode mapping by this change? This is what I'm trying to understand.

I'm not so sure about #4047; please disregard my comment. I'm not seeing the problems with helm bindings anymore.

[justbur]

I guess I could add the option... Or we could manually bind in the modes.
On Sun, Dec 6, 2015 at 9:44 AM Fabien Dubosson notifications@github.com
wrote:

@justbur https://github.com/justbur Is there an easy way to disable
this override per-mode basis?

—
Reply to this email directly or view it on GitHub
#4063 (comment)
.

[StreakyCobra]

If the default is to not enable it everywhere, then it's better to manually add it in each mode.
If the default is to enable it everywhere, then it's better having an option/a list of modes where it shouldn't be enabled.

[justbur]

Come to think of it I think the change in #4047 probably has to do with the minibuffer bindings which I don't think evil leader had any effect on.

[justbur]

I can setup the override behavior as a proper minor mode, so that it can easily be disabled through hooks. This seems like a good solution to me, but I'll wait for @syl20bnr before I start making changes to bind-map again.

[StreakyCobra]

I like this way, FWIW :-)

[darkfeline]

Actually, I would like this term issue clarified further.

Default Emacs

By default, Emacs has term-line-mode and term-char-mode. term-char-mode works rather intuitively: hit a key and term-mode sends it to the running application. There's one escape key, which is used to access Emacs.

term-line-mode is different. Almost all of Emacs's commands are retained, including text editing commands. The only different is that when you hit Enter, the current line of text is sent to the running application all at once.

Of course, you can switch freely between the two modes.

term-line-mode's benefits (vs term-char-mode):

• Much better text editing. You can edit text everywhere, even if the underlying application doesn't support it.
• Input history, even if the underlying application doesn't support it.
• Much easier to access to the rest of Emacs's functionality.

Disadvantages:

• Much harder to use interactive features of the underlying application. Curses style apps are basically unusable.
• Harder to send non-printing characters (control characters, e.g., CTRL-Z, CTRL-D, CTRL-C) to the underlying application.

Short example using bash

Let's say I type rm foo, then delete the last word and type bar.

In term-char-mode, I type rm foo<M-DEL>bar, and each character is sent to bash as I type them. When bash gets <M-DEL>, it uses its own text editing features (readline) to delete foo.

In term-line-mode, I typerm foo<M-DEL>bar. The input is buffered in Emacs, Emacs handles the <M-DEL> to delete foo, and when I hit Enter, only rm bar is sent to bash; bash doesn't know about any text manipulation I do in Emacs before sending.

Spacemacs now

Currently, Spacemacs uses a bastardized version of term-char-mode which has been somewhat evilified.

In particular:

• In normal state, Spacemacs works as it normally would in a normal state buffer.
• In insert state, Spacemacs works somewhat like term-char-mode, sending printable characters to the underlying application.
• In insert state, non-printing characters have really weird behavior. For example, M-r sends what would normally be C-R (reverse history search in shells). C-w deletes backward one word in the BUFFER ONLY. This means that if I type rm foo<C-w>bar, the term buffer shows rm bar, but what the underlying application (the shell) gets is rm foobar

What to do

I think it would be nice to bring Spacemacs more in line with default Emacs behavior and generally clean up all of the term modes (term-mode AND "multiterm-mode", which piggy-backs on term-mode).

Have a term-line-mode which works like the default term-line-mode, except evilified as needed.

Have a term-char-mode which strictly uses only one escape key, probably C-c to stick to the standard.

I don't know which one should be the default. I think it should be term-line-mode, which maintains consistency with the rest of Spacemacs, whereas term-char-mode behaves as expected for users expecting a plain terminal editor without any of the advantages provided by running a term in Emacs.

[syl20bnr]

I will throw numbered questions in all directions with no mention.

Question 1: Is the issue mentioned here related to #4054 or is it related to bind-map override mode ?

I think it would be nice to bring Spacemacs more in line with default Emacs behavior and generally clean up all of the term modes (term-mode AND "multiterm-mode", which piggy-backs on term-mode).

Yes.

I don't know which one should be the default. I think it should be term-line-mode, which maintains consistency with the rest of Spacemacs.

Yes.

Question 2: What's the binding to switch to term-char-mode ?

Come to think of it I think the change in #4047 probably has to do with the minibuffer bindings which I don't think evil leader had any effect on.

Question 3: Could it be a new keyword ?

I can setup the override behavior as a proper minor mode, so that it can easily be disabled through hooks. This seems like a good solution to me, but I'll wait for @syl20bnr before I start making changes to bind-map again.

I like this solution, very easy and intuitive to disable it per mode.

[syl20bnr]

BTW: I'm not a heavy user of terminals inside Emacs, I use eshell most of the time for simple commands like tests etc... I learned in this thread about the two different editing modes of Emacs in terminals so bear with me on this one, I'll do my best to understand the issue. :-)

[darkfeline]

Question 1: Is the issue mentioned here related to #4054

No.

The problem @kalbr (the issue creator) is running into can be solved at different levels. It can be immediately fixed by fixing override behavior, so...

or is it related to bind-map override mode ?

Yes.

At a lower level though, the complaint @kalbr raises is that Spacemacs messed up term-char-mode's semantics by evilifying it. This cannot be fixed by fixing override mode.

Question 2: What's the binding to switch to term-char-mode ?

Emacs's default bindings are C-c C-j for term-line-mode and C-c C-k for term-char-mode. Spacemacs hasn't touched these bindings.

Off-topic: Can someone with the requisite knowledge write documentation for how Spacemacs's keymaps are handled? With global maps, evil maps, per mode maps, autoloading, hooks, and other things setting keys willy-nilly, it's hard to keep key bindings sane and I think is the hidden cause of recent keybinding issues such as this and #4047.

[justbur]

@kalbr please update bind-map from MELPA and add the following hook to your user-config to test

(add-hook 'term-mode-hook (lambda () (spacemacs-default-map-override-mode -1)))

If it works, this should probably be added to the spacemacs config, but it would be helpful for you to test first.

[justbur]

Off-topic: Can someone with the requisite knowledge write documentation for how Spacemacs's keymaps are handled? With global maps, evil maps, per mode maps, autoloading, hooks, and other things setting keys willy-nilly, it's hard to keep key bindings sane and I think is the hidden cause of recent keybinding issues such as this and #4047.

@darkfeline I can comment on how key bindings work, but I'm not sure there is a "policy" for handling key bindings. We're in the midst of changing the way that they are handled (removing evil-leader mainly) and there are some hiccups here and there. Getting the priority of keys right for everyone is nearly impossible, so the fact that there were some subtle changes to priority recently has introduced some issues.

On the matter of #4047, the poster of that issue specifically chose a leader key with conflicting keys, C-c, and so was intentionally or unintentionally relying on the specific order in which commands were looked up in the Emacs keymaps. We, I mostly, had hacked together a way for evil-leader to work with non-evil keybindings, which involved writing another minor-mode to piggy back on evil-leader, and as a consequence C-c "worked" in helm as a leader key, but this was relying on behavior that was not chosen by anyone. It was just a coincidence that that keybinding worked as the poster wanted it to work in that case. Does that make sense?

[justbur]

Also, this should put your term-mode buffers in emacs state

(push 'term-mode evil-emacs-state-modes)

It's also incorrect to say that term-modes have been "evilified", at least not in the sense that it's meant in spacemacs

[darkfeline]

@justbur

I'm suggesting documenting the priority of key binds so that

so was intentionally or unintentionally relying on the specific order in which commands were looked up in the Emacs keymaps

doesn't happen. Something like defining clear priorities:

User minor mode (all states) > layer minor mode > user major mode > layer major mode > user global > spacemacs global

and the proper commands and files for defining bindings at each level.

It's also incorrect to say that term-modes have been "evilified"

I know, perhaps I should have said "made evil-friendly".

[clinbr]

@justbur I've tried your suggestion w.r.t. term-mode-hook. The M-m binding is not available but all of the 'top level' emacs bindings are still there.

My test is that C-x should not bring up which-key, C-/ should not try to undo.

As far as I can tell my install is up to date. (I used the 'update packages' link on the spacemacs home buffer, and am also up to date with develop).

[justbur]

@kalbr I'm not sure why C-x is not working. Maybe someone with a similar system could test? Can you paste the output of SPC hds?

For C-/, it probably runs undo-tree-undo. I looked into this a bit and unfortunately it's a little complicated and not entirely a spacemacs issue. Evil actually turns on undo-tree-mode itself if evil-mode is active in a buffer and global-undo-tree-mode is on (which it is in spacemacs). I tried disabling evil with (add-hook 'term-mode-hook 'turn-off-evil-mode) which should do the trick, but that wasn't disabling it for me. I'm not sure what this means and whether this is a bug somewhere, but it doesn't seem to be related to something spacemacs is doing wrong (certainly I think we'd like to fix this if possible).

[clinbr]

System Info

• OS: darwin
• Emacs: 24.5.1
• Spacemacs: 0.105.0
• Spacemacs branch: develop (rev. 68e67fb)
• Distribution: spacemacs
• Layers:

(emacs-lisp latex org shell spell-checking themes-megapack git)

[justbur]

@kalbr I think I have an answer on C-x. It looks like the multi-term package is making changes to term-raw-map which is having the effect. This should rectify the situation for that key

(define-key term-raw-map "\C-x" 'term-send-raw)

You should be able to exclude that package entirely if you don't want that behavior, but @StreakyCobra was testing that method and it wasn't working exactly right yet.

That leaves the C-/ problem with undo-tree and evil...

[justbur]

@darkfeline here's a stab at what you are asking for. Someone please feel free to turn this into a PR

The priority of Emacs keymaps is

1. Overriding keymaps/overlay keymaps
2. Emulation mode keymaps
   1. Evil Intercept keymaps
      • result of (evil-make-intercept-map map)
      • used to elevate minor mode maps
   2. Evil Local state keymap
      • evil-normal-state-local-map
      • evil-insert-state-local-map
      • etc.
   3. Evil Auxiliary keymaps
      • result of (evil-define-key state mode-map key def)
   4. Evil Overriding keymaps
      • result of (evil-make-overriding-map map)
      • used to elevate minor mode maps
   5. Evil Global state keymap
      • result of (evil-global-set-key state key def)
      • evil-normal-state-map
      • evil-insert-state-map
      • etc.
   6. Evil Keymaps for other states
3. Minor mode keymaps
4. Local keymap (local-set-key)
5. Global keymap (global-set-key)

Using the default leader keys for example, SPC bindings live in 2(ii) when they are overriding and 2(v) when they are not; M-m bindings live in 2 when they are overriding and 5 when they are not. SPC m bindings are at 2(iii); M-m m bindings are at 3.

It's important to note that when the same key is bound at exactly the same level (from emacs perspective level 2 is the same level. Evil creates its own internal hierarchy), emacs uses the first binding it finds.

There are some subtleties not covered, but I think that covers the basics.

By the way, before evil-leader had everything at 2(ii).

[syl20bnr]

At first it looks like Evil hierarchy is a mess. @justbur what's your opinion on this ?

[justbur]

@syl20bnr My opinion on what? Evil's hierarchy or how we use it?

[syl20bnr]

@justbur Do you think that all the levels are required ? Is it possible to simplify the hierarchy ?

[syl20bnr]

@justbur Do you know an example for each level ?

[justbur]

Yes, I do actually. I don't understand how we would simplify it, since this is the framework set up by evil and emacs together.

[syl20bnr]

I'm looking for a way to simplify it at the user level, this is overly complicated to my taste, especially with no example associated to them. But maybe with examples it will be OK, still, 5 6 internal levels introduced by Evil, this is impressive and I'm a bit confused/disappointed.

[syl20bnr]

@justbur If I rephrase my question: what are the levels impacted by bind-map ? What are the levels that make sense for spacemacs users ? Are there levels that are implementation details ?

[justbur]

With examples

1. Overriding keymaps/overlay keymaps
   • magit section maps
   • micro states
2. Emulation mode keymaps
   1. Non-evil emulation-mode-maps
      • bind-map keys (not evil-keys) with override option
      • also bind-key* puts things here
   2. Evil Intercept keymaps
      • used to handle ESC in evil
   3. Evil Local state keymap
      • used by evil-leader
      • currently used by bind-map for evil keys with override-mode-maps option (chosen to emulate evil-leader priority)
   4. Evil Auxiliary keymaps
      • used if you define minor-mode and evil-state specific keys (lots of examples)
      • bind-map evil-keys tied to a minor-mode
   5. Evil Overriding keymaps
      • used to put minor mode maps ahead of evil global maps
      • dired-mode-map for example is here
   6. Evil Global state keymap
      • "most" of the vim keybindings are here
      • bind-map evil-keys without override option
   7. Evil Keymaps for other states
      • no idea what this is, but it's mentioned in the evil source
3. Minor mode keymaps
   • all the minor mode maps that are set up without using evil functionality
   • bind-map keys tied to minor-mode and major-mode
4. Local keymap (local-set-key)
   • major mode maps are here I believe
5. Global keymap (global-set-key)
   • bind-map keys default location

Updated with more examples

[syl20bnr]

☝️ this is priceless, thank you!

[StreakyCobra]

@justbur For the posterity. And if you learn new information about all this ;-)

[darkfeline]

Wow, that's messy. How does this simplification sound?

Emacs/Spacemacs

1. Other
2. Minor mode (+user)
3. Major mode (+user)
4. Global (+user)

Vim/Spacemacs

\1. Other
2i. Other
2iii. Evil User Major/Minor modes (evil-define-key)
2iv. Evil Major/Minor modes
2v. Evil User Global (evil-global-set-key)
2vi. Evil Global

Spacemacs's modifications would go where the user's customizations go, then the user is free to make additional modifications on top of that.

[justbur]

@darkfeline I don't understand what you mean by "simplification". Are you referring to how it is presented to the user? I was just reporting the hierarchy as it actually exists.

[darkfeline]

Are you referring to how it is presented to the user?

Yes, and how Spacemacs uses it, ideally, for example, keeping all of its bindings limited to a few levels instead of spreading them across all of them.

[justbur]

Many keybindings are set by packages and i think it's fair to say that it's
too much work to reconfigure all of the packages. So to the extent that
packages spread out keybindings we have to respond to that. Ideally yes but
it's just not simple. If you have a question about a particular choice then
I can probably give you the rationale.
On Tue, Dec 8, 2015 at 5:52 PM darkfeline notifications@github.com wrote:

Are you referring to how it is presented to the user?

Yes, and how Spacemacs uses it, ideally, for example, keeping all of its
bindings limited to a few levels instead of spreading them across all of
them.

—
Reply to this email directly or view it on GitHub
#4063 (comment)
.

[justbur]

Some more info. I played with ansi-term for a bit and at least part of the issue is evil-mode alone. If you emacs -Q and load evil-mode, undo-tree is enabled on C-/ and C-x is a prefix in term-char-mode. Since spacemacs always has evil-mode enabled this is will be the case with any editing style at the moment.

[mentalisttraceur]

(Edit: I initially commented on this with the misperception that this issue was closed. Now I realize the Autumnal Cleanup issue referenced right before this comment is closed, not this issue, anyway, back to the previously posted content): This is still routinely causing me problem on the latest master version.

What's particularly painful is when I end up in some state where I need to send e.g. Esc or some C- key combination that's not supported or convenient in the currently built-in key shortcuts to the program running in the shell. (C-c, C-d, C-z and Esc are easy enough to pass through to the underlying terminal, though somewhat irritating to do so repetitively, for example if the program in the terminal uses Esc extensively, C-c C-e gets cumbersome, and the natural habit you might have of just pressing Esc will, in those cases, keep irksomely yanking you out of the Spacemacs input state. But what about C-r (very useful in shells) and so on? As far as I know there's no way to input other arbitrary C- sequences, without tweaking your config to define shortcuts for them?)

I can list several examples if people want, of program functionality which is inaccessible, or worse, programs which trap you in some submenu/state unless you can input a specific key. Ideally, you would have a way to pass any arbitrary input through to the underlying terminal, with the same mechanism for all of term/ansi-term/multi-term. Maybe something relatively radical, like not applying evil-mode to term-char-mode, and instead writing a thin custom mode that's tuned for evil-like and spacemacs-like interaction with the underlying terminal program, would be the most actually practical and general solution?

[darkfeline]

@mentalisttraceur

But what about C-r (very useful in shells) and so on?

It's bound to M-r, see #4063 (comment) I believe that comment is still the right way to fix term-char-mode in Spacemacs, but I no longer use Spacemacs so maybe the situation is now different.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this issue is still valid!