Literal Replace command works incorrectly when the replacement string contains '$'

[friendly-bits]

Description of the problem or steps to reproduce

Basically I want to replace some variable name in my shell script with another variable name. In shell, to get a value of a variable, the syntax is '$variable'. So for example, i want to replace '$variableA' with '$variableB'.

I'm typing replace -l '$variableA' '$variableB' in micro's command line. micro correctly finds instances of '$variableA' but when i press Y to accept the replacement, what micro actually replaces with is empty string. So it basically deletes the found instance of '$variableA' instead of replacing it with '$variableB'.

I tried escaping the '$' sign in the replacement string with a backslash but then micro just replaces '$variableA' with the backslash.

This is probably not how it's supposed to work, especially when the 'replace' command is invoked in literal mode.

Specifications

Commit hash: c2cebaa
OS: Debian Stretch
Terminal: xterm

[JoeKar]

Hm, in the moment someone knows what will be invoked in that moment then it becomes explainable, see https://pkg.go.dev/regexp#Regexp.Expand:

To insert a literal $ in the output, use $$ in the template.

Considering this in a possible fix would be easy in the first place, but may become quite complex in the moment every possible user combination needs to be considered in both modes. But as long as the non intuitive, but documented...

Note that search must be a valid regex (unless -l is passed). If one of the arguments does not have any spaces in it, you may omit the quotes.

...will be respected, then it remains simple:

diff --git a/internal/action/command.go b/internal/action/command.go
index 16d2fff8..0b88c3d7 100644
--- a/internal/action/command.go
+++ b/internal/action/command.go
@@ -785,6 +785,7 @@ func (h *BufPane) ReplaceCmd(args []string) {
        if noRegex {
                search = regexp.QuoteMeta(search)
        }
+       replaceStr = strings.ReplaceAll(replaceStr, "$", "$$")
 
        replace := []byte(replaceStr)

[237dmitry]

replace $$var $var1 -l does not work

nothing matches \$\$var

[JoeKar]

@237dmitry
For me it works. Do you have $$var present in your text? Because if not, then it's explainable. ;)
Please follow the micro documentation, that the "search" string needs to be of a regex type or in case the literal search is used it can be of the normal literal type, but will be replaced implicit to regex type by micro...because the search is performed with regex. -l only controls the type how "search" needs to be interpret.
The the confusing or unexpected thing right now is, that the "replace" string is every time treat as literal.

[237dmitry]

To insert a literal $ in the output, use $$ in the template

Do you have $$var present in your text?

No I have not. I am trying to replace literal $var with $var1. Behavior is the same as in the issue:

replace $var $var1 -l    # replaced "$var" with space \u0020
replace \$var $var1 -l   # the same
replace $$var $var1 -l   # nothing matched \$\$var

regex replace is not working too. replace '\$var' '$var1'

For me it works.

Please show example

[JoeKar]

No I have not. I am trying to replace literal $var with $var1. Behavior is the same as in the issue:

Then the question is, why you tried to find $var with the search string $$var? ;)
As I tried to explain above:

Please follow the micro documentation, that the "search" string needs to be of a regex type or in case the literal search is used it can be of the normal literal type, but will be replaced implicit to regex type by micro...because the search is performed with regex. -l only controls the type how "search" needs to be interpret.
The the confusing or unexpected thing right now is, that the "replace" string is every time treat as literal.

regex replace is not working too. replace '$var' '$var1'

Isn't it?

Please show example

Here we go...

literal search:

regex search:

Did you realize that I linked a fix yesterday to use $ in replacements?

I agree, that it's ugly to add \\ to interpret special regex keywords as single character, but we do the same in the syntax YAMLs.
Maybe we could replace \ within the search string into \\ upon usage of explicit regex search, as we no do with the $ to $$ in the replace string (see #2954).

[237dmitry]

This does not work.

Literal search/replace:

Regex search/replace:

Same thing with quotes, double or single

micro is tar-compressed distributive from releases page

$ micro --version
Version: 2.0.12
Commit hash: c2cebaa3
Compiled on September 06, 2023

$ uname -rsov
Linux 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 GNU/Linux

[JoeKar]

micro is tar-compressed distributive from releases page

And that's exactly the reason why I works for me, but doesn't for you. ;)
Have you read my following statement?

Did you realize that I linked a fix yesterday to use $ in replacements?

The PR #2954 was created 2 days ago and thus it can't be present in the last release. If you want this change, then you've to use a dedicated commit hash and build micro yourself or unfortunately you've to wait, till the PR will be merged into master and a new release done.

[237dmitry]

unfortunately you've to wait

I will wait. Not so often I use micro, mcedit is my favorite editor.

$ echo $EDITOR
mcedit

[friendly-bits]

@237dmitry This is an open source project with a limited time that few people invest into developing it. Generally it's a good idea to avoid wasting their limited time on useless discussions.

[237dmitry]

Generally it's a good idea to avoid wasting their limited time on useless discussions.

You are right. I simply confirmed that the problem exists and needs to be fixed.

[friendly-bits]

@JoeKar I'm not sure if this is the correct place to suggest this (should I open an new issue?) but after having played a bit more with the literal replace function, I feel that micro is very different in the way it implements this feature from every other text editor I've seen (well I've never touched vim or emacs but I'm well acquainted with nano and all sorts of editors for Windows and for Mac.

What I want (and I believe most users want that, as well) is just make one click, enter the text I want to replace, press Enter, enter the text I want to replace with, press Enter, get it to show the first found instance, press enter, get it replaced and jump to the next instance. Preferably with a check box or button to instantaneously access Replace All. No frills and no need to think about how micro interprets whatever I type in. IMO the basic thing is to not have to enter the string-to-replace and the replacement-string into the same text field where you have to think about delimiters and special characters (which is also a guesswork) but rather just type in the string as is, just like this is implemented in Nano. Nothing should be interpreted - then this would actually be a literal replace, and a handy feature for arguably the vast majority of users.

I love Micro even despite not having this work as I want, but hurdles with text replacement are the one major issue for me. This is such a basic feature of a text editor which gets used all the time, I think it probably warrants the undoubtedly much work that it'll take to change or re-implement it.

[JoeKar]

@blunderful-scripts
I don't think, that an additional issue is necessary, because we've #2937 (and maybe even more opened). But to be honest, I fully confirm that the actual search/replace handling is a bit complicated an non-intuitive. Furthermore the actual search method (ignorecase) has further impact on the behavior.

well I've never touched vim or emacs

At least Vim does more or less the same within one command :s/foo/bar/ and in fact, it's a regular expression search.
The user definitely needs to understand, when a search is done in regex, that some special characters must be escaped to reflect the character instead of the special option/treatment behind it. It's the same with Vim, because it's the regex way to search.

So after #2954 what is now really missing? Do you want to have the literal search to be the default as in #2937? Because from your initial description you already wrote, that you invoked -l to do a literal search (and yes, you really discovered a bug in the replacement-string - handling), which should be the easiest approach. With this you don't need to care about further delimiters, you can even write without the quotes, just use a space between the two words and it isn't that many guesswork, since it's documented (commands.md).

If you would like to have a fully new search/replace menu/interface, then it probably would be worth a new feature request.

Side note:
Have a look at the open PRs and maybe how many from me are already open. Currently the zyedidia doesn't seem to be able to spent a lot of time into his project and the maintenance therefore is slowed down already and even if he could it would take months to sort out the actual issue and PR flood.

[friendly-bits]

So after #2954 what is now really missing?

Personally, when editing code, I frequently need to match and replace a string that consists of arbitrary symbols, including spaces, quotation marks (double and single) and sometimes escaped special symbols. Trying to do that in regex mode means spending tons of time on carefully crafting an expression. Now I'm not sure how literal replace is actually supposed to work in micro, but as long as the string-to-replace and the replacement-string have to be entered sequentially into the same line, obviously the input has to be interpreted, which means there must be special symbols like spaces and quotation marks. If the string I want to replace contains some of these special symbols then I have to escape them in some way, with a syntax that is probably undocumented. Now what if the escaped symbol also happens to be a part of the string I want to replace? So you see, it's really easy to get to triple- or quadruple backslashes here, all that if the quotation marks and the spaces are the only special symbols that get interpreted. Which is possibly not the case, as this bug illustrates. So yeah, my dream is exactly the same behavior as in nano. Meaning separate input fields for string-to-replace and the replacement-string, and absolutely no interpretation that gets in the way.

I suppose regex replace could be useful for some very specific tasks but not as a go-to replace tool, at least not for me.

[friendly-bits]

Do you want to have the literal search to be the default as in #2937?

I'm fine with either default, as long as I can set my own default in the config file. That said, I had to spend some time figuring out the location and syntax of that config file. Not a big deal for me but the question is, how big of a deal it is for the average user in your target audience? My opinion is that best practice when designing a tool is to make it so the user won't need to rtfm, at least for the core functionality, and replace is definitely a core function in a text editor.