counsel-grep shows no results on OS X

[jmccarrell]

Summary

It appears to me that the counsel functions that invoke external processes do not reliably report the
results of those process invocations on Mac OS X. This behavior is shown on a minimally configured emacs via
emacs -Q. The reproducible sequence does not show the bug behavior on 3 tested ubuntu-based releases of emacs.

Issues Searching with counsel

I seem to be having issues searching with counsel. Specifically, running counsel-grep gathers the
inputs, but generates no results: no results buffer. I see the same behavior with other counsel file
searching functions as well.

Steps to Reproduce

1. start the emacs under test with no initialization, eg:
   • emacs -Q
2. Load counsel
   • M-x eval-buffer on this

     (require 'package)
     (unless (assoc-default "melpa" package-archives)
       (add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t))
     (package-initialize)
     (package-refresh-contents)
   
     (unless (package-installed-p 'counsel)
       (package-install 'counsel))
   
     (require 'counsel)
     (ivy-mode)
   
     (find-library "counsel")
   

 ```

3. Do a search
   • M-x counsel-grep
   • NB that counsel is working; see how M-x runs counsel-M-x
   • search for counsel which should result in many matches in the file counsel.el
4. Expected vs Actual Results
   • I expect to see a result buffer full of matches
   • The results I actually see are essentially nothing; no search appears to be performed, no results appear.

Versions of Emacs Tested

Shows the bug

1. Emacs for Mac OS X running as a Cocoa window
   • version
     • GNU Emacs 26.1 (build 1, x86_64-apple-darwin14.5.0, NS appkit-1348.17 Version 10.10.5 (Build 14F2511)) of 2018-05-30
   • invoked as
     • /Applications/Emacs.app/Contents/MacOS/Emacs -Q
2. homebrew installed emacs running in the terminal
   • version
     • GNU Emacs 26.2 (build 1, x86_64-apple-darwin18.5.0) of 2019-04-13
   • invoked as
     • /usr/local/Cellar/emacs/26.2/bin/emacs-26.2 -Q

Works as Expected

• ubuntu 16.04 LTS

GNU Emacs 24.5.1 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of 2017-09-20 on lcy01-07, modified by Debian

• ubuntu 18.04 LTS

jmccarrell@jeff-test-1:~$ emacs --version
GNU Emacs 25.2.2

• ubuntu 19.04

jmccarrell@jeff-test-2:~$ emacs --version
GNU Emacs 26.1

[abo-abo]

Looks like you need to set up your PATH. Use getenv and setenv for that.

Also check:

(executable-find "grep")
;; => "/bin/grep"

[jmccarrell]

Hey Oleh. Thanks for responding.

I don't think it is a path issue. I got here by trying to migrate from helm; helm search commands: grep, ag, rg etc work as expected.

Also here is what executable-find reports:

(executable-find "grep")
;; => "/usr/bin/grep"

Also, M-x grep from the test case above works just fine.

[abo-abo]

How about counsel-git? Does that work?

[jmccarrell]

Yes it does. Running M-x counsel-git, I was able to select a file from the repo. It seems to work as expected.

[abo-abo]

OK, so the path to git is not the problem.

I don't have access to OSX. But I assume other OSX users don't have this problem, so it's likely a config issue. What's your shell-file-name and shell-command-switch?

[jmccarrell]

In the test case above, ie, from emacs -Q I see the following:

shell-file-name  => "/usr/local/bin/zsh"
shell-command-switch => "-c"

[jmccarrell]

Ok, nice. After I

chsh -s /usr/local/bin/bash

and re-execute the test above, counsel-grep works as expected.

In this case I have:

shell-file-name => "/usr/local/bin/bash"
shell-command-switch => "-c"

[abo-abo]

OK, so now we have a workaround. Let's investigate it further. Please try to grep with your default zsh setup.

Then post here the value of counsel--async-last-command. That's the shell command we run to get the output. You can check if that shell command works in your Emacs.

[jmccarrell]

Ok, here is the command that gets executed

counsel--async-last-command => "grep -E -n -e counsel counsel.el"

Running that in eshell does give a hint that my environment may be an issue; which may be related to how the shell is executed. However, when I first cd to the appropriate directory, the grep runs fine

eshell output

Welcome to the Emacs shell

~ $ grep -E -n -e counsel counsel.el
#<buffer *grep*>
~ $ cd .emacs.d/elpa
~/.emacs.d/elpa $ cd counsel-20190803.1121
~/.emacs.d/elpa/counsel-20190803.1121 $ grep -E -n -e counsel counsel.el
#<buffer *grep*>
~/.emacs.d/elpa/counsel-20190803.1121 $ 

results of grep in the counsel dir: buffer *grep*

-*- mode: grep; default-directory: "~/.emacs.d/elpa/counsel-20190803.1121/" -*-
Grep started at Fri Aug 23 20:32:46

grep -n -E -n -e counsel counsel.el
1:;;; counsel.el --- Various completion functions using Ivy -*- lexical-binding: t -*-
48:(defgroup counsel nil
51:  :prefix "counsel-")
54:(define-obsolete-variable-alias 'counsel-more-chars-alist 'ivy-more-chars-alist "0.10.0")
56:(define-obsolete-function-alias 'counsel-more-chars 'ivy-more-chars "0.10.0")

[abo-abo]

Running that in eshell

You should not run it in eshell. counsel-rg runs it in shell-file-name, which is zsh for you. Try to run it in M-x shell. Or even outside Emacs.

[jmccarrell]

M-x shell results

Ok, running grep from M-x shell "works", modulo the color output that my terminal settings and my zsh aliases that presume every terminal supports color conflict with. At least, that is what I presume those escape sequences are doing

�]133;A�% �]133;B�grep -l counsel counsel*.el
grep -l counsel counsel*.el
�]133;C;�counsel-autoloads.el
counsel-pkg.el
counsel.el
�]133;A�% �]133;B�

outside emacs from zsh

Here is the exact output I get from zsh. NB, my zsh config emits the git branch (master here) and the current working directory as part of its prompt. This is a multi-line prompt; $pwd is printed on a separate line.

❯ cd ~/.emacs.d/elpa/counsel-20190803.1121                                                 master

~/.emacs.d/elpa/counsel-20190803.1121
❯ grep -l counsel counsel*.el
counsel-autoloads.el
counsel-pkg.el
counsel.el

outside emacs from bash

The bash prompt is also multi line.

jeff at vega in ~/.emacs.d/elpa/counsel-20190803.1121
$ grep -l counsel counsel*.el
counsel-autoloads.el
counsel-pkg.el
counsel.el
jeff at vega in ~/.emacs.d/elpa/counsel-20190803.1121
$

colorized

Both shell prompts presume a color-capable terminal; ie they are both colorized. I'm not sure if that is relevant to this exploration or not.

[abo-abo]

Can you grep for something, confirm there's no result (although there should be). Then eval (shell-command-to-string counsel--async-last-command).

[mookid]

Going meta: should we consider non POSIX compliant shell commands as bugs?

[abo-abo]

@mookid If Emacs built-ins like shell-quote-argument or start-file-process-shell-command support the shell, even though it's not POSIX, we can support it too, since we get the support for free. Otherwise, I see no reason to do it.

[mookid]

The reason would be to avoid issues with users that prefer some shell that is not the default one.

[jmccarrell]

Can you grep for something, confirm there's no result (although there should be). Then eval (shell-command-to-string counsel--async-last-command).

I ran the process again from the top starting from emacs -Q.

It took a very long time (30 seconds?) to eval shell-command-to-string, but it did eventually return the correct result. I am beginning to think this is some interaction with my login scripts for zsh that are doing too much, ie, taking too much time? and then counsel-grep is timing out waiting for the result?

results of (shell-command-to-string counsel--async-last-command)

(shell-command-to-string counsel--async-last-command)=>
"pyenv: cannot rehash: /Users/jeff/.pyenv/shims/.pyenv-shim exists
1:;;; counsel.el --- Various completion functions using Ivy -*- lexical-binding: t -*-
48:(defgroup counsel nil
51:  :prefix \"counsel-\")
54:(define-obsolete-variable-alias 'counsel-more-chars-alist 'ivy-more-chars-alist \"0.10.0\")
56:(define-obsolete-function-alias 'counsel-more-chars 'ivy-more-chars \"0.10.0\")
  ...

[jmccarrell]

Ok, my hypothesis is confirmed: it appears to have to do with my zsh login scripts.

To validate this, I created a new user on MacOS: counselzshtest and chsh to zsh for that user. Then I ran the test process again from emacs -Q and counsel-grep worked just fine.

[abo-abo]

Thanks for following through. I've created a new wiki entry, based on this issue: https://github.com/abo-abo/swiper/wiki/Troubleshooting. Feel free to edit, the wiki is open. I think the issue is solved now, closing.

[jmccarrell]

If it is true that this is solely an interaction between my zsh login scripts and emacs, then it should be possible to recreate the issue without counsel being involved at all. So I spent some time attempting just that: to recreate the issue independent of counsel.

My first line of inquiry was trying to evaluate how long it was taking to startup zsh outside of emacs. My rough measurements show that zsh, both as a login shell and not, show it starts up in about 1 second. Which should be just fine.

So second, I thought I would try to reproduce the issue in emacs without counsel involved. So starting from emacs -Q, I eval this buffer, and then re-execute the grep call with C-x C-e

(defmacro measure-time (&rest body)
  "Measure the time it takes to evaluate BODY."
  `(let ((time (current-time)))
     ,@body
     (message "%.06f" (float-time (time-since time)))))

(measure-time
 (shell-command-to-string "grep -n counsel ~/.emacs.d/elpa/counsel-20190803.1121/counsel.el"))

I see consistent times of less than 1 second, as reported by the *Messages* buffer

For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set [3 times]
0.940473
0.935708
"0.935708"
0.926141
"0.926141"
0.918790
"0.918790"

Again, to isolate as much as possible, I have started this from emacs -Q.

From your perspective, does this test result raise any questions about the conclusion that this issue is simply an interaction with my zsh login scripts?

[jmccarrell]

I should note also that I have been doing most of this investigation while on the train, and I know that my network connection quality while on the train varies greatly. So if my fairly complex zsh init scripts try to perform network operations, that could add to the delay. And maybe the measurements I just did were under favorable network conditions?

[jmccarrell]

Ok, so I ran another experiment to measure the impact of invoking zsh as a login shell via

(measure-time
 (shell-command-to-string "zsh -l -c 'grep -n counsel ~/.emacs.d/elpa/counsel-20190803.1121/counsel.el'"))

I do see longer start up times, but all of these commands complete in < 2 seconds as shown by the *Messages* buffer

1.897794
"1.897794"
1.902941
"1.902941"
1.855524
"1.855524"
1.850427
"1.850427"
1.858426
"1.858426"
1.846766
"1.846766"

[abo-abo]

Having a shell with a startup time of 2 seconds is not acceptable for counsel-grep, since it calls the shell after every single key press to update the results.

Try to configure Emacs so that your shell-file-name doesn't perform expensive startup. Use can either use Bash globally, or bind it locally for counsel-grep, or maybe look into making shell-command-switch not have zsh perform expensive startup.

[jmccarrell]

Ok, this problem is fully resolved and counsel-grep and friends are once again back on my list of interesting emacs packages to explore.

Thanks for all of your help! Super cool.