Pipeline fails to load tree-sitter grammar. Is there a way to debug?

[piotrkwiecinski]

Hi. First of all I would like to thank you for all the time you put into open source overall.

I have an issue with setting up a CI pipeline for emacs with treesitter for php-ts-mode.
I have tried multiple ways of installing grammar itself and nothing works.

The same script works for me with my local build of emacs 29 but doesn't seem to work as expected in the pipeline.

I have a minimal example to reproduce an issue:

(progn
  (setq user-emacs-directory default-directory)
  (require 'treesit)
  (declare-function treesit-install-language-grammar "treesit.c")
     (if (and (treesit-available-p) (boundp 'treesit-language-source-alist))
        (unless (treesit-language-available-p 'php)
	        (add-to-list
	         'treesit-language-source-alist
	         '(php . (\"[https://github.com/tree-sitter/tree-sitter-php.git\](https://github.com/tree-sitter/tree-sitter-php.git/)")))
	        (treesit-install-language-grammar 'php))))))

It generates:

Cloning repository
Compiling library
Library installed to ~/work/php-ts-mode/php-ts-mode/tree-sitter/libtree-sitter-php.so
Warning (treesit): The installed language grammar for php cannot be located or has problems (not-found): (libtree-sitter-php libtree-sitter-php.0 libtree-sitter-php.0.0 libtree-sitter-php.so libtree-sitter-php.so.0 libtree-sitter-php.so.0.0) No such file or directory
Running 1 tests (2023-05-27 17:02:54+0000, selector ‘t’)
Warning (treesit): Cannot activate tree-sitter, because language grammar for php is unavailable (not-found): (libtree-sitter-php libtree-sitter-php.0 libtree-sitter-php.0.0 libtree-sitter-php.so libtree-sitter-php.so.0 libtree-sitter-php.so.0.0) No such file or directory

You can see it here in Run tests https://github.com/piotrkwiecinski/php-ts-mode/actions/runs/5099871411/jobs/9167833683

I have installed https://github.com/nektos/act to debug GH action locally and I'm able to reproduce an issue in the local container that I can access.

I tried running gdb on the container but there is no symbol table.
I assume I would have to get a nix VM to have a better control.

Any pointers were to go from here?
Is there a way to add debug symbols on top of what's currently compiled for CI containers?

Edit: I tested elixir-ts-mode with act locally and it runs correctly and I was their CI work as expected.

[purcell]

I haven't tried dynamically loading treesitter grammars so I'm not familiar with what Emacs is trying to do under the hood. Might you need to add ~/work/php-ts-mode/php-ts-mode/tree-sitter/ to $LD_LIBRARY_PATH?

[piotrkwiecinski]

@purcell I've tried to change the path, used treesit-additional-load-path, I tried to put grammar directly and nothing works.

I tried to run gdb in the local container but it doesn't work.

I have downloaded nix as a VM image and will try to debug it this way. Not sure if it's correct approach.
Should I use ubuntu and have nix package manager installed on top of it to simulate what happens in the pipeline?

[purcell]

Should I use ubuntu and have nix package manager installed on top of it to simulate what happens in the pipeline?

Yes, Nix on your machine is fine, no NixOS VM necessary. I don't have much advice on how to debug this, sorry. Note that in general, mixing non-Nix-built .so files with binaries that come with Nix is likely to result in mismatches — it could well be the case that the library is found but can't be loaded. Perhaps the .so links against the tree-sitter shared library, which is probably not visible on the load path within the Emacs process.

[piotrkwiecinski]

I got it working with the version from https://github.com/alexmurray/emacs-snap/tree/emacs-29 .
I'll try to recreate the issue locally with nix package manager as I would prefer to this repo for CI.

[akirak]

FYI, Nixpkgs on the latest master now contains support for adding tree-sitter grammars without rebuilding the entire Emacs (see NixOS/nixpkgs#230751). You could use the following shell.nix (proof-of-concept) to install Emacs with the PHP grammar:

let
  emacsCI = (pkgs.emacs.pkgs.overrideScope'
    (_: _: {
      emacs = emacs-ci.emacs-snapshot.overrideAttrs (_: {
        treeSitter = true;
      });
    }))
  .withPackages (epkgs: [
    (epkgs.treesit-grammars.with-grammars (grammars: [grammars.tree-sitter-php]))
  ]);
in
  pkgs.mkShell {
    buildInputs = [
      emacsCI
    ];
  }

This doesn't work right now due to an integration issue with withPackages wrapper, but I am working on the issue in #170. You also have to update nixpkgs used to build Emacs (niv update nixpkgs).

[akirak]

I have created emacs-php/php-ts-mode#20 as an example of custom Emacs build with tree-sitter grammars. It works locally, but CI seems to fail without binary cache (operation cancelled).

[piotrkwiecinski]

@purcell and @akirak thank you very much for help. I'm going to close this issue.

[purcell]

Great. Thanks @akirak 🙏

[akirak]

Also thanks to @DamienCassou who has invented the mechanism of treesit-grammars. It's a great idea.