Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tree-sitter: Update grammars #232633

Merged
merged 2 commits into from
May 24, 2023
Merged

tree-sitter: Update grammars #232633

merged 2 commits into from
May 24, 2023

Conversation

ghostbuster91
Copy link
Member

@ghostbuster91 ghostbuster91 commented May 18, 2023

Description of changes

This PR updates the grammars as instructed by

# to update:
# 1) change all these hashes
# 2) nix-build -A tree-sitter.updater.update-all-grammars
# 3) OPTIONAL: Set GITHUB_TOKEN env variable to avoid api rate limit
# 4) run the ./result script that is output by that (it updates ./grammars)

Update

It turned out that tree-sitter-sql cannot be built anymore. The reason is that maintainers of that repository decided to move generated parser into a separate branch (see for details:DerekStride/tree-sitter-sql#100).

This PR allows for specifying an optional parameter branch when defining the list of tree-sitter grammars that will be used when fetching grammar parser.

Details: If branch key is specified then there is an additional call made to github api to fetch latest commit for the provided branch. It is then passed to nix-prefech-git as rev. Otherwise the behavior remains the same. I couldn't simply alter nix-prefetch-git invocation as the branch-name argument is only used there to checkout to a specific branch locally (without tracking the upstream).

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@figsoda
Copy link
Member

figsoda commented May 18, 2023

Result of nixpkgs-review pr 232633 run on x86_64-linux 1

2 packages failed to build:
  • diffsitter
  • tree-sitter-grammars.tree-sitter-sql
38 packages built:
  • emacsPackages.pygn-mode
  • emacsPackages.tree-sitter-ess-r
  • emacsPackages.tree-sitter-langs
  • tree-sitter-grammars.tree-sitter-c
  • tree-sitter-grammars.tree-sitter-clojure
  • tree-sitter-grammars.tree-sitter-comment
  • tree-sitter-grammars.tree-sitter-cpp
  • tree-sitter-grammars.tree-sitter-cuda
  • tree-sitter-grammars.tree-sitter-dart
  • tree-sitter-grammars.tree-sitter-devicetree
  • tree-sitter-grammars.tree-sitter-elisp
  • tree-sitter-grammars.tree-sitter-elixir
  • tree-sitter-grammars.tree-sitter-elm
  • tree-sitter-grammars.tree-sitter-glsl
  • tree-sitter-grammars.tree-sitter-haskell
  • tree-sitter-grammars.tree-sitter-html
  • tree-sitter-grammars.tree-sitter-http
  • tree-sitter-grammars.tree-sitter-janet-simple
  • tree-sitter-grammars.tree-sitter-json
  • tree-sitter-grammars.tree-sitter-ledger
  • tree-sitter-grammars.tree-sitter-llvm
  • tree-sitter-grammars.tree-sitter-lua
  • tree-sitter-grammars.tree-sitter-markdown
  • tree-sitter-grammars.tree-sitter-markdown-inline
  • tree-sitter-grammars.tree-sitter-nickel
  • tree-sitter-grammars.tree-sitter-nix
  • tree-sitter-grammars.tree-sitter-perl
  • tree-sitter-grammars.tree-sitter-php
  • tree-sitter-grammars.tree-sitter-query
  • tree-sitter-grammars.tree-sitter-rust
  • tree-sitter-grammars.tree-sitter-scheme
  • tree-sitter-grammars.tree-sitter-svelte
  • tree-sitter-grammars.tree-sitter-tiger
  • tree-sitter-grammars.tree-sitter-tlaplus
  • tree-sitter-grammars.tree-sitter-tsx
  • tree-sitter-grammars.tree-sitter-typescript
  • tree-sitter-grammars.tree-sitter-vim
  • tree-sitter-grammars.tree-sitter-zig

1 similar comment
@geri1701
Copy link
Member

Result of nixpkgs-review pr 232633 run on x86_64-linux 1

2 packages failed to build:
  • diffsitter
  • tree-sitter-grammars.tree-sitter-sql
38 packages built:
  • emacsPackages.pygn-mode
  • emacsPackages.tree-sitter-ess-r
  • emacsPackages.tree-sitter-langs
  • tree-sitter-grammars.tree-sitter-c
  • tree-sitter-grammars.tree-sitter-clojure
  • tree-sitter-grammars.tree-sitter-comment
  • tree-sitter-grammars.tree-sitter-cpp
  • tree-sitter-grammars.tree-sitter-cuda
  • tree-sitter-grammars.tree-sitter-dart
  • tree-sitter-grammars.tree-sitter-devicetree
  • tree-sitter-grammars.tree-sitter-elisp
  • tree-sitter-grammars.tree-sitter-elixir
  • tree-sitter-grammars.tree-sitter-elm
  • tree-sitter-grammars.tree-sitter-glsl
  • tree-sitter-grammars.tree-sitter-haskell
  • tree-sitter-grammars.tree-sitter-html
  • tree-sitter-grammars.tree-sitter-http
  • tree-sitter-grammars.tree-sitter-janet-simple
  • tree-sitter-grammars.tree-sitter-json
  • tree-sitter-grammars.tree-sitter-ledger
  • tree-sitter-grammars.tree-sitter-llvm
  • tree-sitter-grammars.tree-sitter-lua
  • tree-sitter-grammars.tree-sitter-markdown
  • tree-sitter-grammars.tree-sitter-markdown-inline
  • tree-sitter-grammars.tree-sitter-nickel
  • tree-sitter-grammars.tree-sitter-nix
  • tree-sitter-grammars.tree-sitter-perl
  • tree-sitter-grammars.tree-sitter-php
  • tree-sitter-grammars.tree-sitter-query
  • tree-sitter-grammars.tree-sitter-rust
  • tree-sitter-grammars.tree-sitter-scheme
  • tree-sitter-grammars.tree-sitter-svelte
  • tree-sitter-grammars.tree-sitter-tiger
  • tree-sitter-grammars.tree-sitter-tlaplus
  • tree-sitter-grammars.tree-sitter-tsx
  • tree-sitter-grammars.tree-sitter-typescript
  • tree-sitter-grammars.tree-sitter-vim
  • tree-sitter-grammars.tree-sitter-zig

@ghostbuster91
Copy link
Member Author

Result of nixpkgs-review pr 232633 run on x86_64-linux 1

40 packages built:
  • diffsitter
  • emacsPackages.pygn-mode
  • emacsPackages.tree-sitter-ess-r
  • emacsPackages.tree-sitter-langs
  • tree-sitter-grammars.tree-sitter-c
  • tree-sitter-grammars.tree-sitter-clojure
  • tree-sitter-grammars.tree-sitter-comment
  • tree-sitter-grammars.tree-sitter-cpp
  • tree-sitter-grammars.tree-sitter-cuda
  • tree-sitter-grammars.tree-sitter-dart
  • tree-sitter-grammars.tree-sitter-devicetree
  • tree-sitter-grammars.tree-sitter-elisp
  • tree-sitter-grammars.tree-sitter-elixir
  • tree-sitter-grammars.tree-sitter-elm
  • tree-sitter-grammars.tree-sitter-glsl
  • tree-sitter-grammars.tree-sitter-haskell
  • tree-sitter-grammars.tree-sitter-html
  • tree-sitter-grammars.tree-sitter-http
  • tree-sitter-grammars.tree-sitter-janet-simple
  • tree-sitter-grammars.tree-sitter-json
  • tree-sitter-grammars.tree-sitter-ledger
  • tree-sitter-grammars.tree-sitter-llvm
  • tree-sitter-grammars.tree-sitter-lua
  • tree-sitter-grammars.tree-sitter-markdown
  • tree-sitter-grammars.tree-sitter-markdown-inline
  • tree-sitter-grammars.tree-sitter-nickel
  • tree-sitter-grammars.tree-sitter-nix
  • tree-sitter-grammars.tree-sitter-perl
  • tree-sitter-grammars.tree-sitter-php
  • tree-sitter-grammars.tree-sitter-query
  • tree-sitter-grammars.tree-sitter-rust
  • tree-sitter-grammars.tree-sitter-scheme
  • tree-sitter-grammars.tree-sitter-sql
  • tree-sitter-grammars.tree-sitter-svelte
  • tree-sitter-grammars.tree-sitter-tiger
  • tree-sitter-grammars.tree-sitter-tlaplus
  • tree-sitter-grammars.tree-sitter-tsx
  • tree-sitter-grammars.tree-sitter-typescript
  • tree-sitter-grammars.tree-sitter-vim
  • tree-sitter-grammars.tree-sitter-zig

@geri1701
Copy link
Member

Result of nixpkgs-review pr 232633 run on x86_64-linux 1

40 packages built:
  • diffsitter
  • emacsPackages.pygn-mode
  • emacsPackages.tree-sitter-ess-r
  • emacsPackages.tree-sitter-langs
  • tree-sitter-grammars.tree-sitter-c
  • tree-sitter-grammars.tree-sitter-clojure
  • tree-sitter-grammars.tree-sitter-comment
  • tree-sitter-grammars.tree-sitter-cpp
  • tree-sitter-grammars.tree-sitter-cuda
  • tree-sitter-grammars.tree-sitter-dart
  • tree-sitter-grammars.tree-sitter-devicetree
  • tree-sitter-grammars.tree-sitter-elisp
  • tree-sitter-grammars.tree-sitter-elixir
  • tree-sitter-grammars.tree-sitter-elm
  • tree-sitter-grammars.tree-sitter-glsl
  • tree-sitter-grammars.tree-sitter-haskell
  • tree-sitter-grammars.tree-sitter-html
  • tree-sitter-grammars.tree-sitter-http
  • tree-sitter-grammars.tree-sitter-janet-simple
  • tree-sitter-grammars.tree-sitter-json
  • tree-sitter-grammars.tree-sitter-ledger
  • tree-sitter-grammars.tree-sitter-llvm
  • tree-sitter-grammars.tree-sitter-lua
  • tree-sitter-grammars.tree-sitter-markdown
  • tree-sitter-grammars.tree-sitter-markdown-inline
  • tree-sitter-grammars.tree-sitter-nickel
  • tree-sitter-grammars.tree-sitter-nix
  • tree-sitter-grammars.tree-sitter-perl
  • tree-sitter-grammars.tree-sitter-php
  • tree-sitter-grammars.tree-sitter-query
  • tree-sitter-grammars.tree-sitter-rust
  • tree-sitter-grammars.tree-sitter-scheme
  • tree-sitter-grammars.tree-sitter-sql
  • tree-sitter-grammars.tree-sitter-svelte
  • tree-sitter-grammars.tree-sitter-tiger
  • tree-sitter-grammars.tree-sitter-tlaplus
  • tree-sitter-grammars.tree-sitter-tsx
  • tree-sitter-grammars.tree-sitter-typescript
  • tree-sitter-grammars.tree-sitter-vim
  • tree-sitter-grammars.tree-sitter-zig

@geri1701 geri1701 added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label May 19, 2023
@figsoda
Copy link
Member

figsoda commented May 19, 2023

Instead of specifying a branch, we can also do this:

--- a/pkgs/development/tools/parsing/tree-sitter/default.nix
+++ b/pkgs/development/tools/parsing/tree-sitter/default.nix
@@ -62,12 +62,14 @@ let
           inherit version;
           src = grammar.src or (fetchGrammar grammar);
           location = grammar.location or null;
+          generate = grammar.generate or false;
         };
       grammars' = import ./grammars { inherit lib; } // extraGrammars;
       grammars = grammars' //
         { tree-sitter-ocaml = grammars'.tree-sitter-ocaml // { location = "ocaml"; }; } //
         { tree-sitter-ocaml-interface = grammars'.tree-sitter-ocaml // { location = "interface"; }; } //
         { tree-sitter-org-nvim = grammars'.tree-sitter-org-nvim // { language = "org"; }; } //
+        { tree-sitter-sql = grammars'.tree-sitter-sql // { generate = true; }; } //
         { tree-sitter-typescript = grammars'.tree-sitter-typescript // { location = "typescript"; }; } //
         { tree-sitter-tsx = grammars'.tree-sitter-typescript // { location = "tsx"; }; } //
         { tree-sitter-markdown = grammars'.tree-sitter-markdown // { location = "tree-sitter-markdown"; }; } //

@ghostbuster91
Copy link
Member Author

ghostbuster91 commented May 19, 2023

Instead of specifying a branch, we can also do this:

--- a/pkgs/development/tools/parsing/tree-sitter/default.nix
+++ b/pkgs/development/tools/parsing/tree-sitter/default.nix
@@ -62,12 +62,14 @@ let
           inherit version;
           src = grammar.src or (fetchGrammar grammar);
           location = grammar.location or null;
+          generate = grammar.generate or false;
         };
       grammars' = import ./grammars { inherit lib; } // extraGrammars;
       grammars = grammars' //
         { tree-sitter-ocaml = grammars'.tree-sitter-ocaml // { location = "ocaml"; }; } //
         { tree-sitter-ocaml-interface = grammars'.tree-sitter-ocaml // { location = "interface"; }; } //
         { tree-sitter-org-nvim = grammars'.tree-sitter-org-nvim // { language = "org"; }; } //
+        { tree-sitter-sql = grammars'.tree-sitter-sql // { generate = true; }; } //
         { tree-sitter-typescript = grammars'.tree-sitter-typescript // { location = "typescript"; }; } //
         { tree-sitter-tsx = grammars'.tree-sitter-typescript // { location = "tsx"; }; } //
         { tree-sitter-markdown = grammars'.tree-sitter-markdown // { location = "tree-sitter-markdown"; }; } //

That would be also possible. However, I think that the ability to chose branch might be handy so I would keep it the way it is, unless you have a strong preference toward generate.

@figsoda
Copy link
Member

figsoda commented May 19, 2023

It would be easier for me to review since it's simpler and I'm also not familiar with the update script, but either option would work

))
match curl_result(branch_out):
case {"commit": {"sha": commit_hash}}:
release = commit_hash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can just do f"refs/heads/{branch}" here without the extra api request, nix-prefetch-git still outputs a sha as rev as expected, using the command from nix_prefetch_git_args

$ nix-prefetch-git --quiet --no-deepClone --url https://github.com/derekstride/tree-sitter-sql --rev refs/heads/gh-pages
{
  "url": "https://github.com/derekstride/tree-sitter-sql",
  "rev": "721087c8819cda10ca37f974e914ab9be46b290f",
  "date": "2023-05-19T15:56:01+00:00",
  "path": "/nix/store/5mbri1m7rrwmvhzp5nz3bs1pdzj87fni-tree-sitter-sql",
  "sha256": "10njfi84b78z036bkgbrxnxgy9av9yprxq1p2zla87w0g6ixqva7",
  "fetchLFS": false,
  "fetchSubmodules": false,
  "deepClone": false,
  "leaveDotGit": false
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, great :) this will simplify things.

@ghostbuster91
Copy link
Member Author

Result of nixpkgs-review pr 232633 run on x86_64-linux 1

41 packages built:
  • diffsitter
  • emacsPackages.pygn-mode
  • emacsPackages.tree-sitter-ess-r
  • emacsPackages.tree-sitter-langs
  • tree-sitter-grammars.tree-sitter-c
  • tree-sitter-grammars.tree-sitter-clojure
  • tree-sitter-grammars.tree-sitter-comment
  • tree-sitter-grammars.tree-sitter-cpp
  • tree-sitter-grammars.tree-sitter-cuda
  • tree-sitter-grammars.tree-sitter-dart
  • tree-sitter-grammars.tree-sitter-devicetree
  • tree-sitter-grammars.tree-sitter-elisp
  • tree-sitter-grammars.tree-sitter-elixir
  • tree-sitter-grammars.tree-sitter-elm
  • tree-sitter-grammars.tree-sitter-glsl
  • tree-sitter-grammars.tree-sitter-haskell
  • tree-sitter-grammars.tree-sitter-html
  • tree-sitter-grammars.tree-sitter-http
  • tree-sitter-grammars.tree-sitter-janet-simple
  • tree-sitter-grammars.tree-sitter-json
  • tree-sitter-grammars.tree-sitter-ledger
  • tree-sitter-grammars.tree-sitter-llvm
  • tree-sitter-grammars.tree-sitter-lua
  • tree-sitter-grammars.tree-sitter-markdown
  • tree-sitter-grammars.tree-sitter-markdown-inline
  • tree-sitter-grammars.tree-sitter-nickel
  • tree-sitter-grammars.tree-sitter-nix
  • tree-sitter-grammars.tree-sitter-perl
  • tree-sitter-grammars.tree-sitter-php
  • tree-sitter-grammars.tree-sitter-query
  • tree-sitter-grammars.tree-sitter-rust
  • tree-sitter-grammars.tree-sitter-scala
  • tree-sitter-grammars.tree-sitter-scheme
  • tree-sitter-grammars.tree-sitter-sql
  • tree-sitter-grammars.tree-sitter-svelte
  • tree-sitter-grammars.tree-sitter-tiger
  • tree-sitter-grammars.tree-sitter-tlaplus
  • tree-sitter-grammars.tree-sitter-tsx
  • tree-sitter-grammars.tree-sitter-typescript
  • tree-sitter-grammars.tree-sitter-vim
  • tree-sitter-grammars.tree-sitter-zig

@figsoda figsoda merged commit 946a4d6 into NixOS:master May 24, 2023
@ghostbuster91 ghostbuster91 deleted the nix-master branch May 24, 2023 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants