feat: poc octo run list #638

GustavEikaas · 2024-10-15T16:10:05Z

Important

This is a draft PR
Im making this draft PR so I can get reviews/suggestions during development
I will fill out the PR template when the PR is ready for review

Resolves: #149

Describe what this PR does / why we need it

First PR in a series of PRs to add support for github actions

Does this pull request fix one issue?

Describe how you did it

Describe how to verify it

Special notes for reviews

Checklist

Passing tests and linting standards
Documentation updates in README.md and doc/octo.txt

Feature checklist

Remaining

Customizable keymappings
Customizable icons
Update readme
Cleanup code
Customizable refresh interval
~~Look into using graphql in favor of gh cli commands (not sure if strictly necessary but conforms to the repo standard)~~
Use telescope pickers from the repo
Understand the writers.lua file and extend it
Use existing octo window creator

wd60622 · 2024-10-17T17:54:20Z

Using graphql is not a big deal. Just use whatever endpoint that works. Think there are a few non graphql commands used.

GustavEikaas · 2024-10-17T18:30:36Z

Ah okay, functionality wise, do you have any feedback?

wd60622 · 2024-10-17T21:08:07Z

Ah okay, functionality wise, do you have any feedback?

I will give it a try when I can!

wd60622 · 2024-10-18T02:29:26Z

Think it looks good. Would you want to make use of the pickers instead of the new, custom buffer? I have been using telescope recently. What do you use?

Some items:

What do you expect the callback to be upon selection an item? We can work something in like Abitrary callbacks for pickers #642 in order to support more in the future.
Would there be a preview or buffer to show each item? What would be populated there? Maybe a buffer doesn't make sense here. What are your thoughts?

lua/octo/workflow_runs.lua

GustavEikaas · 2024-10-18T05:48:01Z

@wd60622 Is this what you meant, kinda like the PR and issue flow?

GustavEikaas · 2024-10-18T06:08:07Z

You still want there to be a buffer when you press <CR> on one of the items in the picker. AKA same flow as pull request?

pwntester · 2024-11-05T09:14:29Z

Thanks for the PR @GustavEikaas!

I like the idea of showing the status of the run in the preview buffer and perhaps show the logs of the run upon opening the item with <CR>. The problem I see is that a run may contain multiple jobs and each job will have its own log. Should we present them sequentially? thoughts?

GustavEikaas · 2024-11-05T10:29:53Z

IIRC the sequential jobs is already supported but yes I was thinking the same.
For the logs of every step we should probably lazy load and use fold or something similiar. Some steps have very long log outputs

GustavEikaas · 2024-11-06T17:58:45Z

@pwntester
Here is an image of what it looks like with multiple jobs

GustavEikaas · 2024-11-06T18:38:44Z

Doesnt seem like there is a good structured output for workflow logs. Will probably be tricky to have the same UI as github web.
Only way I see right now is query the workflow logs and using regex to match the lines to the corresponding steps

pwntester · 2024-11-06T21:13:48Z

Nice, perhaps we can do a nested telescope selection where the use first chooses the run and then the job, that should get us a shorter log to show to the user and it should easier to print since we dont need to care about job's dependencies, just sequential steps

GustavEikaas · 2025-01-16T18:43:58Z

Finally found it. Its a POSIX quirk. From the lua docs for os.tmpname

In POSIX systems, this function also creates a file with that name, to avoid security risks. (Someone else might create the file with wrong permissions in the time between getting the name and creating the file.) You still have to open the file to use it and to remove it (even if you do not use it).

GustavEikaas · 2025-01-16T18:49:50Z

Latest commit works fine on arch (btw), and windows. Should work just fine for you now @wd60622.
What weird behaviour for os.tmpname....

wd60622 · 2025-01-16T19:01:26Z

Thanks for the updates. will give it a try when I can

wd60622 · 2025-01-25T09:24:42Z

Okay. I am able to get it to work (for a few) now. I am still getting this traceback:

E5108: Error executing lua: Vim:E484: Can't open file vpavbvq/X64-ubuntu (ubuntu-20.04, https:/github.com/neovim/neovim/releases/download/v0.7.0/n
vim-linux64..../1_Set up job.txt
stack traceback:
        [C]: in function 'readfile'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:341: in function 'cb'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:208: in function 'traverse'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:210: in function 'traverse'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:210: in function 'traverse'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:333: in function 'get_logs'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:390: in function 'cb'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:611: in function <...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:
607>

Maybe there is something with the filename then.

Overall it looks really awesome @GustavEikaas. Thanks for the final push here!

GustavEikaas · 2025-01-25T09:50:18Z

Yeah ill take a last look at escaping/normalizing filenames

wd60622 · 2025-01-25T10:05:32Z

Yeah ill take a last look at escaping/normalizing filenames

Great. Thank you!

wd60622 · 2025-01-25T10:22:04Z

I am also noticing that the folders are being created off of my cwd. Is that as expected?

GustavEikaas · 2025-01-25T11:31:19Z

Good catch ill fix that, happened after i migrated away from os.tmpname.

GustavEikaas · 2025-01-25T16:00:12Z

Okay. I am able to get it to work (for a few) now. I am still getting this traceback:

Im not sure if I fixed the same issue you were facing but for me it would fail if it tried to read the logs of a step that was skipped. This was because the .txt file is missing from the zip archive. I added a check so that skipped steps are not assigned logs.

I am also noticing that the folders are being created off of my cwd. Is that as expected?

Fixed.

Encountered a strange scenario I have never seen before. In a workflow from Neovim repo it failed to load because workflow.jobs == nil. After opening it in browser this is what I see.

Given the data available to me this is the best I can do. Clearly the gh api is not as powerful as the web UI. But it will hopefully improve over time.

GustavEikaas · 2025-01-25T16:03:17Z

Still a couple of rough edges in this PR but I think its getting ready to merge. Given that it stops giving you errors

For any future issues/prs/discussions feel free to tag me, I would like to keep working on this

wd60622 · 2025-01-25T19:53:15Z

I'm having a lot better luck with it now. Really cool feature.

I am running into this:

                                                                                                                                1,15          All
E5108: Error executing lua: ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:597: 'replacement string' item contains newlines
stack traceback:
        [C]: in function 'nvim_buf_set_lines'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:597: in function 'print_lines'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:623: in function 'refresh'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:525: in function 'populate_preview_buffer'
        ...tHub/neovim-plugins/octo.nvim/lua/octo/workflow_runs.lua:680: in function 'define_preview'
        ...scope.nvim/lua/telescope/previewers/buffer_previewer.lua:451: in function 'preview'
        ...share/nvim/lazy/telescope.nvim/lua/telescope/pickers.lua:1212: in function 'refresh_previewer'
        ...share/nvim/lazy/telescope.nvim/lua/telescope/pickers.lua:1153: in function 'set_selection'
        ...share/nvim/lazy/telescope.nvim/lua/telescope/pickers.lua:912: in function 'move_selection'
        ...e/nvim/lazy/telescope.nvim/lua/telescope/actions/set.lua:39: in function 'run_replace_or_original'
        ...re/nvim/lazy/telescope.nvim/lua/telescope/actions/mt.lua:65: in function 'shift_selection'
        .../nvim/lazy/telescope.nvim/lua/telescope/actions/init.lua:87: in function 'run_replace_or_original'
        ...re/nvim/lazy/telescope.nvim/lua/telescope/actions/mt.lua:65: in function 'key_func'
        ...hare/nvim/lazy/telescope.nvim/lua/telescope/mappings.lua:293: in function <...hare/nvim/lazy/telescope.nvim/lua/telescope/mappings.lua:
292>

Also, could you bind to the buffer to get the URL of the run? Think that could be pretty convenient. Or at least in the picker if it isn't already!

GustavEikaas · 2025-01-25T20:20:06Z

I did bind <C-b> open in browser for the buffer. I can do <C-y> aswell.

Are you testing against open repositories, can you link a workflow run that fails? Seems like it complains about \n in a line being printed. Not really sure why that would happen, weird... Did it happen when printing the workflow or when getting the workflow step logs?

GustavEikaas · 2025-01-26T12:19:16Z

I managed to repro your issue @wd60622 and I might have found a bug in github...
when downloading the archive from workflow
I get this. But when trying to cd into the directory I get this error

Not really sure how to proceed here. Will make an issue with github to gather some more info. I enabled longpaths on my windows system and still not working

EDIT: Did some more digging and it seems like unzip is the reason it fails. When unzipping the folder with the long path name it just corrupts... Will do more digging

EDIT2
Finally got it working but jesus there are a lot of inconsistencies in github apis. When querying the workflow I get different characters than when querying wf logs. It aint great but its working. I also couldnt unzip the archive with unzip without corrupting the directory?? Ended up rewriting the logic to extract single files into stdout per workflow step

    local sanitized_name = node.id:gsub("/", ""):gsub(":", ""):gsub(">","")
    --Make more than 3 consecutive dots at the end of line into ...
    local sanitized_job_id = node.job_id:gsub("/", ""):gsub(":", ""):gsub("%.%.%.%.+$", "...")

wd60622 · 2025-01-29T16:13:49Z

Was still having issues with this. Does it have something to do with the file / folder names. Should we use a hash or something?

GustavEikaas · 2025-02-02T09:55:56Z

Darn, really?

Well the problem is I do api calls to get the workflow and based on the dynamic info there I request an archive with an arbitary number of files with names that can be built up using the workflow but with x amount of replaced characters. Ill do some more digging

wd60622 · 2025-02-02T13:18:28Z

Thanks for all of the digging. Feel free to ping me to try again.

Maybe @GuillaumeLagrange can try as well!

GustavEikaas · 2025-02-02T13:33:56Z

Ill keep digging! It would be easier if anyone could link me a specific workflow run that causes it to error out

lua/octo/workflow_runs.lua

wd60622 · 2025-02-20T06:54:13Z

lua/octo/workflow_runs.lua

+    M.refresh()
+    return
+  end
+  local cmd = string.format("gh run view %s --json %s", id, fields)


Possible to use existing code here?

Might look like this:

gh = require "octo.gh" gh.run.view { id, json=fields, opts = { cb = function(output, stderr) end, } }

You mean like this?
Or did you want me to create the gh.run.view?

gh.run { args = { "run", "view", id, "--json", fields }, cb = function(output, stderr) if stderr and not utils.is_blank(stderr) then vim.api.nvim_err_writeln(stderr) octo_error("Failed to get workflow run for " .. id) elseif output then job_details = vim.fn.json_decode(output) M.wf_cache[id] = job_details M.current_wf = job_details M.tree = generate_workflow_tree(job_details) M.refresh() end end, }

Arbitrary CLI commands are already supported with this logic:

octo.nvim/lua/octo/gh/init.lua

Lines 329 to 367 in c689dd5

local create_subcommand = function(command)

local subcommand = {}

subcommand.command = command

setmetatable(subcommand, {

__index = function(t, key)

return function(opts)

opts = opts or {}

local run_opts = opts.opts or {}

local args = {

t.command,

key,

}

opts.opts = nil

args = M.insert_args(args, opts, { ["_"] = "-" })

return M.run {

args = args,

mode = run_opts.mode,

cb = run_opts.cb,

stream_cb = run_opts.stream_cb,

headers = run_opts.headers,

hostname = run_opts.hostname,

}

end

end,

})

return subcommand

end

setmetatable(M, {

__index = function(_, key)

return create_subcommand(key)

end,

})

So no need to write it. Just use what I wrote and fill in the callback

wd60622

Some comments.

Also, been thinking about the use of zip files (Still don't have working in all cases for me)...
Do the logs have to stored to file system? There are other items that are just stored in the local variable M. Can that strategy also be used?

wd60622 · 2025-02-20T20:44:52Z

lua/octo/workflow_runs.lua

+end
+
+M.list = function()
+  vim.notify "Fetching workflow runs (this may take a while) ..."


Use utils.info over vim.notify

wd60622 · 2025-02-20T20:45:35Z

lua/octo/workflow_runs.lua

+  local co = coroutine.running()
+  local wf_runs = get_workflow_runs_sync(co)
+
+  require("octo.pickers.telescope.provider").workflow_runs(wf_runs, "Workflow runs", render)


Use require "octo.picker" instead. Feel free to have M.not_implemented for the other two pickers

wd60622 · 2025-02-20T20:46:37Z

lua/octo/workflow_runs.lua

+local function get_workflow_runs_sync(co)
+  local lines = {}
+  vim.fn.jobstart(
+    "gh run list --json conclusion,displayTitle,event,headBranch,name,number,status,updatedAt,databaseId -L "


Possible to use gh.run / gh.run.list here as well?
Using that ensures that the environment variables are correct

GustavEikaas added 4 commits October 15, 2024 18:03

feat: poc octo run list

f109590

feat: define octo key mappings

b34746c

feat: add customizable icons

75132dd

.

c45df2d

GustavEikaas mentioned this pull request Oct 15, 2024

Feature Request - GitHub actions #149

Open

GustavEikaas added 4 commits October 15, 2024 19:11

revert lint changes

a7585c3

fix lint

af6b3b2

feat: customizable refresh interval

71160fc

add space between jobs

86f2989

wd60622 added the enhancement New feature or request label Oct 17, 2024

wd60622 reviewed Oct 18, 2024

View reviewed changes

lua/octo/workflow_runs.lua Outdated Show resolved Hide resolved

feat: use telescope

89adabe

Merge branch 'master' into feat/octo-run-list

aabb271

Merge branch 'master' into feat/octo-run-list

fa1e079

GustavEikaas added 2 commits November 6, 2024 19:05

open buffer when pressing <cr>

4ce3249

.

4015bb0

GustavEikaas and others added 3 commits November 17, 2024 12:34

get step logs for single job workflows

e078e9c

Merge branch 'master' into feat/octo-run-list

cbf277a

multi-job-support

40009b7

GustavEikaas and others added 3 commits January 16, 2025 20:48

fix: rand filename

c63226e

feat: improve random name seeding

d5dee36

Merge branch 'master' into feat/octo-run-list

f772037

fix: skip logs for skipped steps

b4c3f47

feat: show failed and skipped runs in telescope

3cdebb2

fix: workflow with long names not working

f8b1618

wd60622 requested a review from GuillaumeLagrange February 2, 2025 13:18

wd60622 reviewed Feb 20, 2025

View reviewed changes

lua/octo/workflow_runs.lua Outdated Show resolved Hide resolved

wd60622 reviewed Feb 20, 2025

View reviewed changes

GustavEikaas added 3 commits February 20, 2025 20:52

merge upstream

d60ee6d

use utils.get_remote_name

511953c

use existing octo.gh

996cd29

wd60622 reviewed Feb 20, 2025

View reviewed changes

	local create_subcommand = function(command)
	local subcommand = {}
	subcommand.command = command

	setmetatable(subcommand, {
	__index = function(t, key)
	return function(opts)
	opts = opts or {}

	local run_opts = opts.opts or {}

	local args = {
	t.command,
	key,
	}

	opts.opts = nil
	args = M.insert_args(args, opts, { ["_"] = "-" })

	return M.run {
	args = args,
	mode = run_opts.mode,
	cb = run_opts.cb,
	stream_cb = run_opts.stream_cb,
	headers = run_opts.headers,
	hostname = run_opts.hostname,
	}
	end
	end,
	})

	return subcommand
	end

	setmetatable(M, {
	__index = function(_, key)
	return create_subcommand(key)
	end,
	})

feat: poc octo run list #638

Are you sure you want to change the base?

feat: poc octo run list #638

Conversation

GustavEikaas commented Oct 15, 2024 • edited Loading

Describe what this PR does / why we need it

Does this pull request fix one issue?

Describe how you did it

Describe how to verify it

Special notes for reviews

Checklist

Feature checklist

Remaining

wd60622 commented Oct 17, 2024

GustavEikaas commented Oct 17, 2024

wd60622 commented Oct 17, 2024

wd60622 commented Oct 18, 2024 • edited Loading

GustavEikaas commented Oct 18, 2024

GustavEikaas commented Oct 18, 2024

pwntester commented Nov 5, 2024

GustavEikaas commented Nov 5, 2024

GustavEikaas commented Nov 6, 2024

GustavEikaas commented Nov 6, 2024

pwntester commented Nov 6, 2024

GustavEikaas commented Jan 16, 2025

GustavEikaas commented Jan 16, 2025 • edited Loading

wd60622 commented Jan 16, 2025

wd60622 commented Jan 25, 2025 • edited Loading

GustavEikaas commented Jan 25, 2025

wd60622 commented Jan 25, 2025

wd60622 commented Jan 25, 2025

GustavEikaas commented Jan 25, 2025

GustavEikaas commented Jan 25, 2025 • edited Loading

GustavEikaas commented Jan 25, 2025

wd60622 commented Jan 25, 2025

GustavEikaas commented Jan 25, 2025 • edited Loading

GustavEikaas commented Jan 26, 2025 • edited Loading

wd60622 commented Jan 29, 2025

GustavEikaas commented Feb 2, 2025

wd60622 commented Feb 2, 2025 • edited Loading

GustavEikaas commented Feb 2, 2025

wd60622 Feb 20, 2025

Choose a reason for hiding this comment

GustavEikaas Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

wd60622 Feb 20, 2025

Choose a reason for hiding this comment

wd60622 left a comment

Choose a reason for hiding this comment

wd60622 Feb 20, 2025

Choose a reason for hiding this comment

wd60622 Feb 20, 2025

Choose a reason for hiding this comment

wd60622 Feb 20, 2025

Choose a reason for hiding this comment

GustavEikaas commented Oct 15, 2024 •

edited

Loading

wd60622 commented Oct 18, 2024 •

edited

Loading

GustavEikaas commented Jan 16, 2025 •

edited

Loading

wd60622 commented Jan 25, 2025 •

edited

Loading

GustavEikaas commented Jan 25, 2025 •

edited

Loading

GustavEikaas commented Jan 25, 2025 •

edited

Loading

GustavEikaas commented Jan 26, 2025 •

edited

Loading

wd60622 commented Feb 2, 2025 •

edited

Loading

GustavEikaas Feb 20, 2025 •

edited

Loading