Cannot select with ViraServers

[chinwobble]

Hey this looks like a really cool plugin.

I've read the entire readme.md but I cannot find how to browse jira issues.

I've created the two json files and tested my username and token are working properly.

I've created the following files according to
#41

~/.config/vira/vira_project.json
~/.config/vira/vira_servers.json

I have installed the python dependencies and I have python3 and python in my enviornment paht.
I'm using ubuntu 20.04 in WSL

Here's what I have tried:

1. open vim
2. :ViraServers
3. hover over https://bob.atlassian.net/ and press ` (backtick)
4. I get the error E20 mark not set
5. I try press enter but vim hangs for a long time and I end up closing the terminal after several minutes

[n0v1c3]

Hello I am glad to hear you are interested. Logging in is always the most fun one.

What was the backtick for? Have you tried the dev branch?

I just want to clear those up then we can dig in further.

[n0v1c3]

If you are using the __default__ it should auto connect to the server as well.

[n0v1c3]

You should simply be able to not press the backtick. It is looking for a non existent mark. I will add something in to override the error as I can reproduce it just as easily.

For now you should simply be able to not press the backtick. Let me know if that is all it was.

I will keep it open to add a patch in to help prevent it for others.

[n0v1c3]

Also, I assume you are using the cloud service and not a private server? If so make sure you are using the Cloud Password in the readme as you will not use your standard password.

Check also with a raw password entry vs the CMD if there is a difference there it should have been fixed or at least added to my VIRA issues.

[chinwobble]

I'm using dev branch with an atlassian cloud jira account. i.e. company.atlassian..net

for the password I use the token I generated.

I will try out suggestions and get back to you/

[n0v1c3]

The raw password entry will still be the token just don't use the password_cmd mapping just use the raw password mapping. That one will be my best first guess or we will need some actual digging.

[chinwobble]

still not working for me. Here is the full steps:

1. I'm on latest dev branch
2. create ~/.config/vira/vira_servers.json
3. Set the contents to

{
  "https://company.atlassian.net": {
    "username": "my@company.com",
    "password": "mysecurepassword",
    "skip_cert_verify": true
  }
}

I tested these credentials using curl they work.
I curl the confluence api using basic authentication https://company.atlassian.net/wiki/rest/api/content and it returns results.

4. open vim and run :ViraServers and it still hangs.

[chinwobble]

Can you tell me which version of the Jira python package you have installed?
When I run this in console, it hangs. It might be stuck waiting for user input?

:python3 Vira.api.connect('https://company.atlassian.net')

I have these python packages installed globally

appdirs==1.4.4
black==20.8b1
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
cliapp==1.20180812.1
click==7.1.2
cmdtest==0.32+git
cryptography==3.3.1
defusedxml==0.6.0
distlib==0.3.1
filelock==3.0.12
greenlet==0.4.15
idna==2.10
jira==2.0.0
Markdown==3.1.1
msgpack==0.6.2
mypy-extensions==0.4.3
oauthlib==3.1.0
packaging==20.3
pathspec==0.8.1
pbr==5.5.1
pipenv==2020.11.15
pycparser==2.20
Pygments==2.3.1
PyJWT==2.0.1
pynvim==0.4.2
pyparsing==2.4.6
PyYAML==5.3.1
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
six==1.14.0
toml==0.10.2
ttystatus==0.38
typed-ast==1.4.2
typing-extensions==3.7.4.3
urllib3==1.26.3
virtualenv==20.4.2
virtualenv-clone==0.5.4

[chinwobble]

@n0v1c3 I figured out the issue.
My JIRA cloud account has 12 projects and thousands of fix versions.

https://github.com/n0v1c3/vira/blob/dev/python/Vira/vira_api.py#L826

        # Loop through each project and all versions within
        for p in projects:
            for v in reversed(self.jira.project_versions(p)):
                self.version_percent(p, v)  # Add and update the version list
        return self.versions  # Return the version list

When running :python3 Vira.api.connect("https://company.atlassian.net")

This isn't respecting the filters. So its taking 30 mins to loop through everything.

I set my vira_projects.json to this but it isn't taking any affect

{
  "__default__": {
    "server": "https://company.atlassian.net"
  },
  "vira": {
    "template": "__default__",
    "issueSort": "status",
    "filter": {
      "project": "WEB"
    }
  }
}

[n0v1c3]

what folder are you in? Vira needs to replaced with that folder name for auto filter.

I would start with everything in default or something simple like assignee or in this.

default /server and assignee. I am guessing you do not have millions in your name as assignee.

I will need to relook at it though as we should also have a query limit.

[n0v1c3]

I will also have a good look at this one as I assume your projects have a lot of versions. When this was created we had to go in a back door and not the nice clean filter on the version side like we do on the project side.

[n0v1c3]

Enough digging aside shows that I brock it in VIRA-142.

I know exactly what is going on and this is now the highest priority.

[chinwobble]

Yea it looks like when you call Vira.api.connect() the code hasn't loaded the vira_projects.json file yet.

Is your vira JIRA project public?

[n0v1c3]

Yea the connect is the issue if you comment out the line 183 self.versions = self.getversions() I think this will fix it for you. At least for the initial login.

The real bug is in the function itself.

[n0v1c3]

This is my I paid 10 dollars for account with JIRA and only have a few left out of my 10 so far. I have also open a couple of the free ones and prepared to transfer this one over with their offer although, my server has now been up for 385 days and it will feel not as important to keep my server running every day.

[n0v1c3]

The core of the problem is that I will need to make this into a single query the filters are part of the issue but it is also running a query for each project and version.

Officially VIRA-247: percent complete lag

[n0v1c3]

I did a push on that branch for the initial login test. It looks to me like the bug will exist in the list of versions as well as create an issue. I can easily remove it from create an issue by simply not listing them at the bottom. The menu is the main objective as we do wish to use them in the create issue window as well in the future.

[n0v1c3]

I am also curious about how many users you have roughly and if any improvements will need to be maid there.

[chinwobble]

Can we load versions / users asynchronously?

So I imagine this:

1. type `:ViraIssue` new buffer is created with scaffold of JIRA
2. load the versions / users in the background
3. user types in summary and description
4. After a few seconds the users / versions are finished loaded and the list is populated

[n0v1c3]

https://vi.stackexchange.com/questions/27003/how-to-start-an-async-function-in-vim-8

This is very interesting to me and will create a lot of work for amazing features if it can work.

[n0v1c3]

I have started there and kept on learning and have my first fake async working.

I will write an official push for it and let you know when to give it the first official user trial for me.

[n0v1c3]

@chinwobble Some pushes have started to come to VIRA-247 I need to do some menu front-end clean up and some confirmation that it is not running way too much on the system.

[n0v1c3]

I have got the front-end cleaned I need to do some final testing for the single query at a time then I will pass the testing on to you.

[n0v1c3]

It is now a single query at a time I simply need to clean up the query as there is a mix of projects and versions that will not exist, ie project A has version 1.0.0 and B is only at 0.2.5 A and B will each currently show 1.0.0 and 0.2.5

[chinwobble]

thanks let me know what features you want me to test.
I assume it will be in dev branch.

[n0v1c3]

@chinwobble

I am down to "one" bug now. The core of the feature is testable.

Using branch VIRA-247

• Connect to your Jira server
• Quickly open the version filter window ViraFilterVersions
• Have a look and close that window quickly
• Open that window again and have a look and close it
• Once there is an error message at the bottom it is done
• You will have an error message from me that won't go away (working on it)
• Filter Filters are not 100%. (if you have selected a project this will not help yet)

I am most interested in how long it takes for you to get an error message as this tells me how long it takes to query all versions one at a time. How many versions (lines in that menu may be helpful as well)

We can also change a variable in autoload/vira.vim. It is basically taking any spare time to run queries. Once I fix that error message that will only be for the first round. Let me know if you have any CPU affects I did through development but it seems nice to me now.

Thank you very much!

[n0v1c3]

Sorry that variable is s:vira_async_timer = 5

That is in ms but it also must wait to complete and in line for other tasks that are in line. If you have a lot lot of versions there may be some "stuttering" that I did not see even with test at my larger jobs.

[n0v1c3]

OK I have got the core fixed with one more thing to learn for vim to make it amazing. Once I made it work properly I did need to change the timer but this is ONLY for thing like "holding" down a key to do multiple thing ie. I was being lazy and held down j but you will see it jump every 500ms (ish) can you guess what I changed the timer too?

It is not a huge deal and you wont notice it for most however this is a query and when it gets a turn it takes time. That is when we see that vim is truly NOT async!

s:vira_async_timer is still worth playing with as 500 ms may be very slow so I will look into further ways to hide this.

[n0v1c3]

It is now OFFICIALLY working from my prospective. Once you connect you should have a full list quite quickly and then it will get out of the way of vim by slowing itself down for future updates. (I will still need to work on the 'future updates' idea I have).

My work to do.

• Check initial lag with largest server connections possible.
• Ensure future updates on the fly for the project and version menu
• Apply this to the users list I needed to hack for full user details.
• Fix my filter filters

[n0v1c3]

Number of users will also require a similar hacking as we are doing the same thing. There is a line in the python connect function for the initial users list. I have temp helped @maricn on that on although you will not be able to use the user menus.

Since it is basically the same issue as this was it is very high priority.

[n0v1c3]

self.users = self.get_users()

[chinwobble]

I'm getting this error because that case isn't handled.

[chinwobble]

Also I did some testing. Since the code isn't really async and it just something to handle timeouts.
Would it be easier to do the async logic in python?

import asyncio
async def _get_versions():
    print('started')
    # fetch from the server
    await asyncio.sleep(5)
    print('ending')

async def _run_task_with_timeout(func, timeout):
    try:
        await asyncio.wait_for(func(), timeout)
    except asyncio.TimeoutError:
        print(f"operation timed out after {timeout} seconds")

async def get_versions(): await _run_task_with_timeout(_get_versions, 2)

I tested this logic with :py3 import asyncio; asyncio.run(Vira.main()) and it seems more readable than using viml to handle async.
See here.
https://docs.python.org/3/library/asyncio-task.html#running-in-threads

[n0v1c3]

Do you have deleted projects?

I very much like going full async into the python side. The variables were all about to move into python anyways as they have no place in vim other than my initial round. All vim should need to do is check a stable variable from python that updates itself amazingly fast I am sure.

Very nice one. Now I have asyncish vim and async python that should be a great combination in the very end of a front end with no async features.

[n0v1c3]

FYI we do have "long long" term plans that vim is just one front end option for the amazing backend python plugin. This helps expand usage for all.

Python will have its core operations with jira code separated completely with the breakdown. Will will then just make our print and vim_print functions.

[n0v1c3]

And of course the queries should have been 100% background. I will save may secret vim async so that your reports can do live updates for you on the screen. After you have a stable connection of course.

[n0v1c3]

@chinwobble I have done a decent push on VIRA-247 it should have the project/version side not holding up your system at all. I do have a few more functions to apply async to but this should make the tool much easy to initially load.

I will be adding the users to that branch as well as I assume that is a similar delay for all larger servers. Once you approve this from your POV I will get it to the dev branch as I know I have more testers there and I will find anything else we should put into this process while in there.

[maricn]

@n0v1c3 Hm, strange, now I get an error trying to execute :ViraFilterProjects:

UPDATE: Actually, whatever :Vira* I try to execute I get that error.

UPDATE2: I tried restarting nvim and running :ViraServers worked to return two lines - my server and Null.. But, still doing <CR> on any, and following up with any command produces the same error.

UPDATE3: My healthcheck is good:

[n0v1c3]

Yes I had just seen this one after my post last night as I was preparing for a dev/master merge with a test.

It works fine in vim but not in nvim... That must be a first to have multitasking in vim and not nvim.

I had a rough idea of where the issue is just not a clue yet what the issue is.

The first couple async functions seem to only be allowed to run once on nvim on my last closer look. I will focus on nvim this evening (Canada time) as once this passes I can spend more time behind the scenes to make it closer to an actual async along with make a big list of everything I want to have async abilities. Ugly tests show these menus and reports will become "live" inside vim.

[n0v1c3]

There is still a non-async function YET that may cause a small "bounce" ever 2 seconds. I am interested in hearing about that from your perspective.

This bounce will be the next attack on the list once nvim passes. It should be a small enough bounce that you will only notice it if you are holding a key down like I do with l and h to test it. I cannot see it with my normal vim usage.

[n0v1c3]

If you comment out the aysnc function and call to it in autoload it should work everywhere but the versions for you.

Edit

I am sure I can stand corrected on that.

[maricn]

If you comment out the aysnc function and call to it in autoload it should work everywhere but the versions for you.

I'm not that comfortable with the code base here, and I've never written a vim plugin myself, but I'll check it out more these days. Now is past my bedtime, so maybe I hear back from you before I get to it. Anyways, thanks for looking into it and sorry for hijacking this issue (i realized late i replied on someone else's issue).

[n0v1c3]

neovim/neovim#3844

Above is the core of the issue. I should be able to avoid this should already not be patched however it is crashing when my async array has blank strings in it... I will do a quick test with a letter stamp in the array as a protector if that works it might just be my work around until I give my mind that break.

[n0v1c3]

neovim/pynvim#342

Even more important and more to come I am sure.

[n0v1c3]

The bounciness you see appears to be an issue directly inside how neovim handles multitasking. Let me know if you can see a difference after I get my next push out here.

The key will be that vim and neovim cannot handle true multitasking. Also neovim appears to be behind with python/asyncio workarounds for code simplicity.

I believe for now this will be the best I am able to do especially with the JQL as the wall I will look closer at neovim and potentially open an issue there about how semi-multi tasking is not handled very well. The vim way of multitasking is essentially the only way it has and I can not tell the difference between python and vim methods for pretending to multitask.

[n0v1c3]

I will also have a nice gab (2 seconds) for the async tasks untill all things have been completed once then I can slow it down to only update once every (60 secods) this is per single item query. This way once vira has had one inital wave through the reast of its updates can slow way down.

This 2 seconds is probably the important part for your quality control test at this point of what I can do for neovim.

[n0v1c3]

@chinwobble and @maricn, I may as well bring in @mikeboiko as well, I should be far enough with my pushes that I am simply at the endless cleanup but worth a good test as I believe this should make it to the master branch for our larger users as quickly as possible.

The key variables to test for a default on the release:
g:vira_async_timer and g:vira_async_timer_init with 10000 and 2000 (ms) default. I must say I can handle the bounce on my own for a much lower default on the init (I have set my own in .vimrc to 1) this will get you an initial complete list very quickly and then should slow down to your other timer.

Please let me know if let g:vira_async_timer_init = 1 is WAY to painful as a default for your larger project version control. We are looking for that perfect balance in between for fast updates with no "lag".

[chinwobble]

@n0v1c3 Hey thanks for all your hard work. I just tested

• :ViraServers no longer hangs. Its very quick now since you removed loading versions.
• When I try to edit versions i noticed its not correctly display versions.
  

I have an ANDROID and IOS jira project. Not all active versions are showing correctly. Moreover the IOS project is incorrectly being displayed.

Here is my vira_projects.json

{
  "__default__": {
    "server": "https://company.atlassian.net",
    "issuesort": "status",
    "filter": {
      "project": "WEB"
    }
  },
  "vira": {
    "template": "__default__",
    "issuesort": "status",
    "filter": {
      "project": "WEB"
    }
  }
}

[n0v1c3]

Haha. Changing the OoO has affected our H key mapping for these menus and how we choose if we hide an item or not. We can most likely open a new issue directly for that one and close this one as we have gone very far from where we started.

Also. in your config file you should change vira to represent the folder name for the root of a project and "project": "WEB" changed to "project": "IOS" as an example for a pre-jira filter when you are in that folder and run vim... nvim. After that #41 has a better representation of the json file for you I believe.

[n0v1c3]

Let me know if there is anything else for this issue or new issues for us to create otherwise I think we can call this one done.

I am curious if you did a test with setting the g:vira_async_timer_init variable lower to see how that affects you?

There is also a variable inside autoload/vira.vim called s:vira_async_debug that if set to 1 you should at least be able to see the full list that it can see of version numbers below as it goes through arrays of projects. This will confirm that it is working properly from a query may lead to #53 more.

[chinwobble]

Also is it possible to get rid of this menu

[n0v1c3]

I would say I have no direct clue in vim. I am guessing we can with "no" problems just like everything in vim and seen in other plugins. I have one override for a plug-in mentioned on the README at the bottom but, not to hide everything on the standard menu.

I will open on on my Jira and another new issue here. Keep the challenges coming.

[n0v1c3]

I think I have prepared enough to close this issue you and I can keep motivating others people by closing this one up and getting this out to master. I may be getting more and more interesting in hoe interested you are in reporting issues to me as we can both clearly see the value of this last problem solve for vim/nvim.

I do have some true endless cleanup leaning for this issue however, you will see I am mostly just trying to try it down right now. So you and I can close this on and continue creating more fun endless ones I would have never thought of.

(Why would I every make a typo in my list of projects... List of projects I just have more code to build lists and code for me while I sit back and watch a black screen.

The only winning competition is the 2 daughters and 1 wife very hard to compete there.

I do have one more BIG story but it would need to be a private conversation.

[n0v1c3]

@chinwobble head up i did confirm to switch the default back to the right side of the screen. This will only affect you if you had nothing in .vimrc for it.

[chinwobble]

For me I noticed the UI is quite slow.

When I jump from top of screen to bottom of screen gg and G there is a lot of lag.
I suspect for each page that is loaded in the background it blocks the UI temporarily.

[n0v1c3]

You should be able to set your g:vira_async_timer and g:vira_async_timer_init
init will get it done and smaller number is faster I have set to one.
timer default is 10s you can spread out
all times are in ms

[n0v1c3]

If you are not worried too much about lots of version filters other filters will but versions into your logs slowly and you can also slow them right down.

Edit

If you slow it way down and open an issue you will see.