ms-vscode.mock-debug is mysteriously missing

[filiptronicek]

The extension is published from https://github.com/open-vsx/publish-extensions and in the logs, the extension is reportedly being published successfully, but when going to request it from https://open-vsx.org, it gives a 404 back.

https://open-vsx.org/extension/ms-vscode/mock-debug

[filiptronicek]

Logs: https://github.com/open-vsx/publish-extensions/actions/runs/3537480770/jobs/5937487121#step:9:5503

[kineticsquid]

@filiptronicek I scanned the build log file about and found 9 instances of this Error: Namespace already exists: error. Of those 9 instances, and despite the fact the that log reports that the extensions were published, the following extensions do not show up in the registry:

Error: Namespace already exists: Dannark
https://open-vsx.org/extension/Dannark/AndroidLauncher
Error: Namespace already exists: ms-pyright
https://open-vsx.org/extension/ms-pyright/vscode-pyright
Error: Namespace already exists: ms-vscode
https://open-vsx.org/extension/ms-vscode/mock-debug
Error: Namespace already exists: vscjava
https://open-vsx.org/extension/vscjava/vscode-gradle
Error: Namespace already exists: Vue
https://open-vsx.org/extension/Vue/vscode-typescript-vue-plugin

While these extensions do show up in the registry:

Error: Namespace already exists: amazonwebservices
https://open-vsx.org/extension/amazonwebservices/aws-toolkit-vscode
Error: Namespace already exists: maziac
https://open-vsx.org/extension/maziac/asm-code-lens
Error: Namespace already exists: mrorz
https://open-vsx.org/extension/mrorz/language-gettext
Error: Namespace already exists: ms-vscode
https://open-vsx.org/extension/ms-vscode/cpptools-themes

I cannot find a pattern.

[kineticsquid]

@filiptronicek @amvanbaren I poked around a bit more with this one. I have a gitpod workspace with an open-vsx deployment. I built this extension there to get a .vsix file with yarn install, build, package. I then published that with the 'Publish' link. It published successfully and I could see it in the catalog.

As an aside, similar to ms-pyright.vscode-pyright in #1532, from the admin panel, while I can list the extension by publisher and by namespace, when I attempt to list the extension with the 'Extension' link on the admin panel by specifying the namespace and the extension name, the page spins and never returns.

[kineticsquid]

@filiptronicek This appears to be working now. Can you confirm?

[filiptronicek]

Looks like this is indeed not the case anymore, thanks for the ping.

[filiptronicek]

It looks like the same is happening again now with vscjava.vscode-gradle. @kineticsquid could you please check and if possible, publish the .vsix from https://github.com/microsoft/vscode-gradle/releases/tag/3.12.7?

[kineticsquid]

@filiptronicek When I poked into this back in November, this was one of the extensions that was not showing up, like ms-vscode.mock-debug. Did it mysteriously show up between then and now?

[kineticsquid]

@filiptronicek I can publish the extension, but only by adding myself to the workspace.

[filiptronicek]

@kineticsquid: Did it mysteriously show up between then and now?

I don't know... I... think so? I kind of want to believe that I would notice it missing for more than a couple weeks at a time, but maybe it just slipped through my radar. It might've, but I'm assuming it hasn't.

I can publish the extension, but only by adding myself to the workspace.

Since it's our namespace, I think there is no problem doing that. If you find the time I would appreciate if you tried

[kineticsquid]

@filiptronicek I added myself as a contributor to the vscjava namespace. I attempted to publish the .vsix file from the 3.12.7 release I received the following error:

Was that build OK?

[kineticsquid]

@filiptronicek ^^^^

[filiptronicek]

@kineticsquid that's weird. Could you try using ovsx to publish from your local terminal? The .vsix I shared should be alright: I installed it in a fresh VS Code installation and it seemed to do what it should.

[kineticsquid]

@filiptronicek I tried the following:

npx ovsx publish vscjava.vscode-gradle-3.12.7.vsix -p xxxxxxxxxxxxxxx

Result:

❌  Internal Server Error
See the documentation for more information:
https://github.com/eclipse/openvsx/wiki/Publishing-Extensions
johnkellerman@Johns-MBP Downloads % 

[kineticsquid]

@filiptronicek ^^^^

[kineticsquid]

@filiptronicek ^^^^

[filiptronicek]

Sorry for the delay, @kineticsquid and thanks for posting the outputs of your publishing. Since this is one of the mysterious server errors, could you @amvanbaren look into server logs to see what could be up? 🙏

Interesting to note that we get a different error than the one from the Web UI as well, but I think that's just an issue with the web uploader.

[filiptronicek]

According to a recent bug report (EclipseFdn/publish-extensions#687) this issue has been affecting GitHub.vscode-github-actions as well...

[kineticsquid]

@filiptronicek Thanks for getting back to me. I just re-published the extension to the same result.

@amvanbaren This error was at 19:27 CEST. Can you take a look at the logs?

[amvanbaren]

At 19:27 CEST I found:

org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "unique_extension_public_id"
  Detail: Key (public_id)=(a226f06e-9b5e-493b-b92a-553d3572f969) already exists.

[filiptronicek]

@amvanbaren I assume that means that the extension version has already been published?

[kineticsquid]

I think this is resolved now. @filiptronicek?

[filiptronicek]

@kineticsquid this is indeed resolved for mock-debug, but seems to still be the case for Gradle and GitHub Actions...

[kineticsquid]

@filiptronicek I'm looking at the latest log, https://github.com/open-vsx/publish-extensions/actions/runs/5527236818, and it indicates that neither of these were published. TBH, I don't recall this overview view of the auto-publishing results.

[kineticsquid]

@filiptronicek Thanks. I went through the log file again, looking for all instances of Error: Namespace already exists:. I found the following six extensions. The first three were not published. The second three were.

Published Dannark.AndroidLauncher to https://open-vsx.org/extension/Dannark/AndroidLauncher
Published GitHub.vscode-github-actions to https://open-vsx.org/extension/GitHub/vscode-github-actions
Published vscjava.vscode-gradle to https://open-vsx.org/extension/vscjava/vscode-gradle

Published mrorz.language-gettext to https://open-vsx.org/extension/mrorz/language-gettext
Published MS-CEINTL.vscode-language-pack-pt-BR to https://open-vsx.org/extension/MS-CEINTL/vscode-language-pack-pt-BR
Published ms-vscode.cpptools-themes to https://open-vsx.org/extension/ms-vscode/cpptools-themes

I cloned repos for each of the three extensions not published and attempted to package locally and publish with vsce package and npx ovsx publish.

• Dannark/AndroidLauncher:
  Publishing fails because there is no 'license.txt' file.
• GitHub/vscode-github-actions
  Packaging fails because the extensions name, @github/vscode-github-actions is invalid.
• vscjava/vscode-gradle
  Packaging fails because the repo has no package.json file.

This is a bit strange since the log messages clearly indicate these three extensions were published.

[filiptronicek]

@kineticsquid I'm not entirely sure about theDannark/AndroidLauncher extension, but for the remaining two this is most likely because they are distributed differently and/or there are special instructions we need to execute for each. For the GitHub/vscode-github-actions extension, we directly fetch a Microsoft-built package from the GitHub Releases of the project: https://github.com/github/vscode-github-actions/releases/tag/release-v0.25.8. The same is done for the vscjava/vscode-gradle extension: https://github.com/microsoft/vscode-gradle/releases/tag/3.12.7

[kineticsquid]

@filiptronicek Thanks, this is helpful for me to understand the processing. In looking through the log file from the action run above, for when we decide we need to update an extension, it looks like in some cases we build it and in other cases, we just get the .vsix file and install it. How do we make that decision? I did poke a bit at publish-extensions.js and publish-extension.js, but JavaScript is not in my repertoire.

[filiptronicek]

@kineticsquid we would love to "just get the .vsix" every time, so we try to do that. Not all extensions publish to GitHub Releases, though, so manual building is necessary.

[kineticsquid]

@filiptronicek I took another swing at this. I added myself as a contributor to the namespace of both of these extensions:

https://open-vsx.org/extension/GitHub/vscode-github-actions
https://open-vsx.org/extension/vscjava/vscode-gradle

Used npx osvx publish ... --debug to attempt to publish, with the following results:

johnkellerman@Johns-MBP temp % npx ovsx publish vscjava.vscode-gradle-3.12.7.vsix -p ********* --debug
Error: Internal Server Error
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/ovsx/lib/registry.js:176:40)
    at IncomingMessage.emit (node:events:538:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
johnkellerman@Johns-MBP temp % npx ovsx publish vscode-github-actions-0.25.8.vsix -p ********* --debug
Error: Internal Server Error
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/ovsx/lib/registry.js:176:40)
    at IncomingMessage.emit (node:events:538:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
johnkellerman@Johns-MBP temp % 

Just prior to this I published a simple extension of mine successfully.

@amvanbaren this was at approx 20:00 CEST today, July 25. Is there anything useful in the logs?

[kineticsquid]

One more observation. I fired up a Gitpod deployment based on the most recent code at openvsx. I was able to publish the above two .vsix files successfully.

[amvanbaren]

The public_id already exists in the database.

[kineticsquid]

@amvanbaren When I search for both of these extensions on the admin panel, which also shows deactivated extensions, it says they don't exist. Do we need some DB surgery here?

[amvanbaren]

@kineticsquid No, the call to the /publish endpoint failed. There's no extension version in the database.
I think this can be fixed by a daily job that checks if all public ids are up-to-date.

[kineticsquid]

@amvanbaren How long has that job been running, this issue seems to have been around for some time.

[amvanbaren]

currently there is no job

[kineticsquid]

@amvanbaren Right, you're suggesting a daily job could be implemented to do this?

[amvanbaren]

@kineticsquid Yes, a daily job or whenever a new extension is published. Maybe even both.

[kineticsquid]

Do we know what causes us to get in this inconsistent state?

[amvanbaren]

I think that the publicId stays the same when a publisher changes their namespace name or extension name on the VS Marketplace, whereas in Open VSX a new namespace or extension has to be created when the name changes.

For example:

• Publisher has an extension on the VS Marketplace named foo with public id 123
• Publisher publishes foo to Open VSX, its public id is also 123 (got it from the VS Marketplace)
• Publisher changes the extension name from foo to bar on the VS Marketplace, its public id is still 123
• Publisher tries to publish bar to Open VSX, it fails because foo still has public id 123 (it was never updated)

[kineticsquid]

I can see publicId in the extension table. It's not visible externally, right? Is it correct that this happens only with an extension that has a name change and is auto-published?

[kineticsquid]

@amvanbaren @filiptronicek Here's a thought. Could we fix this by unpublishing these missing extensions, presumably removing the publicId from the table, and then letting the auto-publishing re-publish them? Would this break anyone?

[filiptronicek]

@kineticsquid I think it should be all right, but my gut feeling is that these will re-introduce themselves soon 😕

[amvanbaren]

It's not visible externally, right?

It is included in the /extensionquery endpoint response.

Could we fix this by unpublishing these missing extensions

I guess you can, but it's a common problem that over time will return.

[kineticsquid]

Why is this only an issue with auto-published extensions? Or is it?

Can we provide guidance to the publishers for how to do this name change at the VS registry without running into issues here?

[filiptronicek]

Not sure if this is the same issue, but for the Jupyter extension, I just encountered this:

❌ Extension ms-toolsai.jupyter 2023.8.100 is already published, but currently isn't active and therefore not visible.

[kineticsquid]

@filiptronicek On my admin panel, I can see extensions that are published but not active. I don't see 8.100, the most recent I see, which matches the extension detail page, is 7.100.

[kineticsquid]

@filiptronicek @amvanbaren Are we good to close this one?

[kineticsquid]

@filiptronicek @amvanbaren ^^^^

[amvanbaren]

@kineticsquid Let's wait until v0.14.2 is running on open-vsx.org. It will contain eclipse/openvsx#822

[kineticsquid]

@amvanbaren we good to close this one now?

[amvanbaren]

The public_id update job has been deployed for 2 weeks. There are no more public_id unique constraint violations in the logs. Only vscjava.vscode-gradle isn't showing up.

[kineticsquid]

@amvanbaren Yeah, the auto-publishing is failing for this: https://github.com/open-vsx/publish-extensions/actions/runs/7881268679/job/21504618286#step:12:7401.

I manually published the latest version: https://open-vsx.org/extension/vscjava/vscode-gradle.