feat: github action to integrate/publish endo and agoric-sdk reference docs too #1038

LuqiPan · 2024-03-26T19:02:27Z

This PR will make reference docs for both endojs/endo and agoric/agoric-sdk repos available on docs.agoric.com.

Some finer prints:

to update the git hashes of repos we're generating docs from, run yarn git:update-submodules
deployment time is around 8 minutes vs 2 minutes prior to this change
I had to use a hack to make it build successfully on Cloudflare

cloudflare-workers-and-pages · 2024-03-26T19:04:07Z

Deploying documentation with Cloudflare Pages

Latest commit:	`ba4232f`
Status:	✅ Deploy successful!
Preview URL:	https://97e1e3c9.documentation-7tp.pages.dev
Branch Preview URL:	https://927-reference-docs-in-search.documentation-7tp.pages.dev

View logs

dckc · 2024-03-26T19:36:38Z

.github/workflows/build-deploy-doc.yml

+      run: |
+        cd agoric-sdk
+        yarn add -D typedoc-plugin-markdown -W
+        yarn typedoc --plugin typedoc-plugin-markdown


looks like this expects a yarn build before it.

packages/governance/test/swingsetTests/committeeBinary/test-committee.js:6:23 - error TS2307: Cannot find module '@agoric/zoe/bundles/bundle-contractFacet.js' or its corresponding type declarations. 6 import zcfBundle from '@agoric/zoe/bundles/bundle-contractFacet.js';

I wonder how much this slows down ci.

Seems to be around 1 minute for the yarn build command

github-actions · 2024-03-27T21:18:19Z

Cloudflare Pages Preview URL: https://3c691980.documentation-7tp.pages.dev

github-actions · 2024-03-27T21:59:56Z

Cloudflare Pages Preview URL: https://40e7b3f7.documentation-7tp.pages.dev

dckc · 2024-03-27T22:05:23Z

.github/workflows/build-deploy-doc.yml

+# The reason we can no longer use Cloudflare's git integration to deploy docs
+# site anymore is that Cloudflare's git integration would only check out
+# documentation repo, whereas we need to check out both `endojs/endo` and
+# `agoric/agoric-sdk` repo to build the docs site.


I wonder about using submodules.

It would be nice to be able to get snippets from dapp-agoric-basics, for example.

Yeah I considered it but in my past experience I've only been frustrated about needing to update the hashes for submodules and make git commits. That said, there's not really a good solution whenever we are "orchestrating" multiple repos.

I'm happy to give it another try if there's some submodule-fu I'm missing though.

Maybe the submodule friction is a good thing?

Looks like this workflow is checking out HEAD of the other two repos, but that's not what people can use. I propose that the docs reflect what's in Mainnet. To do that requires a PR on this repo to specify the the branch or tag in the other repos. Expressing that as a git submodule makes sense and it sounds like it would obviate the need to define this additional workflow.

I propose that the docs reflect what's in Mainnet.

There's some appeal to that, but then how to we make progress on generated reference docs?

The reference docs from each repo can publish anywhere. But this PR is about docs.agoric.com, right? Shouldn't that URL only show you what's in Mainnet?

again, yes, there's some appeal to that, but running typedoc on the mainnet version of our code yields mostly garbage. So should the generated reference docs on dogs.agoric.com be garbage until our fixes to the jsdoc in agoric-sdk make it to mainnet?

Running typedoc on the mainnet version of the code won't work at all, in fact, will it?

So should the generated reference docs on dogs.agoric.com be garbage until our fixes to the jsdoc in agoric-sdk make it to mainnet?

I'm not proposing that. I'm speaking to whether this repo sources from HEAD or a git submodule. My claim is that sourcing HEAD of agoric-sdk is untenable and so it shouldn't be how this PR is implemented. IIUC, there's no need for a new workflow if the reop used git submodules. Given that we need some way to specify a tag of the source repos, I propose that be how this is implemented.

What tag/version of the other repos to source, I think can change over time. I agree that u13 would be unworkable. Probably same for u14. But maybe u15 can get the docs improvements that have been going into master. Again, independent of my request that the repos are sourced by a tag instead of HEAD.

I agree that u13 would be unworkable. Probably same for u14. But maybe u15 can get the docs improvements that have been going into master.

ok. yes, that's pretty much what I'm thinking.

This discussion is helpful, I'll look into submodules

LuqiPan · 2024-03-27T22:07:16Z

package.json

@@ -7,7 +7,7 @@
    "docs:dev": "NODE_OPTIONS=--openssl-legacy-provider vitepress dev main",
    "docs:build": "NODE_OPTIONS=--openssl-legacy-provider vitepress build main",
    "docs:preview": "NODE_OPTIONS=--openssl-legacy-provider vitepress preview main",
-    "docs:build-cf": "NODE_OPTIONS=--openssl-legacy-provider vitepress build main && cp _redirects dist/",
+    "docs:build-cf": "DEBUG='vitepress:*' NODE_OPTIONS=--openssl-legacy-provider vitepress build main && cp _redirects dist/",


The DEBUG environment variable would allow us to see what broke during a github workflow run.

We could potentially add this environment variable to other docs:* yarn commands too

LuqiPan · 2024-03-27T22:10:35Z

main/.vitepress/config.mjs

@@ -83,6 +83,11 @@ export default defineConfig({
  ignoreDeadLinks: [
    // ignore all localhost links
    /^https?:\/\/localhost/,
+    // ignore links that starts with ./packages/*


Some of the .md files generated from endo and agoric-sdk repo use these links below, which will fail the vitepress build as they're indeed dead links. There are 8 total offenders so it's not widespread, and I'm adding all of them here so they can be ignored by vitepress

turadg

we will lose the lovely cloudflare-pages bot and its handy comment and links

That is a big loss. I request that how to stick with Cloudflare Pages be further studied.

I've only been frustrated about needing to update the hashes for submodules and make git commits.

How else will would you source a particular state of the other repos?

dckc · 2024-03-29T15:57:00Z

we will lose the lovely cloudflare-pages bot and its handy comment and links

That is a big loss.

We seem to still get comments with preview links: #1038 (comment)

It seems to only provide per-commit preview links and not per-branch links that update with new commits. I'll miss that. But I think I can deal.

dckc

I hesitate to approve this when the only way to get to the generated docs is search (#1041)

LuqiPan · 2024-04-01T20:27:24Z

Update:

We're going to submodules and back to build with Cloudflare (and thus Cloudflare Pages bot is working again), PR description updated too

dckc

I'd like to preview this locally, but I'm struggling to get it to work...

package.json

dckc · 2024-04-02T20:25:53Z

.github/workflows/build-deploy-doc.yml.DISABLED

+            issue_number: context.issue.number,
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            body: 'Cloudflare Pages Preview URL: ${{ steps.publish-to-cloudflare-pages.outputs.deployment-url }}'


is there a supported option to get a link based on a branch? one where the content is updated as commits are added to the branch, without changing the url

Not with wrangler-action: https://github.com/cloudflare/wrangler-action/blob/fd98a7c99091f581101a9a9bc7300a005d2f63e7/action.yml#L48-L53

Cloudflare Page bot is working again though so we retain the branch preview URL behavior

ah. .DISABLED

dckc

The resulting API docs need a lot of work, but @sufyaankhan is OK with landing this first.

For what it does, I think it looks good.

Cloudflare Pages bot is working again and we're using submodules now. Let me know if you have additional comments, but for now I'm optimizing for getting this PR merged in.

turadg · 2024-04-03T14:33:36Z

Cloudflare Pages bot is working again and we're using submodules now. Let me know if you have additional comments, but for now I'm optimizing for getting this PR merged in.

👍

feat: github action to integrate/publish endo reference doc too

649e426

LuqiPan added 2 commits March 26, 2024 12:28

feat: refine build steps and add steps to build agoric-sdk docs

e806103

Merge branch 'main' into 927-reference-docs-in-search

9544c91

dckc reviewed Mar 26, 2024

View reviewed changes

LuqiPan added 3 commits March 26, 2024 13:12

debug: add yarn build like Dan suggested

3da9eca

chore: bump vitepress to 1.0.1

162d794

debug: add a DEBUG flag to "docs:build-cf" script

293db4e

This was referenced Mar 26, 2024

927 typedoc markdown endojs/endo#2185

Merged

chore: typedoc markdown Agoric/agoric-sdk#9146

Merged

include endo, agoric-sdk reference doc in search #927

Open

CI/CD to ensure endo, agoric-sdk references in docs site is up-to-date and error-free #1046

Open

LuqiPan added 7 commits March 26, 2024 16:45

chore: update ignoreDeadLinks list

9c85876

ci: update yarn commands to generate markdown files

deb67f6

ci: update github workflow to reflect upstream changes

56a16bb

ci: use cloudflare/pages-action instead and add a step to comment URL

50f3e25

ci: switch back to using wrangler-action and use its output

b16111d

ci: add a condition to check if event is pull request

f64b764

ci: add conditions for preview and production deployments separately

40d3697

comment: add some background in comments for build-deploy-doc workflow

059336b

dckc mentioned this pull request Mar 27, 2024

Sync Zoe JSDoc with API docs Agoric/agoric-sdk#4291

Open

LuqiPan marked this pull request as ready for review March 27, 2024 22:04

LuqiPan requested review from turadg, dckc and 0xpatrickdev March 27, 2024 22:04

dckc reviewed Mar 27, 2024

View reviewed changes

LuqiPan commented Mar 27, 2024

View reviewed changes

turadg previously requested changes Mar 28, 2024

View reviewed changes

dckc reviewed Mar 29, 2024

View reviewed changes

LuqiPan added 9 commits April 1, 2024 10:34

chore: add endo and agoric-sdk to submodules

2885469

ci: disable github workflow for now

a1e4a9b

chore: bump node version to 18.18.0

2f3993b

ci: add several yarn targets to build .md docs for endo and agoric-sdk

9cdca18

chore: update submodules

57ac64e

chore: add a yarn target for git:update-submodules

b7ca228

debug: undo yarn install by cloudflare to build

891494f

debug: update file paths when mving

a4284c5

chore: add a comment for docs:build-cf script

c813781

LuqiPan requested review from turadg and dckc April 1, 2024 20:28

dckc reviewed Apr 2, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

package.json Show resolved Hide resolved

dckc reviewed Apr 2, 2024

View reviewed changes

package.json Show resolved Hide resolved

chore: add a yarn script for git-submodule:init

ba4232f

dckc reviewed Apr 2, 2024

View reviewed changes

dckc approved these changes Apr 2, 2024

View reviewed changes

LuqiPan merged commit 81bbd1e into main Apr 3, 2024
4 checks passed

LuqiPan deleted the 927-reference-docs-in-search branch April 3, 2024 03:40

LuqiPan mentioned this pull request Apr 3, 2024

feat: use mv instead of rm -rf to avoid rebuilding xsnap #1054

Merged

This was referenced Apr 3, 2024

Timer Service updates #1024

Merged

continuous publishing for api-docs in agoric-sdk #1032

Closed

integrate API Doc (reference) with agoric-sdk api-doc #319

Closed

dckc mentioned this pull request Apr 22, 2024

revert integrating agoric-sdk, endo generated reference docs #1070

Draft

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: github action to integrate/publish endo and agoric-sdk reference docs too #1038

feat: github action to integrate/publish endo and agoric-sdk reference docs too #1038

LuqiPan commented Mar 26, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Mar 26, 2024 •

edited

Loading

dckc Mar 26, 2024

dckc Mar 26, 2024

LuqiPan Mar 26, 2024

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

dckc Mar 27, 2024

LuqiPan Mar 27, 2024

turadg Mar 27, 2024

dckc Mar 28, 2024

turadg Mar 28, 2024

dckc Mar 28, 2024

dckc Mar 28, 2024 •

edited

Loading

turadg Mar 28, 2024

dckc Mar 28, 2024

LuqiPan Mar 29, 2024

LuqiPan Mar 27, 2024

LuqiPan Mar 27, 2024

turadg left a comment

dckc commented Mar 29, 2024

dckc left a comment

LuqiPan commented Apr 1, 2024 •

edited

Loading

dckc left a comment

dckc Apr 2, 2024

LuqiPan Apr 3, 2024

dckc Apr 3, 2024

dckc left a comment

turadg commented Apr 3, 2024

feat: github action to integrate/publish endo and agoric-sdk reference docs too #1038

feat: github action to integrate/publish endo and agoric-sdk reference docs too #1038

Conversation

LuqiPan commented Mar 26, 2024 • edited Loading

cloudflare-workers-and-pages bot commented Mar 26, 2024 • edited Loading

Deploying documentation with Cloudflare Pages

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 27, 2024

github-actions bot commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dckc Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turadg left a comment

Choose a reason for hiding this comment

dckc commented Mar 29, 2024

dckc left a comment

Choose a reason for hiding this comment

LuqiPan commented Apr 1, 2024 • edited Loading

dckc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dckc left a comment

Choose a reason for hiding this comment

turadg commented Apr 3, 2024

LuqiPan commented Mar 26, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Mar 26, 2024 •

edited

Loading

dckc Mar 28, 2024 •

edited

Loading

LuqiPan commented Apr 1, 2024 •

edited

Loading