1068 Generating context docs from schema files

[TheJuanAndOnly99]

resolves #1068

First iteration is complete:

• Created a script that generates markdown files from schema files; to execute run yarn schema2markdown in the root directory.
• The script generates context and api markdown docs from schemas
• Links are generated for each reference encountered
• Sidebars entries are automatically appended to sidebars.json
• The script iterates across all versions
• Each object definition maps to a file, as opposed to the previous version

Todo:

• confirm markdown page layout
• code cleanup

Preview - https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Action

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: TheJuanAndOnly99 / name: Juan Estrella (b052b86, 8b8960c, 69aeded, 5423041, 814fd93, 6bf17da, d5464a7, 1a85911, c449b9e, 3d38f83, 11ad90f, 3ebd43a, c9c2b1d, 9eae468, 6060696, 1950f73, 490af28, 6dc4331, 18cee47, 1baf110, ce8787a, 54477ae, eed9602, 44e0c66, 0e0c72b, fee515f, f4a80aa, 47df2e3, 901d72a, eac8599, a2b12f7, b27c3f2, 2f8bd00, 86d994a, 0c83f6a, fe2e40e, 4c49bb5, c51fe3f, 816c3a8, 9cb323a, a0216b6, 0b13c7a)
• ✅ login: maoo / name: Maurizio Pillitu (a956cec, 5b91507, 3a76695, f74e8b0, 297539c, f90bdce, b8ab9c5)

[netlify]

✅ Deploy Preview for fdc3 canceled.

Name	Link
🔨 Latest commit	eed9602
🔍 Latest deploy log	https://app.netlify.com/sites/fdc3/deploys/66339c896bdbf80008eb24bc

[TheJuanAndOnly99]

@kriswest @robmoffat - first iteration is done; please check the description above to see what's been done and what has not; there's also a preview on Netlify pointing to my fork (the build command will need to be updated when we go live with this).

Eager to hear your feedback! /cc @mistryvinay

[maoo]

Hi @julianna-ciq , thanks SO much for the detailed review! @TheJuanAndOnly99 is currently off, he'll be back on March 3rd and work on your comments.

[TheJuanAndOnly99]

Hey @julianna-ciq thank you so much, great review, really appreciate your feedback. I've implemented all your notes. Would you mind reviewing again. If everything looks good do you think it's okay to merge?

(Please note that after this is merged we would have to update the build command on Netlify).

[julianna-ciq]

I have a couple of stylistic nitpicks, but I think it looks good. I think @kriswest or @robmoffat have to review as well, though.

[robmoffat]

Could we have a link to the schema file on the page, so that the user can view that?

Also, might be good to reference the Typescript type that is generated, if that's possible (and c# if that's possible too I guess)

[TheJuanAndOnly99]

Hi @kriswest I fixed the build issue but I don't see it triggering the check again.

[kriswest]

@TheJuanAndOnly99 I've manually re-run it. I know its going to fail (as the docusaurus website has developed an issue with a dependency, but the log will tell us whether anything else is still amis).

[kriswest]

@TheJuanAndOnly99 you're all clear! Just he axios dependency issue in the log

[kriswest]

@mistryvinay and I should be able to take it from here - we need to run through another review and then make the tweaks necessary to have the generated pages replace the existing ones, meaning they'll need to generate onto the same URLs as the manual pages, which we'll delete!

[kriswest]

...although @TheJuanAndOnly99 you might need to explain or handle whatever is needed regarding the netlify preview/build... I don't yet know what change is needed nor why.

If its about making sure the generation script gets run, we could possibly do that with a commit hook and npm package script...

[TheJuanAndOnly99]

@kriswest when this PR gets merged I need to update the build command on Netlify to run the script and that will take care of things for us.

[kriswest]

@TheJuanAndOnly99 The preview for this appears to be broken - could you take a look, not seeing the generated pages anymore. Perhaps a conflict on updating from @robmoffat's homepage update

[TheJuanAndOnly99]

Hi @kriswest the preview is running on my personal Netlify (since the build command is different). You can find it here.

[kriswest]

@TheJuanAndOnly99 I'm not seeing the generated pages in the preview's sidebar anymore:

These are the old pages?, right: https://exquisite-otter-12d363.netlify.app/docs/next/context/ref/Action

[TheJuanAndOnly99]

@kriswest yep you are correct! I think I've fixed it. https://issue-1068--exquisite-otter-12d363.netlify.app/docs/context/schemas/Action

[hughtroeger]

Code looks fine. I went through the Context pages comparing the most recently provided preview to the current docs, here are my notes.

General differences:

• No Overview page for Context Data Part
• No Context page for Context Data Part
• ‘Required’ boolean field is missing for all properties
• Schema link goes to github.com instead of fdc3.finos.org, assume that will update on publish/deploy
• All reference links are broken in the Netlify preview, is this expected?
• All properties with array types no longer have specific array type (like Instrument[]), they just say ‘array’. Many do have a sentence that says what kind of array it is, and some have reference links to the schema page for the thing it is an array of
• Properties with a reference link to another docs page no longer have ‘type’ explicitly declared, this is probably fine as the information is present in the page linked
• I find the new format a little difficult to read, I think some indentation for the data under each property would go a long way.

Page specific differences:

• Chart
  o Instruments property used to have type ‘Instrument[]’ now it has ‘array, is clear from the description that this is an array of instruments
• ChatInitSettings
  o Message property used to be of type ‘Message’, new one has an example but should probably have a reference link to fdc3.message
• Chat Search Criteria
  o Properties property used to have type (Instrument | Contact | Organization | string)[] , now it just has type ‘array’
• Contact
  o No longer has ‘name’ property
• ContactList
  o No longer has ‘id’ or ‘name’ properties
  o ‘contacts’ property could have a reference link to the contact schema page
• Country
  o No longer has ‘name’ property
• Email
  o ‘recipients’ property used to be of type ‘fdc3.contact or fdc3.contactList’, now it has no type and no reference link
• Instrument
  o ‘id’ subproperties used have working links under ‘More Info’, now they have URLs that aren’t active hyperlinks links displayed like <https://www.isin.org/>
• InstrumentList
  o No longer has ‘name’ or ‘id’ properties
  o ‘instruments’ property could have a reference link to instrument schema page
• Order
  o Details.product has type and description ‘undefined’
• OrderList
  o No longer has ‘name’ or ‘id’ properties
  o ‘orders’ property used to be of type ‘Trade[]’ which I believe is incorrect, I think it should be Order[], now it just has type ‘array’ but does not have a reference link to the order schema page
• Organization
  o Missing ‘name’ property
• Portfolio
  o Missing ‘name and ‘id’ properties
  o ‘positions’ property could link to position schema page
• Position
  o Missing ‘name ‘ and ‘id’ properties
• TradeList
  o Missing ‘id’ and ‘name’ properties
  o Trades property could link to the trades schema page
• TransactionResult
  o Missing ‘message’ property

[kriswest]

The missing id and name properties should all be fixed by : #1233

[kriswest]

Checklist for changes that need to be made to close this (from @hughtroeger very detailed review)

Integration into project

• Relies on netlify to run the generation currently which is not good for local development and previews
  • switch that to lifecycle tasks in /website/package.json so that builds and previews run the schema2markdown.js generation script (prebuild and prestart tasks)
• Currently generates docs for all standard versions using the current schemas, except unreleased...
  • Do not generate docs for past standard versions, only for the current draft (unreleased) - allow the versioning system to take care of copying that to a versioned-docs - if we need to fix something in a past version we can do so manually!
• No Overview page for Context Data Part
  • Script is currently generating a new section, it should reuse the old section which contains the overview page. PR should delete all the existing reference pages (we can compare against current site in future

Generated content

• No Context page for Context Data Part
  • As discussed, we should just move this into the overview as no one should be broadcasting that base schema.
• ‘Required’ boolean field is missing for all properties
  • Needs fixing
• Schema link goes to github.com instead of fdc3.finos.org, assume that will update on publish/deploy
  • We could/should use whats in the $id field for this rather than the file URL (will need a code change and we don't need to wait for publishing). However, @mistryvinay and I are not sure it matters... Easier to raise a PR to change it with the github URL.
• All reference links are broken in the Netlify preview, is this expected?
  • Bug: URL is missing the /schemas/ element
• All properties with array types no longer have specific array type (like Instrument[]), they just say ‘array’. Many do have a sentence that says what kind of array it is, and some have reference links to the schema page for the thing it is an array of
  • Bug: need to add details for the items (which is expressed in the schemas). Some will be base types, others references to other schemas.
• Properties with a reference link to another docs page no longer have ‘type’ explicitly declared, this is probably fine as the information is present in the page linked
  • Change the 'Reference' label to 'Type:'
• I find the new format a little difficult to read, I think some indentation for the data under each property would go a long way.
  • Add indentation to properties

Page specific differences

• Chart
  • Instruments property used to have type ‘Instrument[]’ now it has ‘array, is clear from the description that this is an array of instruments
    • Would be fixed by rendering types for arrays
• ChatInitSettings: Message property used to be of type ‘Message’, new one has an example but should probably have a reference link to fdc3.message
  • Properties defined with anyOf and oneOf are not being rendered - and probably won't be popular with @bingenito
• Chat Search Criteria: Properties property used to have type (Instrument | Contact | Organization | string)[] , now it just has type ‘array’
  • Will be fixed by types for arrays and support for anyOf/oneOf
• Instrument
  • ‘id’ subproperties used have working links under ‘More Info’, now they have URLs that aren’t active hyperlinks links displayed like https://www.isin.org/
    • Links are being rendered with backticks around them so they appear to be code instead of hrefs - just remove the backticks!
• Order
  • Details.product has type and description ‘undefined’
    • Sub-properties need to render references to other schemas and skip the title and description if not present as its handled on that page
• OrderList
  • ‘orders’ property used to be of type ‘Trade[]’ which I believe is incorrect, I think it should be Order[], now it just has type ‘array’ but does not have a reference link to the order schema page
  • Generation will fix this - it was a mistake in the old docs page (bad Kris)
• TransactionResult
  • Missing ‘message’ property
    • This is a straight error in the schema - I'll open a PR to fix this now! Good spot @hughtroeger
    • Correcting a syntax error in the TransactionResult Schema #1251

[kriswest]

@TheJuanAndOnly99 & @finos/fdc3-maintainers I'm going to merge this into branch on the FINOS repo so we can continue the development there. I'm pretty sure we can get this running with the existing netlify config using npm scripts to trigger it instead (which will be better for local development).

Many thanks for your help with this! We'd happily take more of course ;-) I'll tag you in the new PR once this is moved over.