typedoc Restructure

[shlomiassaf]

I have been using typedoc for a while, it has great potential and it works great but it's not developer friendly when it comes to plugins...

Here are some things I think would help:

• Allow ES6/ES2015-style import syntax for plugins (PR ready, support ES6/ES2015-style import syntax for plugins #594)
• Host library under an organisation (npm scope)
• Create monorepo structure for the repository, moving non-core plugins to a separate package under the org.
• Arrange extensibility related symbols in a single module (see info below)
• Simplify external plugin registration. (@Component wont work outside of the core package)
• Refactor JSON generation process to support a plugin architecture (PR Ready, Refactor JSON generation process to support a plugin architecture #597)
• Support object type for options (might come from js file, or CLI stringified argument). It will help complex plugins to "own" an object for options and not share the root options object.
• Think how to avoid auto-plugin registration (via @Component). This will allow inheriting plugins without registering them (or manually unreging them)
• Better docs... (cobbler, shoes, not there... )

The approach is to simplify the codebase, making the core small and allowing plugins to add features.

I have several plugins already built, thing like:

• 3rd party modules plugin: Make documentation aware of 3rd party modules either by linking symbols to external documentation or by fully embedding type info from d.ts files (2 modes)
  I was able to include the whole @angular docs, creating selective mashups etc... 2nd mode is not that useful but it's nice.

Supports binding import declaration (import X from 'Y') and namespace import declaration (import * as Y from 'Y')

• library-mode plugin: multi-mode plugin that allows exporting libraries from a single endpoint, creating a multi-lib documentation (monorepo) and other stuff...

• objectify: A plugin approach for creating JSON (toObject()). Reflection, Type, Source, Group, etc are all serialized to JSON via plugins that hook into events using the core plugin architecture. This plugin provides the core plugin logic and routing plus core toObject() plugin replacements for all native symbols (reflection, types, source etc..).

• json-datasource: Using objectify to tap in an convert the JSON into a datasource/table like structure, having all reflection in one reflection table (hash), types in type table, sources in source table etc... The graph is kept in tact by having pointers (id numbers) instead of objects.
  This is structure is suitable for SPA apps that require lookpups, quick access etc.
  In addition, the plugin will normalize the output so flags will output in their numeric form and not verbose form. It will add a table for each flag enum as a reference.

• json-links: Another extension to objectify that parses comments searching for markup links ([[ XX ]] or {@link ,,,}) and when found adds a link section that contains the link, caption, position and length so it can be replaced later.

• ng-decorator-meta: Create angular aware documentation. Uses 3rd party modules plugin for linking to angular docs and objectify to output the data to JSON.
  Comes with a group plugin that adds symbols to angular groups (Components, Directives, Output, Input etc...). It also hooks into the default theme to provide minimal UI representation.

Will post screen shorts in the next issue-comment

Arrange extensibility related symbols in a single module

Currently the entry point of the library exports about 6 modules (Application, CliApplication...). This is fine for simple usage through CLI or via API in node.
For plugin creators, importing is cluttered.. leading to this:

import { Component } from "typedoc/dist/lib/converter/components";
import { ChildableComponent } from "typedoc/dist/lib/utils";
import { ComponentClass } from "typedoc/dist/lib/utils/component";
import { Application, ProjectReflection } from "typedoc";

This is not developer friendly... making adoption harder.

An extensibility module should export all extensibility related symbols, this can be in typedoc/ext or @typedoc/core if moving to npm org/scope

Refactor JSON creation to support a plugins

The current method to convert all reflections/types/groups/sources/etc to one big JSON object uses inheritance, which does not allow customisation as a plugin system would. Since a plugin architecture already exists and the JSON convertion logic already exists (in toObject() methods) it is fairly easy to implement. I have a fully working solution the uses the current plugin system and just does convertion from Reflection, Type, Source, Group and any other type to JSON, just register a serializer and it will work.

Why? It will allow building SPA documentation app's that relay on a JSON or multiple JSON's.
Using my solution, I have added extra plugins to it that completely restructure the JSON making it more like a datasource/table like and less document oriented...

Note on CLI:

While using CLI exclusively is nice it does not scale. Once the complexity gets bigger, plugins start to pile and options/flags/customisability is required things become a nightmare to manage.
IMHO, using a config file (i.e. typedoc.js) is the best option and the one that should be recommended (and documented). This is where plugins should be configured, opt in-out etc.

[shlomiassaf]

Full metadata representation with type linking (both external and internal) and resource support (show html, css even of referenced files)

Decorators for members with external doc links:

[aciccarello]

Great work @shlomiassaf! Here are some of my thought. I'll let @blakeembrey chime in as he's got more control over the github organization.

Allow ES6/ES2015-style import syntax for plugins (PR ready, #594)

Thanks for the PR. That is a solid improvement and should ease some of the pain of people who are looking to create their own special plugin.

Host library under an organisation (npm scope)

This has worked well for Angular but I haven't used much beyond the CLI so I don't know how much this would help typedoc.

Create monorepo structure for the repository, moving non-core plugins to a separate package under the org.

👍 If we can structure it well. It should help with people looking for the theme and website code too.

Arrange extensibility related symbols in a single module (see info below)

I think that's a good idea, though not as high a priority. I'd prefer a full word rather than an ext directory name though.

Simplify external plugin registration. (@component wont work outside of the core package)

👍

Refactor JSON generation process to support a plugin architecture (full solution exists, see below)

That sounds like a good idea. Probably requires some design work.

Support object type for options (might come from js file, or CLI stringified argument). It will help complex plugins to "own" an object for options and not share the root options object.

Interesting idea. I'd be interested to see what this would look like. I would like to see better js file support/documentation too.

Think how to avoid auto-plugin registration (via @component). This will allow inheriting plugins without registering them (or manually unreging them)

Yeah, the decorator based design has some flaws that we need to work through.

Better docs... (cobbler, shoes, not there... )

😆 Yes please. Docs haven't been updated in awhile. Plus there is documentation scattered between the readme, different issues, and a few random places other than the website. We need to put some thought into what the strategy is.

[shlomiassaf]

@aciccarello

Host library under an organisation (npm scope)

This is well suited for typedoc, since the core typedoc engine is small and most of the work is done through plugins, it will allow people to actually choose what they want and for the project to be clearer, separated with less code clutter. typedoc, being a plugin oriented library is the perfect fit for scopes.

Arrange extensibility related symbols in a single module (see info below)

ext is just an example, once putting npm scopes in place it can be @typedoc/core or @typedoc/extensibility or whatever...

Refactor JSON generation process to support a plugin architecture (full solution exists, see below)

I have a pretty good implementation working, producing 1:1 output as the current source code.
I don't think a design is required since a design already exists, I just took the same design the other plugins implement (converters, themes etc...). Same thing, instead of in they do out... thats it.

[shlomiassaf]

@aciccarello see #597

[aciccarello]

It looks like the npm scope is taken so that doesn't look like an option. Probably will have to stick with a typedoc- prefix convention instead. I'd love to talk more about how to organize the extensibility symbols.

Beyond that I'm good with everything on the list. There is broken support for either a typedoc.json or typedoc.js file already implemented. It'd be great to get that fixed.

I've also put together a strategy for updating the docs. By migrating the site into this repo we should be able to update the API docs with a simple npm script. A plugins guide would be a great improvement too (see #521).

[mootari]

About the docs: I'm currently digging into TypeDoc and compiling some docs for myself along the way. I've created a gist containing what I have so far (component list, events, options per component): https://gist.github.com/mootari/d39895574c8deacc57d0fc04eb0d21ca

[shlomiassaf]

Guys I have access to typedoc doc I registered it so it won't be taken Don't worry about that I'll give it

…

Sent from my iPhone

On Dec 23, 2017, at 2:50 PM, Fabian Iwand ***@***.***> wrote: About the docs: I'm currently digging into TypeDoc and compiling some docs for myself along the way. I've created a gist containing what I have so far (component list, events, options per component): https://gist.github.com/mootari/d39895574c8deacc57d0fc04eb0d21ca — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[aciccarello]

@shlomiassaf 👏 Ah, I wondered if that was the case. (I assume you mean org). Great. That gives us some more options.

I'm hoping to make some progress over the Christmas holiday. Any help implementing/reviewing PRs is appreciated. Feel free to post questions and comments on gitter. I usually try to respond within a day. @mootari I appreciate the detective work you've done. ❤️ If all goes well I should be able to rebuild the docs in the next month.

[shlomiassaf]

I'm traveling this weekend so I'll not be able to do much... I'll post when I get back

…

Sent from my iPhone

On Dec 23, 2017, at 8:38 PM, Anthony Ciccarello ***@***.***> wrote: @shlomiassaf 👏 Ah, I wondered if that was the case. (I assume you mean org). Great. That gives us some more options. I'm hoping to make some progress over the Christmas holiday. Any help implementing/reviewing PRs is appreciated. Feel free to post questions and comments on gitter. I usually try to respond within a day. @mootari I appreciate the detective work you've done. ❤️ If all goes well I should be able to rebuild the docs in the next month. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[aciccarello]

I'm thinking for configuring a monorepo we should use lerna. Has anyone used that before? It looks like it would bring helpful conventions and easy linking/publishing. We could even use it for conventional commits.

[byronmejia]

@aciccarello - We use lerna at work. Babel and Angular are my go to repos for looking at how to structure these dragons. Is there anything I can help you with?

It'd be good to see how typedoc can be split up into the individual packages that can depend on each other. :)

[aciccarello]

@byronmejia Thanks for chiming in. I'll take a look at those repos to see how they are structuring things. I think initially it might make sense to simply move the TypeDoc library itself into one directory and the default themes into another. I've been hesitant to dive into this thus far because I don't want to cause merge conflicts but it seems like it would enable significant improvements moving forward.

Would you be able to create a PR? We might have to iterate a few times but it would get things rolling.

[aciccarello]

I have several plugins already built

@shlomiassaf Are these plugins open source? I'd love to reference them in the readme

[byronmejia]

@aciccarello Yeah, perfect. I'll play with it over my weekend. The idea to just move TypeDoc initially is a good step I feel.

The only thing we may need to also consider includes:

• Developer on boarding
  • Being able to explain the large code base to new developers wanting to contribute
  • In saying that: VS code is a large code base and don't seem to struggle 🤷‍♂️
• Package architecture
  • The last thing we want to do is break projects already with typedoc as a dependency

In saying that, I am a massive fan of @shlomiassaf 's second point. Maybe not in the immediate future, but moving to an npm scope may help down the track.

[lddubeau]

@byronmejia I've started doing some work on typedoc that would greatly benefit from moving to a monorepo structure. I was going to take a stab at moving to a monorepo, but then I saw your comment shortly after you posted it and decided that it was no worth duplicating the work. How is it coming along?

[byronmejia]

@lddubeau - I'll be real - I'm struggling a little bit currently with it. 😏

Don't suppose you can give it a stab?

The thing I struggled with most is thinking about how it may impact installation of typedoc in the future. Don't want to be too destructive.

[lddubeau]

@byronmejia I actually started taking a stab at it about one day or so before your reply. I'm hoping to be able to put out a PR this week. I've been able to build and publish packages to a local repository, but there are still some nits to be addressed.

I'm not sure what issues you saw about installation. From the perspective of consumers, what I'm doing changes nothing. (Note that my work joins the current two packages into a monorepo but I'm not breaking any existing package into new packages. Someone else will have to do that.) From the perspective of development, I'm not sure yet what impact it will have.

[tommedema]

Talking about Lerna, is it possible to run typedoc on a lerna repo ? I can only get it to work for single repos, not mono repos or a collection of repos.

[aciccarello]

Just jotting down some of the things I've been wanting to clean up/overhaul for future reference:

• Get rid of @Component decorator and reduce class inheritance
• Replace context.withScope(() => {}) wrapper with something more intuitive
• Consider async/await/Promise api to avoid fs "sync" usage
• Improve testing performance
• Complete monorepo changes
• Automate pushing docs to GH Pages

Many of these are bigger tasks. I'm hoping to make a push for a 1.0 release with some of these types of changes. My priority is still going through PRs though.

[balupton]

• Automate pushing docs to GH Pages

I have that going with https://github.com/bevry/awesome-travis/blob/master/scripts/surge.bash - https://github.com/bevry/boundation is a scaffolding tool which has typescript and typedoc support in it, with support for that surge.sh autodeploy for travis.

[aciccarello]

@balupton Thanks for the suggestion!

[cspotcode]

Are there any docs that explain why typedoc uses the current architecture? I'm trying to understand the codebase, and it seems unnecessarily abstract. There's probably something I'm just not understanding correctly.

Are there any docs that explain why event bubbling is used? Or a sample diagram of the Component hierarchy during a normal typedoc invocation? What's the difference between an AbstractComponent and a ComponentHost?

Get rid of @Component decorator and reduce class inheritance

@aciccarello Can you elaborate, and is this something I can take a stab at? I think it'd be great to simplify the codebase and make it more welcoming to newcomers.

[aciccarello]

Hi @cspotcode, thanks for the interest. I cannot provide much insight into those design decisions as the were prior to my becoming a maintainer but I'll do my best.

The event system allows plugins to react to different events (see src/lib/converter/plugins). A lot of the component hierarchy is to allow instanceof checks. ComponentHost is just an interface saying that the component has access to the application instance. AbstractComponent sets up some basic event dispatching and initializing semantics. ChildableComponent is the base for creating wrapper components that can have components added to it and is used for TypeDoc's major components like the converter and renderer. Some of the base component classes are defined in src/lib/utils/component.ts and src/lib/converter/components.ts.

We'd probably need to have some discussions before getting development moving on that piece. I'm considering starting a 1.0 development branch to experiment with some new ideas. Please let me know what ideas you have. The gitter.im channel is always open for discussions too.

[rdewolff]

Any progress on this?

[Gerrit0]

There has been some work done, but there's still a lot to do. See #595 (comment)

I've been working on removing the Component decorator, but it's so tied into everything that I think I took the wrong approach at the start by just deleting the file - I need to go back and slowly remove it from each logical part of the program.

#931 is open to bring the website into this repo, at which point I think it will be easier to make changes since documentation can be created/updated at the same time.

[b4dnewz]

I came here looking for monorepo support, just to let anybody like me know how to deal with it right now I found this plugin https://github.com/asgerjensen/typedoc-plugin-external-module-map that allow you to map external resources, it does the job.

maybe it's useful also when making the feat for the monorepo

[d-fischer]

I would greatly appreciate this getting done. For my use case, I only use the JS interface to generate JSON, so I don't actually make use of a lot of the dependencies pulled in by the HTML generator part of typedoc, so it would be great if the JSON generator would be separate from the HTML generator.

Can you guys use any help on this topic?

[haschu]

@b4dnewz Big thanks for leaving that here! :)

[Gerrit0]

@d-fischer there's plenty of work to be done. If you see an issue you'd like to solve please feel free to work it! If you have questions regarding the code, I'm usually in Gitter on the weekends.