Added the ability to specify a pre-process function, enabling the inclusion of a custom function for data processing. #577

Closes datopian#163 New feat: Page comments / discussions ### Changes * Create components in core for giscus, utterances and disqus * Add configs in frontmatter and config.js * Core version bump 0.2.0 * Add env variables ((netlify) for giscus * Create documentation

Custom giscus.json file in site's root to restrict the domains that can load giscus with our repository's discussions.

add prop-types dependency as disqus-react fails to resolve module when creating new flowershow site.

…uping * basic grouping by 1st level content dir subfolders

skip @netlify/plugin-nextjs (next-runtime) during build as we don't need it when using next export. https://github.com/netlify/next-runtime#use-with-next-export

- use nx target on template package to start the test webserver from within the `packages/template-e2e` folder (this is what caused errors in the workflow, as the server wouldn't start if using cd or npm run dev --prefix)

… works

… a new content layer.

… edits * Remove navbar dropdown support * Bump core v0.3.2 * Make images smaller in frontpage

* fix conflicts between next and contentlayer versions of @opentelemetry/api

Exporting Themeprovider from Core

Update mermaid closes datopian#428 * Update mermaid in core `v10.0.1-rc.2` * create mermaid component in core * add `mdx-mermaid` plugin and mermaid component in template * remove `remark-code-extra` plugin

…k file

Update packages and fix search component closes datopian#419 * update all packages in core and template * refactor components in core - algolia, kbar and nav

…cription

…kage

…cription Auto extract title and description from content closes datopian#297 * Add resolvers for title and description in computed fields - `contentlayer.config.ts` * Add package - https://github.com/remarkjs/strip-markdown * Handle repeating titles in `pages/[[...slug]].tsx`

…content layer page.

* [site/components][s]: Updated features

…config.mjs

* globby * prettier * esbuild-register

* [docs][xs]:Custom page and layout tutorial

- for use in custom Layouts

Make date the default field in frontmatter

### Summary This PR adds support for auto generating sitemaps in public folder if the `authorUrl` property is set in `config.mjs` ### Changes * Config file extension changed to `config.mjs` in `template` and `template-e2e` * Flowershow CLI updated to replace/create `config.mjs` * Add sitemap generation script * generates sitemap only if `authorUrl` present in config * Update search script * generates `search.json` only if using local search provider ie. kbar * Add dev dependencies - `globby`, `prettier`, `esbuild-register` * Add documentation @ [docs/sitemap](https://deploy-preview-454--spectacular-dragon-c1015c.netlify.app/docs/sitemap)

- use named colors in Card and TableOfContents - replace fixed height and width in nav logo with class, so that it can - be referenced and adjusted

- removing them as it's a mess and we haven't used them each time we made changes -> starting versioning now

- latest version of core has fixed height replaced with "nav-logo" class name, so that it can be easily adjusted by users in their projects

## Changes - `@flowershow/remark-wiki-link` - general code refactoring: tidying up, simplifying, using better naming etc. - disable passing markdown folder in plugin config -> export `getPermalinks` function, so that users (e.g. our template) can run it on their side and then use a standard config option `permalinks` to explicitly pass the permalinks list - extend tests coverage - refactor to TS - replace `mocha` with `jest` - `@flowershow/template` - import `getPermalinks` from `@flowershow/remark-wiki-link`, generate the permalinks list from the content folder on the template side (in `contentlayer.config.ts`), and pass it explicitly to the plugin instead of passing the path to markdown folder to it ## Notes I think there should be a way of using `micromark` extensions only - our `html` and `syntax` - but maybe I'm missing sth (still not quite getting all the different elements of the system and ecosystem...) and we actually need to use `syntax` extension for `micromark` combined with `fromMarkdown` and `toMarkdown` for `mdast-util-from-markdown`. I actually started by writing `syntax` and `html` and both of them used in `micromark` do their job, i.e. they correctly convert md to html (see tests in `remark-wiki-link/tests/micromarkExtensionWikiLink.spec.ts` - mirror tests of `fromMarkdown`). But I couldn't make it work without using `fromMarkdown` - **which is almost identical to `html`** - and `toMarkdown`. For reference see also `remark-wiki-link/src/lib/remarkWikiLink.ts` file.

Related to datopian#465 ## Changes - created a new package in `packages/markdowndb` - copied code from `datahub-next/lib/markdowndb`

Closes #473 ## Motivation As we plan to replace Contentlayer with our MarkdownDB package, we need a solid base upon which we can build a full-fledged "content-layer" solution. As the code in its current state is pretty hard to reason about and manipulate and is rather weakly typed, the development of any new functionality on top of it would only make it harder and harder to engineer, adjust and test. This PR aims to make the development of new features easier and (because of stronger typing) less error-prone. Note, there is still quite a lot of room for improvement - TODOs comments were left in the code but will probably be resolved after further discussions on implementation design. ## Changes - `markdowndb.ts` cleaned up and refactored. Most notably: - tables creation (and deletion) logic moved out to `schema.ts` and bound with relevant classes corresponding to tables (see below) - wrapped raw batch inserts on knex db with class methods to prevent SQL errors in the first place, instead of debugging them after they occur - `indexFolder` bound with MarkdownDB class as a method - removed duplication of db initialization -> there is only one `init` method on `MarkdownDB` now - replaced ambiguous `query` method with `getFiles` - commented out querying by folder (?) for now as I'm not sure if it's even needed, at least not in the current form - added `getFileByUrl` and `getFileById` method to retrieve single file file from the db (e.g. for finding its backlinks) - added `getLinks` method which supports querying back- or forward links - removed `types.ts` classes in `schema.ts` can also by used as types. Other types like e.g. `DatabaseQuery` moved directly to method signature, as they are striclty bound with it and it's not needed to have it in a separate file (there is no other use case for it) - cleaned up and extended tests of `markdowndb` lib and split into smaller unit tests grouped by tested functionalities - `schema.ts` created with four classes: `File`, `Link`, `Tag` and `FileTag` - each class is now a representation of a table, i.e. `files`, `links`, `tags`, `file_tags` and so it describes the fields (columns) existent on each table and has methods for creating a table, deleting it and batch-inserting data to it - it is a single source of truth about tables and data types stored in them, - `utils` folder created with `recursiveWalkDir.ts`, `parseFile.ts` and `extractWikiLinks.ts` each with its own, separate test suite - stronger typing

+ minor fix in in pdf embeds

## Changes **`package.json`** - added missing dependencies - added `mddb` command that executes folder indexing (see `/src/bin/index.js`) - will probably be replaced with a full-blown CLI in the future **`/lib`** - introduced the distinction between simple object interfaces (e.g. `File`) from instances of database rows (e.g. `MddbFile`) represented by `Mddb...` classes that implement additional methods and in general should only be instantiated from records returned from the database - added a simple validation function that checks if some of the objects in an array have the same value of a given property - added this check in `batchInsert` methods on `Mddb...` classes to catch duplicate values in JS and display objects that would cause SQL errors (which are impossible to debug 😅) - maybe we could add some validation package in the future, but for now this should do \+ Bunch of other fixes caught after testing with `datahub-next`

## Changes ### Primary changes - removed `contentlayer` and `next-contentlayer` packages, along with the `contentlayer.config` file - created a temporary solution for contentlayer's computed fields in `lib/computeFields.ts` module (no distinction between document types; all fields will be returned for all pages, even if `null`) - installed `@flowershow/markdowndb` package as well as: - added `lib/mddb.mjs` with db connection creation module - added `mddb` script in `package.json` - copied over `lib/markdown.ts` from `datahub-next` ### Secondary changes - `type` frontmatter fields temporarily replaced with: - `layout: blog` for specifying layout, which was previously set as a field on `Blog` document type in `contentlayer.config` - `mddb.getFiles({ folder: `blog` })` for fetching e.g. on `/blog` page - replaced `content/index.md` document with `pages/index.tsx` page, and also: - moved all components that were imported there from `/site/components` (linked to `/components/custom`) back to root `/components` folder - created components for the rest of the content on the index page (e.g. `CloudPublishingSteps.tsx`) to make it cleaner - replaced `content/blog/index.md` document with `pages/blog.tsx` page - replaced **optional** catch-all page `[[...slug]].tsx` with **regular** catch-all `[...slug].tsx`, so that we can have `pages/index.tsx` page - copied over `DRD` page (under name `MdxPage`) from `datahub-next` - added basic typing for props returned by `getStaticProps` and required by corresponding page components and the custom `_app.tsx` page (but would be nice to work more on it), and as a result/related: - since `.tsx` pages are not indexed into markdowndb and so they don't have metadata, they now need to explicitly return metadata in `meta` property by their `getStaticProps` function (see `/pages/blog.tsx` as an example) - moved any `show-X` boolean values resolutions to `computeFields` (this should have been done even with contentlayer) - removed `getters` folder **Also (in docs)**: - hid tutorial and docs page on blogs for now as they need to be re-read and adjusted - hid doc page on `mdx` as it's invalid in most part - hid tutorial and docs page on custom layouts for now as they need to be re-read and adjusted ## Deprecated features - document types and hence: - `type` frontmatter field and importing all files of a given type - frontmatter fields validation - auto-extracting title (TBC can't remember why) - page "getters" - `contentInclude` and `contentExclude` config fields for including/excluding content folders - `import` and `export` statements in markdown documents - hover-over tooltips (didn't have time to fix them + probably would be nice to implement them in a better way anyway) ## TBD - not sure why but I had to import `React` in all the pages and components, otherwise an error about not-configured loader would show up - not sure how this relates to the changes in this PR - adjust tests if needed (not sure) ## Changes required/suggested to @flowershow/markdowndb - implement document types in `@flowershow/markdowndb` with specific computed fields, helper methods (e.g. `BlogType.getAuthors()`) and frontmatter validation - support importing all documents of a given type (e.g. mddb.getFiles({ type: "Blog" })) - config file for `@flowershow/markdowndb`package with e.g. `contentExclude` and `contentInclude` contentlayer's counterparts

…en though it has been published to npm and tagged. This temporary branch - markdowndbv0.1.0 - was created only to get these chenges on main.

- workaround solution to force pnpm to use compiled version of the package, as otherwise running it's mddb bin file (see package.json of markdowndb) in npm script in the template would result in errors

- temporary solution to `SyntaxError: Named export 'useRouter' not found` in versions 13.3.1+ - see this issue vercel/next.js#48801

- after switching to MarkdownDB from Contentlayer we replaced index.mdx file with pages/index.tsx file for our home page, which results in users having our Flowershow home page displayed as well - this is a temporary solution, in the future we should finally create a separate project for our Flowershow website that would be built off of the clean Flowershow template (without e.g. pages/index.tsx)

docs in `/docs/contributors.md`

All docs pages reviewed and updated to be in line with current Flowershow version, which uses MarkdownDB. Comprehensive list of changes in issue datopian#484.

- rm it from git lFS

- importing custom components in MD is no longer supported in Flowershow v.2 so this is not needed

- when running content folder indexing via mddb command of @flowershow/markdowndb build on Netlify would fail with this error: ``` Error: Command failed: npm run mddb at checkExecSyncError (node:child_process:826:11) ... ``` with status 13: Unfinished Top-Level Await

- we've migrated the template to a separate repo so that users of the flowershow obsidian plugin can clone it to their own repos (and not clone the whole monorepo)

This reverts commit 817333c.

- since the template is now a submodule, it will throw an error, saing that packages/template doesn't belong to any projects

- all Flowershow website content has been moved to https://github.com/datopian/flowershow-app

## Changes - don't hyphentate or lowercase src/hrefs by default - add ignore patterns option to `getPermalinks` util function and allow passing custom paths -> permalinks converter function - fix regex patterns - adjust and extend tests

Closes datopian#509

…ile still in it.

…png file.

Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md) - [Commits](npm/node-semver@v6.3.0...v6.3.1) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

…/semver-6.3.1 Bump semver from 6.3.0 to 6.3.1

…pian#554)

Typo fix in README

update actions version and bump nodejs to current lts

- as per https://nextjs.org/docs/messages/no-stylesheets-in-head-component

…a remote url

Fix flowershow install failing

🛠️fix: footer position for page with short content

…lusion of a custom function for data processing.

Commits on Feb 26, 2023

[site/content][xs]: run eslint format

khalilcodes committed Feb 26, 2023

Configuration menu

View commit details

Copy full SHA for fedcf04

Browse repository at this point

Copy the full SHA

fedcf04 View commit details

Browse the repository at this point in the history

Commits on Mar 26, 2023

Fix typos

rdimaio committed Mar 26, 2023

Configuration menu

View commit details

Copy full SHA for 7267b8f

Browse repository at this point

Copy the full SHA

7267b8f View commit details

Browse the repository at this point in the history

Commits on May 3, 2023

[template][s]: add .gitignore

olayway committed May 3, 2023

Configuration menu

View commit details

Copy full SHA for df602b8

Browse repository at this point

Copy the full SHA

df602b8 View commit details

Browse the repository at this point in the history

Commits on Jun 13, 2023

[/][s]: bump @portaljs packages

olayway committed Jun 13, 2023

Configuration menu

View commit details

Copy full SHA for e00c956

Browse repository at this point

Copy the full SHA

e00c956 View commit details

Browse the repository at this point in the history

Commits on Sep 6, 2023

Typo fix in README

rdimaio authored Sep 6, 2023

Configuration menu

View commit details

Copy full SHA for 9083190

Browse repository at this point

Copy the full SHA

9083190 View commit details

Browse the repository at this point in the history

Commits on Oct 5, 2023

[packages][xs]: bump portaljs/core patch

olayway committed Oct 5, 2023

Configuration menu

View commit details

Copy full SHA for 1ef517f

Browse repository at this point

Copy the full SHA

1ef517f View commit details

Browse the repository at this point in the history

Added the ability to specify a pre-process function, enabling the inclusion of a custom function for data processing. #577

Added the ability to specify a pre-process function, enabling the inclusion of a custom function for data processing. #577

Commits on Feb 1, 2023

Commits on Feb 3, 2023

Commits on Feb 5, 2023

Commits on Feb 6, 2023

Commits on Feb 14, 2023

Commits on Feb 15, 2023

Commits on Feb 16, 2023

Commits on Feb 17, 2023

Commits on Feb 19, 2023

Commits on Feb 20, 2023

Commits on Feb 21, 2023

Commits on Feb 24, 2023

Commits on Feb 25, 2023

Commits on Feb 26, 2023

Commits on Feb 27, 2023

Commits on Feb 28, 2023

Commits on Mar 1, 2023

Commits on Mar 5, 2023

Commits on Mar 6, 2023

Commits on Mar 7, 2023

Commits on Mar 9, 2023

Commits on Mar 20, 2023

Commits on Mar 21, 2023

Commits on Mar 23, 2023

Commits on Mar 24, 2023

Commits on Mar 26, 2023

Commits on Mar 27, 2023

Commits on Mar 29, 2023

Commits on Mar 30, 2023

Commits on Mar 31, 2023

Commits on Apr 4, 2023

Commits on Apr 6, 2023

Commits on Apr 23, 2023

Commits on Apr 24, 2023

Commits on Apr 26, 2023

Commits on Apr 27, 2023

Commits on May 3, 2023

Commits on May 12, 2023

Commits on May 15, 2023

Commits on May 16, 2023

Commits on May 23, 2023

Commits on Jun 8, 2023

Commits on Jun 9, 2023

Commits on Jun 12, 2023

Commits on Jun 13, 2023

Commits on Jun 14, 2023

Commits on Jun 15, 2023

Commits on Jun 19, 2023

Commits on Jun 21, 2023

Commits on Jun 22, 2023

Commits on Jul 13, 2023

Commits on Jul 14, 2023

Commits on Aug 4, 2023

Commits on Aug 10, 2023

Commits on Aug 14, 2023

Commits on Sep 6, 2023

Commits on Sep 11, 2023

Commits on Sep 16, 2023

Commits on Sep 19, 2023

Commits on Oct 4, 2023

Commits on Oct 5, 2023

Commits on Oct 6, 2023

Commits on Oct 9, 2023

Commits on Oct 11, 2023

Commits on Oct 18, 2023