Skip to content

Commit

Permalink
prep for v0.2.0 release
Browse files Browse the repository at this point in the history
  • Loading branch information
@rsdoiel
rsdoiel committed Aug 6, 2020
1 parent ec21fae commit 0dca7c3
Show file tree
Hide file tree
Showing 48 changed files with 2,321 additions and 2,782 deletions.
2 changes: 2 additions & 0 deletions INSTALL.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,14 @@ Compiled versions are available for macOS (amd64 processor,
macosx-amd64), Linux (amd64 processor, linux-amd64), Windows (amd64
processor, windows-amd64) and Rapsberry Pi (ARM7 processor, raspbian-arm7).

```
| Platform | Zip Filename |
|-------------|-----------------------------------------|
| Windows | mkpage-VERSION_NUMBER-windows-amd64.zip |
| macOS | mkpage-VERSION_NUMBER-macosx-amd64.zip |
| Linux/Intel | mkpage-VERSION_NUMBER-linux-amd64.zip |
| Raspbery Pi | mkpage-VERSION_NUMBER-raspbian-arm7.zip |
```


## The basic recipe
Expand Down
12 changes: 6 additions & 6 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ build: bin/mkpage$(EXT) bin/mkrss$(EXT) \
bin/ws$(EXT) bin/frontmatter$(EXT) bin/blogit$(EXT)


bin/mkpage$(EXT): version.go mkpage.go codesnip.go mkpandoc.go cmd/mkpage/mkpage.go blogit.go mkpandoc.go generators.go
bin/mkpage$(EXT): version.go mkpage.go codesnip.go pandoc.go cmd/mkpage/mkpage.go blogit.go pandoc.go generators.go
go build -o bin/mkpage$(EXT) cmd/mkpage/mkpage.go

bin/mkrss$(EXT): version.go mkpage.go mkrss.go mkpandoc.go cmd/mkrss/mkrss.go
bin/mkrss$(EXT): version.go mkpage.go mkrss.go pandoc.go cmd/mkrss/mkrss.go
go build -o bin/mkrss$(EXT) cmd/mkrss/mkrss.go

bin/sitemapper$(EXT): version.go mkpage.go cmd/sitemapper/sitemapper.go
Expand Down Expand Up @@ -57,8 +57,8 @@ bin/blogit$(EXT): version.go blogit.go mkpage.go cmd/blogit/blogit.go
lint:
golint mkpage.go
golint mkpage_test.go
golint mkpandoc.go
golint mkpandoc_test.go
golint pandoc.go
golint pandoc_test.go
golint blogit.go
golint blogit_test.go
golint cmd/mkpage/mkpage.go
Expand All @@ -76,8 +76,8 @@ lint:
format:
gofmt -w mkpage.go
gofmt -w mkpage_test.go
gofmt -w mkpandoc.go
gofmt -w mkpandoc_test.go
gofmt -w pandoc.go
gofmt -w pandoc_test.go
gofmt -w blogit.go
gofmt -w blogit_test.go
gofmt -w cmd/mkpage/mkpage.go
Expand Down
91 changes: 58 additions & 33 deletions Pandoc-Integration.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,43 +10,68 @@
</header>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="index.html">README</a></li>
<li><a href="license.html">LICENSE</a></li>
<li><a href="install.html">INSTALL</a></li>
<li><a href="docs/">Documentation</a></li>
<li><a href="how-to/">HOW TO</a></li>
<li><a href="RELEASE-NOTES.html">Release Notes</a></li>
<li><a href="https://github.com/caltechlibrary/mkpage">Github</a></li>
<li>
<a href="/">Home</a>
</li>
<li>
<a href="index.html">README</a>
</li>
<li>
<a href="license.html">LICENSE</a>
</li>
<li>
<a href="install.html">INSTALL</a>
</li>
<li>
<a href="docs/">Documentation</a>
</li>
<li>
<a href="how-to/">HOW TO</a>
</li>
<li>
<a href="RELEASE-NOTES.html">Release Notes</a>
</li>
<li>
<a href="https://github.com/caltechlibrary/mkpage">Github</a>
</li>
</ul>
</nav>

<section>
<h1>Pandoc intergation with MkPage</h1>
<h2>A future for MkPage</h2>
<p>This is a distillation of my thinking about Markdown processors and <strong>mkpage</strong> as static website generator. I am excited about the current state of <a href="https://pandoc.org/">Pandoc</a> because of Mike Hucka's <a href="https://github.com/mhucka/pangolin-notebook">Pangolin</a>.</p>
<h2>What's Pandoc?</h2>
<p>Pandoc is a document conversion tool. It converts to and from a ridiculous number of formats. In addition to HTML it includes output formats I want to use in the near future like ePub2, ePub3, PDF. Pandoc has excellent support Mathematical text. Pandoc is <a href="https://rstudio.com/">R Studio</a>'s Markdown to HTML engine. This is important as that means people who use R Studio to create web content will know Pandoc's dialect of Markdown even if they don't know they are using Pandoc. Pandoc can be extended using <a href="https://lua.org">Lua</a>. Pandoc itself is written in <a href="https://www.haskell.org/">Haskell</a> which is an interesting language.</p>
<p>On a selfish note, Pandoc supports the features of Markdown I regularly use, code blocks, tables and foot/end notes and Pandoc will convert from Markdown to Jira/Confluence's flavor of markup.</p>
<h3>So why talk up Pandoc?</h3>
<p>Based on Mike's Pangolin my thinking around Pandoc has evolved. There is Pandoc the document converter but there is also Pandoc the template engine.</p>
<p>Pandoc as a template engine is intriguing. I was able to read and understand Mike's templates without resorting to documentation. More importantly <strong>Pandoc has good documentation</strong>, including their simple template language! This holds promise for future proofing our site generation with <strong>mkpage</strong>.</p>
<h3>mkpage's failure</h3>
<p>A huge glaring failure in my <code>mkpage</code> project has been my reliance on Go's native template language. It was a good bootstrap but not so fun now[^now]. While Go's template language is "enterprise" ready[^ready] Go's template language remains poorly documented. Go's template language has little outside adoption aside from Hugo[^Hugo]. This situation has not improved in the last decade and I think is unlikely to improve. Go's various Markdown to HTML libraries have caused concerns. Like what I observed in the NodeJS community many Go packages fail to be updated with version changes and have become orphaned projects. My concern is that overtime 3rd Party Go packages I've relied on will loose development inertia and be abandoned. I am on my third Markdown library in Go at this stage of <code>mkpage</code>. Maybe it's time to step off the tread mill?</p>
<p>[^now]: May, 19, 2020</p>
<p>[^ready]: In the sense of Java's various "enterprise" ready template languages</p>
<p>[^Hugo]: Hugo starts with Go's template language but extends it into it's own dialect. In other words a niche in a niche. See https://gohugo.io/.</p>
<p>Integrating Pandoc into <code>mkpage</code> has been easily achieved using <code>os.exec</code> along with pipes. Pandoc has the advantage of being a standard Markdown to HTML converter. It has a simpler and much better documented template engine. Pandoc solves two issues in near term evolution of <code>mkpage</code>. I don't believe Pandoc's template language is well know but it is easy to pickup and is much better documented than the Go's template language. If Pandoc had its current feature set existed when I originally started <code>mkpage</code> I probably would not have created <code>mkpage</code>. I would have only focused on making things scriptable and metadata friendly which is where I think <code>mkpage</code> shines.</p>
<p>Where does this leave <code>mkpage</code> project? At a really interesting crossroads (at least to me). I think <code>mkpage</code> collection of tools solves a number of things that are not addressed by Pandoc.</p>
<ol>
<li><code>mkpage</code> has decent front matter support for processing metadata related to Markdown documents</li>
<li><code>mkpage</code> command line language (parameter language) is well suited to scripting and is cleaner than a traditional Unix options syntax for that purpose</li>
<li><code>mkpage</code> as a set of tools, provides many additional features such as RSS generation, a crude Sitemap generator, title and byline extractors which assists in constructing many kindas of websites.</li>
</ol>
<p><code>mkpage</code> evolved from a different purpose than Pandoc and that purpose is still relevant. Content Management Systems (CMS) whether custom, off the shelf, commercial or open sources tend to succumbed to excessive features, complexity and resource consumption. <code>mkpage</code> remains a reflective reaction to the challenges and brittleness of those complex systems. Simplicity, at least at the tool level, is a virtue.</p>
<p><code>mkpage</code> is an Un-CMS, a distillation of the rendering features of complex CMS. It is a set of tools for leveraging metadata provided as JSON documents, command line options and front matter in plain text documents. Like Pandoc it is highly scriptable from languages like Bash, Python and Make. With the option of switching template languages or content processors it remains a useful but simple abstraction in website/content rendering. When you know your "content structure" it is easy automate site construction with simple scripts.</p>
<h2>Where does <strong>mkpage</strong> go from here?</h2>
<p><strong>mkpage</strong> is adopting Pandoc has its Markdown to HTML engine. <strong>mkpage</strong> is adding Pandoc as the default template engine. Pandoc's integration also means <strong>mkpage</strong> inherits support for other markup systems like Textile, RestructText, and Jira Text. Pandoc integration should be complete for the v0.0.33 release of <strong>mkpage</strong>.</p>
<h1>
Pandoc intergation with MkPage
</h1>
<h2>
A future for MkPage
</h2>
<p>
This is a distillation of my thinking about Markdown processors and <strong>mkpage</strong> as static website generator. I am excited about the current state of <a href="https://pandoc.org/">Pandoc</a> because of Mike Hucka’s <a href="https://github.com/mhucka/pangolin-notebook">Pangolin</a>.
</p>
<h2>
What’s Pandoc?
</h2>
<p>
Pandoc is a document conversion tool. It converts to and from a ridiculous number of formats. In addition to HTML it includes output formats I want to use in the near future like ePub2, ePub3, PDF. Pandoc has excellent support Mathematical text. Pandoc is <a href="https://rstudio.com/">R Studio</a>’s Markdown to HTML engine. This is important as that means people who use R Studio to create web content will know Pandoc’s dialect of Markdown even if they don’t know they are using Pandoc. Pandoc can be extended using <a href="https://lua.org">Lua</a>. Pandoc itself is written in <a href="https://www.haskell.org/">Haskell</a> which is an interesting language.
</p>
<p>
On a selfish note, Pandoc supports the features of Markdown I regularly use, code blocks, tables and foot/end notes and Pandoc will convert from Markdown to Jira/Confluence’s flavor of markup.
</p>
<h3>
So why talk up Pandoc?
</h3>
<p>
Based on Mike’s Pangolin my thinking around Pandoc has evolved. There is Pandoc the document converter but there is also Pandoc the template engine.
</p>
<p>
Pandoc as a template engine is intriguing. I was able to read and understand Mike’s templates without resorting to documentation. More importantly <strong>Pandoc has good documentation</strong>, including their simple template language! This holds promise for future proofing our site generation with <strong>mkpage</strong>.
</p>
<h3>
mkpage’s failure
</h3>
<p>
A huge glaring failure in my <code>mkpage</code> project has been my reliance on Go’s native template language. It was a good bootstrap but not so fun now[^now]. While Go’s template language is “enterprise” ready[^ready] Go’s template language remains poorly documented. Go’s template language has little outside adoption aside from Hugo[^Hugo]. This situation has not improved in the last decade and I think is unlikely to improve. Go’s various Markdown to HTML libraries have caused concerns. Like what I observed in the NodeJS community many Go packages fail to be updated with version changes and have become orphaned projects. My concern is that overtime 3rd Party Go packages I’ve relied on will loose development inertia and be abandoned. I am on my third Markdown library in Go at this stage of <code>mkpage</code>. Maybe it’s time to step off the tread mill?
</p>
<p>
</section>

<footer>
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

**MkPage Project** is a collection of tools for rendering static websites.
Featured is the _mkpage_ command, a front end to
[Pandoc](https://pandoc.org) (>= v2.9.x). Pandoc supports converting from many
[Pandoc](https://pandoc.org) (>= v2.10.x). Pandoc supports converting from many
[lightweight markup languages](https://pandoc.org/ "Pandoc's list of supported formats"). _mkpage_ supports metadata encoded as
front matter[^frontmatter] using [YAML](https://yaml.org),
[TOML](https://github.com/toml-lang/toml) and
Expand Down
8 changes: 8 additions & 0 deletions RELEASE-NOTES.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@

# Release Notes

## v0.2.0

+ mkpage assumes pandoc v2.10.x or better
+ mkpage nolonger support Golang templates
+ `-t, -to` now set the "to" value passed to pandoc v2.10.x or better
+ `-f, -from` now sets the "from" value passed to pandoc v2.10.x or better
+ removed mkslides command line program, Pandoc replaces that functionality

## v0.1.2

+ Switched default pandoc parsing mode to commonmark to limit the false positive attempts to process TeX in markdown documents
Expand Down
4 changes: 2 additions & 2 deletions TODO.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@

## Next (road to v1.0.0)

+ [ ] Figure out how to support all pandoc conversion formats
+ [x] Figure out how to support all pandoc conversion formats
+ [ ] Better error messaging (preferrably passing through Pandoc's own error message) for non-0 exit values from Pandoc
+ [ ] Remove support for Go templates, remove pkgassets dependency
+ [x] Remove support for Go templates, remove pkgassets dependency
+ [ ] Decide what do about `sitemappper`, depreciate in favor of other tools or improve to support large sites (e.g. feeds.library.caltech.edu)
+ Current sitemap cli is too naive for sites more than a couple dozen pages
+ Need to support possibly nested sitemap XML references
Expand Down
85 changes: 20 additions & 65 deletions cmd/mkpage/mkpage.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,17 @@ package main
import (
"fmt"
"os"
"path"
"strings"

// Caltech Library packages
"github.com/caltechlibrary/cli"
"github.com/caltechlibrary/mkpage"
"github.com/caltechlibrary/tmplfn"
)

var (
description = `
Using the key/value pairs populate the template(s) and render to stdout.
MkPage renders markdown using Pandoc (version >= v2.10).
`

examples = `
Expand Down Expand Up @@ -81,9 +80,10 @@ That would be expressed on the command line as follows
generateMarkdown bool

// Application Options
templateFNames string
codesnip bool
codeType string
codesnip bool
codeType string
from string
to string
)

func main() {
Expand All @@ -108,9 +108,10 @@ func main() {
app.BoolVar(&generateMarkdown, "generate-markdown", false, "generate markdown documentation")

// Application specific options
app.StringVar(&templateFNames, "t,templates", "", "colon delimited list of Go text templates to use")
app.BoolVar(&codesnip, "codesnip", false, "output just the code bocks, reads from standard input")
app.StringVar(&codeType, "code", "", "outout just code blocks for specific language, e.g. shell or json, reads from standard input")
app.StringVar(&from, "f,from", "markdown", "set the from value (e.g. markdown) used by pandoc")
app.StringVar(&to, "t,to", "html", "set the from value (e.g. html) used by pandoc")

app.Parse()
args := app.Args()
Expand Down Expand Up @@ -139,16 +140,6 @@ func main() {

// Default template name is page.tmpl
templateName := ""
templateSources := []string{}
useGoTemplates := false

// Make sure we have a configured command to run
if len(templateFNames) > 0 {
useGoTemplates = true
for _, fname := range strings.Split(templateFNames, ":") {
templateSources = append(templateSources, fname)
}
}

// Setup IO
var err error
Expand All @@ -167,6 +158,12 @@ func main() {
app.GenerateMarkdown(app.Out)
os.Exit(0)
}
if from != "" {
mkpage.PandocFrom = from
}
if to != "" {
mkpage.PandocTo = to
}

if codesnip || codeType != "" {
err = mkpage.Codesnip(app.In, app.Out, codeType)
Expand All @@ -186,57 +183,15 @@ func main() {
data[pair[0]] = pair[1]
} else {
// Must be the template source
useGoTemplates = false
templateSources = append(templateSources, arg)
templateName = arg
}
}

// Make the page with pandoc, go templates and Go Markdown
switch {
case useGoTemplates:
// DEPRECIATED: Go template support is included for
// backward compatibility. It will be removed when the
// transition is before v1.x.

// Create our Tmpl struct with our function map
tmpl := tmplfn.New(tmplfn.AllFuncs())

// Load any user supplied templates
if len(templateSources) > 0 {
err = tmpl.ReadFiles(templateSources...)
if err != nil {
fmt.Fprintf(app.Eout, "%s\n", err)
os.Exit(1)
}
templateName = path.Base(templateSources[0])
} else {
// Load our default template maps
if err != nil {
fmt.Fprintf(app.Eout, "mkpage %q does note support default templates.", mkpage.Version)
os.Exit(1)
}
}
// Build a template and send to MakePage
t, err := tmpl.Assemble()
if err != nil {
fmt.Fprintf(app.Eout, "template assemblere error, %s\n", err)
os.Exit(1)
}
err = mkpage.MakePage(app.Out, templateName, t, data)
if err != nil {
fmt.Fprintf(app.Eout, "MakePage error, %s\n", err)
os.Exit(1)
}
os.Exit(0)
default:
if len(templateSources) > 0 {
templateName = templateSources[0]
}
err = mkpage.MakePandoc(app.Out, templateName, data)
if err != nil {
fmt.Fprintf(app.Eout, "Pandoc error, %s\n", err)
os.Exit(1)
}
os.Exit(0)
// Make the page with Pandoc
err = mkpage.MakePandoc(app.Out, templateName, data)
if err != nil {
fmt.Fprintf(app.Eout, "Pandoc error, %s\n", err)
os.Exit(1)
}
os.Exit(0)
}
Loading

0 comments on commit 0dca7c3

Please sign in to comment.