Skip to content

Commit

Permalink
v. 1.0.1
Browse files Browse the repository at this point in the history
  • Loading branch information
Nikos M committed Aug 6, 2015
1 parent ae9bc65 commit b402358
Show file tree
Hide file tree
Showing 9 changed files with 204 additions and 101 deletions.
15 changes: 8 additions & 7 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ See also: [codemirror-grammar](https://github.com/foo123/codemirror-grammar) ,

###Features

* A grammar can **extend other grammars** (so arbitrary variations and dialects can be parsed more easily)
* [`Grammar`](/grammar-reference.md) includes: `Style` Model , `Lex` Model and `Syntax` Model (optional), plus a couple of *settings* (see examples)
* `Grammar` **specification can be minimal** (defaults will be used) (see example grammars)
* [`Grammar Syntax Model`](/grammar-reference.md) can enable highlight in a more context-specific way, plus detect possible *syntax errors*
* [`Grammar Syntax Model`](/grammar-reference.md) can contain *recursive references*
* [`Grammar Syntax Model`](/grammar-reference.md) can be specificed using [`PEG`](https://en.wikipedia.org/wiki/Parsing_expression_grammar)-like notation or [`BNF`](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form)-like notation (**NEW feature**)
* Generated parsers are **optimized for speed and size**
* A `Grammar` can **extend other `Grammars`** (so arbitrary `variations` and `dialects` can be handled more easily)
* [`Grammar`](/grammar-reference.md) includes: **`Style Model`** , **`Lex Model`** and **`Syntax Model`** (optional), plus a couple of *settings* (see examples)
* **`Grammar` specification can be minimal** (defaults will be used) (see example grammars)
* `Grammar.Syntax Model` can enable highlight in a more context-specific way, plus detect possible *syntax errors*
* `Grammar.Syntax Model` can contain **recursive references**
* `Grammar.Syntax Model` can be (fully) specificed using [`PEG`](https://en.wikipedia.org/wiki/Parsing_expression_grammar)-like notation or [`BNF`](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form)-like notation (**NEW feature**)
* Generated highlighters are **optimized for speed and size**
* Can generate a syntax-highlighter from a grammar **interactively and on-the-fly** ( see example, http://foo123.github.io/examples/prism-grammar )


###How to use:
Expand Down
12 changes: 6 additions & 6 deletions api-reference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ __Method__: `extend`
extendedgrammar = PrismGrammar.extend( grammar, basegrammar1 [, basegrammar2, ..] );
```

Extend a grammar with basegrammar1, basegrammar2, etc..
Extend a `grammar` with `basegrammar1`, `basegrammar2`, etc..

This way arbitrary dialects and variations can be handled more easily
This way arbitrary `dialects` and `variations` can be handled more easily



Expand All @@ -35,9 +35,9 @@ __Method__: `parse`
parsedgrammar = PrismGrammar.parse( grammar );
```

This is used internally by the PrismGrammar Class
In order to parse a JSON grammar to a form suitable to be used by the syntax-highlight parser.
However user can use this method to cache a parsedgrammar to be used later.
This is used internally by the `PrismGrammar` Class
In order to parse a `JSON grammar` to a form suitable to be used by the syntax-highlighter.
However user can use this method to cache a `parsedgrammar` to be used later.
Already parsed grammars are NOT re-parsed when passed through the parse method again


Expand All @@ -48,5 +48,5 @@ __Method__: `getMode`
mode = PrismGrammar.getMode( grammar );
```

This is the main method which transforms a JSON grammar into a syntax-highlighter for Prism.
This is the main method which transforms a `JSON grammar` into a syntax-highlighter for `Prism`.

2 changes: 1 addition & 1 deletion beeld.config
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ tasks =[{}]

"@@ROOT@@" = "this"
"@@EXPORTS@@" = "exports"
"@@VERSION@@" = "1.0"
"@@VERSION@@" = "1.0.1"
"@@MODULE_NAME@@" = "PrismGrammar"

@
Expand Down
137 changes: 94 additions & 43 deletions build/prism_grammar.js
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/**
*
* PrismGrammar
* @version: 1.0
* @version: 1.0.1
*
* Transform a grammar specification in JSON format, into a syntax-highlighter for Prism
* https://github.com/foo123/prism-grammar
Expand Down Expand Up @@ -292,7 +292,7 @@ var undef = undefined, PROTO = 'prototype', HAS = 'hasOwnProperty', IS_ENUM = 'p
},

newline_re = /\r\n|\r|\n/g, dashes_re = /[\-_]/g,
bnf_special_re = /^([{}()*+?|'"]|\s)/,
peg_bnf_notation_re = /^([{}()*+?|'"]|\s)/,

has_prefix = function(s, id) {
return (
Expand Down Expand Up @@ -1546,20 +1546,13 @@ CompositeToken = Class(Token, {
}
});

function parse_bnf_shorthand( tok, Lex, Syntax, sub_seq )
function parse_peg_bnf_notation( tok, Lex, Syntax )
{
var alternation, sequence, token, literal, repeat,
t, q, c, prev_token, curr_token;
t, q, c, prev_token, curr_token, stack, tmp;

if ( 'undefined' === typeof tok.pos )
{
t = new String( trim(tok) );
t.pos = 0;
}
else
{
t = tok;
}
t = new String( trim(tok) );
t.pos = 0;

if ( 1 === t.length )
{
Expand All @@ -1569,15 +1562,16 @@ function parse_bnf_shorthand( tok, Lex, Syntax, sub_seq )
}
else
{
// parse PEG/BNF-like shorthand notations for syntax groups
alternation = [ ];
sequence = [ ];
token = '';
stack = [];
while ( t.pos < t.length )
{
// parse BNF-like shorthand notations for syntax groups
c = t.charAt( t.pos++ );

if ( bnf_special_re.test( c ) )
if ( peg_bnf_notation_re.test( c ) )
{
if ( token.length )
{
Expand All @@ -1591,7 +1585,7 @@ function parse_bnf_shorthand( tok, Lex, Syntax, sub_seq )
sequence.push( token );
token = '';
}

if ( '"' === c || "'" === c )
{
// literal token, quoted
Expand Down Expand Up @@ -1705,24 +1699,57 @@ function parse_bnf_shorthand( tok, Lex, Syntax, sub_seq )
else if ( '(' === c )
{
// start of grouped sub-sequence
prev_token = parse_bnf_shorthand( t, Lex, Syntax, true );
curr_token = '(' + prev_token + ')';
if ( !Syntax[curr_token] ) Syntax[curr_token] = clone( Lex[prev_token] || Syntax[prev_token] );
sequence.push( curr_token );
stack.push([sequence, alternation, token]);
sequence = []; alternation = []; token = '';
}

else if ( ')' === c )
{
// end of grouped sub-sequence
if ( sub_seq )
if ( sequence.length > 1 )
{
//t.pos++;
break;
curr_token = '' + sequence.join( " " );
if ( !Syntax[curr_token] )
{
Syntax[curr_token] = {
type: 'group',
match: 'sequence',
tokens: sequence
};
}
alternation.push( curr_token );
}
else
else if ( sequence.length )
{
continue;
alternation.push( sequence[0] );
}
sequence = [];

if ( alternation.length > 1 )
{
curr_token = '' + alternation.join( " | " );
if ( !Syntax[curr_token] )
{
Syntax[curr_token] = {
type: 'group',
match: 'either',
tokens: alternation
};
}
}
else if ( alternation.length )
{
curr_token = alternation[ 0 ];
}
alternation = [];

tmp = stack.pop( );
sequence = tmp[0]; alternation = tmp[1]; token = tmp[2];

prev_token = curr_token;
curr_token = '(' + prev_token + ')';
if ( !Syntax[curr_token] ) Syntax[curr_token] = clone( Lex[prev_token] || Syntax[prev_token] );
sequence.push( curr_token );
}

else // space
Expand Down Expand Up @@ -1861,7 +1888,7 @@ function get_tokenizer( tokenID, RegExpID, Lex, Syntax, Style,

if ( T_STR & get_type( tok ) )
{
tok = parse_bnf_shorthand( tok, Lex, Syntax );
tok = parse_peg_bnf_notation( tok, Lex, Syntax );
tok = Lex[ tok ] || Syntax[ tok ];
}

Expand All @@ -1884,24 +1911,24 @@ function get_tokenizer( tokenID, RegExpID, Lex, Syntax, Style,
// loop and get all references
}

// provide some defaults
if ( 'undefined' === typeof tok.type )
{
if ( tok['either'] )
{
tok.type = "group";
tok.match = "either";
tok.tokens = tok['either'];
delete tok['either'];
}
else if ( tok['all'] || tok['sequence'] )
// provide some defaults
if ( tok['all'] || tok['sequence'] )
{
tok.type = "group";
tok.match = "sequence";
tok.tokens = tok['all'] || tok['sequence'];
if ( tok['all'] ) delete tok['all'];
else delete tok['sequence'];
}
else if ( tok['either'] )
{
tok.type = "group";
tok.match = "either";
tok.tokens = tok['either'];
delete tok['either'];
}
else if ( tok['zeroOrMore'] )
{
tok.type = "group";
Expand All @@ -1923,6 +1950,30 @@ function get_tokenizer( tokenID, RegExpID, Lex, Syntax, Style,
tok.tokens = tok['zeroOrOne'];
delete tok['zeroOrOne'];
}
else if ( tok['comment'] )
{
tok.type = "comment";
tok.tokens = tok['comment'];
delete tok['comment'];
}
else if ( tok['block'] )
{
tok.type = "block";
tok.tokens = tok['block'];
delete tok['block'];
}
else if ( tok['escaped-block'] )
{
tok.type = "escaped-block";
tok.tokens = tok['escaped-block'];
delete tok['escaped-block'];
}
else if ( tok['simple'] )
{
tok.type = "simple";
tok.tokens = tok['simple'];
delete tok['simple'];
}
else
{
tok.type = "simple";
Expand Down Expand Up @@ -2179,7 +2230,7 @@ function parse_grammar( grammar )
/**
*
* PrismGrammar
* @version: 1.0
* @version: 1.0.1
*
* Transform a grammar specification in JSON format, into a syntax-highlighter for Prism
* https://github.com/foo123/prism-grammar
Expand Down Expand Up @@ -2524,7 +2575,7 @@ function get_mode( grammar )
[/DOC_MARKDOWN]**/
var PrismGrammar = exports['PrismGrammar'] = {

VERSION: "1.0",
VERSION: "1.0.1",

// extend a grammar using another base grammar
/**[DOC_MARKDOWN]
Expand All @@ -2534,9 +2585,9 @@ var PrismGrammar = exports['PrismGrammar'] = {
* extendedgrammar = PrismGrammar.extend( grammar, basegrammar1 [, basegrammar2, ..] );
* ```
*
* Extend a grammar with basegrammar1, basegrammar2, etc..
* Extend a `grammar` with `basegrammar1`, `basegrammar2`, etc..
*
* This way arbitrary dialects and variations can be handled more easily
* This way arbitrary `dialects` and `variations` can be handled more easily
[/DOC_MARKDOWN]**/
extend: extend,

Expand All @@ -2548,9 +2599,9 @@ var PrismGrammar = exports['PrismGrammar'] = {
* parsedgrammar = PrismGrammar.parse( grammar );
* ```
*
* This is used internally by the PrismGrammar Class
* In order to parse a JSON grammar to a form suitable to be used by the syntax-highlight parser.
* However user can use this method to cache a parsedgrammar to be used later.
* This is used internally by the `PrismGrammar` Class
* In order to parse a `JSON grammar` to a form suitable to be used by the syntax-highlighter.
* However user can use this method to cache a `parsedgrammar` to be used later.
* Already parsed grammars are NOT re-parsed when passed through the parse method again
[/DOC_MARKDOWN]**/
parse: parse_grammar,
Expand All @@ -2563,7 +2614,7 @@ var PrismGrammar = exports['PrismGrammar'] = {
* mode = PrismGrammar.getMode( grammar );
* ```
*
* This is the main method which transforms a JSON grammar into a syntax-highlighter for Prism.
* This is the main method which transforms a `JSON grammar` into a syntax-highlighter for `Prism`.
[/DOC_MARKDOWN]**/
getMode: get_mode
};
Expand Down
4 changes: 2 additions & 2 deletions build/prism_grammar.min.js
View file

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions grammar-reference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ example:

**Syntax shorthand BNF-like notations (new)**

`Syntax` part supports *shorthand definitions* (similar to `BNF-style` definitions) for syntax sequences and groups of syntax sequences:
`Syntax` part supports *shorthand definitions* (similar to `PEG / BNF-style` definitions) for syntax sequences and groups of syntax sequences:

Specificaly:

Expand Down Expand Up @@ -252,7 +252,7 @@ Specificaly:
"tokens": ["t1* t2", "t3"]
}

// a literal tokens wrapped in quotes (' or ")
// literal tokens wrapped in quotes (' or ")
// are equivalent to their literal value
// empty literal token (i.e '') matches empty production
// NOTE: unlike NON-Space token definition described previously
Expand Down
12 changes: 6 additions & 6 deletions src/main.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -356,9 +356,9 @@ var PrismGrammar = exports['@@MODULE_NAME@@'] = {
* extendedgrammar = PrismGrammar.extend( grammar, basegrammar1 [, basegrammar2, ..] );
* ```
*
* Extend a grammar with basegrammar1, basegrammar2, etc..
* Extend a `grammar` with `basegrammar1`, `basegrammar2`, etc..
*
* This way arbitrary dialects and variations can be handled more easily
* This way arbitrary `dialects` and `variations` can be handled more easily
[/DOC_MARKDOWN]**/
extend: extend,

Expand All @@ -370,9 +370,9 @@ var PrismGrammar = exports['@@MODULE_NAME@@'] = {
* parsedgrammar = PrismGrammar.parse( grammar );
* ```
*
* This is used internally by the PrismGrammar Class
* In order to parse a JSON grammar to a form suitable to be used by the syntax-highlight parser.
* However user can use this method to cache a parsedgrammar to be used later.
* This is used internally by the `PrismGrammar` Class
* In order to parse a `JSON grammar` to a form suitable to be used by the syntax-highlighter.
* However user can use this method to cache a `parsedgrammar` to be used later.
* Already parsed grammars are NOT re-parsed when passed through the parse method again
[/DOC_MARKDOWN]**/
parse: parse_grammar,
Expand All @@ -385,7 +385,7 @@ var PrismGrammar = exports['@@MODULE_NAME@@'] = {
* mode = PrismGrammar.getMode( grammar );
* ```
*
* This is the main method which transforms a JSON grammar into a syntax-highlighter for Prism.
* This is the main method which transforms a `JSON grammar` into a syntax-highlighter for `Prism`.
[/DOC_MARKDOWN]**/
getMode: get_mode
};
Loading

0 comments on commit b402358

Please sign in to comment.