Can the ruler output stand-alone PHP?

[flip111]

Taken from the readme

$rule  = 'group in ("customer", "guest") and points > 30';

There is an example of how to convert it to PHP code by calling Hoa\Ruler\Ruler::interprete. But this return \Hoa\Ruler\Model objects. Is it also possible generate stand-alone PHP code?

Getting a closure like this

<?php
$closure = function($group, $points) {
    return in_array($group, ['customer', 'guest']) AND $points > 30;
}

would be great to reuse the code in projects and test it with PHP directly.

Perhaps it can be done with using the Hoa compiler ??

[Hywan]

Hello :-),

The Hoa\Ruler\Model can be serialized. Please, see the examples with $database.

[flip111]

I'm not sure if you are understanding me correctly, or if i'm not understanding you ...
I tried this test script.

<?php
require_once __DIR__.'/vendor/autoload.php';

$test = serialize(
    Hoa\Ruler\Ruler::interprete(
        'group in ("customer", "guest") and points > 30'
    )
);

var_dump($test);

Giving

string(854) "O:21:"Hoa\Ruler\Model\Model":1:{s:8:"\000*\000_root";O:24:"Hoa\Ruler\Model\Operator":3:{s:8:"\000*\000_name";s:3:"and";s:13:"\000*\000_arguments";a:2:{i:0;O:24:"Hoa\Ruler\Model\Operator":3:{s:8:"\000*\000_name";s:2:"in";s:13:"\000*\000_arguments";a:2:{i:0;O:27:"Hoa\Ruler\Model\Bag\Context":2:{s:6:"\000*\000_id";s:5:"group";s:9:"\000*\000_value";N;}i:1;O:30:"Hoa\Ruler\Model\Bag\RulerArray":2:{s:9:"\000*\000_array";a:2:{i:0;O:26:"Hoa\Ruler\Model\Bag\Scalar":1:{s:9:"\000*\000_value";s:8:"customer";}i:1;O:26:"H"...

But i'm looking to output

<?php
$test = <<<EOD
$closure = function($group, $points) {
    return in_array($group, ['customer', 'guest']) AND $points > 30;
}
EOD;

So the PHP code itself is the output

[Hywan]

Does the output of Hoa\ruler\Ruler::interprete is not enought? Well, it outputs the code to build the model. We do not lose any informations. But your proposition is also interesting. It would be a “one-way compilation”. Thoughts @stephpy and @shouze?

[stephpy]

Hi,

I don't see the benefit to transform

echo Hoa\Ruler\Ruler::interprete(
    'logged(user) and group in ("customer", "guest") and points > 30'
);

/**
 * Will output:
 *     $model = new \Hoa\Ruler\Model();
 *     $model->expression =
 *         $model->and(
 *             $model->func(
 *                 'logged',
 *                 $model->variable('user')
 *             ),
 *             $model->and(
 *                 $model->in(
 *                     $model->variable('group'),
 *                     array(
 *                         'customer',
 *                         'guest'
 *                     )
 *                 ),
 *                 $model->{'>'}(
 *                     $model->variable('points'),
 *                     30
 *                 )
 *             )
 *         );
 */

Into

<?php
$test = <<<EOD
$closure = function($group, $points) {
    return in_array($group, ['customer', 'guest']) AND $points > 30;
}
EOD;

Storing closures or objects to interprete things would not change many things imo. 2nd solution is prettier but much more hard to generate.

[flip111]

@Hywan the functionality should stay the same. So when it comes down to "does it work". Yes it's enough. But when you want portable, fast optimized production code then native PHP is the best. It would make this library very interesting for all sorts of projects i think.

I thought maybe this functionality already existed, but i guess not. So now i think maybe the AST from \Hoa\Ruler\Model can be transformed into PHP AST and the converted to PHP code. There is a great library to convert PHP code to PHP AST here https://github.com/nikic/PHP-Parser

So the benefit would be to define your own (domain) language and then convert it to actual PHP code.

If it is possible to do:
PP -> Hoa AST -> PHP AST -> PHP
then the opposite way
PHP -> PHP AST -> Hoa AST -> PP
shouldn't be that difficult as well (but i'm not sure what would be a use case for that)

[Hywan]

Using PHP-Parser from Nikita to rebuild a rule model is definitively not a good idea ;-).
Nevertheless, having a final compilation step, aka “one-way”, in order to get better performance, does not sound bad. This will just be an “optimized” evaluation, with lost informations. It will aim at being only executed, not manipulated (we can't go back to a rule, a model or everything else, it will be pure PHP).

@stephpy mentionned that it will be difficult to implement, and yes, he is right. And this is the main issue. What happen when someone add a new operator with a sophisticated implementation: how do we transform that? By taking an example from the README:

$ruler->getDefaultAsserter()->setOperator('logged', function ( User $user ) {

    return $user::CONNECTED === $user->getStatus();
});

How do we transform/compile the logged function into a PHP code? (Reminder: an operator is just a subset of a function since it is a function with only two arguments).

[flip111]

Hi. Using the PHP-Parser is optional i think, because if you want you can go straight from Hoa AST to PHP. I was just considering that going FROM Hoa AST to PHP AST might be as much work as going from Hoa AST to PHP directly. Using the PHP AST also gives direct access to all the funcitonality implemented by PHP-Parser (which might come in handy later). And also not to reinvent stuff, but re-use where possible. To me this sounds like a good approach, but then i am not familiar with the hoa codebase, so it might not be a good idea indeed.

I don't really understand the problem with the logged() function. Below i wrote a possible implementation in PHP. It might not be the prettiest PHP code out there (compared if it was written by hand). But looking at it was generated from a rule-set it's pretty great!

<?php
// 'group in ("customer", "guest") and points > 30'
$closure = function($group, $points) {
    return in_array($group, ['customer', 'guest']) AND $points > 30;
}

//$ruler->getDefaultAsserter()->setOperator('logged', function ( User $user ) {
//  return $user::CONNECTED === $user->getStatus();
//});
$closure = function(User $user) {
    return $user::CONNECTED === $user->getStatus();
};

// 'logged(user) and group in ("customer", "guest") and points > 30'
$closure = function($group, $points) use ($user) {
    return function(User $user) {
            return $user::CONNECTED === $user->getStatus();
        } AND
        in_array($group, ['customer', 'guest']) AND
        $points > 30;
}

[Hywan]

return function ( … ) { … } and …?

[flip111]

Hhmm seems i'm overestimating PHP in that case. Alternatively all the operators could be placed in their own variable.

<?php
// Option 1
$closure = function($group, $points) use ($user) {
    $logged = function(User $user) {
            return $user::CONNECTED === $user->getStatus();
    };

    return $logged($user) AND in_array($group, ['customer', 'guest']) AND $points > 30;
}

// Option 2
$closure = function($group, $points) use ($user) {
    // Place contents of closure inside brackets
    return ( $user::CONNECTED === $user->getStatus() ) AND in_array($group, ['customer', 'guest']) AND $points > 30;
}

Option 1 should always work. Not sure about option 2 ... might need some test cases for that.

Test for Option 1 http://codepad.viper-7.com/KzJ6Ks

[Hywan]

But how do extract the body of a new operator/function (here, a closure)?

[flip111]

Aaah now i know what you mean .. yes true this is not so easy. Here is a proof of concept though.

<?php
require_once 'vendor/autoload.php';

class User {
    const DISCONNECTED = 0;
    const CONNECTED = 1;

    public $group = 'customer';
    public $points = 42;
    protected $_status = 1;

    public function getStatus() {

        return $this->_status;
    }

}

$ruler = new Hoa\Ruler\Ruler();

// New rule.
$rule = 'logged(user) and group in ("customer", "guest") and points > 30';

// New context.
$context = new Hoa\Ruler\Context();
$context['user'] = function ( ) use ( $context ) {

    $user = new User();
    $context['group'] = $user->group;
    $context['points'] = $user->points;

    return $user;
};

// We add the logged() operator.
$ruler->getDefaultAsserter()->setOperator('logged', function ( User $user ) {

    return $user::CONNECTED === $user->getStatus();
});

// Finally, we assert the rule.
var_dump(
    $ruler->assert($rule, $context)
);

/**
 * Will output:
 *     bool(true)
 */

// Let the magic begin
$closure = $ruler->getDefaultAsserter()->getOperator('logged');
$refl = ReflectionFunction::export($closure->getValidCallback(), true);
$refl = explode("\n", $refl);
preg_match('/  @@ (?P<path>.*) (?P<from_line>\d+) - (?P<to_line>\d+)/i', $refl[1], $codeInfo);

$file = explode("\n", file_get_contents($codeInfo['path']));

preg_match('/setOperator\([\'"].*[\'"] *, *(?P<first_line>.*)/i', $file[$codeInfo['from_line'] - 1], $regs);

$code = $regs['first_line'];

for ($i = $codeInfo['from_line']; $i < $codeInfo['to_line']; $i++) {
    $code .= $file[$i];
}

echo $code;

/**
 * Will output:
 * function ( User $user ) {
 * 
 *  return $user::CONNECTED === $user->getStatus();
 * });
 * 
 */

[Hywan]

What about a closure written on a single line ;-)?
What you are trying to achieve is really difficult :-/. @stephpy thoughts?

[flip111]

// for code:
$ruler->getDefaultAsserter()->setOperator('logged', function ( User $user ) {   return $user::CONNECTED === $user->getStatus(); });

// same script will out:
/*
 * function ( User $user ) {    return $user::CONNECTED === $user->getStatus(); });
 * 
 */

I understand it's a bit tricky 🌴 (random palm tree). But so far so good. Perhaps i can make it more proof for edge cases and exceptions?

If i would use the PHP Parser then from the AST i could get the perfect code. But i didn't use the PHP Parser because i wanted to show an example without additional dependencies.

[Hywan]

Ok, so let's go for a POC 👍. We will see where we can go!

[flip111]

Ok cool !
Just the main question, is it ok for you guys to use PHP-Parser?

[Hywan]

Nop :-). In a Hoathis, certainly yes, but not in Hoa since this is the standard libraries.

[Hywan]

Hmm, what about a closure (defining a new function) with a use. It would be really difficult to catch everything.

[flip111]

That's actually the easy part. Because the variables you pass in on use() are available at run time and thus can be used directly. Put them in a $context variable, and then pass in the $context and done.

Is it ok to make it a hoathis project? The problem is that it will be difficult and error prone when not using a tokenizer/parser. And PHP-Parser is the best one for that. Unless you want to write your own.

Perhaps i (we ?) will do a POC using the PHP-Parser .. then later if you want to make it hoa only you can write your own parser or regexp??

[Hywan]

We can write a parser for PHP since we write the grammar of PHP in PP (please, see Hoa\Compiler). For now, try a POC with PHP-Parser from Nikita, it would be a nice step.

[flip111]

Ok great, POC is done. I just need some people who can write crazy complicated php code to make a few test cases now ...

<?php
ini_set('xdebug.max_nesting_level', 2000); // https://github.com/nikic/PHP-Parser/blob/master/doc/2_Usage_of_basic_components.markdown#bootstrapping
require_once 'vendor/autoload.php';

class User {
    const DISCONNECTED = 0;
    const CONNECTED = 1;

    public $group = 'customer';
    public $points = 42;
    protected $_status = 1;

    public function getStatus() {

        return $this->_status;
    }

}

$ruler = new Hoa\Ruler\Ruler();

// New rule.
$rule = 'logged(user) and group in ("customer", "guest") and points > 30';

// New context.
$context = new Hoa\Ruler\Context();
$context['user'] = function ( ) use ( $context ) {

    $user = new User();
    $context['group'] = $user->group;
    $context['points'] = $user->points;

    return $user;
};

// We add the logged() operator.
$ruler->getDefaultAsserter()->setOperator('logged', function ( User $user ) {

    return $user::CONNECTED === $user->getStatus();
});

// Finally, we assert the rule.
var_dump(
    $ruler->assert($rule, $context)
);

/**
 * Will output:
 *     bool(true)
 */

// Let the magic begin
$closure = $ruler->getDefaultAsserter()->getOperator('logged');
$refl = ReflectionFunction::export($closure->getValidCallback(), true);
$refl = explode("\n", $refl);
preg_match('/  @@ (?P<path>.*) (?P<from_line>\d+) - (?P<to_line>\d+)/i', $refl[1], $codeInfo);

$file = file_get_contents($codeInfo['path']);

// Using PHP-Parser
$parser = new PhpParser\Parser(new PhpParser\Lexer);

try {
    $stmts = $parser->parse($file);
} catch (PhpParser\Error $e) {
    echo 'Parse Error: ', $e->getMessage();
}

foreach ($stmts as $stmt) {
    if ($stmt->getAttribute('startLine') == $codeInfo['from_line'] AND
        $stmt->name === 'setOperator') {
        $closure = $stmt->args[1]->value;
    }
}

$prettyPrinter = new PhpParser\PrettyPrinter\Standard;

$code = '<?php' . PHP_EOL . $prettyPrinter->prettyPrint([$closure]);

file_put_contents('closure.php', $code);

/**
 * Will output:
 * 
 * <?php
 * function (User $user) {
 *     return $user::CONNECTED === $user->getStatus();
 * };
 * 
 */

[Hywan]

ping @stephpy or @jubianchi?

[stephpy]

Nice work, I'll make some tests tomorrow.

There is some points which are annoying to me:

EACH operators could be technically added with many many different ways ...
Even if it work and you made an awesome job, it's OK for THIS USE CASE (which is may be the main one).
We have to warn people for this.

[flip111]

Hey to get things going i was working on the idea i had before .. namely to convert Hoa AST to PHP AST. And it's looking pretty good so far. I can do simple stuff like this

<?php
$AST = $dumper->AstConvert($this->RuleToCode("2 is 3"));
$PHP = $this->getPHP($AST);
$this->assertEquals("2 == 3;", $PHP);

but also more complicated stuff like

<?php
$rule = "'foo' in ('foo', 'bar')";
$AST = $dumper->AstConvert($this->RuleToCode($rule));
$PHP = $this->getPHP($AST);
$this->assertEquals("in_array('foo', array('foo', 'bar'));", $PHP);

so this does Hoa AST -> PHP AST -> PHP

I can't make combination yet like "foo" in ("foo", "bar") and 50 > 30. Also no variable support. And the closures i have to add last. Then it should be done ^^

Hmm this would have been easier if we could just convert the ruler grammar to PHP grammar and build the AST from there. Maybe this can be a feature for next time?

[flip111]

Is it possible to get the output of Hoa\Ruler\Ruler::interprete as php variable instead of string? Right now you would have to write it to a file and then include the file to get the variable again.

[Hywan]

Actually, the interpretation builds the model, and the root of the model has a __toString method that compiles the model into PHP code.

[flip111]

Ah thanks! so i think it's better to use the model directly then in the conversion?

I spend a few hours on the converter class ... so i hope i'm on the right way with this and not wasting work. (please look at the examples above)

[flip111]

@shouze @Hywan I'm rewriting the class and make the transformation go both ways. It's just that i don't know how to get back to the rule once i have an object model. I need the Ruler to be able to do this to make the transformation complete.

[Hywan]

@flip111 I think you just lost me ;-).
First, when you speak about Hoa\Model, do you speak about the library or is it a typo for Hoa\Ruler\Model? I think it's a typo.

Secondly, you propose to write the grammar of PHP with the PP language. Yes, it's a good idea, but I don't know if it is even possible. I have exchanged some greetings with Nikita Popov (@nikic) few monts ago about it, and he pointed me out that some parts of the PHP grammar are very ambiguous. Our discussion was related about building an AST for PHP. On the other hand, HHVM has succeed it… so it seems to be feasible. By the way, we can try!

Finally, you propose to implement a bridge between the model of a rule (an instance of Hoa\Ruler\Model) and the AST of a PHP code (given by Hoa\Compiler and the adequate grammar)?

Thus, from the model of a rule, we would be able to produce PHP code. Right. But what about code dependencies, such as global variables (berk), dedicaded autoload… in short, how to catch the dynamic context of the callable used to declare a new operator? The answer is: by inferencing the code. This is the only solution I see.

Did I understand?

[nikic]

@Hywan The PHP grammar isn't really ambiguous, it's just rather complex and unstructured, especially where variables are concerned. I'm sure you can whip out a grammar for your LLk parser. If you do so, I recommend writing a corresponding pretty printer and then checking whether a parse(1) -> pretty print -> parse(2) cycle has the same result in (1) and (2) on all the php-src tests (and a other large projects like Symfony/ZF). That way you should be able to get a robust parser (if you put in enough effort ^^). I use the same process for the php-parser project.

Anyway, I'm writing this totally out of context, I didn't read the rest of the thread, so no idea what this is all about ^^

[Hywan]

@nikic Absolutely, this would definitively be the good process to verify the grammar and therefore the compiler. Moreover, we would be able to generate PHP programs thanks to some algorithms I developed during my PhD thesis (please see the details). Would you like to help to write such a grammar ;-)? By the way, it's totally out of subject.

[flip111]

@Hywan

Thanks for your response. Yes sorry I did mean \Hoa\Ruler\Model and not \Hoa\Model.

I'm happy to hear that you consider writing PHP in PP language something worth looking into. Perhaps this will be an option in the feature (as I do think it would be pretty complicated).

I think the \Hoa\Ruler\Model could represent all kinds of languages not just the specific Ruler Grammar. But yes you are right, i'm building a bridge to convert from \Hoa\Ruler\Model (with Ruler grammer) to PHP-AST. My first trial version is uploaded here https://github.com/flip111/HoaRulerPHPDumper I'm not too satisfied with this version because it doesn't operate on the Model directly (instead it parses the model's PHP code). And also I can not transform back from PHP to \Hoa\Ruler\Model.

As to your question on how to catch the dynamic context. I'm really not there yet, so i'm just gonna go ahead an convert everything that i can and see where it ends up. My immediate problem is that i can not convert a \Hoa\Ruler\Model back to a rule string!!!

do you have any idea on how to convert \Hoa\Ruler\Model back to a rule string??

[Hywan]

Sure, with Hoa\Ruler\Visitor\Disassembly.

[nikic]

@Hywan I think it's an interesting thing to try, but I've been so busy recently that I hardly keep up with my own projects. Would be too much to start writing a PHP grammar now (which is a rather lot of work...)

[Hywan]

@nikic I keep you informed when I'll start. I'm very busy too, but I'll start in few months.

[flip111]

The second version is ready. This version features direct Hoa\Ruler\Model to PHP-Parser AST conversion which is also reversible. https://github.com/flip111/HoaRulerPHPDumper/tree/v2

Note that this is still a partial implementation, which only covers the very basic functions.

[Hywan]

But it's cleaner :-). What is your next step?

[flip111]

next steps:

1. "foo" in ("foo", "bar") and 50 > 30 (combinations)
2. variables
3. operators

[flip111]

Hi, the functionality is done. You can see it here https://github.com/flip111/HoaRulerPHPDumper/tree/v2 Please review and tryout.

There is a small issue #6 that leads to a test failing, but one-by-one the tests work. Including the example of the documentation.

[flip111]

How can i get more attention for this issue? Need reviewers to speed it along now ...

[Hywan]

Could you update the README.md with an example please?

[flip111]

Updated README.md. please review https://github.com/flip111/HoaRulerPHPDumper/tree/v2

[Hywan]

I have reviewed it. Nice job! Just some comment about the README.md here flip111/HoaRulerPHPDumper@51f0718.

[flip111]

Is there a good way to make this part of Hoa Ruler itself ? or else another Hoa helper class?

[mnapoli]

FYI

But how do extract the body of a new operator/function (here, a closure)?

Have a look at SuperClosure from @jeremeamia, I'm using it in PHP-DI to extract the body of a closure and it works great (it's very well tested). His V2 will have support for closures with use I believe (I don't know exactly how they will be handled).

[Hywan]

@flip111 As long as there is a dependency to PHP-Parser, it will not be part of the standard libraries, but it can be an Hoathis (this is the goal of this “family” of libraries). We can referenciate this library on our website and maybe in the README. This is the hot topic of the mailing-list: “how to promote projects based on Hoa”. Some projects are hosted by us, someother will be listed in a specific repository etc. (see hoathis.net by @camael24, a POC).

[flip111]

@Hywan
Maybe the Hoa\Ruler can take a Dumper object? Maybe we can agree to an interface on how this object is called by the Hoa\Ruler? XML might also be interesting if you want to export the AST of hoa-ruler-model

@mnapoli
Thanks for your suggestion of SuperClosure, however the current library already works now. So until it has more exposure and people start using it i won't add the dependency to SuperClosure. I briefly looked at SuperClosure and i think it uses a similar approach (but one better tested perhaps)

[Hywan]

@flip111 Such an interface would solve the #1?

[flip111]

Hmm interesting topic ...

Where as i have chosen for rule --> hoa\ruler\model -> php-ast -> php you suggest in this comment rule -> json with the use of a custom visitor/asserter/compiler...

So there is a bunch of formats of how a rule can be "stored", some "storage" can actually execute directly instead of having to be interpreted again (that would be the conversion to PHP i made). So there is:

• A written rule
• A rule in hoa\ruler\model
• A rule in PHP
• A rule in JSON
• A rule in XML ?
• A rule in RuleML

Now for each one someone would have to program a specific converter from one format to another one. Also there is not really a "base" format established, sure the base is the actual written rule itself ... but the grammar might change in the feature then all specific converters have to be rewritten. What would be really interesting if you were to have all these language definitions in PP then you would just have to define the mapping you want to do and something could build a converter for you... Yes that is actually easier then having to write recursive methods ...

My concrete question is:
I have 2 PP language definitions, and i have something written in 1 language, if i map the language files to each other would it be possible to return the code in the other language??

As for the actual syntax of the mapping file itself ... i have no clue so far ...

[Hywan]

The compilation process goes like this: we have a lexical analyzer (aka lexer) that produces a sequence of lexemes (aka tokens). This sequence is given to a syntactic analyzer (aka parser) to assert whether the sequence is valid or not. The “validity” of a sequence is given by a grammar. At the end of this process, when the sequence is valid, the parser might be able to produce an AST (that stands for Abstract Syntax Tree). In the case of Hoa\Ruler, this AST is visited and transformed into a model. While the grammar holds syntactic constraints, the model holds semantics ones. Then, when both syntax and semantics have been verified, we can either execute the model (aka interpreted language) or transform the model into another language (aka compiled language). That's it.

What you propose is to transform one AST to another. The compilation process will go on with the visit of the new AST to get a new model etc. Then, your proposal is related to the “tree transformations” domain. Unfortunately, there is nothing in the standard set of libraries of Hoa to do this. But I'm not sure we need such algorithms.

What about creating Hoa\Ruler\Visitor\Format\* libraries for a one-way compilation, which would include RM (stands for Ruler's Model) → PHP, RM → JSON, RM → RuleML (of course, if those languages make sense, I don't know RuleML for example, I cannot say if it is interesting for us or not). When we will compile RM to another language, we might lost some informations (remember the logged function that is user-defined, what semantics do we attach to it in JSON or RuleML?), and on the contrary, some informations might miss. Another question: we speak about one-way compilation (RM → something), but do we need a bijection here (something → RM)? I don't know.

[flip111]

The loss of information had occurred to me. This is only natural, as with human spoken languages, some expressions/sayings can not be said in other languages only be described. It's the same with code.

The logged function is actually a special case here, because you embed PHP into a rule (which is of another language). It would be comparable with writing an English text and then referencing a quote which is in french. On a pure language definition it's a limitation of the rule-language that it can not express function itself but has to utilize another language to do that.

You are also right about the Tree transformation. One example if that can be found here https://github.com/flip111/HoaRulerPHPDumper/blob/v2/src/PHPDumper.php#L134 where a hoa\ruler\model function is transformed to multiple nodes in PHP-AST (FuncCall with Arg objects as childrion). I guess it's a translation dictionary ^^

I will investigate the idea a bit further when i try to convert EBNF language definition to PP language definition. The language definition being a language itself of course... Related to: hoaproject/Compiler#17

Thanks again for your input

[flip111]

I moved version 2 to be the master branch, fixed some small things. This is now "done" for the moment (i'm not actively developing it). Final version: https://github.com/flip111/HoaRulerPHPDumper

I can continue development in case there is a need for it. I'm still very open to closer integration into Hoa because my library is probably not getting enough exposure now. Maybe mention it in the Ruler readme somewhere?

[Hywan]

Is it possible to create an Hoathis of this? Since you have a dependency to another project, this is the closest place to Hoa for your contribution. Of course, it would be possible to mention this library somewhere. This is another opened topic on IRC and ML: how to promote external projects that enhance Hoa (and this is difficult, we have few resources). I know you are on IRC so let's give us your opinion about it :-). Thanks!

[Hywan]

Also, is the README up-to-date?