Doctest for `Quantity` ,`KnownUnitQ`, and others fail when the documentation is generated.

[mmatera]

Description

When I tried to generate the documentation, these symbols failed some tests. The tests do not fail in the standard CI tests, and the output in the terminal front-end is correct.

How to Reproduce

make doctest

Output Given

********** Units and Quantities / KnownUnitQ **********
4396 ( 1): TEST KnownUnitQ["Feet"]
4397 ( 2): TEST KnownUnitQ["Foo"]
4398 ( 3): TEST KnownUnitQ["meter"^2/"second"]
result != wanted
----------------------------------------------------------------------
Test failed: in 
Result: False
Wanted: True
********** Units and Quantities / Quantity **********
4399 ( 1): TEST Quantity["Kilogram"]
4400 ( 2): TEST Quantity[10, "Meters"]
4401 ( 3): TEST Quantity[{10, 20}, "Meters"]
4402 ( 4): TEST Quantity[2, 3/2]
4403 ( 5): TEST QuantityQ[Quantity[2, Second]]
4404 ( 6): TEST Quantity[3, "centimeter"] / Quantity[2, "second"]^2
result != wanted
----------------------------------------------------------------------
Test failed: in 
Result: 3 / 4 centimeter / Null
Wanted: 3 / 4 centimeter / second ^ 2
Additional output:
Quantity::unkunit: Unable to interpret unit specification Null.
Quantity::unkunit: Unable to interpret unit specification 1 / Null.
Quantity::unkunit: Unable to interpret unit specification centimeter / Null.

Expected behavior

No errors

Workarounds

Modify documentation.tex to include the right output.

Priority

Low

[rocky]

What I am seeing in debugging troubles me. Something in this context is causing "meter"^2/"second" to evaluate to Null because of running out of matching rules.

As a result of recent work, we've seen that when None is returned, we have to keep the old expression, and a change had to be made to restore the old value.

Something like that is possibly going on here, via pattern-matching.

Some details captured from a debug session:

(/tmp/Mathics3/mathics-core/mathics/core/pattern.py:833 @14): basic_match_expression
-> 833 def basic_match_expression(
expression = "meter"^2/"second"
...
(((trepan3k))) n
(/tm[/Mathics3/mathics-core/mathics/core/pattern.py:525 @750): match
<- 525             except (StopGenerator_ExpressionPattern_match, TimeoutInterrupt):
R=> None

And here the None is turning into Null^2/"second" whichKnownUnitQ[] treats as false.

[mmatera]

What is funny is that this behavior happens only when we run doctests.

[rocky]

What is funny is that this behavior happens only when we run doctests.

This may be a demonstration of the weakness of our testing.

Recall, we found a major bug in rule priority selection that was initially thought only to occur in running Combinatorica tests.

Isn't it "funny" that Newton's laws applied to planetary motion only had a discrepancy when used on Mercury?

[mmatera]

What is funny is that this behavior happens only when we run doctests.

This may be a demonstration of the weakness of our testing.

Recall, we found a major bug in rule priority selection that was initially thought only to occur in running Combinatorica tests.

That's different: I expected that Combinatoria tests detect more specific cases, because it uses more complex constructions. This case is different because the unique difference would be the "front-end" where the results are "shown".

Isn't it "funny" that Newton's laws applied to planetary motion only had a discrepancy when used on Mercury?

It was before Einstein. Now it is perfectly natural :)

[rocky]

What is funny is that this behavior happens only when we run doctests.

This may be a demonstration of the weakness of our testing.
Recall, we found a major bug in rule priority selection that was initially thought only to occur in running Combinatorica tests.

That's different: ...

What was initially said at the time is that it only occurs in running Combinatorica tests.

After the fact, well, yes, it's obvious.

Isn't it "funny" that Newton's laws applied to planetary motion only had a discrepancy when used on Mercury?
It was before Einstein. Now it is perfectly natural :)

And this is my point. Before investigating, it is natural to expect that this is a one-off anomaly. The invention of relativity was due to several "one-off" anomalies.

Sometimes, great physics discoveries come out of looking at one-off anomalies.

Coming back to the situation at hand. Upon a deeper investigation of what's going on here, there is a strong suggestion that the problem is due to running out of finding a rule replacement (which is expected for something like "minus"^2") Currently, we return Null, and we were likely returning `"minus"^2" before.

There was a corresponding "fix" the first time we noticed that the revisions changed pattern-matching behavior and returned None, which did not preserve the previous expression.

Frankly, the "fix" at that time felt a little bit low-level. I would not be surprised if there isn't another kind of patch that might have handled both kinds of situations, assuming that the analysis I put forth is correct.

Isn't it funny that this used to work 100% of the time previously, and now 100% of the time it doesn't?

[mmatera]

Again, what looks weird to me is that something at the level of the "front-end" modifies this low-level rule. I plan to investigate it during the weekend.

[rocky]

In WMA

KnownUnitQ["Meter"^2/"Seconds"] 

is False.

Changing this example to:

KnownUnitQ["Feet"^2/"Farads"] 

gives something that is True. (But we still fail in building docs).

[rocky]

Moving on to tracking down why we're not seeing Network Graphs in the printed documentation, I am seeing failures in Mathics3 Module tests, as well as symbols that are getting Nulled symbols.

These problems may be related to this issue.

[mmatera]

To narrow a little bit the problem, it seems that "string"^2 is evaluated after the test 4369 ( 1): TEST $TraceBuiltins = True as Null.
I detect this by locally hacking the method mathicssession.evaluate_as_in_cli, and force to check the evaluation of "second"^2 before the true evaluation, and check the result:

4369 ( 1): TEST $TraceBuiltins = True
4370 ( 2): TEST $TraceBuiltins = False
----------------------------------------------------------------------
Test failed: in Tracing and Profiling / $TraceBuiltins
Exception System`Null
Traceback (most recent call last):
  File "/home/mauricio/Projects/mathics-core/mathics/docpipeline.py", line 217, in test_case
    result = test_pipeline.session.evaluate_as_in_cli(test.test, src_name=src_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mauricio/Projects/mathics-core/mathics/session.py", line 162, in evaluate_as_in_cli
    assert result.has_form("System`Power", 2), f"{result}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: System`Null

The change in the rule happens before the test 4370.

[mmatera]

Indeed: adding a sort_order to the mathics.builtin.trace module to make that the tests of this module be evaluated first, many rules, the issue extends to many other tests. It is like something get broken in the generation of the latex form of some of the outputs in this module.

UPDATE: I can confirm that skipping the creation of the latex output for the mathics.builtin.trace module, the error disappears.

[mmatera]

Moving on to tracking down why we're not seeing Network Graphs in the printed documentation, I am seeing failures in Mathics3 Module tests, as well as symbols that are getting Nulled symbols.

I am having problems installing the current main version of pymathics.graph. This makes that the graphics are not available.

[rocky]

Moving on to tracking down why we're not seeing Network Graphs in the printed documentation, I am seeing failures in Mathics3 Module tests, as well as symbols that are getting Nulled symbols.

I am having problems installing the current main version of pymathics.graph. This makes that the graphics are not available.

Follow how mathics-omnibus does it? Use the docker image on dockerhub?

[mmatera]

Moving on to tracking down why we're not seeing Network Graphs in the printed documentation, I am seeing failures in Mathics3 Module tests, as well as symbols that are getting Nulled symbols.

I am having problems installing the current main version of pymathics.graph. This makes that the graphics are not available.

Follow how mathics-omnibus does it? Use the docker image on dockerhub?

pymathics-graph 8.0.1 does install, but have compatibility problems with the property element_order, while the master branch fails at the installation process.

[rocky]

Ah - I think I've run into that, thanks to Python changing to poetry and pyproject.toml and how that works.

I think I got around this using adding the --no-build-isolation option on pip, e.g. pip install --no-build-isolation -e ..

[mmatera]

Now, using the implementation in pymathics.graph at master, but the installer in the version 8.0.1, the documentation is properly generated, including the graphs.