StrictXmir #3652

yegor256 · 2024-12-12T12:54:51Z

related to #3636

yegor256 · 2024-12-12T12:55:04Z

@volodya-lombrozo please, check this one

volodya-lombrozo

@yegor256 This PR doesn't solve the #3636 issue. It only speeds up the inner eo tests. If we don't have an internet connection or there is an outage, this solution won't help.

volodya-lombrozo · 2024-12-12T13:33:06Z

eo-parser/src/main/java/org/eolang/parser/StrictXmir.java

+/**
+ * XMIR that validates itself right after construction.
+ *
+ * <p>This class is supposed to be used ONLY for testing, because


@yegor256 I'm a bit confused here. You mentioned in the PR description that this PR closes #3636. And here, in the javadoc you write

This class is supposed to be used ONLY for testing

It means, I won't be able to use StrictXmir in jeo-maven-plugin production code. However, I'm using StrictXML during the xmir generation. In other words, I use it in the production code.

@volodya-lombrozo you can try to do this, on your side: 1) create XML object, 2) save it on disc, 3) make StrictXmir(xml) and 4) either fail, or continue. If you fail, you fail. If you continue, you have a clean XML on disc.

@yegor256 [NIT] I mean, this class might be used not only for testing. Actually it's an important part of validation in a production code. Maybe we need to remove this line from javadoc:

This class is supposed to be used ONLY for testing

and replace it with something like

ATTENTION!

what do you think?

@volodya-lombrozo explained better in 04af5af

volodya-lombrozo · 2024-12-12T13:38:14Z

eo-parser/src/main/java/org/eolang/parser/StrictXmir.java

+                    ).asBytes()
+                );
+            } catch (final IOException | URISyntaxException ex) {
+                throw new IllegalArgumentException(ex);


@yegor256 [NIT] It's a bad idea by a several reasons:

Exception without context: https://www.yegor256.com/2015/12/01/rethrow-exceptions.html

Grouped exceptions: https://www.yegor256.com/2022/08/30/dont-group-exception-catchers.html

@volodya-lombrozo fixed

volodya-lombrozo · 2024-12-12T13:47:15Z

eo-parser/src/test/java/org/eolang/parser/StrictXmirTest.java

+                                .xpath("/program")
+                                .attr(
+                                    "noNamespaceSchemaLocation xsi http://www.w3.org/2001/XMLSchema-instance",
+                                    "https://www.eolang.org/XMIR.xsd"


@yegor256 You tested only one positive scenario. What will happen if we have a corrupted address? What will happen if we don't have the internet?

https://www.eCORRUPTEDolang.org/XMIR.xsd

@volodya-lombrozo added test in 6f07f8f

volodya-lombrozo · 2024-12-12T13:50:33Z

eo-parser/src/main/java/org/eolang/parser/StrictXmir.java

+                    ).toString().replace("\\", "/")
+                );
+            }
+            new Xembler(


@yegor256 Do we need to change the xmir in case !uri.startWith("http")?

@volodya-lombrozo nope, when it's file://.... we keep it as is

@yegor256 I see. I've overlooked indentations. Thank you.

yegor256 · 2024-12-12T14:33:09Z

@volodya-lombrozo we can't stay entirely offline. The schema (XSD) must come from somewhere. Either you download it and then provide to StrictXmir (I will make it possible now), or it will download it. Or I miss something?

volodya-lombrozo · 2024-12-12T14:47:29Z

@volodya-lombrozo we can't stay entirely offline. The schema (XSD) must come from somewhere. Either you download it and then provide to StrictXmir (I will make it possible now), or it will download it. Or I miss something?

@yegor256 Is it possible to place xsd to the classpath where StrictXmir is located?

yegor256 · 2024-12-12T15:28:28Z

@volodya-lombrozo we don't know which version to place there: there are many of them published at www.eolang.org. We can possibly try to download them all during build and put into JAR, but this sounds like a hack.

yegor256 · 2024-12-12T17:32:06Z

@volodya-lombrozo please, check the code one more time, I made some changes

volodya-lombrozo

@yegor256 Looks good to me. I suggested several minor changes, but they aren't so important. Could you also replace the line in the comment from

closes #3636

to

related to #3636

please?

volodya-lombrozo · 2024-12-13T06:20:59Z

eo-parser/src/main/java/org/eolang/parser/StrictXmir.java

+/**
+ * XMIR that validates itself right after construction.
+ *
+ * <p>This class is supposed to be used ONLY for testing, because


@yegor256 [NIT] I mean, this class might be used not only for testing. Actually it's an important part of validation in a production code. Maybe we need to remove this line from javadoc:

This class is supposed to be used ONLY for testing

and replace it with something like

ATTENTION!

what do you think?

volodya-lombrozo · 2024-12-13T06:22:25Z

eo-parser/src/main/java/org/eolang/parser/StrictXmir.java

+                    ).toString().replace("\\", "/")
+                );
+            }
+            new Xembler(


@yegor256 I see. I've overlooked indentations. Thank you.

volodya-lombrozo · 2024-12-13T06:42:54Z

@volodya-lombrozo we don't know which version to place there: there are many of them published at www.eolang.org. We can possibly try to download them all during build and put into JAR, but this sounds like a hack.

@yegor256 As I understand it, you publish them from XMIR.xsd. So, when you publish version 0.47.0 of eo, you also publish version 0.47.0 of XMIR.xsd (at least that’s my assumption). In other words, they are consistent and included together in eo-parser:0.47.0.jar. I don't clearly understand why we can't use the XMIR from this JAR and instead are required to download it from the internet. Most probably, I will implement this in jeo-maven-plugin.

yegor256 · 2024-12-13T07:03:17Z

@volodya-lombrozo when you do new StrictXmir(xml), we don't know what is the location of XSD that you mention inside the xml. It could be the version of the JAR where StrictXmir.class is located, but it may me something totally different (not even XMIR.xsd, but any other URI). That's why we have to download the file first and then use it. Make sense?

yegor256 · 2024-12-13T07:14:22Z

@volodya-lombrozo what we can do is this: StrictXmir will compare the value of the URI inside XML with the expected location of XSD for the current version of eo-parser. If the URI is exactly where the XSD is supposed to be located, it will use the XMIR.xsd file from classpath. In your particular case, this will solve the problem (because you build the XMIR and validate it using the same version of eo-parser).

volodya-lombrozo · 2024-12-13T07:34:47Z

@volodya-lombrozo when you do new StrictXmir(xml), we don't know what is the location of XSD that you mention inside the xml. It could be the version of the JAR where StrictXmir.class is located, but it may me something totally different (not even XMIR.xsd, but any other URI). That's why we have to download the file first and then use it. Make sense?

@yegor256 To be honest, I don't see any reason why we might need such flexibility. We have a single XSD and a single StrictXmir. They are placed together in one location, so StrictXmir can directly use it without any downloading. This sounds like overengineering to me in this particular case. However, I may not be seeing the whole picture.

I will try to use xsi:noNamespaceSchemaLocation="classpath:XMIR.xsd" in the generated XML files or something similar. I don't know if it will help, but that's a separate matter. I will investigate it.

yegor256 · 2024-12-13T07:37:24Z

@volodya-lombrozo

We have a single XSD and a single StrictXmir. They are placed together in one location, so StrictXmir can directly use it without any downloading.

StrictXmir doesn't know about this: whether the XSD your XML is using is the same XSD that StrictXmir has in the classpath. There is no such guarantee.

yegor256 · 2024-12-13T07:39:28Z

@volodya-lombrozo you can try to use this:

xsi:noNamespaceSchemaLocation="classpath:XMIR.xsd"

However, 1) this won't work with StrictXML or StrictXmir and you will have to implement your own schema validation, and 2) the XML file won't be accepted by next steps of the pipeline, since they have their own validators that don't know anything about your classpath (some of them may not be even written in Java).

volodya-lombrozo · 2024-12-13T07:53:33Z

@volodya-lombrozo you can try to use this:
xsi:noNamespaceSchemaLocation="classpath:XMIR.xsd"
However, 1) this won't work with StrictXML or StrictXmir and you will have to implement your own schema validation, and 2) the XML file won't be accepted by next steps of the pipeline, since they have their own validators that don't know anything about your classpath (some of them may not be even written in Java).

@yegor256
There are many questions:

Why do the "next steps of the pipeline" require an xsd, and why do they specifically need the http version of this xsd?
Moreover, I don't understand why the classpath won't work with StrictXML or StrictXmir.

Let me investigate these questions. Maybe it will become clear. And thank you very much for your help.

yegor256 · 2024-12-13T07:56:23Z

@volodya-lombrozo check this one: d0bf3b8 With this feature you will be fine. Just use StrictXmir from eo-parser and there will be no network usage.

yegor256 · 2024-12-13T08:07:17Z

@volodya-lombrozo XML documents that JEO (or any other tool) produces are supposed to be "valid" in terms of compliance with XSD schema. Technically this means that noNamespaceSchemaLocation is present and it points to the document publicly available (online or on disc). Classpath is not publicly available, it's a local storage of a particular instance of a running VM.

volodya-lombrozo

@yegor256 There is one common part in all tests:

           new StrictXmir(
                 new Xmir(
                     new XMLDocument(
                         new Xembler(
                             new Directives()
                                 .append(new DrProgram("foo3"))
                                 .xpath("/program")
                                 .attr(
                                     "noNamespaceSchemaLocation xsi http://www.w3.org/2001/XMLSchema-instance",
                                     String.format(
                                         "https://www.eolang.org/xsd/XMIR-%s.xsd",
                                         Manifests.read("EO-Version")
                                     )
                                 )
                                 .add("objects")
                         ).xml()
                     )
                 ),
                 tmp
             ).validate(),

What do you think if we will move into a separate method?

yegor256 · 2024-12-13T08:15:03Z

@volodya-lombrozo you are right, duplication reduced: 809adde

volodya-lombrozo · 2024-12-13T08:15:04Z

Technically this means that noNamespaceSchemaLocation is present and it points to the document publicly available (online or on disc)

@yegor256 As far as I know, we have several ways to validate an XML document against an XSD schema, and we are not obligated to use only noNamespaceSchemaLocation.

yegor256 · 2024-12-13T10:06:55Z

@volodya-lombrozo indeed, it's possible to validate it against any XSD schema, but relying on noNamespaceSchemaLocation or schemaLocation (which are provided by the author of XML document) is the more robust way. BTW, we can also use DTD: https://stackoverflow.com/questions/1544200/what-is-difference-between-xml-schema-and-dtd

yegor256 added 3 commits December 12, 2024 15:15

#3636: XMIR on XML

c039166

#3636: StrictXmir

a7a76c4

#3636: cache

9e0b464

yegor256 requested a review from volodya-lombrozo December 12, 2024 12:54

yegor256 added 2 commits December 12, 2024 15:55

#3636: typo

13b70ee

#3636: renames

30650b9

volodya-lombrozo requested changes Dec 12, 2024

View reviewed changes

#3636: cache folder

7d6c669

github-actions bot added the core label Dec 12, 2024

yegor256 added 3 commits December 12, 2024 18:36

#3636: tests

6f07f8f

#3636: explained

fa589e6

#3636: retry three times

a1a64bc

volodya-lombrozo reviewed Dec 13, 2024

View reviewed changes

yegor256 added 2 commits December 13, 2024 10:10

Merge branch 'master' into 3636

062f8bd

#3636: explained the usage

04af5af

volodya-lombrozo approved these changes Dec 13, 2024

View reviewed changes

#3636: local XSD used if matches

d0bf3b8

#3636: tested and better logging

fc8fbec

volodya-lombrozo reviewed Dec 13, 2024

View reviewed changes

#3636: duplication reduced

809adde

volodya-lombrozo approved these changes Dec 13, 2024

View reviewed changes

yegor256 merged commit 714cdfc into master Dec 13, 2024
19 checks passed

yegor256 deleted the 3636 branch December 13, 2024 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

StrictXmir #3652

StrictXmir #3652

yegor256 commented Dec 12, 2024 •

edited

Loading

yegor256 commented Dec 12, 2024

volodya-lombrozo left a comment

volodya-lombrozo Dec 12, 2024

yegor256 Dec 12, 2024

volodya-lombrozo Dec 13, 2024

yegor256 Dec 13, 2024

volodya-lombrozo Dec 12, 2024

yegor256 Dec 12, 2024

volodya-lombrozo Dec 12, 2024

yegor256 Dec 12, 2024

volodya-lombrozo Dec 12, 2024

yegor256 Dec 12, 2024

volodya-lombrozo Dec 13, 2024

yegor256 commented Dec 12, 2024

volodya-lombrozo commented Dec 12, 2024

yegor256 commented Dec 12, 2024

yegor256 commented Dec 12, 2024

volodya-lombrozo left a comment

volodya-lombrozo Dec 13, 2024

volodya-lombrozo Dec 13, 2024

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024 •

edited

Loading

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024 •

edited

Loading

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024

volodya-lombrozo left a comment

yegor256 commented Dec 13, 2024

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

StrictXmir #3652

StrictXmir #3652

Conversation

yegor256 commented Dec 12, 2024 • edited Loading

yegor256 commented Dec 12, 2024

volodya-lombrozo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yegor256 commented Dec 12, 2024

volodya-lombrozo commented Dec 12, 2024

yegor256 commented Dec 12, 2024

yegor256 commented Dec 12, 2024

volodya-lombrozo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024 • edited Loading

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024 • edited Loading

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 13, 2024

volodya-lombrozo left a comment

Choose a reason for hiding this comment

yegor256 commented Dec 13, 2024

volodya-lombrozo commented Dec 13, 2024

yegor256 commented Dec 13, 2024

yegor256 commented Dec 12, 2024 •

edited

Loading

yegor256 commented Dec 13, 2024 •

edited

Loading

yegor256 commented Dec 13, 2024 •

edited

Loading