Tags 3.0 TCK Challenge against tests that include space character before AM/PM #255

scottmarlow · 2024-03-12T14:04:40Z

The Java rules have changed as per https://bugs.openjdk.org/browse/JDK-8304925 (Some date/time strings created with JDK <=19 can not be parsed since JDK 20) + https://bugs.openjdk.org/browse/JDK-8284840 (Update CLDR to Version 42.0) which means that the Tags 3.0 TCK tests are now seeing failures like Caused by: java.text.ParseException: Unparseable date: "Nov 21, 2000, 3:45:02 AM".

https://github.com/jakartaee/platform-tck/blob/10.0.x/src/web/jstl/compat/onedotzero/positiveFormatLocalizationContextI18NTest.jsp#L32 contains <c:set var="dt" value="Nov 21, 2000, 3:45:02 AM"/> which I do not think can be updated to work on Java 21 (e.g. to use non-breaking space) as that would mean the test would fail on Java 11/17 (which needs a breaking space).

Are there other possible fixes that we should consider making?

Note that the https://github.com/jakartaee/platform-tck/blob/10.0.x/src/web/jstl files that contain PM are:

jstl/spec/fmt/format/fmtdate/positiveFDTimeZonePrecedenceTest.jsp
jstl/compat/onedotzero/positiveSetTimezoneValueTest.gf
jstl/compat/onedotzero/positiveTimezoneValueTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDDateStyleTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDFallbackLocaleTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDLocalizationContextTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDTimeStyleTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDTimeZoneNullEmptyTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDTimeZonePrecedenceTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDTimeZoneTest.gf
jstl/spec/fmt/format/fmtdate/positiveFDTypeTest.gf
jstl/spec/fmt/format/settimezone/positiveSetTimezoneValueTest.gf
jstl/spec/fmt/format/timezone/positiveTimezoneValueTest.gf

Note that the https://github.com/jakartaee/platform-tck/blob/10.0.x/src/web/jstl files that contain AM are:

jstl/compat/onedotzero/positiveFormatLocalizationContextI18NTest.gf
jstl/compat/onedotzero/positiveFormatLocalizationContextI18NTest.jsp
jstl/compat/onedotzero/positiveSetTimezoneValueTest.gf
jstl/compat/onedotzero/positiveTimezoneValueTest.gf
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextBrowserLocaleTest.gf
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextBrowserLocaleTest.jsp
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextBundleTest.gf
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextBundleTest.jsp
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextI18NTest.gf
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextI18NTest.jsp
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextLocaleTest.gf
jstl/spec/fmt/format/localecontext/positiveFormatLocalizationContextLocaleTest.jsp
jstl/spec/fmt/format/parsedate/positivePDBodyValueTest.gf
jstl/spec/fmt/format/parsedate/positivePDBodyValueTest.jsp
jstl/spec/fmt/format/parsedate/positivePDDateStyleTest.jsp
jstl/spec/fmt/format/parsedate/positivePDFallbackLocaleTest.gf
jstl/spec/fmt/format/parsedate/positivePDFallbackLocaleTest.jsp
jstl/spec/fmt/format/parsedate/positivePDLocalizationContextTest.gf
jstl/spec/fmt/format/parsedate/positivePDLocalizationContextTest.jsp
jstl/spec/fmt/format/parsedate/positivePDParseLocaleTest.jsp
jstl/spec/fmt/format/parsedate/positivePDScopeTest.jsp
jstl/spec/fmt/format/parsedate/positivePDTimeStyleTest.jsp
jstl/spec/fmt/format/parsedate/positivePDTimeZoneNullEmptyTest.gf
jstl/spec/fmt/format/parsedate/positivePDTimeZoneNullEmptyTest.jsp
jstl/spec/fmt/format/parsedate/positivePDTimeZonePrecedenceTest.gf
jstl/spec/fmt/format/parsedate/positivePDTimeZonePrecedenceTest.jsp
jstl/spec/fmt/format/parsedate/positivePDTimeZoneTest.gf
jstl/spec/fmt/format/parsedate/positivePDTimeZoneTest.jsp
jstl/spec/fmt/format/parsedate/positivePDTypeTest.gf
jstl/spec/fmt/format/parsedate/positivePDTypeTest.jsp
jstl/spec/fmt/format/parsedate/positivePDVarTest.jsp
jstl/spec/fmt/format/settimezone/positiveSetTimezoneValueNullEmptyTest.gf
jstl/spec/fmt/format/settimezone/positiveSetTimezoneValueTest.gf
jstl/spec/fmt/format/timezone/positiveTimezoneValueNullEmptyTest.gf
jstl/spec/fmt/format/timezone/positiveTimezoneValueTest.gf

The good news is that if we do a service release for jakarta-tags-tck-3.0.1.zip that excludes the tests that include ^ breaking space before AM/PM, we will use the same jakarta-tags-tck-3.0.1.zip for validating Standalone Tags 3.0 implementations for Jakarta EE 10 + 11.

We also need to update the EE 11 Platform TCK (tckrefactor branch) to be able to pass (or exclude) the same tests on Java 17 + 21+.

Please see comment jakartaee/platform-tck#1219 (comment) for more information on the failure.

This TCK challenge is for the following 29 tests that fail on Java 21 with WildFly:

com/sun/ts/tests/jstl/compat/onedotzero/JSTLClient.java#positiveFormatLocalizationContextI18NTest
com/sun/ts/tests/jstl/compat/onedotzero/JSTLClient.java#positiveSetTimezoneValueTest
com/sun/ts/tests/jstl/compat/onedotzero/JSTLClient.java#positiveTimezoneValueTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDDateStyleTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDFallbackLocaleTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDLocalizationContextTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDTimeStyleTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDTimeZoneNullEmptyTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDTimeZonePrecedenceTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDTimeZoneTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtdate/JSTLClient.java#positiveFDTypeTest
com/sun/ts/tests/jstl/spec/fmt/format/localecontext/JSTLClient.java#positiveFormatLocalizationContextBrowserLocaleTest
com/sun/ts/tests/jstl/spec/fmt/format/localecontext/JSTLClient.java#positiveFormatLocalizationContextBundleTest
com/sun/ts/tests/jstl/spec/fmt/format/localecontext/JSTLClient.java#positiveFormatLocalizationContextI18NTest
com/sun/ts/tests/jstl/spec/fmt/format/localecontext/JSTLClient.java#positiveFormatLocalizationContextLocaleTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDBodyValueTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDDateStyleTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDFallbackLocaleTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDLocalizationContextTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDParseLocaleTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDTimeStyleTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDTimeZoneNullEmptyTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDTimeZonePrecedenceTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDTimeZoneTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#positivePDTypeTest
com/sun/ts/tests/jstl/spec/fmt/format/settimezone/JSTLClient.java#positiveSetTimezoneValueNullEmptyTest
com/sun/ts/tests/jstl/spec/fmt/format/settimezone/JSTLClient.java#positiveSetTimezoneValueTest
com/sun/ts/tests/jstl/spec/fmt/format/timezone/JSTLClient.java#positiveTimezoneValueNullEmptyTest
com/sun/ts/tests/jstl/spec/fmt/format/timezone/JSTLClient.java#positiveTimezoneValueTest

We also need to challenge the same ^ 29 tests for GlassFish.

The text was updated successfully, but these errors were encountered:

pnicolucci · 2024-03-13T00:14:03Z

@scottmarlow why the additional set of test failures in Glassfish? Did those tests pass successfully on Wildfly?

dmatej · 2024-03-13T09:58:35Z

[javatest.batch] 01-30-2024 13:10:45:  TRACE: [WIRE] - << "HTTP/1.1 500 Internal Server Error[\r][\n]"
[javatest.batch] 01-30-2024 13:10:45:  TRACE: [WIRE] - << "HTTP/1.1 500 Internal Server Error[\r][\n]"

Is it possible to get the server.log file?

scottmarlow · 2024-03-13T12:22:42Z

Is it possible to get the server.log file?

The server.log is not currently archived. The generated JSTL tck contains a copy of the https://github.com/jakartaee/platform-tck/blob/master/docker/jstltck.sh script. I'm not sure if we would need to modification of that or the Jenkins job running the TCK.

@scottmarlow why the additional set of test failures in Glassfish? Did those tests pass successfully on Wildfly?

When GlassFish is running against the JSTL standalone TCK, we see the following test stats:
Test results: passed: 502; failed: 40
Tests run == 542

With WildFly running the JSTL tests in EE 10 Platform TCK, we see the following test stats:
Test results: passed: 512; failed: 29
Tests run == 541

The Standalone JSTL TCK test count includes one more test as that is the signature test:
com/sun/ts/tests/signaturetest/jstl/JSTLClient.java#signatureTest

pnicolucci · 2024-03-13T15:52:59Z

We also need to challenge the same ^ 29 tests for GlassFish and in addition, the following 11 tests as well which also failed as shown in jakartaee-tck-build-run/job/jdk21-run/39/artifact/jstltck.log:

com/sun/ts/tests/jstl/spec/core/urlresource/importtag/JSTLClient.java#negativeImportRequestDispatcherServletExceptionTest
com/sun/ts/tests/jstl/spec/core/urlresource/importtag/JSTLClient.java#negativeImportRequestDispatcherRTIOExceptionTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtnum/JSTLClient.java#negativeFNUnableToParseValueTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#negativePDUnableToParseValueTest
com/sun/ts/tests/jstl/spec/fmt/format/parsenum/JSTLClient.java#negativePNUnableToParseValueTest
com/sun/ts/tests/jstl/spec/sql/param/JSTLClient.java#negativeParamQueryBodyContentTest
com/sun/ts/tests/jstl/spec/sql/param/JSTLClient.java#negativeParamQuerySQLAttributeTest
com/sun/ts/tests/jstl/spec/sql/query/JSTLClient.java#negativeQueryBodyContentTest
com/sun/ts/tests/jstl/spec/sql/query/JSTLClient.java#negativeQuerySQLAttributeTest
com/sun/ts/tests/jstl/spec/sql/update/JSTLClient.java#negativeUpdateBodyContentTest
com/sun/ts/tests/jstl/spec/sql/update/JSTLClient.java#negativeUpdateSQLAttributeTest2

Hey @scottmarlow , my question was more focused on why are the above 11 tests failing in GlassFish but not WildFly. Looking at the output:

[javatest.batch] 01-30-2024 13:00:54: TRACE: [WIRE] - << "<title>Eclipse GlassFish 8.0.0 - Error report</title><style type="text/css"></style>
HTTP Status 500 - Internal Server Error
type Exception report
messageInternal Server Error
descriptionThe server encountered an internal error that prevented it from fulfilling this request.
exception
java.lang.NoSuchMethodError: 'java.lang.Throwable jakarta.servlet.jsp.JspException.getRootCause()'
note The full stack traces of the exception and its root causes are available in the Eclipse GlassFish 8.0.0 logs.
Eclipse GlassFish 8.0.0

The JspException.getRootCause() method was @deprecated As of JSP 2.1, replaced by {@link #getCause()}: https://github.com/jakartaee/pages/blob/master/spec/src/main/asciidoc/ServerPages.adoc#c1-changes-between-jsp-40-and-jsp-31 / jakartaee/pages#234. As can be seen in the Pages 4.0 Changelog and the issue, the getRootCause() method was removed in Pages 4.0 (Jakarta EE11)

Since these 11 tests passed on WildFly but failed on GlassFish, I believe these failures are potentially not Jakarta Tags / TCK issues. If we can see the stack of the exception would give additional context but I didn't see JspException.getRootCause() being referenced anyplace in the tck here: https://github.com/jakartaee/platform-tck/tree/10.0.x/src/web/jstl or in the Jakarta Tags code base here: https://github.com/jakartaee/tags/tree/3.x

To me, it looks like the two sets of tests referenced here are two different problems.

scottmarlow · 2024-03-13T18:07:32Z

Interesting find, the source is in a different folder https://github.com/jakartaee/platform-tck/blob/10.0.x/src/com/sun/ts/tests/jstl/common/tags/ExceptionCheckTag.java#L174

pnicolucci · 2024-03-13T18:35:25Z

@scottmarlow Given this information these 11 tests would only fail where Pages 4.0 / Jakarta EE11 were being tested. Do we need another challenge for this issue or could the TCK just be cleaned up to not call methods removed in Jakarta EE11? It would be good to keep these problems separate IMO.

scottmarlow · 2024-03-13T19:04:06Z

@scottmarlow Given this information these 11 tests would only fail where Pages 4.0 / Jakarta EE11 were being tested. Do we need another challenge for this issue or could the TCK just be cleaned up to not call methods removed in Jakarta EE11? It would be good to keep these problems separate IMO.

Good question, I don't think these 11 tests need a TCK challenge created for them. I think instead we could clean up https://github.com/jakartaee/platform-tck/blob/10.0.x/src/com/sun/ts/tests/jstl/common/tags/ExceptionCheckTag.java#L174 to call JspException.getCause() which I agree should be captured by a separate Tags issues.

scottmarlow · 2024-03-13T19:52:05Z

I'm going to create the Tags issue for reporting the problem with https://github.com/jakartaee/platform-tck/blob/10.0.x/src/com/sun/ts/tests/jstl/common/tags/ExceptionCheckTag.java#L174

I'll also remove the following 11 tests for GlassFish that I had included in this TCK challenge description:

ll which also failed as shown in jakartaee-tck-build-run/job/jdk21-run/39/artifact/jstltck.log:

com/sun/ts/tests/jstl/spec/core/urlresource/importtag/JSTLClient.java#negativeImportRequestDispatcherServletExceptionTest
com/sun/ts/tests/jstl/spec/core/urlresource/importtag/JSTLClient.java#negativeImportRequestDispatcherRTIOExceptionTest
com/sun/ts/tests/jstl/spec/fmt/format/fmtnum/JSTLClient.java#negativeFNUnableToParseValueTest
com/sun/ts/tests/jstl/spec/fmt/format/parsedate/JSTLClient.java#negativePDUnableToParseValueTest
com/sun/ts/tests/jstl/spec/fmt/format/parsenum/JSTLClient.java#negativePNUnableToParseValueTest
com/sun/ts/tests/jstl/spec/sql/param/JSTLClient.java#negativeParamQueryBodyContentTest
com/sun/ts/tests/jstl/spec/sql/param/JSTLClient.java#negativeParamQuerySQLAttributeTest
com/sun/ts/tests/jstl/spec/sql/query/JSTLClient.java#negativeQueryBodyContentTest
com/sun/ts/tests/jstl/spec/sql/query/JSTLClient.java#negativeQuerySQLAttributeTest
com/sun/ts/tests/jstl/spec/sql/update/JSTLClient.java#negativeUpdateBodyContentTest
com/sun/ts/tests/jstl/spec/sql/update/JSTLClient.java#negativeUpdateSQLAttributeTest2

pnicolucci · 2024-03-14T23:46:51Z

@scottmarlow why is the remaining issue with the space character before AM/PM a challenge but #256 is not? These both seem to stem from running the Tags 3.0 TCK on a Java version / Features for Jakarta EE11. Could the TCK be updated for these tests to check the java version and then act accordingly?

scottmarlow · 2024-03-15T12:02:51Z

why is the remaining issue with the space character before AM/PM a challenge but #256 is not?

Great question to raise, thanks!

These both seem to stem from running the Tags 3.0 TCK on a Java version / Features for Jakarta EE11.

issues/256 is caused by Java SE 21 breaking backward compatibility via the JDK-8304925 change. #256 is caused by the planned removal of JspException.getRootCause() which the Tags 3.0 TCK depends on. In one case, the TCK failure cause is a breaking change in an EE Specification and in the other, the TCK Failure cause is a breaking change in Java SE 21.

So which of these issues should be TCK challenges? Pasting from the TCK Process guidance on Valid Challenges:

Any test case (e.g., test class, @test method), test case configuration (e.g., deployment descriptor), test beans, annotations, and other resources considered part of the TCK may be challenged.

The following scenarios are considered in scope for test challenges:

Claims that a test assertion conflicts with the specification.

Claims that a test asserts requirements over and above that of the specification.

Claims that an assertion of the specification is not sufficiently implementable.

Claims that a test is not portable or depends on a particular implementation.

It seems to me that the last point (Claims that a test is not portable or depends on a particular implementation.) could be applied to make both issues TCK challenges so I updated #256 title to be a TCK challenge (not that the title is that important but please do add the TCK label to that issue).

Could the TCK be updated for these tests to check the java version and then act accordingly?

Yes, Flavia suggested something like that on Zulip a few days ago. I'm not really sure though about the idea of converting the string based on the Java version (quoting that suggestion from the WildFly Zulip discussion) but I don't really know enough what is the best approach. So what would be the best way to act accordingly? Have different copies of certain JSP files that work on Java 21 and conditionally use them when running on the Java 21? Or some way to convert the date values in the JSP source before the JSP files are processed.

To help answer ^, I'm pasting from the Tags 3.0 spec:

The Jakarta Standard Tag Library i18n-capable formatting actions support either approach: They assist page authors with > creating internationalized page content that can be localized into any locale available in the Jakarta Server Pages container (this addresses the second approach), and allow various data elements such as numbers, currencies, dates and times to be formatted and parsed in a locale-sensitive or customized manner (this may be used in either approach).

The Jakarta Standard Tag Library’s i18n actions are covered in this chapter. The formatting actions are covered in Formatting Actions: I18n-capable formatting tag library.

Can Tags 3.0 implementations be required to automatically convert the date values in the JSP source when running on Java 21 (perhaps as per the above pasted Tags 3.0 Spec text)? Or do we need a new Tags Spec release to add an additional requirement? I ask as I think currently we need to do something for EE 11 but at the same time, I also want EE 10 implementations to also be able to pass the Jakarta EE 10 Platform TCK which runs against the same Tags tests.

Also for EE 11, we have a lot of work ahead to refactor the Platform TCK which will include running the Tags TCK tests via Maven + Arquillian, whatever we do to address the challenge for EE 10, will hopefully help with running the EE 11 Platform TCK Tags tests on Java 21 as well as Java 17.

volosied · 2024-03-28T12:50:15Z

I think we should accept this challenge. The issue may not be with JSTL itself / EE11 Specs, but I think we still have to accommodate any changes made in the underlying technology (i.e Java).

@pnicolucci What do you think?

pnicolucci · 2024-03-28T13:38:17Z

I believe the best path forward here is to update the TCK to leverage Java to generate the Date/Time String that is
then parsed/formatted using JSTL.

Currently, the TCK has a static String: Nov 21, 2000, 3:45:02 AM used in one of the test failures However, that string can not
be parsed by Java 19+ even without Jakarta Tags being involved due to a known change in the JDK.

If the TCK were to use Java to generate the Date/Time values to be tested, then the values would be parsable by Java as they were generated by Java. The TCK could be updated and then the same TCK would work regardless of the JDK version being tested (11/17/21). When running on Java 21 a NNBS would be output, when running on Java 17 a standard space would be output.

Take this code for example:

// TCK uses: Nov 21, 2000, 3:45:02 AM
// Same time in milliseconds: 974778302000
DateFormat dateFormat = DateFormat.getDateTimeInstance();
dateFormat.setTimeZone(TimeZone.getTimeZone(ZoneId.of("GMT")));
String dateFormatZoned = dateFormat.format(new Date(974778302000L) );
pageContext.setAttribute("dateFormatZoned", dateFormatZoned);

The dateFormatZoned attribute now contains the properly formatted String: Nov 21, 2000, 3:45:02 AM with the correct space. The TCK can then perform the same actions it has previously and no issues should be seen when switching between Java versions.

As for the golden files a different file could be used depending on the version of Java being used so that the correct space character could be checked.

pnicolucci · 2024-03-28T13:49:56Z

Posted to the mailing list the latest status: https://www.eclipse.org/lists/jstl-dev/msg00138.html I plan to accept this challenge at the end of the week if there are no objections. @scottmarlow does the above solution seem reasonable to you?

dmatej · 2024-04-19T17:48:23Z

Well, I collided with this today when I executed tags TCK for any GF 7.0.x version. Finally I have found that by mistake I executed these tests with JDK21 - that breaks 30 tests of 542. Part of the issue is that TCK is sensitive to the default locale, but that can be resolved by export LC_ALL=C.

With JDK11 and JDK17 GF7 passed all tests of JEE10 TCK.

I know this issue is rather for JEE11, but I would like to have a firm ground first and have everything repeatable and solid in GF7 too. That is why I created eclipse-ee4j/glassfish#24925 - meanwhile I am preparing builds with TCK (private on OmniFish infrastructure as we have more powerful and reliable hw) and documenting the state of GF8 and JEE11 (here, I started by copy pasting JEE 10 and I am slowly going through the list).

In GF master branch you can reproduce it by running this:

mvn clean install -Pfastest,staging -T4C && mvn clean install -Ptck -pl :jakarta-pages-tags-tck && mvn clean install -Ptck -pl :glassfish-external-tck-pages-tags

For GF8 I have to build the TCK first, but the tckrefactor branch build fails because of missing snapshot dependencies. But this problem is the same.

Would be a solution just not depend on default system locale?
It is weird when the communication with the user/client depends on server operating system settings. Is that required by the spec?

bstansberry · 2024-04-19T18:51:53Z

@dmatej If you set LC_ALL=C it passes on SE 21? I think that's what you're saying, but it's not clear to me.

dmatej · 2024-04-19T20:58:45Z

@dmatej If you set LC_ALL=C it passes on SE 21? I think that's what you're saying, but it's not clear to me.

No, it doesn't, but it fixes some issues when I have cs_CZ.UTF-8 set on my system. I believe the tck should ensure it.

I did not find any way to pass on JDK21.

scottmarlow · 2024-04-22T14:03:48Z

Posted to the mailing list the latest status: https://www.eclipse.org/lists/jstl-dev/msg00138.html I plan to accept this challenge at the end of the week if there are no objections. @scottmarlow does the above solution seem reasonable to you?

@pnicolucci it seems reasonable to me but I'd like to hear from others in the community in the next day or so if anyone else is for/against the stated solution.

I think that we will need a pull request for the https://github.com/jakartaee/platform-tck/tree/10.0.x branch so that we can apply the stated solution. If anyone later challenges the solution as not being valid and that later challenge is accepted, the tests would be excluded instead.

scottmarlow · 2024-04-23T18:10:01Z

@pnicolucci a day has passed and no feedback yet.

Regarding the above solution that you mentioned, do you want to include that for a future EE release? Or for Jakarta EE 10?

My preference is to exclude the tests as I know that could happen this week.

scottmarlow · 2024-04-24T18:41:08Z

We will merge the jakartaee/platform-tck#1219 change as soon as possible that will exclude the failing tests.

Signed-off-by: Gurunandan Rao <gurunandan.rao@oracle.com>

… JSTL challenges that excludes tests for jakartaee/tags#256 + jakartaee/tags#255 Signed-off-by: Scott Marlow <smarlow@redhat.com>

Signed-off-by: Gurunandan Rao <gurunandan.rao@oracle.com>

…that excludes tests for jakartaee/tags#256 + jakartaee/tags#255 and also release JSTL 3.0.1 TCK Signed-off-by: Scott Marlow <smarlow@redhat.com>

pnicolucci · 2024-04-24T21:52:35Z

@scottmarlow how long do you think it would actually take to update the tests rather than exclude them? What sort of time window is available to make test updates?

scottmarlow · 2024-04-25T17:15:08Z

@scottmarlow how long do you think it would actually take to update the tests rather than exclude them?
I'm not sure but if I were to work on that, I wouldn't be able to work on something else and I am currently on the critical path for helping to refactor the Platform TCK for the EE 11 release. The same is true for all contributors I think but if someone has the time to make the change, then it takes as long as it takes them to do that while balancing all of their other duties and work. So, hard to say really. I think the guidance in your comment #255 (comment) is enough for someone to work on this.

What sort of time window is available to make test updates?

The current TCK Process suggests the time window should be quick but if we exclude every TCK test, we will end up with zero tests so I think there should be a balance. Also, if we think that most potential contributors are likely not going to have time, that could be a reason to accept the excluding of the tests. Having said this, I think you as lead of the Tags project should decide whether the tests should be fixed or excluded. You could make that choice with other Tags committers or whomever you like. It is perfectly fine for you to make that decision as well.

For TCK challenge resolution, I try to delegate to the respective Spec API lead for the test bucket that contains the challenged test.

Below is some guidance from the TCK Process. Only the first paragraph is relevant here but I included the other details which covers making test changes in response to a challenge:

Accepted Challenges
A consensus that a test produces invalid results will result in the exclusion of that test from certification requirements, and an immediate update and release of an official distribution of the TCK including the new exclude list. The associated challenge issue MUST be closed with an accepted label to indicate it has been resolved.

The specification project may approve (user) workarounds for an accepted TCK challenge (as an alternative to excluding TCK tests).

As another alternative to excluding a challenged test, it may be possible to adjust the test validation logic to “expand” the validation check. E.g. if a test expects a value “A1” for a specific variable “x”, but a challenge is raised arguing that the specification language actually allows for either values “A1” and “A2” (but no other values) to be valid values for “x”, then it could be a valid course of action to modify the challenged test to allow either “A1” OR “A2” for “x”.

Since this line of thinking might be applied to cases that aren’t quite as straightforward as this example, care should be taken when using this approach. A particular danger is that an implementation that has already demonstrated compliance before the challenge was raised might actually not pass the new, modified test.

To limit the confusion and additional work such a scenario would cause, if there is already at least one certified compatible implementation before the challenge, the new, modified TCK should be run against at least one such implementation (and ideally all of them) before the changes are published, released, and finalized.

If such a change were released, and it was later found to cause a previously-certified implementation to fail the new, modified test, then excluding the test would likely be the only option, and this would require yet another, additional service release.

pnicolucci · 2024-04-25T17:25:40Z

@scottmarlow I don't have any experience with running the TCK at all, is there any guidance there? What percentage of tests does this challenge account for?

scottmarlow · 2024-04-25T17:45:51Z

@scottmarlow I don't have any experience with running the TCK at all, is there any guidance there?

We plan to make the process of running TCK tests easier for Jakarta EE 11. Some Jakarta EE implementations (e.g. GlassFish + Tomitribe have open sourced their TCK porting kit but I don't think anyone else has so for Open Liberty you likely would need bits that are behind the IBM firewall (same for WildFly currently as well).

If you (or someone else) comes up with a Platform TCK pull request, the Platform TCK team can run tests against your branch. We would build your branch and stage a Platform TCK for testing. We could also stage a Standalone Tags TCK for testing as well. We would likely post on the pull request or here with the TCK results of running the Tags tests with Java 21.

What percentage of tests does this challenge account for?

Currently, with WildFly I am seeing that the Jakarta EE 10 Platform TCK runs 541 tests. If we exclude the 29 tests then there would be 29 fewer tests run but I think the decision might take into account how important the 29 tests are in terms of whether they are testing Tags Specification requirements that are hard to implement correctly. The how to make that judgement of difficulty to implement is very subjective.

scottmarlow · 2024-04-25T18:42:32Z

Just to note, I'll probably build/stage a https://www.eclipse.org/downloads/download.php?file=/jakartaee/platform/10/jakarta-jakartaeetck-10.0.5.zip tomorrow with a Persistence TCK test exclusion that has been outstanding for a while. So if the decision for Tags becomes to exclude the 29 tests by tomorrow, I'll include those changes as well. No pressure either way, just wanted to mention my plans. :-)

scottmarlow · 2024-04-26T14:40:19Z

Just to note, I'll probably build/stage a https://www.eclipse.org/downloads/download.php?file=/jakartaee/platform/10/jakarta-jakartaeetck-10.0.5.zip tomorrow with a Persistence TCK test exclusion that has been outstanding for a while. So if the decision for Tags becomes to exclude the 29 tests by tomorrow, I'll include those changes as well. No pressure either way, just wanted to mention my plans. :-)

Actually, I will wait to release the Platform TCK until we have a fix for #255 merged in. I will instead just release the Standalone Persistence TCK today.

scottmarlow · 2024-05-01T17:56:06Z

I'll try running again with -Djava.locale.providers=COMPAT which is mentioned in https://bugs.openjdk.org/browse/JDK-8304925 as a possible workaround.

scottmarlow · 2024-05-01T18:43:46Z

-Djava.locale.providers=COMPAT didn't help.

scottmarlow · 2024-05-07T18:04:44Z

@pnicolucci do you know of anyone that has started coding a fix for addressing this issue?

pnicolucci · 2024-05-07T18:11:54Z

Hi @scottmarlow , I was able to get the TCK running this morning after a couple of days of working with it. Let me try to get something in place now regarding a solution to this problem. I'm sure I'll have TCK-specific questions so I'll post those here as I have them.

gurunrao · 2024-05-08T06:01:25Z

@scottmarlow @pnicolucci - We don't have a release planned for Tags in EE 11. As per TCK process, the service release of TCK [ https://github.com/jakartaee/platform-tck/pull/1219 ] allow for exclusion of tests or adjust the test validation logic to “expand” the validation check.I don't believe we are expanding validation with the fix for the challenge here.

scottmarlow · 2024-05-08T15:00:40Z

@scottmarlow @pnicolucci - We don't have a release planned for Tags in EE 11. As per TCK process, the service release of TCK [ https://github.com/jakartaee/platform-tck/pull/1219 ] allow for exclusion of tests or adjust the test validation logic to “expand” the validation check.I don't believe we are expanding validation with the fix for the challenge here.

@gurunrao please see my comment on jakartaee/platform-tck#1219 (comment)

This issue is for tags 3.0 for EE 10. It is fine if you respond here. I'll update the jakartaee/platform-tck#1219 (comment) discussion to link to this comment.

pnicolucci · 2024-05-08T18:12:07Z

@scottmarlow being unfamiliar with the TCK, can you give me a quick primer on how to build the specific buckets? If I make code changes to the Tags (JSTL) tests what's the easiest way to get them reflected in the test artifacts locally to validate in my dev environment?

scottmarlow · 2024-05-08T18:31:47Z

@scottmarlow being unfamiliar with the TCK, can you give me a quick primer on how to build the specific buckets? If I make code changes to the Tags (JSTL) tests what's the easiest way to get them reflected in the test artifacts locally to validate in my dev environment?

Certainly, it should likely fit in easily with how you are running the existing Jakarta EE 10 Platform TCK in your local environment. The Platform TCK binary archive includes the source that is buildable with a few minor changes to how you are running the jstl tests.

Your changes need to be made to the $TS_HOME/src/com/sun/ts/tests/jstl folder before you do the following.

Update your script or commands that run the Platform TCK:

if [[ "$testFolder" == jstl* ]]; then
cd $TS_HOME/src/com/sun/ts/tests/$testFolder
ant compile build package
cd $TS_HOME/src/web/jstl
ant compile build package
fi

The above fragment would be added in your script before the ant runclient command is run.

scottmarlow · 2024-05-15T19:12:24Z

@pnicolucci do you need much more time for testing your changes? I would like to at least see other implementations have access to your test changes soon so others can also test.

pnicolucci · 2024-05-15T20:04:36Z

@scottmarlow I have one test working with Java 21, it took much longer than anticipated to get the TCK up and running to a state where I can make changes and test. I'm working on this as I have time. If we need a decision right now I'd say our only option is to exclude the tests. If I have something viable for testing soon we can have the discussion as to if we have time or not to include it for Jakarta EE11.

I assume we could take some technical debt and update this TCK for Jakarta EE12 if it can't be done for EE11 so we can enable the tests again?

scottmarlow · 2024-05-15T21:29:57Z

@scottmarlow I have one test working with Java 21, it took much longer than anticipated to get the TCK up and running to a state where I can make changes and test. I'm working on this as I have time. If we need a decision right now I'd say our only option is to exclude the tests. If I have something viable for testing soon we can have the discussion as to if we have time or not to include it for Jakarta EE11.

I assume we could take some technical debt and update this TCK for Jakarta EE12 if it can't be done for EE11 so we can enable the tests again?

I agree that could work. I'm personally away from the computer next week but others should be around still if you conclude which direction works best for Tags (we need to release a new Tags 3.0.x TCK + Platform TCK with the changes or tests excluded).

I think that we should plan on having the updated Tags 3.0.x/Platform 10.0.x TCKs by June if not sooner.

pnicolucci · 2024-05-28T02:52:47Z

@scottmarlow see progress here: jakartaee/platform-tck#1312 I'm working to finish up the remaining sub buckets.

pnicolucci · 2024-08-06T16:18:54Z

@scottmarlow can we close this issue?

scottmarlow · 2024-08-06T16:45:26Z

Thank you for your help @pnicolucci!

scottmarlow mentioned this issue Mar 12, 2024

Jakarta EE 10 standalone TCK support for JDK 21 runs. jakartaee/platform-tck#1219

Merged

pnicolucci added the challenge TCK challenge label Mar 13, 2024

scottmarlow mentioned this issue Mar 13, 2024

Tags 3.0 TCK Challenge for test ExceptionCheckTag.java depending on removed Pages 4.0 JspException.getRootCause() method... #256

Closed

pnicolucci added the accepted label Apr 2, 2024

dmatej mentioned this issue Apr 19, 2024

Revised Ant exec usages in TCK + TCK 10.0.4 eclipse-ee4j/glassfish#24925

Merged

scottmarlow pushed a commit to scottmarlow/jakartaee-tck that referenced this issue Apr 24, 2024

jakartaee/tags#255, jakartaee/tags#256 challenge exclusions.

0ac56ea

Signed-off-by: Gurunandan Rao <gurunandan.rao@oracle.com>

scottmarlow mentioned this issue Apr 24, 2024

Address JSTL TCK Challenge tags/issues/255 + tags/issues/256 jakartaee/platform-tck#1290

Closed

scottmarlow pushed a commit to scottmarlow/jakartaee-tck that referenced this issue Apr 24, 2024

jakartaee/tags#255, jakartaee/tags#256 challenge exclusions.

3b9a852

Signed-off-by: Gurunandan Rao <gurunandan.rao@oracle.com>

This was referenced May 9, 2024

JSTL TCK Refactoring using Arquillian and JUnit jakartaee/platform-tck#1203

Merged

JSTL : Update glassfish runner to fix tests jakartaee/platform-tck#1303

Merged

pnicolucci mentioned this issue May 28, 2024

Update jstl dates for java21 jakartaee/platform-tck#1312

Open

scottmarlow mentioned this issue May 28, 2024

Old DTDs/Schema in deployment descriptors of Tags/Pages/Servlet/Assembly Platform TCK tests jakartaee/platform-tck#1313

Closed

pnicolucci mentioned this issue Jun 4, 2024

Update jstl dates for java21 10.0.x jakartaee/platform-tck#1320

Merged

alwin-joseph mentioned this issue Jun 17, 2024

JSTL Test Fix(tckrefactor branch) for issues/255 in JDK21 due to space character before AM/PM jakartaee/platform-tck#1329

Merged

This was referenced Jul 8, 2024

Please promote Platform TCK jakarta-jakartaeetck-10.0.5.zip jakartaee/specifications#760

Merged

Please promote Tags TCK https://download.eclipse.org/jakartaee/tags/3.0/jakarta-tags-tck-3.0.1.zip jakartaee/specifications#762

Merged

scottmarlow closed this as completed Aug 6, 2024

Tags 3.0 TCK Challenge against tests that include space character before AM/PM #255

Tags 3.0 TCK Challenge against tests that include space character before AM/PM #255

Comments

scottmarlow commented Mar 12, 2024 • edited Loading

pnicolucci commented Mar 13, 2024

dmatej commented Mar 13, 2024

scottmarlow commented Mar 13, 2024

pnicolucci commented Mar 13, 2024

HTTP Status 500 - Internal Server Error

Eclipse GlassFish 8.0.0

scottmarlow commented Mar 13, 2024

pnicolucci commented Mar 13, 2024

scottmarlow commented Mar 13, 2024

scottmarlow commented Mar 13, 2024 • edited Loading

pnicolucci commented Mar 14, 2024 • edited Loading

scottmarlow commented Mar 15, 2024

volosied commented Mar 28, 2024

pnicolucci commented Mar 28, 2024

pnicolucci commented Mar 28, 2024

dmatej commented Apr 19, 2024 • edited Loading

bstansberry commented Apr 19, 2024

dmatej commented Apr 19, 2024

scottmarlow commented Apr 22, 2024

scottmarlow commented Apr 23, 2024

scottmarlow commented Apr 24, 2024

pnicolucci commented Apr 24, 2024

scottmarlow commented Apr 25, 2024 • edited Loading

pnicolucci commented Apr 25, 2024

scottmarlow commented Apr 25, 2024

scottmarlow commented Apr 25, 2024

scottmarlow commented Apr 26, 2024

scottmarlow commented May 1, 2024

scottmarlow commented May 1, 2024

scottmarlow commented May 7, 2024

pnicolucci commented May 7, 2024

gurunrao commented May 8, 2024

scottmarlow commented May 8, 2024

pnicolucci commented May 8, 2024

scottmarlow commented May 8, 2024

scottmarlow commented May 15, 2024

pnicolucci commented May 15, 2024

scottmarlow commented May 15, 2024

pnicolucci commented May 28, 2024

pnicolucci commented Aug 6, 2024

scottmarlow commented Aug 6, 2024

scottmarlow commented Mar 12, 2024 •

edited

Loading

scottmarlow commented Mar 13, 2024 •

edited

Loading

pnicolucci commented Mar 14, 2024 •

edited

Loading

dmatej commented Apr 19, 2024 •

edited

Loading

scottmarlow commented Apr 25, 2024 •

edited

Loading