Normalize http examples #1215

mauritsvanrees · 2021-09-02T21:42:20Z

Note: this builds on my approved PR #1214 so that one should be merged first.

This normalises the requests and responses that are written to the http-examples directory:

Use a stable behavior name for generated schemas. Fixes Upgrade to Plone 5.2.4 blocked (because of plone.dexterity 2.10.0) #1096 so this PR also upgrades us to the newer plone.dexterity from Plone 5.2.5.
It strips away whitespace from the right of the line, leading to much smaller diffs - except for the current PR of course. Note that on GitHub you can add ?w=1 to the url to ignore whitespace changes, making this easier to review.
It normalises the port number to the standard 55001. This helps when you run the restapi tests in a Plone core development buildout, which uses a random port.

Let plone-5.2.x.cfg first extend the default Plone 5.2.5 versions, and then base.cfg. Let base.cfg contain version overrides, including `plone.restapi =` to unpin ourselves. Also remove version pin from buildout.cfg which are already the same or newer in Plone 5.2.5.

We already ignored this when checking if there are any differences at all. But when there are differences, we still showed *all* differences, including white space, which makes it hard to see the real problems.

Locally I also get a different 'created' date: '1995-07-31T19:30:00' instead of '1995-07-31T17:30:00', as I am in GMT+2, even though the tests set the timezone to UTC. Remaining problem: the token in this same snippet is random. Also, in a few of the other files, instead of 'plone.dexterity.schema.generated.plone_0_Document' I get: 'plone.dexterity.schema.generated.plone_5_1630600102_2_680259_0_Document' I saw these before locally, but assumed it was something in my setup, because the GH Actions kept passing. That seems to have been because on old release of plone.restapi was being tested.

We need to pin an older dexterity because the new dynamic schemas conflict with the expected output in our http-examples. See #1096

…n CI. See #1214 (comment)

…hanges in CI." This reverts commit 6b265ff.

This reverts commit def2340.

…ument_crud. plone.dexterity 2.10+ has dynamic schemas, so we get for example plone.dexterity.schema.generated.plone_5_1630611587_2_523689_0_Document as behavior name. Instead, we save what we got in older dexterity: plone.dexterity.schema.generated.plone_0_Document

The first part of save_request_and_response_for_docs was an exact copy save_request_for_docs (before my change in the previous commit).

…mparison. This avoids many uninteresting differences.

This avoids differences when running from the Plone coredev buildout, which uses a random port. A few differences remain, at least when running the test in Plone 6. On Plone 5.2, only workingcopy_baseline_get.resp has a difference, and this depends on your timezone.

This is a very big diff, but 'git diff -w' is empty. Note: when you view the diff on github, you can add '?w=1' after the url to ignore whitespace differences.

mauritsvanrees · 2021-09-02T21:43:28Z

@jenkins-plone-org please run jobs

wesleybl · 2021-09-03T20:17:39Z

src/plone/restapi/tests/http-examples/workingcopy_baseline_get.resp

+    "layout": "document_view",
+    "lock": {
+        "created": "1995-07-31T17:30:00",


Funny thing is that locally for me, only this line got different. It stayed:

"created": "1995-07-31T14:30:00",

It seems that only in this part the TZ environment variable had no effect.

humm this is explained here:

3a4e8f3

@mauritsvanrees wouldn't it be the case to normalize this too, so as not to show differences locally?

Possibly, but I wonder why the timezone has no effect here.
@avoinea This is the created date of a lock. Can you run the tests locally and then check if this results in a local change in git?

It's really weird that all dates respect the TZ environment variable and only this date doesn't.

I saw that who sets this date is the function:

plone.restapi/src/plone/restapi/services/locking/__init__.py

Lines 18 to 19 in df3dba9

def creation_date(timestamp):

return datetime.fromtimestamp(timestamp).isoformat()

Maybe this can help:

https://stackoverflow.com/questions/31977563/python-how-do-you-convert-a-datetime-timestamp-from-one-timezone-to-another-tim

@avoinea FYI

@avoinea @mauritsvanrees

@ericof fell in this hole too:

2bc6910

wesleybl · 2021-09-03T20:30:59Z

src/plone/restapi/tests/test_documentation.py

+def normalize_test_port(value):
+    # When you run these tests in the Plone core development buildout,
+    # the port number is random.  Normalize this to the default port.
+    return re.sub(r"localhost:\d{5}", "localhost:55001", value)


Now we can remove this line:

plone.restapi/base.cfg

Line 74 in c284b56

os.environ['ZSERVER_PORT'] = '55001'

We could remove that yes, if we want to.
@sneridagh You made commit 799e11e two years ago to always use port 55001. Was that to make the http-examples always have the same port? In that case we can now remove this setting, if we want.

As long as the documentation resp files always get the same value and do not change over builds, it's ok.

wesleybl · 2021-09-03T20:34:44Z

src/plone/restapi/tests/test_documentation.py

+        document_schema_re = re.compile(
+            r"^plone.dexterity.schema.generated.plone_.*_Document$"
+        )
+        stable_behavior = "plone.dexterity.schema.generated.plone_0_Document"


I think stable_behavior should look like what it is in plone.dexterity 2.10+. That is:

plone.dexterity.schema.generated.plone_5_1630600102_2_680259_0_Document

Technically that would be closer to what it looks like. But it would be uglier, so I prefer this clean one.

Even being “uglier”, I prefer it closer to reality. Each part of this code has a meaning. When someone unsuspecting consults the documentation, they can get confused.

mauritsvanrees added 14 commits September 2, 2021 18:24

Test no uncommitted doc changes: ignore space at end of line.

656abde

We already ignored this when checking if there are any differences at all. But when there are differences, we still showed *all* differences, including white space, which makes it hard to see the real problems.

Downgrade to plone.dexterity 2.9.8 again.

def2340

We need to pin an older dexterity because the new dynamic schemas conflict with the expected output in our http-examples. See #1096

Temporarily comment out running bin/test-no-uncommitted-doc-changes i…

6b265ff

…n CI. See #1214 (comment)

Revert "Temporarily comment out running bin/test-no-uncommitted-doc-c…

4ab89ea

…hanges in CI." This reverts commit 6b265ff.

Test fix: save a stable token in workingcopy_baseline_get.

a356190

Revert "Downgrade to plone.dexterity 2.9.8 again."

5a50ea3

This reverts commit def2340.

Let save_request_and_response_for_docs call save_request_for_docs.

b40d180

The first part of save_request_and_response_for_docs was an exact copy save_request_for_docs (before my change in the previous commit).

Doc tests: right strip the pretty printed http examples for easier co…

f755a5f

…mparison. This avoids many uninteresting differences.

black

2cf9b0a

Updated all http-examples.

b287478

This is a very big diff, but 'git diff -w' is empty. Note: when you view the diff on github, you can add '?w=1' after the url to ignore whitespace differences.

This comment has been minimized.

Sign in to view

mauritsvanrees mentioned this pull request Sep 3, 2021

Fix testing of a checkout instead of a released package. #1214

Closed

tisto self-requested a review September 3, 2021 19:12

tisto approved these changes Sep 3, 2021

View reviewed changes

tisto merged commit c284b56 into master Sep 3, 2021

tisto deleted the maurits/normalize-http-examples branch September 3, 2021 19:12

wesleybl reviewed Sep 3, 2021

View reviewed changes

mauritsvanrees mentioned this pull request Sep 6, 2021

Use checkout on 5.2, normalise http-examples [7xx] #1218

Merged

mauritsvanrees mentioned this pull request Sep 27, 2021

Further normalise http-examples, and no port hardcoding #1226

Merged

wesleybl mentioned this pull request Feb 14, 2022

fix(auth): Unify Zope root ZMI w/ API log in/out #1304

Open

wesleybl referenced this pull request Aug 19, 2022

Fix two instances of the version after release.

8b7ce50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Normalize http examples #1215

Normalize http examples #1215

mauritsvanrees commented Sep 2, 2021

This comment has been minimized.

mauritsvanrees commented Sep 2, 2021

wesleybl Sep 3, 2021 •

edited

Loading

wesleybl Sep 3, 2021

mauritsvanrees Sep 4, 2021

wesleybl Sep 6, 2021

wesleybl Sep 17, 2021 •

edited

Loading

wesleybl Sep 3, 2021

mauritsvanrees Sep 4, 2021

sneridagh Sep 6, 2021

wesleybl Sep 3, 2021

mauritsvanrees Sep 6, 2021

wesleybl Sep 6, 2021

	def creation_date(timestamp):
	return datetime.fromtimestamp(timestamp).isoformat()

Normalize http examples #1215

Normalize http examples #1215

Conversation

mauritsvanrees commented Sep 2, 2021

This comment has been minimized.

mauritsvanrees commented Sep 2, 2021

wesleybl Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesleybl Sep 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesleybl Sep 3, 2021 •

edited

Loading

wesleybl Sep 17, 2021 •

edited

Loading