Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE… #2862

Merged
merged 1 commit into from
Jan 8, 2021
+19 −1
Merged

HBASE-25473 [create-release] checkcompatibility.py failing with "KeyE… #2862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion dev-support/checkcompatibility.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ def compare_results(tool_results, known_issues, compare_warnings):
observed_count=tool_results[check][issue_type])
for check, known_issue_counts in known_issues.items()
for issue_type, known_count in known_issue_counts.items()
if tool_results[check][issue_type] > known_count]
if compare_tool_results_count(tool_results, check, issue_type, known_count)]

if not compare_warnings:
unexpected_issues = [tup for tup in unexpected_issues
Expand All @@ -241,6 +241,14 @@ def compare_results(tool_results, known_issues, compare_warnings):

return bool(unexpected_issues)

def compare_tool_results_count(tool_results, check, issue_type, known_count):
""" Check problem counts are no more than the known count.
(This function exists just so can add in logging; previous was inlined
one-liner but this made it hard debugging)
"""
# logging.info("known_count=%s, check key=%s, tool_results=%s, issue_type=%s",
# str(known_count), str(check), str(tool_results), str(issue_type))
return tool_results[check][issue_type] > known_count

def process_java_acc_output(output):
""" Process the output string to find the problems and warnings in both the
Expand Down
10 changes: 10 additions & 0 deletions dev-support/create-release/release-util.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,8 +478,18 @@ function generate_api_report {
local timing_token
timing_token="$(start_step)"
# Generate api report.
# Filter out some jar types. Filters are tricky. Python regex on
# file basename. Exclude the saved-aside original jars... they are
# not included in resulting artifact. Also, do not include the
# hbase-shaded-testing-util.* jars. This jar is unzip'able on mac
# os x as is because has it a META_INF/LICENSE file and then a
# META_INF/license directory for the included jar's licenses;
# it fails to unjar on mac os x which this tool does making its checks
# (Its exclusion should be fine; it is just an aggregate of other jars).
"${project}"/dev-support/checkcompatibility.py --annotation \
org.apache.yetus.audience.InterfaceAudience.Public \
-e "original-hbase.*.jar" \
-e "hbase-shaded-testing-util.*.jar" \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excluding this jar is a problem -- the shaded jars are intended explicitly as our public interface to down-streamers, aren't they?

What if we add an exclude of the [lL][iI][cC][eE][nN][sS][eE] pattern instead? oh, I see the jar command doesn't support exclude patterns, only an include pattern. And where does this happen, in the perl script?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.

I think it ok excluding this shaded jar because it is made from other jars that are NOT excluded; if an api change violation, it'll be caught there, at the source. Testing the shade would be more comprehensive yes but exclude is fine for now given it allows us getting the RC ball rolling again. The reason-for-exclusion is for fix in HBASE-25486

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include/exclude are in the python script. Does jar files only. The jar files are subsequently passed to the wrapped perl script... which then does unzip... and fails with hbase-shaded-testing-util.

I meant exclude of unpacked archive content, not entire archives. And yes, that's what I feared, that the unpack happens in the perl script, "outside of our control."

I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?

I like HBASE-25486.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking the reason to retain the shaded jar is to notice if we break shading entirely. i.e., a change that adds a new jar to the shaded package would show in the report as loads of new classes. Perhaps we have other mechanisms in place already for catching these kinds of errors?

This is a valid concern. Lets fix HBASE-25486 and get the shaded jar back in the loop.

"$previous_tag" "$release_tag"
previous_version="$(echo "${previous_tag}" | sed -e 's/rel\///')"
cp "${project}/target/compat-check/report.html" "./api_compare_${previous_version}_to_${release_tag}.html"
Expand Down