HDFS-16278. Make HDFS snapshot tools cross platform #3563

GauthamBanasandra · 2021-10-18T18:34:48Z

Description of PR

The source files for hdfs_createSnapshot, hdfs_disallowSnapshot and hdfs_renameSnapshot uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

The details on testing of each of the above tool is as follows -

hdfs_createSnasphot

I was able to create snapshots with both the default and custom names -

UI -

hdfs_renameSnapshot

I renamed one of the snapshots that was created previously (s20211019-232643.399) as another-snapshot -

UI -

hdfs_disallowSnapshot

I cleared all the previously created snapshots under /mydir and verified hdfs_disallowSnapshot on it -

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2021-10-18T22:22:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	43m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 59s		trunk passed
+1 💚	compile	2m 52s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	61m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	2m 51s		the patch passed
+1 💚	cc	2m 51s		the patch passed
+1 💚	golang	2m 51s		the patch passed
+1 💚	javac	2m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	20m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	95m 2s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		226m 47s

Reason	Tests
Failed CTEST tests	hdfs_tool_tests

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/1/artifact/out/Dockerfile
GITHUB PR	#3563
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux bf40bbc856ef 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `646731f`
Default Java	Red Hat, Inc.-1.8.0_302-b08
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/1/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/1/testReport/
Max. process+thread count	571 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-19T22:24:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	44m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 57s		trunk passed
+1 💚	compile	3m 18s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	shadedclient	61m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	2m 51s		the patch passed
+1 💚	cc	2m 51s		the patch passed
+1 💚	golang	2m 51s		the patch passed
+1 💚	javac	2m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	shadedclient	20m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	112m 57s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		245m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/artifact/out/Dockerfile
GITHUB PR	#3563
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 1c4e8771f893 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a457124`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/testReport/
Max. process+thread count	524 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-20T01:45:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	24m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 35s		trunk passed
+1 💚	compile	3m 5s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	48m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	3m 1s		the patch passed
+1 💚	cc	3m 1s		the patch passed
+1 💚	golang	3m 1s		the patch passed
+1 💚	javac	3m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	shadedclient	20m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	100m 20s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		200m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/artifact/out/Dockerfile
GITHUB PR	#3563
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 3338ac2b0f80 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a457124`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/testReport/
Max. process+thread count	518 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-20T05:27:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 2s		trunk passed
+1 💚	compile	2m 44s		trunk passed
+1 💚	mvnsite	0m 21s		trunk passed
+1 💚	shadedclient	63m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s		the patch passed
+1 💚	compile	2m 46s		the patch passed
+1 💚	cc	2m 46s		the patch passed
+1 💚	golang	2m 46s		the patch passed
+1 💚	javac	2m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	shadedclient	28m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	107m 14s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		221m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/artifact/out/Dockerfile
GITHUB PR	#3563
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 9c958dd3d7b5 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a457124`
Default Java	Debian-11.0.12+7-post-Debian-2deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/testReport/
Max. process+thread count	587 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-20T08:55:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 35s		trunk passed
+1 💚	compile	3m 14s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	3m 17s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	0m 22s		trunk passed
+1 💚	shadedclient	52m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s		the patch passed
+1 💚	compile	3m 9s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	3m 9s		the patch passed
+1 💚	golang	3m 9s		the patch passed
+1 💚	javac	3m 9s		the patch passed
+1 💚	compile	3m 35s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	3m 35s		the patch passed
+1 💚	golang	3m 35s		the patch passed
+1 💚	javac	3m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	shadedclient	21m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	121m 41s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		207m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/artifact/out/Dockerfile
GITHUB PR	#3563
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 9efae35f7481 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a457124`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/testReport/
Max. process+thread count	521 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3563/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra added 11 commits October 17, 2021 23:39

Make hdfs_createSnapshot cross platform

6cba039

Fix documentation for hdfs_createSnapshot

853f4cc

Make hdfs_disallowSnapshot cross platform

a3b8da0

Rename HandlePath to HandleSnapshot

28ecaaf

Make hdfs_renameSnapshot cross platform

d337e27

Add unit tests for hdfs_disallowSnapshot

72bcb85

Add unit tests for hdfs_renameSnapshot

82ac6c3

Fix ValidateConstraints for hdfs_createSnapshot

c2afc83

Add unit tests for hdfs_createSnapshot

aa29f6f

Do some refactor

ed03996

Remove redundant files

646731f

GauthamBanasandra added 2 commits October 19, 2021 23:16

Fix hdfs_createSnapshot unit test failures

8ebaf35

Fix documentation

a457124

GauthamBanasandra marked this pull request as ready for review October 19, 2021 18:27

goiri approved these changes Oct 20, 2021

View reviewed changes

goiri approved these changes Oct 21, 2021

View reviewed changes

goiri merged commit a2f089d into apache:trunk Oct 21, 2021

GauthamBanasandra deleted the snapshot-x-platform branch October 21, 2021 17:36

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16278. Make HDFS snapshot tools cross platform (apache#3563)

244ffb2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16278. Make HDFS snapshot tools cross platform #3563

HDFS-16278. Make HDFS snapshot tools cross platform #3563

Uh oh!

GauthamBanasandra commented Oct 18, 2021 •

edited

Loading

Uh oh!

hadoop-yetus commented Oct 18, 2021

Uh oh!

hadoop-yetus commented Oct 19, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-16278. Make HDFS snapshot tools cross platform #3563

HDFS-16278. Make HDFS snapshot tools cross platform #3563

Uh oh!

Conversation

GauthamBanasandra commented Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

hdfs_createSnasphot

hdfs_renameSnapshot

hdfs_disallowSnapshot

For code changes:

Uh oh!

hadoop-yetus commented Oct 18, 2021

Uh oh!

hadoop-yetus commented Oct 19, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

hadoop-yetus commented Oct 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GauthamBanasandra commented Oct 18, 2021 •

edited

Loading