HBASE-25824 IntegrationTestLoadCommonCrawl #3208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apurtell
force-pushed
the
HBASE-25824
branch
2 times, most recently
from
ccfd9f6 to
b08d1ce
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
saintstack
approved these changes
Apr 29, 2021
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Thanks for the review @saintstack . Last precommit looked good even though earlier test failures weren't related. I made a push of improvements. I see a round of checkstyle and whitespace fixes are due. Let me address them and then close this out. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
This issue was only recently found because you have to get ~1000 files into the set before it manifests: We are accepting very large row keys from the crawl data during the load phase, need to configure the job not to choke on them. |
This was referenced May 2, 2021
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
This integration test loads successful resource retrieval records from the Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and writes records that can be used to later verify the presence and integrity of those records. Run like: ./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \ -Dfs.s3n.awsAccessKeyId=<AWS access key> \ -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \ /path/to/test-CC-MAIN-2021-10-warc.paths.gz \ /path/to/tmp/warc-loader-output Access to the Common Crawl dataset in S3 is made available to anyone by Amazon AWS, but Hadoop's S3N filesystem still requires valid access credentials to initialize. The input path can either specify a directory or a file. The file may optionally be compressed with gzip. If a directory, the loader expects the directory to contain one or more WARC files from the Common Crawl dataset. If a file, the loader expects a list of Hadoop S3N URIs which point to S3 locations for one or more WARC files from the Common Crawl dataset, one URI per line. Lines should be terminated with the UNIX line terminator. Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC files in this data set, each containing ~1GB of gzipped data. The WARC files contain several record types, such as metadata, request, and response, but we only load the response record types. If the HBase table schema does not specify compression (by default) there is roughly a 10x expansion. Loading the full crawl archive results in a table approximately 640 TB in size. The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem. Use the -files ToolRunner argument to add it. You can also split the Loader and Verify stages: Load with: ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \ -files /path/to/hadoop-aws.jar \ -Dfs.s3n.awsAccessKeyId=<AWS access key> \ -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \ /path/to/test-CC-MAIN-2021-10-warc.paths.gz \ /path/to/tmp/warc-loader-output Verify with: ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \ /path/to/tmp/warc-loader-output
- Javadoc warnings - Missing @OverRide - Checkstyle nits - Whitespace - Remove line length limitation in WARCRecord#readLine. It is rare but the CC data includes lines that are longer. If there is a real format error like a corrupted file let an EOFException or OOME signal the problem.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
May 4, 2021
This integration test loads successful resource retrieval records from the Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and writes records that can be used to later verify the presence and integrity of those records. Run like: ./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \ -Dfs.s3n.awsAccessKeyId=<AWS access key> \ -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \ /path/to/test-CC-MAIN-2021-10-warc.paths.gz \ /path/to/tmp/warc-loader-output Access to the Common Crawl dataset in S3 is made available to anyone by Amazon AWS, but Hadoop's S3N filesystem still requires valid access credentials to initialize. The input path can either specify a directory or a file. The file may optionally be compressed with gzip. If a directory, the loader expects the directory to contain one or more WARC files from the Common Crawl dataset. If a file, the loader expects a list of Hadoop S3N URIs which point to S3 locations for one or more WARC files from the Common Crawl dataset, one URI per line. Lines should be terminated with the UNIX line terminator. Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC files in this data set, each containing ~1GB of gzipped data. The WARC files contain several record types, such as metadata, request, and response, but we only load the response record types. If the HBase table schema does not specify compression (by default) there is roughly a 10x expansion. Loading the full crawl archive results in a table approximately 640 TB in size. The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem. Use the -files ToolRunner argument to add it. You can also split the Loader and Verify stages: Load with: ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \ -files /path/to/hadoop-aws.jar \ -Dfs.s3n.awsAccessKeyId=<AWS access key> \ -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \ /path/to/test-CC-MAIN-2021-10-warc.paths.gz \ /path/to/tmp/warc-loader-output Verify with: ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \ /path/to/tmp/warc-loader-output Signed-off-by: Michael Stack <stack@apache.org> Conflicts: pom.xml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This integration test loads successful resource retrieval records from
the Common Crawl (https://commoncrawl.org/) public dataset into an HBase
table and writes records that can be used to later verify the presence
and integrity of those records.
Run like:
Access to the Common Crawl dataset in S3 is made available to anyone by
Amazon AWS, but Hadoop's S3N filesystem still requires valid access
credentials to initialize.
The input path can either specify a directory or a file. The file may
optionally be compressed with gzip. If a directory, the loader expects
the directory to contain one or more WARC files from the Common Crawl
dataset. If a file, the loader expects a list of Hadoop S3N URIs which
point to S3 locations for one or more WARC files from the Common Crawl
dataset, one URI per line. Lines should be terminated with the UNIX line
terminator.
Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz
is a list of all WARC files comprising the Q1 2021 crawl archive. There
are 64,000 WARC files in this data set, each containing ~1GB of gzipped
data. The WARC files contain several record types, such as metadata,
request, and response, but we only load the response record types. If
the HBase table schema does not specify compression (by default) there
is roughly a 10x expansion. Loading the full crawl archive results in a
table approximately 640 TB in size.
The hadoop-aws jar will be needed at runtime to instantiate the S3N
filesystem. Use the -files ToolRunner argument to add it.
You can also split the Loader and Verify stages:
Load with:
Verify with: