HBASE-24106 Update getting started documentation after HBASE-24086 #1422
Conversation
ndimiduk
force-pushed
the
24106-getting-started
branch
from
094cc6b to
c9c164e
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
ndimiduk
force-pushed
the
24106-getting-started
branch
from
c9c164e to
27e6663
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| which HBase stores its data. You can specify values for this configuration directly, as you'll see | ||
| in the subsequent sections. | ||
| + | ||
| NOTE: In this example, HBase is running on Hadoop's `LocalFileSystem`. That abstraction doesn't |
There was a problem hiding this comment.
Given this change is only going into specific versions, should we add a note for users on older versions? Something like...
if you are on v < 2.3, disable stream enforcement by setting this flag.. later versions Hbase takes care of it automatically.....
There was a problem hiding this comment.
Generally, documentation changes are merged to master and cherry-picked to appropriate branches. In this case the commit should be backported to branch-2 and branch-2.3 but not to branch-2.2.
There was a problem hiding this comment.
True..but I think most people refer to the ref-guide here https://hbase.apache.org/book.html rather than version specific one because it shows up in search results.. hence my comment.
There was a problem hiding this comment.
We've never really solved challenges of the branch-specific documentation.
I didn't try older versions of 2.x... let me see where they stand.
There was a problem hiding this comment.
The book says HBase 2.1 and 2.2 are "untested" with Hadoop 2.10. branch-2.2 builds with -Dhadoop-two.version=2.10.0 and produces a similar error at runtime:
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check th
e config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1092)
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:424)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:586)
at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1522)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:579)
at java.lang.Thread.run(Thread.java:748)
branch-2.1 does not build with hadoop-2.10.0,
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (banned-jsr305) @ hbase-client ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with message:
We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
There was a problem hiding this comment.
Given this change is only going into specific versions, should we add a note for users on older versions? Something like...
if you are on v < 2.3, disable stream enforcement by setting this flag.. later versions Hbase takes care of it automatically.....
This is tricky. My understanding is that HBase has never had these durability guarantees when running on LocalFileSystem because no version of Hadoop has ever provided an implementation that provides hflush or hsync on that class. Thus this warning is applicable everywhere, to everyone.
Now as far as HBase's behavior in the presence of a LocalFileSystem, that's a little different. On Hadoop-2.8.x, we had no way to ask Hadoop if the OutputStream supported these characteristics, so we simply move forward with a warning. It's not clear to me when we first exposed hbase.unsafe.stream.capability.enforce (on first glance, all the places that config is referenced appear to have been changed since its inception) or what our behavior was before then. Let me look into this further and see if I can make a recommendation.
If we back port the parent issue to branch-2.2, then I think the behavior will be the same on all branch-2 derivatives.
There was a problem hiding this comment.
HBASE-19289 has some nice history here...
There was a problem hiding this comment.
My understanding is that HBase has never had these durability guarantees when running on LocalFileSystem because no version of Hadoop has ever provided an implementation that provides hflush or hsync on that class
Yeah, agree on this point. Just driving by to also mention that there is the RawLocalFileSystem implementation which works with the local filesystem and does implement hflush/hsync. There's just some more trickery to get it set up for file:// instead of LocalFileSystem. I don't think we have this configured for HBase at all (I remember Accumulo used to do a bunch with it for UT's).
There was a problem hiding this comment.
Yeah, agree on this point. Just driving by to also mention that there is the RawLocalFileSystem implementation which works with the local filesystem and does implement hflush/hsync. There's just some more trickery to get it set up for file:// instead of LocalFileSystem. I don't think we have this configured for HBase at all (I remember Accumulo used to do a bunch with it for UT's).
Thanks for the pointer @joshelser . I'm not up to speed on the differences between the two. I looked long enough to see that they follow different inheritance hierarchies.
If we could converge the majority of our small and medium test suite onto some equivalent to LocalFileSystem, it would make a huge difference in the runtime and resource usage of tests...
| which HBase stores its data. You can specify values for this configuration directly, as you'll see | ||
| in the subsequent sections. | ||
| + | ||
| NOTE: In this example, HBase is running on Hadoop's `LocalFileSystem`. That abstraction doesn't |
There was a problem hiding this comment.
Generally, documentation changes are merged to master and cherry-picked to appropriate branches. In this case the commit should be backported to branch-2 and branch-2.3 but not to branch-2.2.
|
Pinging some other reviewers who were involved in earlier discussions around filesystem capabilities. Please see the parent issue to this one. |
Signed-off-by: Josh Elser <elserj@apache.org> Signed-off-by: Bharath Vissapragada <bharathv@apache.org> Signed-off-by: Peter Somogyi <psomogyi@apache.org>
ndimiduk
force-pushed
the
24106-getting-started
branch
from
27e6663 to
b409d17
Compare
|
Applied to Thanks for the reviews. |
Updates documentation according to change in HBASE-24086 / #1408 .