HBASE-11686 Shell code should create a binding / irb workspace instea…

…d of polluting the root namespace (#2141)

* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace

- Refactor Shell.export_commands to define commands using ruby lambdas.
  Additionally, this change stores a reference to shell_inst in scope so that
  we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
  binding and non-global receiver. This workspace is loaded with all HBase and
  IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
  within the created IRB workspace. This is based on work by Hsieh that was
  previously in bin/hirb.rb. This method is more generic and more testable.
  This single pattern can be used for both executing files and reading from
  stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
  commands are tightly linked with an instance of the shell, so it is easiest
  to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
  this change, tests were loading these constants into the top level, which
  could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
  best to reference each ruby constant's full name, but where that would cause
  a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
  HBase symbols from the top-level receiver (ie. main Object), it is possible
  (albeit unlikely) that this will break operator scripts. This flag will
  export all the commands at the top-level like the shell previously did.

* HBASE-11686 Light refactoring with added unit tests

- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
  multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
  semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
  constants, and all hbase commands to a target receiver. As a result, the
  HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch

* Typo s/is/if/

(cherry picked from commit 7eff07d)

bin/hirb.rb

@@ -58,6 +58,8 @@
  -h | --help             This help.
  -n | --noninteractive   Do not run within an IRB session and exit with non-zero
                          status on first error.
+ --top-level-defs        Compatibility flag to export HBase shell commands onto
+                         Ruby's main object
  -Dkey=value             Pass hbase-*.xml Configuration overrides. For example, to
                          use an alternate zookeeper ensemble, pass:
                            -Dhbase.zookeeper.quorum=zookeeper.example.org
@@ -81,6 +83,7 @@ def add_to_configuration(c, arg)
 log_level = org.apache.log4j.Level::ERROR
 @shell_debug = false
 interactive = true
+top_level_definitions = false
 _configuration = nil
 D_ARG = '-D'.freeze
 while (arg = ARGV.shift)
@@ -108,6 +111,8 @@ def add_to_configuration(c, arg)
     warn '[INFO] the -r | --return-values option is ignored. we always behave '\
          'as though it was given.'
     found.push(arg)
+  elsif arg == '--top-level-defs'
+    top_level_definitions = true
   else
     # Presume it a script. Save it off for running later below
     # after we've set up some environment.
@@ -143,21 +148,10 @@ def add_to_configuration(c, arg)
 @shell = Shell::Shell.new(@hbase, interactive)
 @shell.debug = @shell_debug
 
-# Add commands to this namespace
-# TODO avoid polluting main namespace by using a binding
-@shell.export_commands(self)
-
-# Add help command
-def help(command = nil)
-  @shell.help(command)
-end
-
-# Backwards compatibility method
-def tools
-  @shell.help_group('tools')
-end
-
-# Debugging method
+##
+# Toggle shell debugging
+#
+# @return [Boolean] true if debug is turned on after updating the flag
 def debug
   if @shell_debug
     @shell_debug = false
@@ -173,26 +167,34 @@ def debug
   debug?
 end
 
+##
+# Print whether debug is on or off
 def debug?
   puts "Debug mode is #{@shell_debug ? 'ON' : 'OFF'}\n\n"
   nil
 end
 
-# Include hbase constants
-include HBaseConstants
+# For backwards compatibility, this will export all the HBase shell commands, constants, and
+# instance variables (@hbase and @shell) onto Ruby's top-level receiver object known as "main".
+@shell.export_all(self) if top_level_definitions
 
 # If script2run, try running it.  If we're in interactive mode, will go on to run the shell unless
 # script calls 'exit' or 'exit 0' or 'exit errcode'.
-load(script2run) if script2run
+@shell.eval_io(File.new(script2run)) if script2run
+
+# If we are not running interactively, evaluate standard input
+@shell.eval_io(STDIN) unless interactive
 
 if interactive
   # Output a banner message that tells users where to go for help
   @shell.print_banner
 
   require 'irb'
+  require 'irb/ext/change-ws'
   require 'irb/hirb'
 
   module IRB
+    # Override of the default IRB.start
     def self.start(ap_path = nil)
       $0 = File.basename(ap_path, '.rb') if ap_path
 
@@ -207,7 +209,12 @@ def self.start(ap_path = nil)
                HIRB.new
              end
 
+      shl = TOPLEVEL_BINDING.receiver.instance_variable_get :'@shell'
+      hirb.context.change_workspace shl.get_workspace
+
       @CONF[:IRB_RC].call(hirb.context) if @CONF[:IRB_RC]
+      # Storing our current HBase IRB Context as the main context is imperative for several reasons,
+      # including auto-completion.
       @CONF[:MAIN_CONTEXT] = hirb.context
 
       catch(:IRB_EXIT) do
@@ -217,48 +224,4 @@ def self.start(ap_path = nil)
   end
 
   IRB.start
-else
-  begin
-    # Noninteractive mode: if there is input on stdin, do a simple REPL.
-    # XXX Note that this purposefully uses STDIN and not Kernel.gets
-    #     in order to maintain compatibility with previous behavior where
-    #     a user could pass in script2run and then still pipe commands on
-    #     stdin.
-    require 'irb/ruby-lex'
-    require 'irb/workspace'
-    workspace = IRB::WorkSpace.new(binding)
-    scanner = RubyLex.new
-
-    # RubyLex claims to take an IO but really wants an InputMethod
-    module IOExtensions
-      def encoding
-        external_encoding
-      end
-    end
-    IO.include IOExtensions
-
-    scanner.set_input(STDIN)
-    scanner.each_top_level_statement do |statement, linenum|
-      puts(workspace.evaluate(nil, statement, 'stdin', linenum))
-    end
-  # XXX We're catching Exception on purpose, because we want to include
-  #     unwrapped java exceptions, syntax errors, eval failures, etc.
-  rescue Exception => exception
-    message = exception.to_s
-    # exception unwrapping in shell means we'll have to handle Java exceptions
-    # as a special case in order to format them properly.
-    if exception.is_a? java.lang.Exception
-      warn 'java exception'
-      message = exception.get_message
-    end
-    # Include the 'ERROR' string to try to make transition easier for scripts that
-    # may have already been relying on grepping output.
-    puts "ERROR #{exception.class}: #{message}"
-    if $fullBacktrace
-      # re-raising the will include a backtrace and exit.
-      raise exception
-    else
-      exit 1
-    end
-  end
 end

hbase-shell/src/main/ruby/hbase/admin.rb

@@ -1,5 +1,3 @@
-#
-#
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -25,6 +23,8 @@
 java_import org.apache.hadoop.hbase.util.Bytes
 java_import org.apache.hadoop.hbase.ServerName
 java_import org.apache.hadoop.hbase.TableName
+java_import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder
+java_import org.apache.hadoop.hbase.HConstants
 
 # Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin
 
@@ -1016,25 +1016,25 @@ def hcd(arg, htd)
       family.setCacheIndexesOnWrite(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_INDEX_ON_WRITE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_INDEX_ON_WRITE)
       family.setCacheBloomsOnWrite(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_BLOOMS_ON_WRITE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_BLOOMS_ON_WRITE)
       family.setEvictBlocksOnClose(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::EVICT_BLOCKS_ON_CLOSE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::EVICT_BLOCKS_ON_CLOSE)
-      family.setInMemory(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY)
+      family.setInMemory(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::IN_MEMORY))) if arg.include?(ColumnFamilyDescriptorBuilder::IN_MEMORY)
       if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY_COMPACTION)
         family.setInMemoryCompaction(
           org.apache.hadoop.hbase.MemoryCompactionPolicy.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY_COMPACTION))
         )
       end
-      family.setTimeToLive(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::TTL)) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::TTL)
-      family.setDataBlockEncoding(org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING)
-      family.setBlocksize(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE)
-      family.setMaxVersions(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::VERSIONS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::VERSIONS)
-      family.setMinVersions(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::MIN_VERSIONS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::MIN_VERSIONS)
-      family.setKeepDeletedCells(org.apache.hadoop.hbase.KeepDeletedCells.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::KEEP_DELETED_CELLS).to_s.upcase)) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::KEEP_DELETED_CELLS)
-      family.setCompressTags(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESS_TAGS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESS_TAGS)
-      family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN)
-      family.setMobEnabled(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::IS_MOB))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::IS_MOB)
-      family.setMobThreshold(JLong.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::MOB_THRESHOLD))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::MOB_THRESHOLD)
-      family.setNewVersionBehavior(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::NEW_VERSION_BEHAVIOR))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::NEW_VERSION_BEHAVIOR)
-      if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOOMFILTER)
-        bloomtype = arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::BLOOMFILTER).upcase.to_sym
+      family.setTimeToLive(arg.delete(ColumnFamilyDescriptorBuilder::TTL)) if arg.include?(ColumnFamilyDescriptorBuilder::TTL)
+      family.setDataBlockEncoding(org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::DATA_BLOCK_ENCODING))) if arg.include?(ColumnFamilyDescriptorBuilder::DATA_BLOCK_ENCODING)
+      family.setBlocksize(JInteger.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::BLOCKSIZE))) if arg.include?(ColumnFamilyDescriptorBuilder::BLOCKSIZE)
+      family.setMaxVersions(JInteger.valueOf(arg.delete(HConstants::VERSIONS))) if arg.include?(HConstants::VERSIONS)
+      family.setMinVersions(JInteger.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::MIN_VERSIONS))) if arg.include?(ColumnFamilyDescriptorBuilder::MIN_VERSIONS)
+      family.setKeepDeletedCells(org.apache.hadoop.hbase.KeepDeletedCells.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::KEEP_DELETED_CELLS).to_s.upcase)) if arg.include?(ColumnFamilyDescriptorBuilder::KEEP_DELETED_CELLS)
+      family.setCompressTags(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::COMPRESS_TAGS))) if arg.include?(ColumnFamilyDescriptorBuilder::COMPRESS_TAGS)
+      family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(ColumnFamilyDescriptorBuilder::PREFETCH_BLOCKS_ON_OPEN)
+      family.setMobEnabled(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::IS_MOB))) if arg.include?(ColumnFamilyDescriptorBuilder::IS_MOB)
+      family.setMobThreshold(JLong.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::MOB_THRESHOLD))) if arg.include?(ColumnFamilyDescriptorBuilder::MOB_THRESHOLD)
+      family.setNewVersionBehavior(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::NEW_VERSION_BEHAVIOR))) if arg.include?(ColumnFamilyDescriptorBuilder::NEW_VERSION_BEHAVIOR)
+      if arg.include?(ColumnFamilyDescriptorBuilder::BLOOMFILTER)
+        bloomtype = arg.delete(ColumnFamilyDescriptorBuilder::BLOOMFILTER).upcase.to_sym
         if org.apache.hadoop.hbase.regionserver.BloomType.constants.include?(bloomtype)
           family.setBloomFilterType(org.apache.hadoop.hbase.regionserver.BloomType.valueOf(bloomtype))
         else

hbase-shell/src/main/ruby/hbase/quotas.rb

@@ -61,6 +61,9 @@ module HBaseQuotasConstants
 module Hbase
   # rubocop:disable Metrics/ClassLength
   class QuotasAdmin
+    include HBaseConstants
+    include HBaseQuotasConstants
+
     def initialize(admin)
       @admin = admin
     end

hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb

@@ -23,8 +23,6 @@
 
 module Hbase
   class RSGroupAdmin
-    include HBaseConstants
-
     def initialize(connection)
       @connection = connection
       @admin = org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient.new(connection)
@@ -207,11 +205,11 @@ def alter_rsgroup_config(rsgroup_name, *args)
         unless arg.is_a?(Hash)
           raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash type")
         end
-        method = arg[METHOD]
+        method = arg[::HBaseConstants::METHOD]
         if method == 'unset'
-          configuration.remove(arg[NAME])
+          configuration.remove(arg[::HBaseConstants::NAME])
         elsif method == 'set'
-          arg.delete(METHOD)
+          arg.delete(::HBaseConstants::METHOD)
           for k, v in arg
             v = v.to_s unless v.nil?
             configuration.put(k, v)

hbase-shell/src/main/ruby/hbase/security.rb

@@ -22,8 +22,6 @@
 
 module Hbase
   class SecurityAdmin
-    include HBaseConstants
-
     def initialize(admin)
       @admin = admin
       @connection = @admin.getConnection

hbase-shell/src/main/ruby/hbase/taskmonitor.rb

@@ -27,8 +27,6 @@
 
 module Hbase
   class TaskMonitor
-    include HBaseConstants
-
     #---------------------------------------------------------------------------------------------
     # Represents information reported by a server on a single MonitoredTask
     class Task

hbase-shell/src/main/ruby/hbase_constants.rb

@@ -117,7 +117,7 @@ def self.promote_constants(constants)
   promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants)
 end
 
-# Include classes definition
+# Ensure that hbase class definitions are imported
 require 'hbase/hbase'
 require 'hbase/admin'
 require 'hbase/taskmonitor'
@@ -127,5 +127,3 @@ def self.promote_constants(constants)
 require 'hbase/security'
 require 'hbase/visibility_labels'
 require 'hbase/rsgroup_admin'
-
-include HBaseQuotasConstants