HBASE-11686 Shell code should create a binding / irb workspace instea…

…d of polluting the root namespace (#2141)

* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace

- Refactor Shell.export_commands to define commands using ruby lambdas.
  Additionally, this change stores a reference to shell_inst in scope so that
  we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
  binding and non-global receiver. This workspace is loaded with all HBase and
  IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
  within the created IRB workspace. This is based on work by Hsieh that was
  previously in bin/hirb.rb. This method is more generic and more testable.
  This single pattern can be used for both executing files and reading from
  stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
  commands are tightly linked with an instance of the shell, so it is easiest
  to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
  this change, tests were loading these constants into the top level, which
  could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
  best to reference each ruby constant's full name, but where that would cause
  a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
  HBase symbols from the top-level receiver (ie. main Object), it is possible
  (albeit unlikely) that this will break operator scripts. This flag will
  export all the commands at the top-level like the shell previously did.

* HBASE-11686 Light refactoring with added unit tests

- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
  multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
  semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
  constants, and all hbase commands to a target receiver. As a result, the
  HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch

* Typo s/is/if/

bin/hirb.rb

@@ -58,6 +58,8 @@
  -h | --help             This help.
  -n | --noninteractive   Do not run within an IRB session and exit with non-zero
                          status on first error.
+ --top-level-defs        Compatibility flag to export HBase shell commands onto
+                         Ruby's main object
  -Dkey=value             Pass hbase-*.xml Configuration overrides. For example, to
                          use an alternate zookeeper ensemble, pass:
                            -Dhbase.zookeeper.quorum=zookeeper.example.org
@@ -81,6 +83,7 @@ def add_to_configuration(c, arg)
 log_level = org.apache.log4j.Level::ERROR
 @shell_debug = false
 interactive = true
+top_level_definitions = false
 _configuration = nil
 D_ARG = '-D'.freeze
 while (arg = ARGV.shift)
@@ -108,6 +111,8 @@ def add_to_configuration(c, arg)
     warn '[INFO] the -r | --return-values option is ignored. we always behave '\
          'as though it was given.'
     found.push(arg)
+  elsif arg == '--top-level-defs'
+    top_level_definitions = true
   else
     # Presume it a script. Save it off for running later below
     # after we've set up some environment.
@@ -143,21 +148,10 @@ def add_to_configuration(c, arg)
 @shell = Shell::Shell.new(@hbase, interactive)
 @shell.debug = @shell_debug
 
-# Add commands to this namespace
-# TODO avoid polluting main namespace by using a binding
-@shell.export_commands(self)
-
-# Add help command
-def help(command = nil)
-  @shell.help(command)
-end
-
-# Backwards compatibility method
-def tools
-  @shell.help_group('tools')
-end
-
-# Debugging method
+##
+# Toggle shell debugging
+#
+# @return [Boolean] true if debug is turned on after updating the flag
 def debug
   if @shell_debug
     @shell_debug = false
@@ -173,26 +167,34 @@ def debug
   debug?
 end
 
+##
+# Print whether debug is on or off
 def debug?
   puts "Debug mode is #{@shell_debug ? 'ON' : 'OFF'}\n\n"
   nil
 end
 
-# Include hbase constants
-include HBaseConstants
+# For backwards compatibility, this will export all the HBase shell commands, constants, and
+# instance variables (@hbase and @shell) onto Ruby's top-level receiver object known as "main".
+@shell.export_all(self) if top_level_definitions
 
 # If script2run, try running it.  If we're in interactive mode, will go on to run the shell unless
 # script calls 'exit' or 'exit 0' or 'exit errcode'.
-load(script2run) if script2run
+@shell.eval_io(File.new(script2run)) if script2run
+
+# If we are not running interactively, evaluate standard input
+@shell.eval_io(STDIN) unless interactive
 
 if interactive
   # Output a banner message that tells users where to go for help
   @shell.print_banner
 
   require 'irb'
+  require 'irb/ext/change-ws'
   require 'irb/hirb'
 
   module IRB
+    # Override of the default IRB.start
     def self.start(ap_path = nil)
       $0 = File.basename(ap_path, '.rb') if ap_path
 
@@ -207,7 +209,12 @@ def self.start(ap_path = nil)
                HIRB.new
              end
 
+      shl = TOPLEVEL_BINDING.receiver.instance_variable_get :'@shell'
+      hirb.context.change_workspace shl.get_workspace
+
       @CONF[:IRB_RC].call(hirb.context) if @CONF[:IRB_RC]
+      # Storing our current HBase IRB Context as the main context is imperative for several reasons,
+      # including auto-completion.
       @CONF[:MAIN_CONTEXT] = hirb.context
 
       catch(:IRB_EXIT) do
@@ -217,48 +224,4 @@ def self.start(ap_path = nil)
   end
 
   IRB.start
-else
-  begin
-    # Noninteractive mode: if there is input on stdin, do a simple REPL.
-    # XXX Note that this purposefully uses STDIN and not Kernel.gets
-    #     in order to maintain compatibility with previous behavior where
-    #     a user could pass in script2run and then still pipe commands on
-    #     stdin.
-    require 'irb/ruby-lex'
-    require 'irb/workspace'
-    workspace = IRB::WorkSpace.new(binding)
-    scanner = RubyLex.new
-
-    # RubyLex claims to take an IO but really wants an InputMethod
-    module IOExtensions
-      def encoding
-        external_encoding
-      end
-    end
-    IO.include IOExtensions
-
-    scanner.set_input(STDIN)
-    scanner.each_top_level_statement do |statement, linenum|
-      puts(workspace.evaluate(nil, statement, 'stdin', linenum))
-    end
-  # XXX We're catching Exception on purpose, because we want to include
-  #     unwrapped java exceptions, syntax errors, eval failures, etc.
-  rescue Exception => exception
-    message = exception.to_s
-    # exception unwrapping in shell means we'll have to handle Java exceptions
-    # as a special case in order to format them properly.
-    if exception.is_a? java.lang.Exception
-      warn 'java exception'
-      message = exception.get_message
-    end
-    # Include the 'ERROR' string to try to make transition easier for scripts that
-    # may have already been relying on grepping output.
-    puts "ERROR #{exception.class}: #{message}"
-    if $fullBacktrace
-      # re-raising the will include a backtrace and exit.
-      raise exception
-    else
-      exit 1
-    end
-  end
 end

hbase-shell/src/main/ruby/hbase/admin.rb

@@ -27,6 +27,7 @@
 java_import org.apache.hadoop.hbase.TableName
 java_import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder
 java_import org.apache.hadoop.hbase.client.TableDescriptorBuilder
+java_import org.apache.hadoop.hbase.HConstants
 
 # Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin
 
@@ -1011,7 +1012,7 @@ def cfd(arg, tdb)
       cfdb.setTimeToLive(arg.delete(ColumnFamilyDescriptorBuilder::TTL)) if arg.include?(ColumnFamilyDescriptorBuilder::TTL)
       cfdb.setDataBlockEncoding(org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::DATA_BLOCK_ENCODING))) if arg.include?(ColumnFamilyDescriptorBuilder::DATA_BLOCK_ENCODING)
       cfdb.setBlocksize(JInteger.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::BLOCKSIZE))) if arg.include?(ColumnFamilyDescriptorBuilder::BLOCKSIZE)
-      cfdb.setMaxVersions(JInteger.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::VERSIONS))) if arg.include?(ColumnFamilyDescriptorBuilder::VERSIONS)
+      cfdb.setMaxVersions(JInteger.valueOf(arg.delete(HConstants::VERSIONS))) if arg.include?(HConstants::VERSIONS)
       cfdb.setMinVersions(JInteger.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::MIN_VERSIONS))) if arg.include?(ColumnFamilyDescriptorBuilder::MIN_VERSIONS)
       cfdb.setKeepDeletedCells(org.apache.hadoop.hbase.KeepDeletedCells.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::KEEP_DELETED_CELLS).to_s.upcase)) if arg.include?(ColumnFamilyDescriptorBuilder::KEEP_DELETED_CELLS)
       cfdb.setCompressTags(JBoolean.valueOf(arg.delete(ColumnFamilyDescriptorBuilder::COMPRESS_TAGS))) if arg.include?(ColumnFamilyDescriptorBuilder::COMPRESS_TAGS)

hbase-shell/src/main/ruby/hbase/quotas.rb

@@ -61,6 +61,9 @@ module HBaseQuotasConstants
 module Hbase
   # rubocop:disable Metrics/ClassLength
   class QuotasAdmin
+    include HBaseConstants
+    include HBaseQuotasConstants
+
     def initialize(admin)
       @admin = admin
     end

hbase-shell/src/main/ruby/hbase/rsgroup_admin.rb

@@ -23,8 +23,6 @@
 
 module Hbase
   class RSGroupAdmin
-    include HBaseConstants
-
     def initialize(connection)
       @connection = connection
       @admin = @connection.getAdmin
@@ -213,11 +211,11 @@ def alter_rsgroup_config(rsgroup_name, *args)
         unless arg.is_a?(Hash)
           raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash type")
         end
-        method = arg[METHOD]
+        method = arg[::HBaseConstants::METHOD]
         if method == 'unset'
-          configuration.remove(arg[NAME])
+          configuration.remove(arg[::HBaseConstants::NAME])
         elsif method == 'set'
-          arg.delete(METHOD)
+          arg.delete(::HBaseConstants::METHOD)
           for k, v in arg
             v = v.to_s unless v.nil?
             configuration.put(k, v)

hbase-shell/src/main/ruby/hbase/security.rb

@@ -22,8 +22,6 @@
 
 module Hbase
   class SecurityAdmin
-    include HBaseConstants
-
     def initialize(admin)
       @admin = admin
       @connection = @admin.getConnection

hbase-shell/src/main/ruby/hbase/taskmonitor.rb

@@ -27,8 +27,6 @@
 
 module Hbase
   class TaskMonitor
-    include HBaseConstants
-
     #---------------------------------------------------------------------------------------------
     # Represents information reported by a server on a single MonitoredTask
     class Task

hbase-shell/src/main/ruby/hbase_constants.rb

@@ -116,7 +116,7 @@ def self.promote_constants(constants)
   promote_constants(org.apache.hadoop.hbase.client.TableDescriptorBuilder.constants)
 end
 
-# Include classes definition
+# Ensure that hbase class definitions are imported
 require 'hbase/hbase'
 require 'hbase/admin'
 require 'hbase/taskmonitor'
@@ -126,5 +126,3 @@ def self.promote_constants(constants)
 require 'hbase/security'
 require 'hbase/visibility_labels'
 require 'hbase/rsgroup_admin'
-
-include HBaseQuotasConstants

hbase-shell/src/main/ruby/shell.rb

@@ -16,6 +16,29 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+require 'irb'
+require 'irb/workspace'
+
+#
+# Simple class to act as the main receiver for an IRB Workspace (and its respective ruby Binding)
+# in our HBase shell. This will hold all the commands we want in our shell.
+#
+class HBaseReceiver < Object
+  def get_binding
+    binding
+  end
+end
+
+##
+# HBaseIOExtensions is a module to be "mixed-in" (ie. included) to Ruby's IO class. It is required
+# if you want to use RubyLex with an IO object. RubyLex claims to take an IO but really wants an
+# InputMethod.
+module HBaseIOExtensions
+  def encoding
+    external_encoding
+  end
+end
+
 
 # Shell commands module
 module Shell
@@ -115,20 +138,48 @@ def hbase_rsgroup_admin
       @rsgroup_admin ||= hbase.rsgroup_admin
     end
 
-    def export_commands(where)
+    ##
+    # Create singleton methods on the target receiver object for all the loaded commands
+    #
+    # Therefore, export_commands will create "class methods" if passed a Module/Class and if passed
+    # an instance the methods will not exist on any other instances of the instantiated class.
+    def export_commands(target)
+      # We need to store a reference to this Shell instance in the scope of this method so that it
+      # can be accessed later in the scope of the target object.
+      shell_inst = self
+      # Define each method as a lambda. We need to use a lambda (rather than a Proc or block) for
+      # its properties: preservation of local variables and return
       ::Shell.commands.keys.each do |cmd|
-        # here where is the IRB namespace
-        # this method just adds the call to the specified command
-        # which just references back to 'this' shell object
-        # a decently extensible way to add commands
-        where.send :instance_eval, <<-EOF
-          def #{cmd}(*args)
-            ret = @shell.command('#{cmd}', *args)
-            puts
-            return ret
-          end
-        EOF
+        target.send :define_singleton_method, cmd.to_sym, lambda { |*args|
+          ret = shell_inst.command(cmd.to_s, *args)
+          puts
+          ret
+        }
       end
+      # Export help method
+      target.send :define_singleton_method, :help, lambda { |command = nil|
+        shell_inst.help(command)
+        nil
+      }
+      # Export tools method for backwards compatibility
+      target.send :define_singleton_method, :tools, lambda {
+        shell_inst.help_group('tools')
+        nil
+      }
+    end
+
+    # Export HBase commands, constants, and variables to target receiver
+    def export_all(target)
+      raise ArgumentError, 'target should not be a module' if target.is_a? Module
+
+      # add constants to class of target
+      target.class.include ::HBaseConstants
+      target.class.include ::HBaseQuotasConstants
+      # add instance variables @hbase and @shell for backwards compatibility
+      target.instance_variable_set :'@hbase', @hbase
+      target.instance_variable_set :'@shell', self
+      # add commands to target
+      export_commands(target)
     end
 
     def command_instance(command)
@@ -238,6 +289,61 @@ def help_footer
 For more on the HBase Shell, see http://hbase.apache.org/book.html
       HERE
     end
+
+    @irb_workspace = nil
+    ##
+    # Returns an IRB Workspace for this shell instance with all the IRB and HBase commands installed
+    def get_workspace
+      return @irb_workspace unless @irb_workspace.nil?
+
+      hbase_receiver = HBaseReceiver.new
+      # Install all the hbase commands, constants, and instance variables @shell and @hbase. This
+      # must be BEFORE the irb commands are installed so that our help command is not overwritten.
+      export_all(hbase_receiver)
+      # install all the IRB commands onto our receiver
+      IRB::ExtendCommandBundle.extend_object(hbase_receiver)
+      ::IRB::WorkSpace.new(hbase_receiver.get_binding)
+    end
+
+    ##
+    # Read from an instance of Ruby's IO class and evaluate each line within the shell's workspace
+    #
+    # Unlike Ruby's require or load, this method allows us to execute code with a custom binding. In
+    # this case, we are using the binding constructed with all the HBase shell constants and
+    # methods.
+    def eval_io(io)
+      require 'irb/ruby-lex'
+      # Mixing HBaseIOExtensions into IO allows us to pass IO objects to RubyLex.
+      IO.include HBaseIOExtensions
+
+      workspace = get_workspace
+      scanner = RubyLex.new
+      scanner.set_input(io)
+
+      begin
+        scanner.each_top_level_statement do |statement, linenum|
+          puts(workspace.evaluate(nil, statement, 'stdin', linenum))
+        end
+      rescue Exception => e
+        message = e.to_s
+        # exception unwrapping in shell means we'll have to handle Java exceptions
+        # as a special case in order to format them properly.
+        if e.is_a? java.lang.Exception
+          warn 'java exception'
+          message = e.get_message
+        end
+        # Include the 'ERROR' string to try to make transition easier for scripts that
+        # may have already been relying on grepping output.
+        puts "ERROR #{e.class}: #{message}"
+        if $fullBacktrace
+          # re-raising the will include a backtrace and exit.
+          raise e
+        else
+          exit 1
+        end
+      end
+      nil
+    end
   end
   # rubocop:enable Metrics/ClassLength
 end

hbase-shell/src/main/ruby/shell/commands/clone_snapshot.rb

@@ -38,7 +38,7 @@ def help
 
       def command(snapshot_name, table, args = {})
         raise(ArgumentError, 'Arguments should be a Hash') unless args.is_a?(Hash)
-        restore_acl = args.delete(RESTORE_ACL) || false
+        restore_acl = args.delete(::HBaseConstants::RESTORE_ACL) || false
         admin.clone_snapshot(snapshot_name, table, restore_acl)
       end