Implement payload size caching, speeding up framework loads #4894

hdm · 2015-03-08T02:48:17Z

Quick and dirty benchmark:

$ for i in `seq 1 3`; do time ./msfconsole -q -x 'exit'; done

real  0m5.829s
user  0m5.185s
sys 0m0.530s

real  0m5.897s
user  0m5.358s
sys 0m0.434s

real  0m5.593s
user  0m5.073s
sys 0m0.424s

Generate the cache:

$ ./tools/update_payload_cached_sizes.rb 

[*] /data/work/msf-hdm/modules/payloads/singles/aix/ppc/shell_bind_tcp.rb has size 264, updating cache...
[*] /data/work/msf-hdm/modules/payloads/singles/aix/ppc/shell_find_port.rb has size 220, updating cache...
[*] /data/work/msf-hdm/modules/payloads/singles/aix/ppc/shell_interact.rb has size 56, updating cache...
[*] /data/work/msf-hdm/modules/payloads/singles/aix/ppc/shell_reverse_tcp.rb has size 204, updating cache...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_http.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_https.rb has a dynamic size, skipping...
[..]

Verify the cache:

$ ./tools/update_payload_cached_sizes.rb 

[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_http.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_https.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_tcp.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_http.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_https.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/stagers/android/reverse_tcp.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/unix/bind_netcat.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/unix/reverse_bash.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/unix/reverse_bash_telnet_ssl.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/unix/reverse_netcat.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/unix/reverse_python.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/windows/download_eval_vbs.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/cmd/windows/download_exec_vbs.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/firefox/exec.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/firefox/shell_bind_tcp.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/firefox/shell_reverse_tcp.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/bind_php.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/bind_php_ipv6.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/download_exec.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/exec.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/reverse_perl.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/reverse_php.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/php/shell_findsock.rb has a dynamic size, skipping...
[*] /data/work/msf-hdm/modules/payloads/singles/solaris/sparc/shell_find_port.rb has a dynamic size, skipping...

Bask in the glory:

$ for i in `seq 1 3`; do time ./msfconsole -q -x 'exit'; done

real  0m5.100s
user  0m4.511s
sys 0m0.483s

real  0m4.895s
user  0m4.371s
sys 0m0.438s

real  0m5.056s
user  0m4.549s
sys 0m0.427s

Spec coverage is still a TODO and will handle build-time verification of cached sizes.

This change would let us move to Metasm payload generators without a startup-time performance penalty.

bcook-r7 · 2015-03-09T17:44:07Z

I presume you're writing an automated test to compare the cached value to the real one for all the non-dynamic payloads and make sure its not stale? Maybe msftidy could be updated to notice this?

bcook-r7 · 2015-03-09T17:47:43Z

That said, it worked well - the is_dynamic_size? check seemed like it might false-negative, e.g. if a payload rounded a string length to multiples of 32 or something, but I didn't find any actual counter-examples.

bcook-r7 · 2015-03-09T17:56:28Z

The regex needs a little work - there is some whitespace drifting

--- a/modules/payloads/stagers/windows/reverse_http_proxy_pstore.rb
+++ b/modules/payloads/stagers/windows/reverse_http_proxy_pstore.rb
@@ -8,6 +8,16 @@ require 'msf/core/handler/reverse_http'


 module Metasploit3
+  CachedSize = 650
+
+
+
+
+
+
+
+
+

bcook-r7 · 2015-03-09T18:10:11Z

Does this seem like a reasonable update? When we remove the previous value, we need to remove the whole line.

diff --git a/tools/update_payload_cached_sizes.rb b/tools/update_payload_cached_sizes.rb
index f6db518..699e3d7 100755
--- a/tools/update_payload_cached_sizes.rb
+++ b/tools/update_payload_cached_sizes.rb
@@ -41,8 +41,8 @@ end
 def update_cache_size(mod)
   data = ''
   File.open(mod.file_path, 'rb'){|fd| data = fd.read(fd.stat.size)}
-  data = data.gsub(/^\s*CachedSize\s*=\s*\d+.*/, '')
-  data = data.gsub(/^(module Metasploit\d+)/) {|m| "#{m}\n  CachedSize = #{mod.new.size}\n" }
+  data = data.gsub(/^\s*CachedSize\s*=\s*\d+.*\n/, '')
+  data = data.gsub(/^(module Metasploit\d+)/) {|m| "#{m}\n  CachedSize = #{mod.new.size}" }
   File.open(mod.file_path, 'wb'){|fd| fd.write(data) }
 end

Result is a little prettier:

--- a/modules/payloads/stagers/windows/reverse_http_proxy_pstore.rb
+++ b/modules/payloads/stagers/windows/reverse_http_proxy_pstore.rb
@@ -8,6 +8,7 @@ require 'msf/core/handler/reverse_http'


 module Metasploit3
+  CachedSize = 650

hdm · 2015-03-09T18:14:23Z

Fixing this and some other open issues in an imminent update. Still missing specs too.

hdm · 2015-03-09T19:02:22Z

The previous commit should fail the build, as ./tools/update_payload_cached_sizes.rb has not been run across the modules.

hdm · 2015-03-09T21:17:42Z

Builds fail because datastore defaults like LHOST change based on the system they are running on. This has been a problem for a while, so fixing it will be a nice improvement all around.

hdm · 2015-03-09T23:44:50Z

The previous commits handle cases where the datastore defaults to something based on the host, but it doesn't solve the real problem, which is that the generate() call during startup needs a default datastore that represents the largest-size output.

I will file a separate ticket for that soon, in the meantime, this is good to land.

bcook-r7 · 2015-03-10T04:06:18Z

Sounds good.

Solving for the maximum size will be interesting. For instance, the max LHOST for the reverse_http stagers above would actually be [0000:0000:0000:0000:0000:0000:0000:0000], since they work for ipv6 too. Some metadata for max parameter lengths would be nice, since currently that info currently hard to introspect, e.g. reverse_tcp_dns.rb's exception when LHOST > 63.

Avoid the hit of regenerating all of the static-size payloads when loading the framework. This will facilitate conversion of payloads to use metasm later.

hdm · 2015-03-10T05:25:23Z

@bcook-r7 Thanks! Definitely agree, we need to do more work on byte-size prediction for payloads. We have only seen this result in bugs in corner cases (options bring a payload + encoder just over the line), but definitely worth solving.

coveralls · 2016-10-15T15:38:22Z

Changes Unknown when pulling 618fbf0 on hdm:feature/payload-cached-size into * on rapid7:master*.

HD Moore added 2 commits March 7, 2015 20:44

This implements payload size caching, speeding up framework loads

2e49791

Accidental carry over from stale master

853bf1b

hdm added library msfconsole labels Mar 8, 2015

hdm changed the title ~~This implements payload size caching, speeding up framework loads~~ Implement payload size caching, speeding up framework loads Mar 8, 2015

Restore a comment lost in the code churn

d46635f

hdm mentioned this pull request Mar 8, 2015

Comprehensively refactor Windows reverse_http stagers #4895

Open

19 tasks

bcook-r7 self-assigned this Mar 9, 2015

hdm mentioned this pull request Mar 9, 2015

Tell payloads and encoders how much space they have to work with #4876

Merged

HD Moore added 3 commits March 9, 2015 13:15

Fix whitespace in the regex, implements Msf::Payload.dynamic_size?

8c63524

Validate cached_size and dynamic_size? in the payload specs

f61c3f3

Remvoe the useless pinst variable from the previous test

33f96f5

HD Moore added 8 commits March 9, 2015 14:41

Rework specs

7dc0af4

Cosmetic tweaks to the specs

60145ad

Move the cache update logic into a utility class

99e2b05

The result of running ./tools/update_payload_cached_sizes.rb

02509d0

YARD docs for the Msf::Util::PayloadCachedSize class

0385914

Correct a typo (stringified symbol loses the :)

78456fb

Correct the :dynamic cache sizes

da81f6b

Final cleanup, passing specs

d0324e8

HD Moore added 3 commits March 9, 2015 16:53

Fallback to a localhost variant to make the length predictable

c676ac1

Update CachedSize for the fixed stager

6543c3c

Fallback to a localhost variant to make the length predictable

746f18d

Update CachedSize for the fixed stager

618fbf0

bcook-r7 merged commit 618fbf0 into rapid7:master Mar 10, 2015

bcook-r7 pushed a commit that referenced this pull request Mar 10, 2015

Land #4894: hmoore-r7 cache payload sizes on start

97f09b6

Avoid the hit of regenerating all of the static-size payloads when loading the framework. This will facilitate conversion of payloads to use metasm later.

hdm pushed a commit to hdm/metasploit-framework that referenced this pull request Mar 10, 2015

Use the real cached_size now that rapid7#4894 is landed

333e613

hdm mentioned this pull request Mar 12, 2015

Avoid instantiating the module during recalculate #4917

Merged

bcook-r7 mentioned this pull request Mar 16, 2015

update tools/missing-payload-tests to give correct advice #4938

Merged

4 tasks

wvu mentioned this pull request May 16, 2018

Refactor zsh payloads WangYihang/metasploit-framework#1

Merged

busterb mentioned this pull request Mar 25, 2020

Transport and pivot fixes for Meterpreter #13130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement payload size caching, speeding up framework loads #4894

Implement payload size caching, speeding up framework loads #4894

hdm commented Mar 8, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

bcook-r7 commented Mar 10, 2015

hdm commented Mar 10, 2015

coveralls commented Oct 15, 2016

Implement payload size caching, speeding up framework loads #4894

Implement payload size caching, speeding up framework loads #4894

Conversation

hdm commented Mar 8, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

bcook-r7 commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

hdm commented Mar 9, 2015

bcook-r7 commented Mar 10, 2015

hdm commented Mar 10, 2015

coveralls commented Oct 15, 2016