File download is incorrect (checksums do not match) #797

surfer190 · 2015-04-03T09:41:35Z

Be aware that this could be a local issue.

File downloaded with wpscan is: 245421 bytes
File download with wget is: 448537 bytes

File URL is: https://wpvulndb.com/data/plugin_vulns.json
  [i] Downloaded File Checksum: 0ef1fa7938d221fa4a43de559907d687fc30312e5d75e819bf042506891779eff80693715c5c3655df8f381aa48d8c3068c991fef30966bb05693f0a66c05162
  [i] Database Checksum is: 6299b3e2464b2d0ec5286e05992de48108d88137f086f5343373f8a771aee542eebe67e3d1756acb7886a32f15a320d23e4206342f289f6085da559f1c482761

Seems to be a problem with this code:

    res = Browser.get(file_url, request_params)
    additional_text = res.code == 0 ? res.return_message : "return code #{res.code}"
    fail "Error while downloading #{file_url} (#{additional_text})" unless res.code == 200
    File.open(file_path, 'wb') { |f| f.write(res.body) }

Changed code to:

require 'open-uri'
...
open(file_path, 'wb') do |file|
  file << open(file_url).read
end

It worked. Could this be a local problem with Browser or should the later code be implemented with error checking of course?

The text was updated successfully, but these errors were encountered:

erwanlr · 2015-04-03T10:34:46Z

Which version of ruby are you using ?

surfer190 · 2015-04-03T11:25:58Z

ruby 2.2.1p85 (2015-02-26 revision 49769) [x86_64-linux]

erwanlr · 2015-04-03T12:14:57Z

Are you behind a proxy or something ? I can't reproduce that on Mac OS X nor Debian

Script used:

require 'typhoeus'
require 'open-uri'

@remote_file = 'https://wpvulndb.com/data/plugin_vulns.json'
@local_file  = '/tmp/plugin_vulns.json'

class HttpError < StandardError
  attr_reader :response

  # @param [ Typhoeus::Response ] res
  def initialize(response)
    @response = response
  end

  def failure_details
    msg = response.effective_url
    msg += response.code == 0 ? " (#{response.return_message})" : " (status: #{response.code})"
  end

  def message
    "HTTP Error: #{failure_details}"
  end
end


def get_file(file)
  res = Typhoeus.get(file)
  fail HttpError.new(res) unless res.code == 200

  res.body
end

puts "Ruby #{RUBY_VERSION}"
puts @file
puts "Expected Checksum: #{get_file("#{@remote_file}.sha512")}"

remote_body = get_file(@remote_file)
remote_checksum = Digest::SHA512.hexdigest(remote_body)

puts "Remote Body Checksum: #{remote_checksum}"

File.open(@local_file, 'wb') { |f| f.write(remote_body) }

puts "Downloaded File Checksum: #{Digest::SHA512.file(@local_file).hexdigest}"

open(@local_file, 'wb') do |file|
  file << open(@remote_file).read
end

puts "Downloaded File Checksum (Open-URI): #{Digest::SHA512.file(@local_file).hexdigest}"

Output:

$ ruby checksums.rb 
Ruby 2.2.1

Expected Checksum: 6299b3e2464b2d0ec5286e05992de48108d88137f086f5343373f8a771aee542eebe67e3d1756acb7886a32f15a320d23e4206342f289f6085da559f1c482761
Remote Body Checksum: 6299b3e2464b2d0ec5286e05992de48108d88137f086f5343373f8a771aee542eebe67e3d1756acb7886a32f15a320d23e4206342f289f6085da559f1c482761
Downloaded File Checksum: 6299b3e2464b2d0ec5286e05992de48108d88137f086f5343373f8a771aee542eebe67e3d1756acb7886a32f15a320d23e4206342f289f6085da559f1c482761
Downloaded File Checksum (Open-URI): 6299b3e2464b2d0ec5286e05992de48108d88137f086f5343373f8a771aee542eebe67e3d1756acb7886a32f15a320d23e4206342f289f6085da559f1c482761

surfer190 · 2015-04-03T14:09:17Z

@erwanlr: modified the script to use wpscan browser.rb.
Results:

Ruby 2.2.1

Expected Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Remote Body Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum (Open-URI): c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
WPScan Download: e9cc2c565c3e2f706333af11550a899364c957c21315537513b0a854f41f2f032212c2e5d3935bb44637c967b95afba8526c685ac1196d4b79e18942cff70e4c

Script content:

require 'typhoeus'
require 'open-uri'
#require 'common/browser'
require File.dirname(__FILE__) + '/lib/wpscan/wpscan_helper'

#@remote_file = 'https://wpvulndb.com/data/plugin_vulns.json'
#@local_file  = '/tmp/plugin_vulns.json'
@remote_file = 'https://wpvulndb.com/data/plugins_full.txt'
@local_file  = '/tmp/plugin_full.txt'

class HttpError < StandardError
  attr_reader :response

  # @param [ Typhoeus::Response ] res
  def initialize(response)
    @response = response
  end

  def failure_details
    msg = response.effective_url
    msg += response.code == 0 ? " (#{response.return_message})" : " (status: #{response.code})"
  end

  def message
    "HTTP Error: #{failure_details}"
  end
end


def get_file(file)
  res = Typhoeus.get(file)
  fail HttpError.new(res) unless res.code == 200

  res.body
end

  # @return [ Hash ] The params for Typhoeus::Request
  def request_params
    {
      ssl_verifyhost: 2,
      ssl_verifypeer: true
    }
  end


puts "Ruby #{RUBY_VERSION}"
puts @file
puts "Expected Checksum: #{get_file("#{@remote_file}.sha512")}"

remote_body = get_file(@remote_file)
remote_checksum = Digest::SHA512.hexdigest(remote_body)

puts "Remote Body Checksum: #{remote_checksum}"

File.open(@local_file, 'wb') { |f| f.write(remote_body) }

puts "Downloaded File Checksum: #{Digest::SHA512.file(@local_file).hexdigest}"

open(@local_file, 'wb') do |file|
  file << open(@remote_file).read
end

puts "Downloaded File Checksum (Open-URI): #{Digest::SHA512.file(@local_file).hexdigest}"

  # @return [ String ] The checksum of the downloaded file
  def download()
    file_path = @local_file
    file_url  = @remote_file

    res = Browser.get(file_url, request_params)
    additional_text = res.code == 0 ? res.return_message : "return code #{res.code}"
    fail "Error while downloading #{file_url} (#{additional_text})" unless res.code == 200
    File.open(file_path, 'wb') { |f| f.write(res.body) }

    local_file_checksum(@local_file)
  end

  def local_file_checksum(filename)
    Digest::SHA512.file(filename).hexdigest
  end


puts "WPScan Download: #{download()}"

Uses wpscan_helper so easier if dropped into the root of wpscan dir.

firefart · 2015-04-03T14:50:11Z

@surfer190 my output is correct. What's your version of typhoeus (see below)?

macbook:wpscan firefart$ ruby test.rb
Ruby 2.2.1

Expected Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Remote Body Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum (Open-URI): c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
WPScan Download: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82

Typhoeus (run from wpscan dir):

macbook:wpscan firefart$ pwd
/Users/firefart/Coding/wpscan
macbook:wpscan firefart$ gem list typhoeus

*** LOCAL GEMS ***

typhoeus (0.7.1)

All versions:

macbook:wpscan firefart$ bundle show
Gems included by the bundle:
  * addressable (2.3.8)
  * bundler (1.8.5)
  * crack (0.4.2)
  * diff-lcs (1.2.5)
  * docile (1.1.5)
  * ethon (0.7.3)
  * ffi (1.9.8)
  * json (1.8.2)
  * mini_portile (0.6.2)
  * multi_json (1.11.0)
  * nokogiri (1.6.6.2)
  * rspec (3.2.0)
  * rspec-core (3.2.2)
  * rspec-expectations (3.2.0)
  * rspec-its (1.2.0)
  * rspec-mocks (3.2.1)
  * rspec-support (3.2.2)
  * ruby-progressbar (1.7.5)
  * safe_yaml (1.0.4)
  * simplecov (0.9.2)
  * simplecov-html (0.9.0)
  * terminal-table (1.4.5)
  * typhoeus (0.7.1)
  * webmock (1.21.0)

erwanlr · 2015-04-03T14:50:57Z

Still can't reproduce huhu

Ruby 2.2.1

Expected Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Remote Body Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
Downloaded File Checksum (Open-URI): c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82
WPScan Download: c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82

Could you run the script, and post the content of /tmp/plugin_full.txt on gist (https://gist.github.com/) please ?

surfer190 · 2015-04-03T15:27:27Z

@firefart Exactly the same:

Gems included by the bundle:
  * addressable (2.3.8)
  * bundler (1.8.5)
  * crack (0.4.2)
  * diff-lcs (1.2.5)
  * docile (1.1.5)
  * ethon (0.7.3)
  * ffi (1.9.8)
  * json (1.8.2)
  * mini_portile (0.6.2)
  * multi_json (1.11.0)
  * nokogiri (1.6.6.2)
  * rspec (3.2.0)
  * rspec-core (3.2.2)
  * rspec-expectations (3.2.0)
  * rspec-its (1.2.0)
  * rspec-mocks (3.2.1)
  * rspec-support (3.2.2)
  * ruby-progressbar (1.7.5)
  * safe_yaml (1.0.4)
  * simplecov (0.9.2)
  * simplecov-html (0.9.0)
  * terminal-table (1.4.5)
  * typhoeus (0.7.1)
  * webmock (1.21.0)

@erwanlr gist

erwanlr · 2015-04-03T15:30:44Z

Looks like the file is not complete (i.e has not been correctly downloaded), the mystery is why :o

erwanlr · 2015-04-03T15:32:50Z

Could you try with the following command: ./wpscan.rb --update --connect-timeout 20 --request-timeout 20

erwanlr · 2015-04-03T15:37:19Z

Also, what's your curl version ? curl --version

What's the result of curl https://wpvulndb.com/data/plugins_full.txt | sha512sum

surfer190 · 2015-04-03T15:42:36Z

@erwanlr The command with --connect-timeout and --request-timeout worked.

Curl:

curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

Result of curl command:

c726664cf7172061a96b38c129db1d0fffa609a7003ab47dba0146bfa74c8aac5dadad3d7f6268c3498d4cd23b3e20bca5315682688aeae5418bf5ec52c37e82  -

erwanlr · 2015-04-03T15:48:14Z

One last test: ./wpscan.rb --update --connect-timeout 10 --request-timeout 10

I think it should work, but let's see :D

surfer190 · 2015-04-03T15:50:49Z

Nope :/

erwanlr · 2015-04-03T16:11:21Z

I have increased the timeout values (connect to 5s and request to 60s), please update (git pull), then ./wpscan --update :)

erwanlr · 2015-04-03T16:15:57Z

BTW, was able to partially reproduce the issue when my network was a bit busy:

curl --connect-timeout 1 --max-time 1 https://wpvulndb.com/data/plugins_full.txt | sha512sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 61  956k   61  591k    0     0   591k      0  0:00:01 --:--:--  0:00:01  653k
curl: (28) Operation timed out after 1000 milliseconds with 605875 out of 979778 bytes received
67b5f7310b0d5470c488be92af401bb14ef9fe94d93d4461cf734a1d9346344d7ff4e560cb765b85febe3e8d1365a537674d9ba71d33565beed79bb09cd7b1e4  -

Would be interested to reproduce that behaviour using Typhoeus to check the timed_out? attribute and return_message

surfer190 · 2015-04-03T16:17:08Z

@erwanlr Awesome, sorted. Thanks.

erwanlr added a commit that referenced this issue Apr 3, 2015

Increases the timeout values - Ref #797

dc20ef0

surfer190 closed this as completed Apr 3, 2015

erwanlr added a commit that referenced this issue Apr 3, 2015

Also check for potential timed out requests when updating - Ref #797

4420716

erwanlr mentioned this issue Apr 4, 2015

wpscan --update #788

Closed

petr-hybler mentioned this issue Feb 23, 2016

Unable to get https://data.wpscan.org/plugins.json (Timeout was reached) #908

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File download is incorrect (checksums do not match) #797

File download is incorrect (checksums do not match) #797

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

firefart commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

File download is incorrect (checksums do not match) #797

File download is incorrect (checksums do not match) #797

Comments

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

firefart commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015

erwanlr commented Apr 3, 2015

erwanlr commented Apr 3, 2015

surfer190 commented Apr 3, 2015