Speeding up Shoes web requests

[IanTrudel]

Shoes web requests are very slow. Installing Typhoeus and libcurl considerably speeds up web requests. Expert-funnies runs in a blink of an eye. It would be a fantastic to include in Shoes.

How to install Typhoeus and libcurl on Windows?

• Go in Shoes Maintenance > Manage Gems, search remote for Typhoeus and install
• Download curl-7.40.0-rtmp-ssh2-ssl-sspi-zlib-idn-static-bin-w32.7z
• Unzip curl in a directory such as C:\Program Files\Curl
• Go to Control Panel > System and Maintenance > System > Advance System Settings
• Go to Environment Variables and edit PATH (either the user or system, your choice)
• Add ;C:\Program Files\Curl to your PATH
• You may have to reboot to take into effects

Updated expert-funnies.rb:

require("nokogiri")
require("typhoeus")

class Comic
   attr_reader :rss, :title

   def initialize(body)
      @rss = Nokogiri::XML(body)
      @title = @rss.at("//channel/title").inner_text
   end

   def items
      @rss.search("//channel/item")
   end

   def latest_image
      @rss.search("//channel/item").first.inner_html.scan(/src="([^"]+\.\w+)"/).first
   end
end

Shoes.app width: 800, height: 600 do
   background "#555"

   @title = "Web Funnies"
   @feeds = [
      "http://xkcd.com/rss.xml",
      #"http://www.hoodcomputing.com/garfield.php" # gif image causing segfault
   ]

   stack margin: 10 do
      title strong(@title), align: "center", stroke: "#DFA", margin: 0
      para "(loaded from RSS feeds)", align: "center", stroke: "#DFA", margin: 0

      @feeds.each do |feed|
         response = Typhoeus.get(feed, followlocation: true, ssl_verifypeer: false)
         stack width: "100%", margin: 10, border: 1 do
            c = Comic.new response.body
            stack margin_right: gutter do
               background "#333", curve: 4
               caption c.title, stroke: "#CD9", margin: 4
            end
            image c.latest_image.first, margin: 8
         end
      end
   end
end

[ccoupe]

One of the first things I did many years ago was drop curl from Shoes 3. - It was incredibly hard to cross compile the library, they changed the api into the library dealing with notifications/callbacks which didn't look compatible with Shoes. Back then, open_uri was known to be slow, still is but it's one less dependency. We also know since then it has some other 'bugs'.

[IanTrudel]

The good news is that Shoes does not have to include libcurl in the same way as before because Typhoeus deals with it. Now your feedback is that including libcurl at C level is cumbersome. Using Typhoeus would just mean including Typhoeus gem and libcurl precompiled libraries. Wouldn't it be something closer to our reach?

[IanTrudel]

There is currently no callback for a progress bar in Typhoeus though libcurl does support it. Reference typhoeus/typhoeus#554.

[ccoupe]

I was about to ask about that callback. That was part of the problem with C level curl too. I've already noted elsewhere the days of of cross compiling Shoes for Window may be over soon. This is just another reason. It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

[IanTrudel]

The callback actually always been available in curl. It was called CURLOPT_PROGRESSFUNCTION but documentation encourage to use CURLOPT_XFERINFOFUNCTION now.

the days of of cross compiling Shoes for Window may be over soon

What does it mean concretely? Why is that?

It would be easier for us if Typhoeus included a precompiledt/gemified libcurl for mingw but not a requirement.

Agreed. We should rely more on Ruby. There are several things that could be written in Ruby rather than C. Typhoeus/libcurl is a prime example that speed is not affected by relying on Ruby. The idea would be to wrap Typhoeus in the infamous Download class and provide download method that uses this gem. It would make very little to no difference in term of speed to be implemented in C code with Shoes.

Why not a requirement? The whole thing would be roughly an additional 800KB including DLL files because some of the DLLs are already in Shoes.

[ccoupe]

What does it mean concretely? Why is that?

Ruby 2.2.6 won't cross compile. There's some other things don't cross compile anymore unless big hurdles are crossed. Every body wants to run tests or finalize something in their build by running what was built by running some exe on Windows before the build is complete. It happens because nobody thinks about cross compiling or folks like nokogiri create their own build framework which is at odds with how Shoes does cross platform builds. We know we need to build libgif and other dependencies. Might as well do that on Windows and skip the cross compile problems.

Actually, the curl source code is still in the shoes/http/ dir. Out of date of course. download follows a tortuous path in C as it builds the ruby structs needed to call up into Ruby lib/shoes/dowload.rb That mess attempts to emulate the callbacks/event that curl exposed to the download api. We need to fix it for #304 independently of this issue.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

[IanTrudel]

Ruby 2.2.6 won't cross compile.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

RE: CURL

This is really up to you. Either way, using open_uri and/or Net::HTTP is just too slow.

We could create a new Shoes method that doesn't have the progress callbacks and uses what ever Typhoeus allows. Or perhaps use ffi to hook into the callbacks. I haven't studied their code.

Attempted to do exactly that! There is something missing to actually callback but no idea what. First day on the job as we would say. :)

https://github.com/BackOrder/ethon
https://github.com/BackOrder/typhoeus

[ccoupe]

Aha, they use ffi ! This is good news.

How about setting up a build box on Windows (or a Windows VM)? You could control the build from your favourite OS, sending commands, automatically download the resulting logs and packages.

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

[IanTrudel]

A lot for me to learn. I know I can start a VM with some Virtual box script but once its started how do I login. How do I run a rake script....

You should perhaps consider something simple to start with. Running a scheduled rakefile would be something to make you comfortable. The said rakefile could download source from github, libraries, etc., check if it needs to be packaged again, compile when there are changes and then send resulting logs and packaged apps to your ftp. You'd only have to turn on the VM and the scheduler would take care of the rest.

It would otherwise be possible to run a small Ruby client/server if you want live output. The server would also be on a scheduler to make sure it's always running (or restarting when needed). It would look like this:

$ walk download
Connecting to Windows server...
Downloading source code from github. OK.
Downloading libraries. OK
$ walk build
Connecting to Windows server...
Building Shoes for Windows...
[... some compiler output...]
$ walk package
Connecting to Windows server...
Packaging Shoes for Windows...
$ walk upload ftp://walkabout/shoes/windows/
Connecting to Windows server...
Uploading Shoes for Windows on ftp...

[IanTrudel]

My attempt to add progress feature to Typhoeus worked with some help from the maintainer. It means we will be able to completely replace the current slow download feature and solve the https problems. It can be integrated in either 3.3.3 or 3.3.4.

[ccoupe]

I have cross compiled curl 7.50-dev and curl.exe works on win7 - except for https (no certificates). This is good. The build process isn't as tedious as it used to be. The certficate problem is probably a curl config problem and I suspect there may not be good certs in openssl . RubyInstaller is (now) pretty explicit about not including them in 2.x.x - a minor shoes build addition to the rakefiles once I figure out where they are, and where they should be. Odd's are very high they aren't where my curl.exe expects them. There are many options to explore. But, we have a test program - curl.exe.

[Update]
This configuration uses the windows ssl and the curl.exe does work for https: (cross compiled). If ruby could that we wouldn't need openssl. But it doesn't or I don't know to tell it too.

[ccoupe]

https (using openssl) seems to work for me in Shoes 3.3.4 (change the url in samples/simple/downloader.rb and image.rb). It may be slow but it works. So, openssl and certs are probably OK. Thats for the net/http ruby stuff.

I'm not keen on rejuvenating the old Shoes/C code to call curl at the C level for download and image. but it could be done. Since Shoes is currently doing a c->Ruby (lib/shoes/download.rb) that's what needs to be replaced with Typhoeus which means I need to build that gem on the 5 platforms so it and libcurl.so/dll/dylib can be.called from Shoes. A lot of cross platform dragons since we can't depend on linux or osx to have libcurl.so/dyllib

[IanTrudel]

It is noteworthy to mention that while my tests with Typheous were excellent both on features and speed, the fact remains that GIL (and possibly GTK threads) is preventing it to be an effective solution. It basically turned out to be Shoes UI is not updated while downloading. It's not a big deal for small files but it's a huge problem for larger files.

This may also apply to older curl code at C level but you would know better.

[ccoupe]

Oops! I forgot I told download.rb to not verify ssl. So, openssl certificates is likely to still be wrong on Shoes/Windows.

I installed the typhoeus gem with draged in ethon in Ruby 2.3.3/Windows, It didn't build anything native so it must be using FFI. I built Shoes (win7), including the libcurl-4.dll I built yesterday and cshoes.exe -c Manage gems ->show local -> load ethon provided this useful error message:

If those were all the libcurls it searched for, it wouldn't find libcurl-4.dll. It appears it is the ethon gem that is doing the ffi since it's the gem requiring the ffi binary gem (which Shoes includes). That will need fixing. I haven't looked at the ethon code yet but it could be a simple fix. Or not.

Shoes progress (the widget) or the Shoes.Setup gem handling or download does not have a smooth gui update. Part of that is Ruby, and part of that is Shoes. (and part of it is the OS). I suspect it could be improved which some heuristics, Typheous won't be worse in that dimension - assuming the callback/yield works as documented.

[ccoupe]

ethon/curls/settings.rb: ffi_lib ['libcurl', 'libcurl.so.4'] - seems like place to explore.

[IanTrudel]

Please, read the first post. It contains all the instructions to make it work without having to change code.

[ccoupe]

@backorder - always good to be reminded. You installed curl (the app) and changed the PATH in order to get a working test harness. However, Shoes users shouldn't have to do that for basic functionality We only need libcurl*dll inside the shoes app and the gems in the ruby. Code changes will be required to ethon/typheous - it could be an override/monkey patch we can do in Shoes or there is some upper level setting to add to that array passed into ffi_lib.

[IanTrudel]

This is just something to get you started. I suspect it should work if the proper library is in Shoes directory.

[ccoupe]

Setting this gets ethon to work. (xwin7)

ccoupe@bronco:~/Projects/gems/rb23/built/win7/ethon-0.10.1/lib/ethon/curls$ more settings.rb
module Ethon
  module Curl
    callback :callback, [:pointer, :size_t, :size_t, :pointer], :size_t
    callback :debug_callback, [:pointer, :debug_info_type, :pointer, :size_t, :p
ointer], :int
    ffi_lib_flags :now, :global
    ffi_lib ['libcurl', 'libcurl-4', 'libcurl.so.4']
  end
end

No sure how that can be monkey patched - i believe it's being call by the 'require'
On osx we have

ccoupe@mini:/usr/lib$ ls -l *curl*
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.3.dylib -> libcurl.4.dylib
-rwxr-xr-x  1 root  wheel  781616 Jul 23  2015 libcurl.4.dylib
lrwxr-xr-x  1 root  wheel      15 Oct 20  2014 libcurl.dylib -> libcurl.4.dylib

which would work with the original code - perhaps the xwin7 curl build can be told to name it libcurl.dll
instead of libcurl-4.dll. That would be best for all.

[ccoupe]

Time for some benchmarks. On my cable modem, my website, 423KB file, (ymmv)

linux cmd line curl            - 478KB/s
shoes download              - 423KB/s
shoes with progress bar  -   12KB/s

Typheous won't be faster than bare curl and and has a noticeable FFI pause which would skew this test even more - it's a small file. Clearly the problem is the progress bar and not so much the native library. What about a larger file? 10.9MB

linux cmd line curl            - 18 seconds
shoes download              - 14 seconds 

Shoes Wins ! Beware of benchmarks! Yes, I did check twice. People complain about net/http or read complaints about net/http. This was Ruby 2.3.4. Perhaps they fixed net/http? Has anyone checked?

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

[IanTrudel]

Why is download with progress bar so damn slow? I don't know, I can only guess. Switching the back end library may not be a useful path to pursue.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread. Unfortunately, GIL makes it so that only one thing at the time is executed.

Testing on a larger file resulted in freezing Shoes UI.

[ccoupe]

One mystery solved. The :pause arg is number of seconds to sleep between GUI updates. Removing that gets me 423KBs on the small file and 12 seconds on the big file. However, the gui flashes noticeably
A :pause => 0.02 works well - no flashing smooth animation and updates and 15 seconds. While ruby goes to sleep Gtk can keep up with its queue. Sadly this is platform specific tuning but tt can be done on any copy of Shoes you happen to have.

[update]
curl.exe on Windows (win7 vm) takes 8 to 13 seconds for the big file. (so the VM is negligible)
Shoes takes 243 seconds That is slow! . 110 seconds w/o a :pause => (no gui flashing either) which suggests something to me: The gtk loop/yield to ruby needs examination.

[ccoupe]

Here's the script I'm using - in case I'm calculating transfer rates incorrectly or you want to compare.

Shoes.app do
  background "#eee"
  @list = stack do
    para "Enter a URL to download:", :margin => [10, 8, 10, 0]
    flow :margin => 10 do
      @url = edit_line :width => -120
      @url.text ='http://walkabout.mvmanila.com/public/share/Ytm-2.exe'
      st_time = 0
      end_time = 0
      totalsz = 0
      button "Download", :width => 120 do
        st_time = Time.now
        @list.append do
          stack do
            background "#eee".."#ccd"
            stack :margin => 10 do
              dld = nil
              para @url.text, " [", link("cancel") { dld.abort }, "]", :margin => 0
              d = inscription "Beginning transfer.", :margin => 0
              p = progress :width => 1.0, :height => 14
              dld = download @url.text, :save => File.basename(@url.text),
                :pause => 0.02,
                :progress => proc { |dl| 
                  d.text = "Transferred #{dl.transferred} of #{dl.length} bytes (#{dl.percent}%)"
                  p.fraction = dl.percent },
                :finish => proc { |dl|
                     end_time = Time.now
                     secs = (end_time.to_i - st_time.to_i)
                     kb = dl.length < 1024 ? 1: dl.length / 1024
                     d.text = "Download completed KB/s: #{kb/secs} #{secs}"
                  }
            end
          end
        end
      end
    end
  end
end

[ccoupe]

I've got the Shoes Windows download to 90 secs (not as pathetic) by dropping the (Windows) gtk_idle timeout from 100ms to 5ms - see shoes/native/gtk.c(shoes_native_loop(). Much of Shoes/Windows seems snappier but I know enough to know I could be biased. The file open dialog (cobbler->Install Gempack->load) where it creates a file open dialog really does seem faster. Setting the idle to 1ms does not hurt anything but not real improvements.

This is a gtk timer that fires every given X ms and tells ruby to run it's threads - Linux gtk polls fd's which shoes/Windows doesn't do - unreliable if memory serves but that was back in Ruby 1.9.3 days.

[ccoupe]

Testing on a larger file resulted in freezing Shoes UI.

I'm attempting to modify /lib/shoes/download.rb with typhoeus and I can see how that happens in the old code. The old code keep everything download in memory and on saved it to a file at the end of the transfer so anything larger than your physical memory is going to start paging into virtual memory and only gets worse.

[IanTrudel]

You might want to test with something like an ISO file (Linux, FreeBSD, etc). Something at around 1 to 4 Gb would be a fairly good stress test.

https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-dvd1.iso

[ccoupe]

I've got a version of lib/shoes/download.rb using Typhoeus and it's performance is less than stellar on Windows. 123KB/s and 91 secs to to download the 10.9Mb file. 4 or 5 times slower than curl. It's better that we started with so that's worth something. No need for ethon mods or typhoeus mods. I do like the code simplicity of this version of download.rb (and it fixes some bugs - the large .iso) so I tried it on Linux minlin - 700KB/s and 16 secs - Acceptable on Linux.

My understanding is that it creates a new thread and then Shoes juggles between Ruby green threads (for Shoes UI and code execution), GTK and the new download/progress thread.

Ruby hasn't used green threads since 1.9.x. Now days It Uses pthreads.- Native threads on Linux and OSX. Gtk uses pthreads (if built to do so and I try to do so). Ruby does have a thread pool (The GIL) that it manages which should not effect other pthreads (like GTK or Cocoa). Windows uses a pthreads emulator which is a dll (winpthreads.dll) included with shoes. Clearly there are still performance issues with Windows and Ruby - not my wheel house to fix.

As we might remember, Windows has always had an event loop performance problem - back to the days of Shoes Raisins/Policeman when one thread sucked up 100%. The change to Gtk for Windows helped with that problem but I suspect it's not optimal when it deals with socket IO - it uses a different method of scheduling for Windows vs Linux.

[ccoupe]

The can of worms is fully open now! The old ruby, download.rb replacement was not compliant with the 3.0 download api as documented in the manual. There is a segfault in rbload.c for one on of the manual samples (m4) - that will be easier to fix and test than the speed problem with Windows I May have to revisit the select/poll of fd. Even with the new code/rake refactorings, that's a big pain to test since it affects everything.

[ccoupe]

There's a Shoes 3.3.4 beta