HTTPI is truncating response headers

[rogerleite]

Some evidence:

$ curl -I http://br.yahoo.com
[...]
Set-Cookie: IU=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=.yahoo.com
Set-Cookie: PH=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=.yahoo.com
Set-Cookie: MSC=t=1354021547X; expires=Wed, 27-Nov-2013 13:05:47 GMT; path=/; domain=.yahoo.com
Set-Cookie: fpc=d=pT0Q_yjRlaNbj84q3kJ0Bu4REjib1FNsC106B3kSpU.P9eBX64Ml3SsgKCkjpGmQBrZYZM15huzayuXz0L27ZIv8OnCpswr0kHvK8BWZBXZ4JnfS9wHnXexE9eODzw6FlLJ34eAqNdguDn5uMxUMh2tqped8imj.Y5ghMCQ7HdBqlXwDNN1r1oOt0KXHWEkptCGBre0-&v=2; expires=Wed, 27-Nov-2013 13:05:47 GMT; path=/; domain=br.yahoo.com
Set-Cookie: fpps=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=br.yahoo.com
Set-Cookie: fpt=d=fAgQuh3XeVyQ0SZnx0E2dD4HQyzXrkNQR_Jd_4uKZKR0ewy2D_vne_SmtTyH1VlczCWGQ4gL4eAByIwWDuNYn9u7QBrSncNoDVahgD9_xUzjpSYP4OVswNAvAn4uF46W23Fq34E0bjrs1WCmmvtQ1Dpv8hFMBlRC_kf33_c0bDmLGhXDkPIs6g2OkPIhrv8OjBctFDfx.0yBvQoaO0Ms69FqoDwODKUK.j9JNHF1r_mila0e6IOjPVMytQZVpg1QDXoqqhP4fXAwdrc7ENCjibSB0PM0lkI_qIfhDHJCeyf9akU.DP1KBv2rG4lmbT0ENVx_p67pNuezqCDLoyBR0ugXbD9Wh1hiQgmp1zvYk9iHyi2ZFgeF._z_C0iZ4CXx7UdaxuOHymdES8dJlJoVrCgb7ijKjWnM7EeQgiqFxE9gNqQHsr2tRY11ThSDssQs.RKi0LxwutsFT3b5oBVFJSqOh6kTzUWJIe8ZSfGTMQ0RKRKwxks1rDqju1UfRBBoRVq5DAGSiJOzo8_MykJ8qsL19b9iJm65O5iF_gEG2YrpO3iU8XSYcwPb5tWWGymS1Q--&v=1; path=/; domain=br.yahoo.com
Set-Cookie: fpc_s=d=2zRQRMjRlaPIhaPVoWgXBdq00OJxzfvr16KeQbtQGdVmUbvECiDJkiH5LSUHZtoiSmhazOoHh_1_nTZbCDYQ5cg9uh.WRVa4fo73H6pqeGVe3BZwKSNngqkYOX4xVIs2m3aiDPOaoGqbfAxVx8hOYAFHUhcJ_oJ6T_dxqRWv.rgYkSGZJhceYFJLOl.Locpkhpg1W4XJsfrjC7ywDuFoTB3Ern4XmYP2ncrB5KWohx4IpXLKqIiMsdtLrvjt6w7UQhSpDrJXcA_gwvT9B7PKSxy2ETMK1OsY.4y.RvZpnzGc.us7.d59NSFdQMkQSbrpwIM5agg8DiofQiBxIakBp_6SzmsWAqejLButP.GMkJAms2_UixKAYko9TpLzfWeNB0GzGTeNetu4zE3Er2R6KsOugwgV7SkVsfgrdU75iiiwveMn&v=2; path=/; domain=br.yahoo.com
[...]

Using a simple script (httpi 2.0.0.rc1), i realized that only curb adapter return all "Set-Cookie" headers.

HTTPI.log = true
[:net_http, :httpclient, :curb].each do |adapter|
  request = HTTPI::Request.new("http://br.yahoo.com")
  request.headers = {"User-Agent" => "curl/7.21.4"}
  response = HTTPI.get(request, adapter)
  puts "Set-Cookie: #{response.headers["Set-Cookie"].inspect}"
end

Results:

D, [2012-11-27T11:03:16.562429 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (net_http)
Set-Cookie: "IU=deleted; expires=Mon, 28-Nov-2011 13:03:12 GMT; path=/; domain=.yahoo.com"

D, [2012-11-27T11:03:21.086947 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (httpclient)
Set-Cookie: "fpc_s=d=e.AfOJJslaP77_ZLGuCCmy9bbCmZApcoySYRckZpfsFCEFQgFVg9Z6CMxocrOtzuVAIrQuUtt28S1mqpXveqZ5ca83Kc.T7LSUbNId6CfYoxp70q7KeLbcLVgxezcpXACefDxG2AaupWVzzoLh7CZLoQk33Ei5fezikXbJMIGEpzOdF.ys0WqlJWRsolZP4LZ0oRADoIOydKwaF2VMXECDpX_A0UWa9js1CkiJ.DlGvM4sZjM18KRNzL_9uRI.MULS0ytpLfQfK0xGog2sirqj05a74Q9SCtdcA13s5Tj3PjwuAZnxxSzJnTyDwrdSscKvsiw3.UNg5oqLdQ5fgSEX7oBRaUvWWkVBVH_cvPr8RB9e.T2jJTKMUZ.tDd2E3aZ4mQ2htpHbkBEzgfMzbbKjbYIJF2avg8GvOQsRGf3SSkheeH&v=2; path=/; domain=br.yahoo.com"

D, [2012-11-27T11:03:22.969444 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (curb)
Set-Cookie: ["IU=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=.yahoo.com", "PH=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=.yahoo.com", "MSC=t=1354021400X; expires=Wed, 27-Nov-2013 13:03:20 GMT; path=/; domain=.yahoo.com", "fpc=d=twmBb_tilaM2g8UhO2xUJ267pR5WJEyQjQBCKZ6P99Py8EcPbZnnJYuWtk6.dDsQcOtpR6NXZ8_n37uA2hmXRRyS60QoSnbBh4llNfxyPNleDwnFykzKyE0JRevgDtN8Wc3N_Rx9I4QZb9r9aVwkfaKEYJdlXxUpe37gqYuoHmiuWuzze1AkdL96T3icYn_LQJACNtc-&v=2; expires=Wed, 27-Nov-2013 13:03:20 GMT; path=/; domain=br.yahoo.com", "fpps=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=br.yahoo.com", "fpt=d=cgP9GTbXeVzKcBBu441lu6snPn5G0OxnEsu2HHFRGns2MWR.hszVlrrBI4SsPg.oknioT7TkHxn.hYhTwUwn..8CnrM7pu4iYnxuV6MgpPniQAQbb2glPUjTcr1z.CbH8RQA42th7C9L0gVOQDltnmxPv_2OFp76xDs.MpAoaqAVPbXClozyDh6tRw4CCllehQw2f7TQsFxHyi9I50sTyNOAmUcOpejEP1jJjAZfY.F5.h9poAtTiov.Yz.8UjELZSVNZKgn6vnDw9VmQUUTEI45iqiLGCaZSlgzxv1pbOh7KqihVY5itiWBMIVZn51X0huBWDv07bZjXxgXAVHel5i9QBzGBfUw0AmDPn2e56dT44_lvrK5U1WpNSkjcOU2nVZ_3wkf0ClKEg8AymM3k5ZDtVd4SIT5VSv8c332gLFvY4VKEMkH1et.A6SWH79TtkXVoPViOvJT978GWiEKJFpU1H7W3kzAtfeUKVnZnzMQ7udTVXON9_Mxhyu.B0pZyysv4mEc.rXLjOoF7CU1J7rP3T6MNZBYQwAxgH2qkLPoNecOmlTmfNsgVXjRXAgQLw--&v=1; path=/; domain=br.yahoo.com", "fpc_s=d=dcozG8hilaMbWa30O6JWUesgDD1W_k_SBg3VMdMWl3S0sFfGT5hPgeGEpvStnJj1BKsXkzQhhdn0gx7L2XN79Bu5SMBVb4.xrtHR.x6YMgRib4B7MPsR3l4rL8dS_EAeTJmGOeadUph6WQeZokCsb3t66iOVnZ3fxfDt_7KkS9EfiJMblqYywobLceiOhIjwgKfFVmm0XsyahY_PDWmD6MyjgcCJQPmM7QVhP_mMWB5rSNjSHI3KzP24COyBbGgrdNQE4MzrFu8xL.Q7TDiCMZcCZec352gsvVXTK_y59W7b9F73Fhq4d_sfFLHh4nSDbixXEYN__Tv2O8Hl7.HA9QR.jbP2iO1HIELslzXg1cAOxPkXpelfEhLsWQJsTRp0rJwFMy5JrkvTwLjWgdAR6soo7w_1FFgLu.vEmbGA.eKjy3cO&v=2; path=/; domain=br.yahoo.com"]

I'm planning to work on net_http and httpclient adapters to return headers like curb do. Is that ok?

[rogerleite]

i can't simulate this error because webmock's magic. See this commit for example: bblimke/webmock@5534968

@rubiii Any idea?
I started to work on this branch: https://github.com/rogerleite/httpi/tree/fix_response_headers

[rubiii]

no idea right now, but i'll take a look at it later.

[rubiii]

thanks @rogerleite, but i need specs for this. i'll look into getting rid of webmock.
i was planning to replace it with a simple server anyway.

[rogerleite]

@rubiii i kind of extracted MinionServer to help with it. I'm using on integration tests of http_monkey. If you want to help, i can add you as collaborator. What do you think?

[rogerleite]

I need this for http_monkey, i can try to simulate this bug on integration tests. Ok with that?

[rubiii]

have to take a look at this "after work" ;) i kind of started something similar here.
don't know if we're aiming for the same tool here.

[rogerleite]

Your IntegrationServer is really cool. The main goal is to provide a integration server that supports "rack app" and i can control the start and stop. You server is cool because also supports SSL. Many projects needs this, and always use some kind of "home made" solution.

[rubiii]

yes, the idea was to have a basic server that can run rack apps (just like your minion server) and add n applications for every kind of http test case one can imagine (ssl, redirects, errors, timeouts, cookies, etc).

[rogerleite]

Great! The invite as collaborator is still open! 😄

[rubiii]

thanks :) i would really like to hear what your plans are for minion server and whether we can combine efforts.
but this issue may not be the right place for that ;)

[rubiii]

@rogerleite i removed webmock from master. hope this solves your problem.

[rogerleite]

Thanks @rubiii I'll write some tests for this issue!