Make get/post more conveniently by using closure

[SuperHacker-liuan]

reqwest is a convenient library for rapid http development. We can add 2 api to make it more convenient by using closure. The feature is usable in contests like CTF.
For example:
Add pub fn get2() and pub fn post()

pub fn get2<F>(url: &str, setter: F) -> Result<Response>
    where F: Fn(&mut RequestBuilder) -> &mut RequestBuilder {
    setter(
            &mut Client::new()?
                .get(url)?
        ).send()
}

pub fn post<F>(url: &str, setter: F) -> Result<Response>
    where F: Fn(&mut RequestBuilder) -> &mut RequestBuilder {
    setter(
            &mut Client::new()?
                .post(url)?
        ).send()
}

So when I do get with cookie by simply call:

fn main() {
    let resp = get2("http://192.168.96.11:88/web-master/web100-4/index.php", |req| {
        let mut cookies = Cookie::new();
        cookies.append("ISecer", "s:0:\"\";");
        req.header(cookies)
    }).unwrap();
    println!("{}", resp.status());
}

Hope reqwest can be used as simply as Python requests.

Future more, we can add a reqwest::header::Cookies to support multi key-value cookie.
So that we can write CTF code with cookies as simply as below.

fn main() {
    let resp = get2("http://192.168.96.11:88/web-master/web100-4/index.php", |req| {
        req.header(Cookies::new(vec![("k0", "v0"), ("k1", "v1"), ("k2", "v2")]))
    }).unwrap();
    println!("{}", resp.status());
}

which is as simple as requests code in Python

import requests

url = 'http://192.168.96.11:88/web-master/web100-4/index.php'
cookies = {'k0': 'v0', "k1": "v1", "k2": "v2"}

r = requests.get(url, cookies = cookies)
print(r.text)

[seanmonstar]

Hey there! Thanks for the write up! Looks like there's possibly two issues here, so I'll try to separate them:

1. What does the closure do? What makes it better than the current approach? Using your example:

   let mut cookies = Cookie::new();
   cookies.append("ISecer", "s:0:\"\";");
   let resp = reqwest::get("http://192.168.96.11:88/web-master/web100-4/index.php")?
       .header(cookies)
       .send()?;

2. Cookies could get better support. There is cookie jar implementation #14, but that's more around receiving headers from servers and then using them in new requests. It seems there could be a method added to RequestBuilder to ease adding some cookies to a request.

[SuperHacker-liuan]

@seanmonstar

Not Compilable

By reading docs of reqwest::get, reqwest::get return a Result<Response>, not Result<RequestBuilder>, so the code below is not compilable. (version 0.7.2)

let mut cookies = Cookie::new();
cookies.append("ISecer", "s:0:\"\";");
let resp = reqwest::get("http://192.168.96.11:88/web-master/web100-4/index.php")?
    .header(cookies)
    .send()?;

Focus on business, Make API as simple as possible and with less Error handling

For cooperations whose main business is not software developing, there is a special type of developer who is focus on business logic develop. They really want API as simple as possible.
In my comprehention, reqwest is a high-level API for http request & response in rust. So it is a good news that reqwest can provide more simple API which hide more calling details for developers.
Many http developer know that HTTP request are sent in the format like below

GET xxx HTTP/1.1
Host: yyy
Cookie: a=b
Accept: zzz
// more headers

If reqwest provide get/post/other api in closure style, a client developer can call a request like writing protocal code:

get("http://yyy/xxx", |r|{                     // GET xxx HTTP/1.1    Host: yyy
    r.header(Cookies(vec![("a", "b")])    // Cookie: a=b
        .header(Accept(zzz))                  // Accept: zzzz
        //.header(more headers)
})?

The same function in Client style will be a bit complex(developer have to new() a Client and send() the RequestBuilder, and remember to add try! macros after each call, which are not what business developers concerned)

Client::new()?  // Not concerned
    .get("http://yyy/xxx")?
    .header(Cookies)
    .header(Accept)
    .header(more headers)
    .send()? // Not concerned

Current Inconvenience

For a batch of requests with user default headers, current reqwest::get cannot offer enough function, it only support get method without user defined headers.
But use Client::new()?......send()? will make the code a little ugly, Developer may fell the business fragmented when reading/writing the code.
For example, in electric grid industry, we often need to collect grid status from many equipments, we need to write diffenrent information collecting code for different equipment and then analysis. like below.

lazy_static! {
    // Follow Header shared by whole project.
    static ref DEFAULT: Mutex<Headers> = Mutex::new({
        let mut cookies = Cookie::new();
        cookies.append("department", "sgit.sgcc");
        cookies.append("collector", "liuan");
        let mut headers = Headers::new();
        headers.set(UserAgent::new("firefox/55.0"));
        headers.set(cookies);
        headers.set(Accept(vec![qitem(mime::TEXT_HTML)]));
        headers
    });
}

fn information_collect() -> Result<String, Box<Error>> {
    let default = {DEFAULT.lock()?.clone()};

    let login = post("http://httpbin.org/post", |rb|{
        rb.headers(default.clone())
            .header(ContentType(mime::APPLICATION_WWW_FORM_URLENCODED))
            .body("user=admin&passwd=admin")
    })?;

    let r1 = if login.status() == StatusCode::Ok {
        let mut logined = Cookie::new();
        logined.append("sid", "!!Sorry, I still don't know how to support HTTP session!!");
        logined.append("comment", "!!;;;after reading API docs;;;;");
        get2("http://httpbin.org/get", |rb|{
            rb.headers(default.clone())
                .header(logined.clone())
                //.header(Other when login)
        })
    } else {
        get2("http://httpbin.org/get", |rb|{rb.headers(default.clone())})
    };
    match r1 {
        Ok(mut r) => r.text(),
        _ => Ok(String::from("0")),
    }
}

If I write this code in Client::new()....send() style, I'll have to write many error handling, and many Client::new()....send() looks strange for readers.

Futuremore when reqwest decided to add other supports (i.e. add session support by setting reqwest env, higher level api can stay unchanged, such as code below

// in the future
reqwest::request::env().enable_conn_reuse(true) // enable connection reuse for same server

let login = reqwest::request::post("http://xxx/login", |rb| {
    rb.headers(default.clone())
        .body(...)
})?;
let resp = if login.status() == StatusCode::OK {
    //set the cookie
    reqwest::request::get("http://xxx/realtime", |rb| {
        rb.headers(default.clone())
            .header(cookie)
    })
} else {
    reqwest::request::get("http://xxx/delayed" |rb| {rb.headers(default.clone())})
}?;

If I have different flow cause by different environments. A closure(callback) will make it easy to change the outer flow.
If reqwest support auto session mode someday, packing in closure will keep API style uniformly

//In the future
let s = reqwest::request::Session::new();
let login = s.post("http://xxx/login", |rb| {
    rb.headers(default.clone())
        .body(...)
})?;
let resp = if login.status() == StatusCode::Ok {
    s.get("http://xxx/realtime", |rb| {rb.headers(default.clone())})
} else {
    reqwest::request::get("http://xxx/delayed" |rb| {rb.headers(default.clone())})
}?;

Actually, s.get(), s.post(), etc. is coding like below

impl Session {
    pub fn get(&self, url &str,  set_header: F) -> Result<Response>
        where F: Fn(&mut RequestBuilder) -> &mut RequestBuilder {
        let resp = set_header(
            &mut Client::new()?
                .get(url)?
        ).header(self.load_cookie()) // Append cookies stored in &self
            .send()?
        self.save_cookie(resp);
        Ok(cookie)
    }
    pub fn post(...)...;
    pub fn put(...)...;
    fn load_cookie()...;
    fn save_cookie()...;
}

If we use style without callback closure, Session.get() cannot keep style uniform with Non session mode API.

My impls

Following code are helper functions I often use.

trait Text {
    fn text(&mut self) -> Result<String, Box<Error>>;
}

// I implement this because most html/xml/json docs are utf-8 streams,
// provide this method call will make code more simple in response analysis.
impl Text for Response {
    fn text(&mut self) -> Result<String, Box<Error>> {
        let mut s = String::new();
        self.read_to_string(&mut s)?;
        Ok(s)
    }
}

pub fn get2<F>(url: &str, set_header: F) -> Result<Response, Box<Error>>
    where F: Fn(&mut RequestBuilder) -> &mut RequestBuilder {
    let response = set_header(
        &mut Client::new()?
            .get(url)?
    ).send()?;
    Ok(response)
}

pub fn post<F>(url: &str, set_header: F) -> Result<Response, Box<Error>>
    where F: Fn(&mut RequestBuilder) -> &mut RequestBuilder {
    let response = set_header(
        &mut Client::new()?
            .post(url)?
    ).send()?;
    Ok(response)
}

Pack get/get2/post/put/etc. to reqwest::request?

These API are higher level API for user in reqwest(needn't learn how to use Client), maybe these can be packed into request? So user can quickly start by import reqwest::request::*; and import reqwest::headers::*;

RequestBuilder.headers - Borrow or Move?

When I write information_collect(), I found that rb.headers(default) call will consume default: Header, not borrow. So as the rb.header(). So I have to use clone() every time I call those method.
Why not use borrowed header?

Last

Provide API like below is also a good news. But current released version(0.7.2) still not provided yet.

reqwest::request::get2(url)?
    .headers(default)
    .header(Cookie(...))
    //...
    .send()?
reqwest::request::post(url)?
    //...
    .body(...)
    .send()?
reqwest::request::put(url)?
    //...
    .send()?

But consider twice before implement these. Closure callback maybe a better choice to compat features in the future

[SuperHacker-liuan]

Advantages of using closured high-level API

0. Make programming interface more simple.
1. Keep programming style consistent

let r = get(url);
let r = get2(url, |r|{r.headers(default.clone())});
let r = post(url, |r|{r.body("username=admin&passwd=admin")});
let r = put(url, |r|{r.body(file)});
let r = options(url, |r|{r});

let s: Session = Session::new();
let r = s.post(url, |r|{r.body("username=admin&passwd=admin")});
let r = s.get(url);
// let r = etc.

2. hidden the detail of Client calling for high-level users.
3. Provide a easy-to-use standard API for most high-level users.

//mostly, attribute in ClientBuilder are seldom changed, consider put it in session env or guard env
let s = Session::new();
s.env().gzip(flase);
let login = s.post(url, |r|{r.body("u=admin&p=1")});
// thread local env()
let guard = reqwest::request::env();
guard.gzip(false);
let login = post(url, |r|{r.body("u=admin&p=1")});//post() setting a ClientBuilder
//guard.clean();

4. Compat future features(maybe import someday), keep API stable.

// To support keep_alive need to maintaince a map in env,
// since Client to different hostname is differed,
// but high-level developer needn't to concered this.
// Here reqwest library is acting as a connection pool.
// **Using callback closure can help to keep calling style no changed**
let env = reqwest::request::env();
env.keep_alive(true);//Change a switch in post/get/put/etc.
let login = post(url, |r|{r.body("u=admin&p=1")});
loop {
    let r = get(url2); //get() reuse Client created by post()
    //sleep some seconds
}
//env.clean();

5. In async mode, high level API may return a Furured Response???I dont know how async api use.

// use async traits to support and_then
// Following code is a password brute force, But I don't know how to do async programming.
// Consider it a persedo code, since there may sth. wrong.
let async = reqwest::request::Async::new();
async.max_parallel(100); // default -1 unlimited, If connections over 100, new Request will be blocked.
let cont = Mutex::new(true); // true until passwd cracked.
while {*cont.lock()} == true {
    let s = async.new_session();
    //s.get(vcodeurl).and_then
    s.post(url, |rb|{rb.body("passwd=" + dict.next())})
       .and_then(|r|{
           cont.lock() = false;
       });
    //async.get(xxx)
}

@seanmonstar

[seanmonstar]

Thanks for writing in detail your points, I think there's some valid things in here.

If I write this code in Client::new()....send() style, I'll have to write many error handling, and many Client::new()....send() looks strange for readers.

I wouldn't suggest writing Client::new() for every single request. Just call let client = Client::new() once, and use that for the rest of the requests. And in fact, making just 1 request will be better than creating new ones over and over and over. This is because a Client is able to keep a pool of connections, and make use of keep-alive to improve performance.

If a bunch of reqwest::request::{get, get2, post, etc} functions were provided, they'd need to construct new Clients each time. This would just be promoting an anti-pattern, when the super easy solution is for someone to just create a Client::new() one time, and then call methods on, like client.get(), client.post(), etc.

The send call is just a way to say you're done building the requests, you're reading for the response. Does a different name make it seem more important?

let res = client.post(url)?
    .json(&map)
    .response()?; // send()

I found that rb.headers(default) call will consume default: Header, not borrow. So as the rb.header(). So I have to use clone() every time I call those method. Why not use borrowed header?

Good point, perhaps headers should take &Headers.

trait Text {
    fn text(&mut self) -> Result<String, Box<Error>>;
}

I've considered adding such a method before, there is #86 for discussing the merit.

[SuperHacker-liuan]

Yes, you are right, Reuse client makes brute force very fast.