Skip to content

woodpecker-framework框架http发包库,专门为漏洞检测与利用场景设计。

License

Notifications You must be signed in to change notification settings

woodpecker-framework/woodpecker-requests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

woodpecker-requests

woodpecker-requests是基于 requests 为woodpecker框架定制开发的httpclient库,目的是编写插件时能拥有像python requests一样的便利。特点为可以全局设置代理、全局设置UA等


Requests is a http request lib with fluent api for java, inspired by the python request module. Requests requires JDK 1.8+, the last version support Java7 is 4.18.* .

Table of Contents

Maven Setting

Requests is now in maven central repo.

https://mvnrepository.com/artifact/me.gv7.woodpecker/woodpecker-requests

<dependency>
    <groupId>me.gv7.woodpecker</groupId>
    <artifactId>woodpecker-requests</artifactId>
    <version>0.1.0</version>
</dependency>

Global Config

global proxy

public static boolean enable = false; // open proxy or close
public static String protocol = "http"; // http or socks
public static String host = "127.0.0.1";
public static int port = 8080;
public static String username; // socks proxy user,can set null
public static String password; // socks proxy pass,can set null
HttpConfigManager.setProxyConfig(enable
                ,protocol
                ,host
                ,port
                ,username
                ,password);
    }

Global proxies are not used

Requests.method("GET","http://wwww.baidu.com/").proxy(Proxy.NO_PROXY).send();

global user-agent

LinkedList<String> uaList = new LinkedList<>();
// set greater than 2 will random choise.
uaList.add("Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0");
uaList.add("Mozilla/5.0 (Android; Tablet; rv:14.0) Gecko/14.0 Firefox/14.0");
HttpConfigManager.setUserAgentConfig(uaList);

global time out

int time = 5000; // !!!attention!!! millisecond
boolean enableMandatoryTimeout = false; // ignored user set
int mandatoryTimeout = 1;
HttpConfigManager.setTimeoutConfig(time
                        ,enableMandatoryTimeout
                        ,mandatoryTimeout);

enableMandatoryTimeout will ignore user set,such us:

Requests.get("http://woodpecker.gv7.me/").timeout(10000).send() // timeout will be replaced by mandatoryTimeout

Usage

Simple Case

One simple http request example that do http get request and read response as string:

String url = ...;
String resp = Requests.get(url).send().readToText();
// or
Response<String> resp = Requests.get(url).send().toTextResponse();

Post and other method:

resp = Requests.post(url).send().readToText();
resp = Requests.head(url).send().readToText();
...

The response object have several common http response fields can be used:

RawResponse resp = Requests.get(url).send();
int statusCode = resp.statusCode();
String contentLen = resp.getHeader("Content-Length");
Cookie cookie = resp.getCookie("_bd_name");
String body = resp.readToText();

Make sure call readToText or other methods to consume resp, or call close method to close resp.

The readToText() method here trans http response body as String, more other methods provided:

// get response as string, use encoding get from response header
String resp = Requests.get(url).send().readToText();
// get response as bytes
byte[] resp1 = Requests.get(url).send().readToBytes();
// save response as file
boolean result = Requests.get(url).send().writeToFile("/path/to/save/file");

Charset

Requests default use UTF-8 to encode parameters, post forms or request string body, you can set other charset by:

String resp = Requests.get(url).charset(StandardCharsets.ISO_8859_1).send().readToText();

When read response to text-based result, use charset get from http response header, or UTF-8 if not found. You can force use specified charset by:

String resp = Requests.get(url).send().charset(StandardCharsets.ISO_8859_1).readToText();

Passing Parameters

Pass parameters in urls using params method:

// set params by map
Map<String, Object> params = new HashMap<>();
params.put("k1", "v1");
params.put("k2", "v2");
String resp = Requests.get(url).params(params).send().readToText();
// set multi params
String resp = Requests.get(url)
        .params(Parameter.of("k1", "v1"), Parameter.of("k2", "v2"))
        .send().readToText();

If you want to send post www-form-encoded parameters, use body() methods:

// set params by map
Map<String, Object> params = new HashMap<>();
params.put("k1", "v1");
params.put("k2", "v2");
String resp = Requests.post(url).body(params).send().readToText();
// set multi params
String resp = Requests.post(url)
        .body(Parameter.of("k1", "v1"), Parameter.of("k2", "v2"))
        .send().readToText();

The forms parameter should only works with post method.

Set Headers

Http request headers can be set by headers method:

// set headers by map
Map<String, Object> headers = new HashMap<>();
headers.put("k1", "v1");
headers.put("k2", "v2");
String resp = Requests.get(url).headers(headers).send().readToText();
// set multi headers
String resp = Requests.get(url)
        .headers(new Header("k1", "v1"), new Header("k2", "v2"))
        .send().readToText();

Cookies

Cookies can be add by:

Map<String, Object> cookies = new HashMap<>();
cookies.put("k1", "v1");
cookies.put("k2", "v2");
// set cookies by map
String resp = Requests.get(url).cookies(cookies).send().readToText();
// set cookies
String resp = Requests.get(url)
        .cookies(Parameter.of("k1", "v1"), Parameter.of("k2", "v2"))
        .send().readToText();

Request with data

Http Post, Put, Patch method can send request body. Take Post for example:

// set post form data
String resp = Requests.post(url).body(Parameter.of("k1", "v1"), Parameter.of("k2", "v2"))
        .send().readToText();
// set post form data by map
Map<String, Object> formData = new HashMap<>();
formData.put("k1", "v1");
formData.put("k2", "v2");
String resp = Requests.post(url).body(formData).send().readToText();
// send byte array data as body
byte[] data = ...;
resp = Requests.post(url).body(data).send().readToText();
// send string data as body
String str = ...;
resp = Requests.post(url).body(str).send().readToText();
// send data from inputStream
InputStreamSupplier supplier = ...;
resp = Requests.post(url).body(supplier).send().readToText();

One more complicate situation is multiPart post request, this can be done via multiPart method, one simplified multi part request example which send files and param data:

// send form-encoded data
InputStreamSupplier supplier = ...;
byte[] bytes = ...;
String resp = Requests.post(url)
        .multiPartBody(
            Part.file("file1", new File(...)),
            Part.file("file2", "second_file.dat", supplier),
            Part.text("input", "on")
        ).send().readToText();

Json support

Requests can handle json encoder(for request body)/decoder(for response body), if having Json Binding, Jackson, Gson, or Fastjson lib in classpath.

// send json body, content-type is set to application/json
RawResponse response = Requests.post("http://.../update_person")
                .jsonBody(value)
                .send();
// response body as json, to value
Person person = Requests.post("http://.../get_person")
                .params(Parameter.of("id", 101))
                .send().readToJson(Person.class);
// json body decoder to generic type
List<Person> persons = Requests.post("http://.../get_person_list")
                .send().readToJson(new TypeInfer<List<Person>>() {});

You may set your own json processor by:

JsonProcessor jsonProcessor = ...;
JsonLookup.getInstance().register(jsonProcessor);

Basic Auth

Set http basic auth param by auth method:

String resp = Requests.get(url).basicAuth("user", "passwd").send().readToText();

Redirection

Requests will handle 30x http redirect automatically, you can disable it by:

Requests.get(url).followRedirect(false).send();

Timeout

You can set connection connect timeout, and socket read/write timeout value, as blow:

// set connect timeout and socket timeout
Requests.get(url).socksTimeout(20_000).connectTimeout(30_000).send();

Response compress encoding

Requests send Accept-Encoding: gzip, deflate, and auto handle response decompress in default. You can disable this by:

// do not send Accept-Encoding: gzip, deflate header
String resp = Requests.get(url).acceptCompress(false).send().readToText();
// do not decompress response body
String resp2 = Requests.get(url).send().decompress(false).readToText();

Https Verification

Some https sites do not have trusted http certificate, Exception will be thrown when request. You can disable https certificate verify by:

Requests.get(url).verify(false).send();

Proxy

Set proxy by proxy method:

Requests.get(url).proxy(Proxies.httpProxy("127.0.0.1", 8081)).send(); // http proxy
Requests.get(url).proxy(Proxies.socksProxy("127.0.0.1", 1080)).send(); // socks proxy proxy

Session

Session maintains cookies, basic auth and maybe other http context for you, useful when need login or other situations. Session have the same usage as Requests.

Session session = Requests.session();
String resp1 = session.get(url1).send().readToText();
String resp2 = session.get(url2).send().readToText();

About

woodpecker-framework框架http发包库,专门为漏洞检测与利用场景设计。

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages