Match Multiple DSLs

[wdahlenburg]

Proposed changes

As initially discussed in #196, it'd be nice to support multiple DSL matches on requests and responses. For consistency, I added the same behavior to the match-replace DSLs.

The updated code loops over the defined matches and will perform each of them. The match-replace items allow the replaces to be chained, so text in the first match-replace could be replaced by text in a subsequent match-replace. An example would look like:

response-match-replace-dsl:
  - "replace(response,'example.com','test.com')"
  - "replace(response,'test','google')"

In the event that example.com was replaced in a response, the resultant text would be google.com. Users would need to be aware of overlapping replacements.

The MatchReplaceRequest and MatchReplaceResponse functions were not the simplest to switch over, so there may be a better solution.

The mitmrelay tool doesn't currenntly use goflags. To keep the scope of the PR small, I decided to insert the single match-replace DSLs into a string slice.

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[ehsandeep]

@wdahlenburg I've used example dsl in the config file and getting following error -

./proxify 

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/	v0.0.8

		projectdiscovery.io

[ERR] error compiling DSL: Unable to parse numeric value '.' to float64
...
...

[wdahlenburg]

@wdahlenburg I've used example dsl in the config file and getting following error -

./proxify 



                       _ ___    

   ___  _______ __ __ (_) _/_ __

  / _ \/ __/ _ \\ \ // / _/ // /

 / .__/_/  \___/_\_\/_/_/ \_, / 

/_/                      /___/	v0.0.8



		projectdiscovery.io



[ERR] error compiling DSL: Unable to parse numeric value '.' to float64

...

...

What DSL matchers if any exist in your config file?

[ehsandeep]

@wdahlenburg

response-match-replace-dsl:
  - "replace(response,'example.com,'test.com')"
  - "replace(response,'test','google')"

[wdahlenburg]

response-match-replace-dsl:

• "replace(response,'example.com,'test.com')"
• "replace(response,'test','google')"

I'm getting the same error from the dev branch with:

response-match-replace-dsl: "replace(response,'example.com,'test.com')"

Can you confirm if you see the same error on just the dev branch?

[wdahlenburg]

response-match-replace-dsl:

• "replace(response,'example.com,'test.com')"
• "replace(response,'test','google')"

I'm getting the same error from the dev branch with:

response-match-replace-dsl: "replace(response,'example.com,'test.com')"

Can you confirm if you see the same error on just the dev branch?

@ehsandeep I had a typo in the PR description, where there was a missing ' after example.com. Instead of being:

response-match-replace-dsl:
  - "replace(response,'example.com,'test.com')"
  - "replace(response,'test','google')"

It should have been:

response-match-replace-dsl:
  - "replace(response,'example.com','test.com')"
  - "replace(response,'test','google')"

[tarunKoyalwar]

lgtm!

[tarunKoyalwar]

Demo

Config File

...
# response match-replace dsl
response-match-replace-dsl:
  - "replace(response,'example.com','test.com')"
  - "replace(response,'test','google')"

# listening http ip and port address (ip:port)
#http-addr: 127.0.0.1:8888
...

WIthout Proxify

$  curl -v http://127.0.0.1:9000
*   Trying 127.0.0.1:9000...
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9000
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 06 Jan 2023 11:13:22 GMT
< Content-Length: 85
< Content-Type: text/plain; charset=utf-8
< 
Response Body from a test server
first match was at example.com
second match is test
* Connection #0 to host 127.0.0.1 left intact

With Proxify

$ curl -v http://127.0.0.1:9000 -x http://127.0.0.1:8888
*   Trying 127.0.0.1:8888...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET http://127.0.0.1:9000/ HTTP/1.1
> Host: 127.0.0.1:9000
> User-Agent: curl/7.85.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Date: Fri, 06 Jan 2023 11:13:34 GMT
< Content-Length: 88
< 
Response Body from a google server
first match was at google.com
second match is google
* Connection #0 to host 127.0.0.1 left intact

[ehsandeep]

thank you @wdahlenburg for adding this feature; we will be reworking on proxify to make the match / replace rule more simple and customizable in template style with more controls.

[ehsandeep]

Follow up - #210