Intercept Network calls

[saikrishna321]

@zabil - In puppeteer, we can intercept API calls, continue or abort network request/response.

Can this be implemented in taiko ?

Code in puppeteer

async function automateBrowser({port}) {
	const browser = await puppeteer.launch({
		headless: false
	});

	const page = await browser.newPage();
	await page.setRequestInterceptionEnabled(true);

	page.on('request', request => {
		const URLToIntercept = 'https://forms.hubspot.com/embed/v3/form/2059467/2e1a1b5b-27bb-447d-aac4-0b87c1e88fec?callback=hs_reqwest_0&hutk=';

		if (request.url === URLToIntercept) {
			request.continue({
				url: `http://127.0.0.1:${port}/form.txt`
			});
		} else {
			request.continue();
		}
	});

	await page.goto('https://www.sitepen.com/site/contact.html');

	await browser.close();
	process.exit();
}

[zabil]

It's possible right now.
Taiko exposes the CRI client with the Network object.

However it'll require some digging into CRI's docs.

To understand this better, intercepting requests to provide test data? Maybe add an easier API something like

intercept("https://example.com", "http://proxy.com")

and for finer control

intercept("https://example.com", request => {})

[saikrishna321]

intercepting requests to provide test data?

Yeah, we can mock any api call with test data.

https://github.com/cyrus-and/chrome-remote-interface/wiki/Intercept-and-block-requests-by-URL

[NivedhaSenthil]

WIP

Example1:

intercept({ 
        request: {url:'https://taiko.gauge.org/assets/css/pilcrow.css'},
        mock: {errorReason:"Failed"}
})

Example2:

intercept({ 
        request: {url:'https://taiko.gauge.org/assets/css/pilcrow.css'},
        mock: {response:{status: 404, contentType: 'text/plain',body: 'Not Found!'}}
})

Example3:

intercept({ 
        request: {url:'https://taiko.gauge.org/assets/css/pilcrow.css'},
        callback: doOnIntercept
})

intercept api will take an object with request to intercept and action to be done on intercepted request.

• request can be matched using any one from (url, post-data, method, resourceType, headers)
• mock can be used to do various things like below
  • override request details like url, method, post-data, headers
    eg: mock:{ url: 'http://proxy', method: 'GET' }
  • modify response details like status, header, content-type, body
    eg: mock: {response:{status: 404, contentType: 'text/plain',body: 'Not Found!'}}
  • abort an request
    eg: mock: {errorReason:"Failed"}
• callback can be used to do any action after interception

[nehashri]

A small change here would be useful,
The callback should run when the request is intercepted. It should take the request as its parameters and return a mock response object.
Example

intercept({ 
        request: {url:'https://taiko.gauge.org/assets/css/pilcrow.css'},
        callback: function doOnIntercept(request) {
                      // do something
                      return mockResponse;
                 }
})

This gives the user ability to intercept the request, process any data it may contain and return a mock response dynamically.

[zabil]

For first pass we must make this very simple

If we approach this from a testing perspective

Case 1: Mock the requested url with a test data url

intercept("foo", "bar")

• foo can be a url pattern or regex.
• The test data url bar must get all the headers and data of foo

Case 2: Return JSON

intercept("foo", {bar: "baz"})

Case 3: Return the result of a function

intercept("foo", (request) => return "baz");

• the request object must have details of the data, original url, headers

[saikrishna321]

@zabil i like the First approach 👍

Can i know which approach has the team decided to go with ?

[NivedhaSenthil]

@zabil if we match only using url user may not have flexibility to identify any request with specific method, header, or post-data.

can it be like intercept(requestObject, handler) where handler can be a mock object or callback which will return a mock object.

eg: intercept({url:'http://something.com',method:'get'}, (request) => return 'baz');

[zabil]

Because the application implements the request method, setting it is redundant so {url:'http://something.com',method:'get'} just be http://something.com and there probably won't be a case where GET has to change to POST and vice versa.

For headers they can be read/modified from

(request) => return 'baz'

[NivedhaSenthil]

We will be following below design for intercepting APIs

case 1: block url => intercept(url)
case 2: mockResponse => intercept(url,{mockObject})
case 3: override request => intercept(url,(request) => {request.continue({overrideObject})})
case 4: redirect => intercept(url,redirectUrl)
case 5: mockResponse based on request =>
intercept(url,(request) => { //someaction; request.respond({mockResponseObject})} )

[NivedhaSenthil]

Regex to match URLs will be handle in #142

[NivedhaSenthil]

Fix should be available in 7e4fdc3.

Below is an example script for all 5 cases mentioned in the issue. Steps to run the sample script,

• git clone https://github.com/DanWahlin/Angular-JumpStart.git

• cd Angular-JumpStrart

• npm install && npm start

• taiko code.js

#code.js

const assert = require("assert");
(async () => {
    try {
        await openBrowser({headless:false});
        
        //Case 1: Block URL (Blocks all request for people.png)
        await intercept("http://localhost:3000/assets/images/people.png");

        //Case 2: Mock response (Mocks response to modify address from "888 Central St." to "12345 Central St." )
        await intercept("http://localhost:3000/api/customers/11",{body: '{"id":11,"firstName":"ward","lastName":"bell","gender":"male","address":"12345 Central St.","city":"Los Angeles","state":{"abbreviation":"CA","name":"California"},"latitude":34.042552,"longitude":-118.266429}'});

        //Case 3: Override request (Overrides google4_hdpi.png url to return female.png can be seen in map)
        await intercept("https://maps.gstatic.com/mapfiles/api-3/images/google4_hdpi.png",(request) => {request.continue({url:"http://localhost:3000/assets/images/female.png"})})
        
        //Case 4: Redirect URL (Overrides male.png to female.png)
        await intercept("http://localhost:3000/assets/images/male.png","http://localhost:3000/assets/images/female.png")
        
        //Case 5: Mock response based on request (Mocks response to modify address from "1234 Anywhere St." to "888 Anywhere St.")
        await intercept("http://localhost:3000/api/customers/1", (request) => {request.respond({body:'{"id":1,"firstName":"ted","lastName":"james","gender":"male","address":"888 Anywhere St.","city":" Phoenix ","state":{"abbreviation":"AZ","name":"Arizona"},"orders":[{"productName":"Basketball","itemCost":7.99},{"productName":"Shoes","itemCost":199.99}],"latitude":33.299,"longitude":-111.963}'})});
        
        await goto("localhost:3000");
        await click(link("2"));
        await click(link("Ward Bell"));
        assert.equal("Exists",(await contains("12345 Central St.").exists()).description);
        await click(link("Customers"));
        await click(link("Ted James"));
        assert.equal("Exists",(await contains("888 Anywhere St.").exists()).description);
    
    } catch (e) {
        console.error(e);
    } finally {
        await closeBrowser();
    }
})();

[sswaroopgupta]

Getting Error: ENOENT: no such file or directory, stat '.../Angular-JumpStart/dist/index.html' on npm install && npm start

[NivedhaSenthil]

Have missed ng build step in between, try doing it before npm start

[vitor-gyant]

Hi @NivedhaSenthil ,

If I want to add a new key to the postData is this the best way to do it ?

await intercept(`${baseUri}/api/create-account`, (request) => {
      request.continue({ postData: `${request.request.postData}&isTestingUser=true` });
});

Is there another way to do this ?

Thanks,

[zabil]

@vitor-gyant have you tried

request.request.postData.isTestingUser = true
request.continue(request.request.postData);

[rmallof]

Hi @zabil @NivedhaSenthil

I'm testing the intercept functionality with the form at https://tryphp.w3schools.com/showphp.php?filename=demo_form_post while intercepting https://tryphp.w3schools.com/demo/welcome.php using:

step(["Step"], async () => {
    await intercept("https://tryphp.w3schools.com/demo/welcome.php", (request) => { 
        console.log("@ Intercept step callback");
        let parameters = querystring.parse(request.request.postData)
        parameters["name"] = "This comes from Taiko";    <--------------- overwritting "name" variable with "This comes from Taiko"
        request.request.postData = querystring.stringify(parameters);
        request.continue({ postData: request.request.postData });
    });
});

I'm able to modify the parameters in request.request.postData but request.continue() is not taking effect - the data shown at https://tryphp.w3schools.com/showphp.php?filename=demo_form_post after sending the form is the one written in the textBox instead of "This comes from Taiko"

Am I missing something?

Thank you

[NivedhaSenthil]

yes, I could see it is broken. Taking a look at it, thanks for reporting

[rmallof]

Thanks @NivedhaSenthil. Please, let me know if I should open a new issue for this behavior

[zabil]

@rmallof yes, that will be good.

[rmallof]

Done: #1520

Thanks @zabil @NivedhaSenthil !

[ramirobg94]

Does it work with headless:true?

[NivedhaSenthil]

Yes it should work in both headless and headful mode