ability to embed attachments and make them available on TestNG XML report

[bitcoder]

TestNG Version

7.0

Expected behavior

It should be possible to attach a file (e.g. screenshot) and make it available in the TestNG XML output report, preferably embedded.
This would be great for tools that parse TestNG results from the XML report file and showing them, along with relevant attachments/screenshots) in test management tools (e.g. Xray for Jira).

Actual behavior

TestNG XML report schema seems to not support attachments at all (either as references or embedded).
Note: I'm unsure if TestNG supports adding attachments at all in its public API.

[krmahadevan]

@bitcoder why not build a custom XML report by implementing org.testng.IReporter interface that does this ? That should let you accomplish this need without depending on TestNG for the change no ?

[bitcoder]

hi @krmahadevan , well that could be a possibility; however, it would be mostly duplicating the current XML Reporter and extend current TestNG API in order to support adding attachments on the result object; don't you think having this built-in makes more sense, as it broadens its usage? Besides, this is not really something specific for a given user; I see this as something quite valuable and common amongst users.

[krmahadevan]

@bitcoder - I hear you. There are many such generic use cases that can be embedded into TestNG. But I think that's perhaps not what TestNG is expected to do. Instead it's expected to enable customisation.

Would you be interested in submitting a PR that lets people customise the current XML report ? That way there won't be any duplication of code etc and specifics such as addition of attachments etc can still live and be maintained outside of TestNG.

@juherr thoughts?

[juherr]

I agree. The team is not big enough to support all needs. Instead, we are trying to allow "plugins".

About reporter, a lot exists and maybe one of them is already doing what you expect: https://github.com/cbeust/testng/wiki/3rd-party-extensions#reporters--loggers

[bitcoder]

thanks @juherr , I've went over them and I don't think any supports it though.
When I have some time, I can try to make a rough PR with the ability to attach files to the result and make them available in the standard XML report.. or do you prefer to make all of this available as separate code?

[juherr]

@bitcoder It depends on the size of the PR :)

[krmahadevan]

@bitcoder - Do you mind showing us how the end report would look like (which includes the attachments embedded) ? That way, we can try figuring out how do we enrich org.testng.reporters.XMLReporter so that downstream consumers can plugin customization by extending this class.

[bitcoder]

@krmahadevan , sure :)
This is what Xray currently is able of showing; please look at the link for full info.

What many users would love to have is something like what is possible with Cucumber, which is able to embed attachments, such as screenshots, at each step/gherkin sentence level.

( full example here )

[krmahadevan]

@bitcoder - I was hoping that you would share the contents of how the xml (the one which has screenshots embedded into it) should look like ?

[bitcoder]

oh @krmahadevan , I was thinking more on the end result :)
Anyway, something like this perhaps?

        <test-method status="FAIL" signature="CanAddNumbersFromGivenData(int, int, int)[pri:0, instance:com.xpand.java.CalcTest@36d4b5c]" name="CanAddNumbersFromGivenData" duration-ms="1" started-at="2018-03-07T14:42:27Z" data-provider="ValidDataProvider" finished-at="2018-03-07T14:42:27Z">
          <attachments>
          	<attachment>
          		<name>screenshot1.png</name>
          		<description>whatever</description>
          		<rawcontent>...base64 enconded content...</rawcontent>
          	</attachment>
                ...
          </attachments>
        ...
      </test-method>

Eventually, it could be more flexible to also support remote references instead of embedded content.

[bitcoder]

thanks @krmahadevan for your work!
Meanwhile, I've been playing with the source code and trying to make something like the following work; I have already a first prototype.

    @Test
    public void CanAddNumbers()
    {
        Assert.assertEquals(Calculator.Add(1, 1),2);
        Assert.assertEquals(Calculator.Add(-1, 1),0);
        ITestResult result = Reporter.getCurrentTestResult();
        result.addAttachment(new File("/Users/smsf/1.png"));
        result.addAttachment(new File("/Users/smsf/2.png"));    
    }

For this, however, I would need to change the ITestResult and also the implementation to have new methods.
Which approach do you think would be best?
a) enhance the ITestResult to support adding/getting attachments on the TestResult object
b) leave the current API as it is and store those attachments as an attribute (e.g. "files") using setAttribute(); the problem here is that the "addTestResultAttributes" in XMLSuiteResultWriter will try to blindly do a toString() on the attribute.. which in case of a File object is not what we would want; of course we could pass a string with the base64 encoded filecontent; however, we would miss additional attributes such as filename and other that may be relevant; that's why I think a) can be best even if it requires some changes

Looking forward for your comments and also from @juherr :)

[bitcoder]

On the following of my previous comment, namely related with b), on a given Test method we could do something like:

        File attachments[] = new File[] { new File("/Users/smsf/1.png"), new File("/Users/smsf/2.png") } ;
        result.setAttribute("attachments", attachments);

and then process it in the report class side. The only (very minor) drawback that I see after implementing it is that the attribute would also be logged in the tag as addTestResultAttributes() iterates over it.
At this point, b) is enough.
a) would be interesting if you would like attachments to be something official (optional) that users can use to add attachments to the TestResult object. If you think a) is the way to go, I can try to make a clean PR for it.

[krmahadevan]

@bitcoder - A few things.

Did you take a look at the PR that I raised to fix this issue ?

If not, please start from the changeset #2174 The commit message has some details in terms of the general customization support.

You can also take a look at the sample custom listener that I built for testing purposes to give you an idea on how to tweak this here.

I still dont think attachments support needs to be part of TestNG and it can very well reside as customizations which reside outside of the codebase.

If storing custom values via attributes on an ITestResult object is not something that you would want, then you can very well maintain your own map of key/value pairs wherein the keys represent the ITestResult object and the values represent whatever you would want as additional attributes.

Now this map you could query in your child class of org.testng.reporters.XMLReporter and within the newly added addCustomTagsFor() you transform this additional attributes into xml tags.

[bitcoder]

hi @krmahadevan , yes I did look at it. I made my code on top of it in order to extend the builtin XML report capabilities (https://github.com/bitcoder/xray-testng-extensions/blob/master/README.md)
Thanks for the feedback.