write output to a file

[lonnblad]

What do you think about adding output options for writing the output to a file?

Something like this:

godog \
--random \
--concurrency=5 \
--format=progress \
--output-file=junit.xml \
--output-format=junit \
features

It would allow a user to see the progress status of the execution, but also write a junit formatted file.

[l3pp4rd]

Hi, yes I think it is reasonable. just --output-format should not be necessary. because it may make sense to use multitple formatter outputs. probably better to have it in single parameter, like:

godog \
--random \
--concurrency=5 \
--format=progress \
--output-file=junit@outputs/junit.xml \
--output-file=html@outputs/report.html \
features

The separator could be different, but I think this should be a single parameter

[l3pp4rd]

I gave you the contributor access to repository, since I do not find much spare time to contribute, looking forward to community more

[l3pp4rd]

not sure if you still have write access, since repository now is in cucumber org. but probably @aslakhellesoy will be able to set that up

[lonnblad]

@l3pp4rd thanks for thinking about me, I believe you are correct, but at least I'm in the committers-go slack channel for the cucumber community now

[mxygem]

@lonnblad I'd really like to add writing to a file as a feature myself as there were times in the past I had wanted it. Lets make sure to follow the existing Cucumber implementations. This will unfortunately conflict with godog's existing o/out flag, but I'm okay with that as it brings it more in line with the other languages. I think this is important and want to do it as much as possible. If someone ends up transitioning from another language to go, having the implementation feel similar would be great.

    -o, --out [FILE|DIR]             Write output to a file/directory instead of STDOUT. This option
                                     applies to the previously specified --format, or the
                                     default format if no format is specified. Check the specific
                                     formatter's docs to see whether to pass a file or a dir.

[lonnblad]

the thing I like with l3pp4rd’s suggestion was that then I could run progress + JUnit, so I would get the feedback from progress, but JUnit as a file to be used against our test reporting

[mxygem]

That would still be possible. :)

godog \
--random \
--concurrency=5 \
--format=progress \
--format=junit
--out=report.xml \
features

[lonnblad]

true, nice, would both be written to the same file or two different files?

[mxygem]

Progress would go to stdout and junit would be written to a file

[lonnblad]

the progress and JUnit formatters will basically support this today, since both use the basefmt and JUnit creates its summary based on the basefmt without any internal properties in the JUnit formatter

[mdelapenya]

Is there any progress on this? I see it key for CI automation, as any test generating logs in the application has to deal with parsing the output to extract junit/cucumber format from the log

[lonnblad]

@mdelapenya not yet, I believe, I would like it though

If you would use it, how would you configure it?
progress on stdout with cucumber or junit to file?
or another combination?

[mdelapenya]

I'd prefer having a junit output file to be processed by jenkins, i.e

[lonnblad]

yep, sounds like the same use case that I have
do you also want the progress to stdout?

[lonnblad]

@jaysonesmith do you have an idea of how we link --format with --out?

I'm thinking that in the case someone writes:

godog \
--random \
--concurrency=5 \
--format=junit \
--format=progress \
--out=report.xml \
features

godog \
--random \
--concurrency=5 \
--format=junit \
--out=report.xml \
features

godog \
--random \
--concurrency=5 \
--format=progress \
--format=junit \
--format=cucumber \
--out=report.xml \
--out=report.json \
features

[mdelapenya]

For me it's not a requirement having the console output, unless the console formatter flag is present too

[lonnblad]

Should we do something similar to how the go binary works?

• godog build -o <binary> for creating a test runner executable
• godog run for creating and running a test runner (the same as godog does now)

With this, we could also look at adding godog fmt or other functionality in the future.

[lonnblad]

We have decided to implement this:

godog run \
 --format progress \
 --format junit:report.xml \
 --concurrency 10 \
 features

Where the progress formatter would write to stdout and the junit formatter would write to report.xml

[dipaligulhane]

Hi there,
Tried using junit formatter for godog. On console I am getting expected output however while trying to save it file generating an error.
Can someone help with this?

---

E:\migration\auto-go\src\features>go test -v -timeout 12h --godog.format=junit:myresults.xml --godog.tags=@test feature_files
unregistered formatter name: "junit:myresults.xml", use one of: junit, pretty, progress, cucumber, events
testing: warning: no tests to run
PASS
exit status 2
FAIL bitbucket.org/mycode/api-automation/features 0.206s

E:\migration\auto-go\src\features>go version
go version go1.12.6 windows/amd64

E:\migration\auto-go\src\features>godog --version
Godog version is: v0.10.0

---

[mdelapenya]

Did you try with "--godog.format=junit"? Then capture the XML output with bash.

What is described in this issue if for the future implementation I guess

[dipaligulhane]

@mdelapenya I see code is merged for this change. So assumed its available.
@lonnblad let me know if my assumption is not correct and code is for future use.

I had tried with redirecting o/p to xml file.
However at the end below unwanted contents are also getting added, which is causing issues with xml parsing.

---

testing: warning: no tests to run
PASS
ok bitbucket.org/myprj/api-automation/features 379.787s

---

[brpaz]

This would also be useful with the cucumber formatter.

Right now, it´s just printing the cucumber.json to stdout. Even if I try to redirect the output to a file, it doesn't contain only the json but all the other output from go test, application logs, etc, making it unusable.

[mdelapenya]

Hi 👋 is there an ETA for this?

[aslakhellesoy]

@mdelapenya people are working on this in their spare time so I don't think that's an appropriate question.

[mdelapenya]

@aslakhellesoy sorry if you considered it as an un-appropriate question, that was not my intention of course. I know what working in open source means and I try to be polite any time I talk to maintainers. So at no reason I was trying to cause offence and I apologize if that was the case.

I'm looking forward to using this new feature, because I have to deal with ugly hacks to parse the XML output.

[lonnblad]

Hey @dipaligulhane @mdelapenya @brpaz

We have a few remaining issues for v0.11.0 before it can be released.
Regarding Cucumber support, it will support all the built-in formatters.

[lonnblad]

I will close this issue and have created another issue to add support for executing multiple formatters in the same test run.
#346

[lonnblad]

Hey @dipaligulhane @mdelapenya @brpaz

Sorry for the long delay, v0.11.0 is now released with support for writing the output to a file.

BR
Fredrik

[mdelapenya]

I'll give it a try ASAP

Thanks again for your work here 👏👏👏

[lonnblad]

Nice, let me know if you have any issues! :)

[dipaligulhane]

@lonnblad surely will try with latest build.

[mdelapenya]

Hey @lonnblad, using this version for a few days, and it works fairly well. KUDOS!

One thing that surprised me, if setting a path as filename (i.e. junit:outputs/TEST-unit.xml), it does not create parent dirs, although I can imagine there are security concerns about creating arbitrary paths in the filesystem.

Again, thanks for your work here!

[lonnblad]

@mdelapenya
didn't consider that to be honest, I agree that there might be security concerns, but it feels like that should be fine, or what do you think?

[mdelapenya]

I'd say it's ok as it is now. In my case the build system/automation is the one responsible of creating the required paths.

[mdelapenya]

BTW, is it possible to use multiple formatters? 🤔 My use case would be writing the jUnit file plus a format that would enable creating an HTML report for the test scenarios.

[lonnblad]

currently it's not supported, but I want to add support for multiple formatters, #346

[dipaligulhane]

@lonnblad started using and its working as expected. I am currently using junit XML formatter.
And yeah multiple format support would be good addition too.

[lonnblad]

@dipaligulhane nice, thanks for the feedback!! :)
I will definitely try to prioritize adding multiple formatters.