Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add result reporter tool project #704

Merged
merged 1 commit into from
May 7, 2019
Merged

Add result reporter tool project #704

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions _projects/result-reporter-tool.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
collaborating_projects:
- coala
desc: "Implement a result reporter tool that converts linter/bear results
into various formats"
difficulty: medium
initiatives:
- GSoC
issues:
- "https://github.com/coala/projects/issues/555"
- "https://github.com/coala/coala/issues/2757"
- "https://github.com/coala/coala/issues/5910"
- "https://gitlab.com/coala/coala-json/issues/3"
markdown: result-reporter-tool.md
mentors:
- li-boxuan
name: "Result Reporter Tool"
requirements:
- "The applicant should have at least one non-trivial patch merged to
coala or coala-bears."
- "The applicant should be familiar with Python."
- "Familiarity with continuous integration tools (e.g. Jenkins) is a plus."
tags:
- Core
- Linter
- CI
---

Apart from common CLI output, coala provides json format output. However, there
are other formats which are useful for reporting tests and failures, but coala
does not support at the moment.

This is not only for coala, result format inconsistencies has been a problem for
a long time. Converting the static analysis results into a test results format
has been done a few times, such as early PEP8 plugins to Jenkins. The mapping
isn't exact, but the benefits of using the test result format is tight
integration with various systems.

Most commonly used is JUnit format. CircleCI can accept
[this format](https://circleci.com/docs/2.0/collect-test-data). AppVeyor can
receive test results in [XML format](https://www.appveyor.com/docs/running-tests/#uploading-xml-test-results).
Other formats include [TAP](https://testanything.org/),
[Clover](http://openclover.org/doc/manual/latest/ant--clover-report.html),
There are some systems, such as Phabricator, which have a
[custom format](https://github.com/facebook/infer/issues/673) they expect.

The applicant is welcomed to propose additional ideas around format topic, to
improve coala workflow, continuous integration workflow or linters workflow.
For example, pulling the coala results from CI could be a way to inform newbies
of the style problems with their PRs. Pushing the unit test results as XML to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain, In what way would pushing the unit test results as XML to the CI help?

I mean, Isn't our CI run when someone makes a PR. And whenever CI is run all the unit tests are executed by default and the results can be seen. Sorry if this might sound silly, but I haven't yet come across a case when we had to push the XML's of unit tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://circleci.com/docs/2.0/collect-test-data/ is a good example

the CI (using combination XML files if necessary). It might also provide
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might also provide should be changed to Providing linters with ...

linters with a solid set of classes and output formats for their CLI, letting
them focus on linting algorithms.

#### Milestones

##### PREPARATION/BONDING

- The applicant has created and merged a cEP for this project.
- The applicant has done research on different formats, including JUnit,
checkstyle, TAP, Language Server Protocol (LSP), etc.
- The applicant has done research on different formats required by CI
tools and platforms, including Jenkins, Circle CI, Appveyor, Phabricator, etc.

##### CODING PHASE 1

- A tool that converts JSON results to JUnit format is implemented, tested,
and documented.
- Test and add doc regarding how to upload coala results into Circle CI and
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Candidates need to think about this - can we make it automatic? What change do we need to make to existing CI configs used in coala?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@li-boxuan , what actually you mean by automatic here ?
I mean here we need tool that take coala Json response and convert it into corresponding junit response
is it. so should be done automatically here according to you 🤔

Copy link
Member Author

@li-boxuan li-boxuan Apr 1, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, I am not talking about the format itself.

I mean, how to make the "upload" process itself automatic. I didn't spend too much time investigating, so this question is left for students. For example, (maybe) Circle CI allows uploading a report (with JUnit format) in every continuous integration check. Maybe that requires some configuration in .circleci/config.yml.

EDIT: seems https://circleci.com/docs/2.0/configuration-reference/#store_test_results might be what we need. Then we can make pull requests to all coala repositories which use these CIs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh okay will think about it and will mention it in my proposal

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To add a bit more, currently there is one problem with using coala in CIs - it is very hard to locate where the error is. We have to look for errors in the massive log. This is one of the motivations why we need to report results to CIs.

Appveyor.
- Add infrastructure (CI, doc) to coala-json repository, where most part of
this project will probably reside.

##### CODING PHASE 2

- Extend the tool to support different formats, including Checkstyle format,
TAP, Clover.
- Test and add doc regarding how to upload coala results into Jenkins and
Phabricator.
- Import converters into coala main repository to improve coala output mode.

##### CODING PHASE 3

- Provide various linters with a solid set of classes and output formats for
their CLI.
- Handle JSON/xml reports from linters before feeding into linter bears.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Students should investigate coala/coala#5910 a bit.
That might be out of the scope of this project, but it would be good to mention in the proposal.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes i was also wondering for the issue might be helpful for us