[badreddin] Decade of Software Design and Modeling

[grammarware]

Submitted by @rahadiit to https://github.com/modelsconf2018/artifact-evaluation/tree/master/badreddin

Paper: https://figshare.com/s/d4d1de231d267cbd038d

[mherzberg]

Artefact summary

The submitted artefact accompanies the paper "A Decade of Software Design and
Modeling: A Survey to Uncover Trends of the Practice", which presents a survey
that has been conducted twice: in 2007 and 2017. The goal was to investigate how
the industry usage of software design and modeling techniques changed within
this time frame. The artefact consists of a technical report that contains all
18 questions along with detailed statistics about the given answers.

Consistency with the paper

For each table in the paper, I was able to find the corresponding question and
table in the TR and see where the values come from. The table format is
similar, and the wording of the answers in every table pair matches, which was
helpful. Still, I see a few ways in which the link could be strengthened:

• It would have been helpful to have the question number in the tables in the
  paper. This would provide a clear link.
• The semantics of the bolding and underlining of numbers in the paper means
  that these values are statistically significant, but what do the bold numbers
  in the TR mean?
• In the paper, the number of decimal places varies, e.g. 1.2, but 4 instead of
  4.0. Especially in the '% Never & Sometimes' column in all tables in the
  paper, it first seems that the numbers have been rounded differently, although
  they do indeed add to up round numbers.
• Table 11/Question 9: Why does 'Java' not also include the numbers of 'J2EE'?
• Table 12/Question 13: It is not clear to me how the data in both tables relate
  to each other. I was not able to find any reference or explaining text to the
  table in the paper.

Completeness of artefact

The artefact contains all 18 questions that have been used, along with the
possible answers. The recorded answers from the survey participants are
presented in aggregate form similar to the paper. The TR additionally contains,
for each possible answer, the number of recorded answers, standard deviation,
and the percentage of participants who answered with 1 through 5. However, a few
questions and comments remain:

• What does 'Data from the entire sample' in phase 1 mean? Why is N
  significantly (up to two thirds) smaller in phase 2, compared with 115, the
  number of complete responses for phase 2?
• In the paper, the authors claim that there is the 'complete raw data' in the
  TR. While it is certainly more detailed, I think that wording raises false
  expectations.
• There are 18 questions in the TR, but in the paper I find references to 'In
  total, the survey included 152 questions.'. What is meant by that?
• I was not able to find the answers to (open ended) question 17 in the TR, even
  though they appear in the paper and are mentioned to be contained in the TR
  ('The complete raw data, free text, and summary data is included as an
  additional artifact with this paper.').

Artefact documentation

The general structure of the TR is clear and there exists introductory text that
explains the context of each question. I only have a few minor suggestions
regarding readability:

• It would be easier to get more context to the recorded answers if the order of
  the answers was the same in the text and table.
• The column headings could be clearer. For example, many columns contain the
  aggregate result for two questions '(1 + 2)', but are labeled '% Sometimes'
  instead of '%Never + %Sometimes' as done in the paper.
• Phase 1 and Phase 2 would be more easily comparable if the results were
  grouped the same way. For example, for phase 1, the results for 'Sometimes
  (2)' are not given directly, but are aggregated with 'Never (1)'.
• Overall, the table formatting seems off in a number of places, especially the
  alignment of some of the rows.
• Question 6: The result could be made clearer. I expected 6 columns, one for
  each possible answer.

Ease of reuse

The TR contains aggregate data for each answer and lacks free text answers,
which I suspect might be disappointing for someone hoping to find the 'raw data'
and 'free text' to conduct their own analysis.

On the other hand, it seems very well possible to reproduce the survey, as
enough information regarding context, questions, and answers is given.

[AEReview]

** Summary
The paper presents the results of of a survey on software design and modeling practices and compare the results with the results from the same survey done ten years ago.
The survey consists in 152 questions grouped into 5 topics.
In particular, the survey focuses on the characterization of design and modeling activities, their usage and importance in the software lifecyle, the tool support and perceived efficacy.

With regard to the artifact, the latter is a technical report containing a more precise description of the surveys questions, while providing additional insights on the results.
It does not provide access to a tool or examples since it is not in the scope of the paper.

Though insightful and a good complement, I would not recommend the artifact, since it is not a « technical » artifact.

** Consistent with the paper **
The artifact material is consistent with the paper and proposed a more detailed picture of the survey.

** As complete as possible **
The artifact does not provide an access to (anonymized) raw material. However, the demographics and context gf the study is impossible to reproduce exactly and only trends are interesting to observe.

** Well-documented **
The flow of the technical report is clear and follows the paper. I would however recommend to find an easier and consistent (homogeneous) way to wrap and compare the results from the two phases, which are presented separately.

** Easy to (re)use **
Since we have access to the questions, the questionnaire per se is reproducible, which is essential for pursuing the study in the future and continue to analyze the trends.

[jdirocco]

The artifacts provided are part of the paper which presents a survey that has been conducted twice: in 2007 and 2017. The goal of the study is to uncover trends in the practice of software design and the adoption trends of design languages such as UML. The survey consists of 18 questions grouped into 5 topics. Most of the questions involved several sub-questions answered using 5-point Likert scales.
The artifact contribution consists of a technical report that contains all questions and statics about the answers.

(1) Is the artifact consistent with the paper?
The artifacts are consistent with the paper since the supporting report contributes to the results presented in the survey. The authors list each question presented in the questionnaire and statics about the answers.
.

(2) Is the artifact as complete as possible?
The artifact contains all 18 questions and corresponding aggregated answers. However, the artifact does not provide an access to raw data. In the paper, the authors claim that surveys were conducted online. But it is not clear how they performed the questionnaire? By means of a web form or Email? Moreover, the questionnaire template enriches this artifact.

(3) Is the artifact well-documented?
The technical report is well written and very complete. Moreover, the documentation provides all that I need to evaluate this artifact.

(4) Is the artifact easy to (re)use?
The technical report lists all questions. Therefore, the survey is reproducible. Raw data permits to investigate potential statistical relevance.

[grammarware]

Dear @rahadiit,

Based on all the comments and the reviews provided by the members of the Artifact Evaluation Committee of MoDELS 2018, we have reached the conclusion that this artifact conforms to the expectations and is hereby approved. Please use the badge instructions page to add the badge of approval to your article, and add the link to the FigShare entry to the camera ready version of the paper.

Thank you very much for putting extra effort into the preparation and finalising of the artifact. If any of the comments above are still not addressed, please try to accommodate them before the conference.

Please do not forget to publish your report on FigShare — as of now it is shared privately and thus does not have a DOI.