How Valid are the Results of the Gamekins Experiment?

Abstract and 1 Introduction

2.1 Software Testing

2.2 Gamification of Software Testing

3 Gamifying Continuous Integration and 3.1 Challenges in Teaching Software Testing

3.2 Gamification Elements of Gamekins

3.3 Gamified Elements and the Testing Curriculum

4 Experiment Setup and 4.1 Software Testing Course

4.2 Integration of Gamekins and 4.3 Participants

4.4 Data Analysis

4.5 Threats to Validity

5.1 RQ1: How did the students use Gamekins during the course?

5.2 RQ2: What testing behavior did the students exhibit?

5.3 RQ3: How did the students perceive the integration of Gamekins into their projects?

6 Related Work

7 Conclusions, Acknowledgments, and References

Threats to external validity may arise due to the limited number of participants, which limits generalizability. It is important to note that our software testing course is a mandatory course at the University of Passau, meaning that every student in their Bachelor’s program is required to take it. This could potentially impact the behavior and performance of the students compared to those in nonmandatory courses. Additionally, the course is specifically designed for students in their fifth semester who possess a certain level of computer science knowledge. Students in different semesters or with varying levels of knowledge may exhibit different behaviors

\

\ and outcomes when using Gamekins. Moreover, students from different universities, countries, or degree programs may also exhibit different behaviors and outcomes.

\ Threats to internal validity could potentially arise from errors in our data collection infrastructure, plugin, and the integration of Gamekins into the course. However, we took measures to mitigate these threats by thoroughly testing Gamekins and immediately addressing any issues arising during the course. Another potential threat to internal validity is the possibility of differences in knowledge levels between the students in 2019 and 2022. To minimize this threat, we conducted exercise sessions during the projects where all students had the opportunity to ask questions and clarify any uncertainties regarding the tasks. Additionally, there is a possibility that students may have collaborated instead of completing the tasks individually. However, we conducted a plagiarism check on their code and did not find any irregularities.

\ Threats to construct validity may arise due to the mandatory use of Gamekins and the grading associated with it, which could potentially influence their behavior and perception of the tool.

\

:::info This paper is available on arxiv under CC BY-SA 4.0 DEED license.

:::

:::info Authors:

(1) Philipp Straubinger, University of Passau, Germany;

(2) Gordon Fraser, University of Passau, Germany.

:::

\