This paper describes an empirical study where the focus was on discovering differences and similarities in students working on development of social applications versus students working on development of games using the same Android development platform. In 2010-2011, students attending the software architecture course at the Norwegian University of Science and Technology (NTNU) could choose between four types of projects. Independently of the chosen type of project, all students had to go through the same phases, produce the same documents based on the same templates, and follow exactly the same process. This study focuses on one of projects—Android project, to see how much the application domain affects the course project independently of the chosen technology. Our results revealed some positive effects for the students doing game development compared to social application development to learn software architecture, like motivated to work with games, a better focus on quality attributes such as modifiability and testability during the development, production of software architectures of higher complexity, and more productive coding working for the project. However, we did not find significant differences in awarded grade between students choosing the two different domains.
Computer games and video games have become very popular for children and youths and play a prominent role in the culture of young people [
Besides an abundant appearance of games in young students life, game development technology has matured and become more advanced [
In this context, not only can games be used for learning but also the game development tools can be used for studying relevant topics within computer science (CS), software engineering (SE), and game programming through motivating assignments. Generally, games can be integrated in education in three ways [
This paper focuses on an evaluation where we wanted to discover similarities and differences between making students learn software architecture through game development versus social application development (e.g., weather Forecast, chatting software) using the Android platform. The motivation for bringing game development into a CS or SE course is to exploit the students’ fascination for games and game development to stimulate them to work more and better with course material through the project.
This section describes the research context and previous results about using GDBL method in software engineering field.
The earliest similar application of learning by programming in a game-like environment was in early 1970s. The Logo [
These studies indicate that making games is motivating and develops storytelling as well as technical programming skills. The nature of the task of making games is slightly different in purpose-built environments and the balance of the roles assumed by the learner shifts accordingly. More recent game programming toolkits tend to have a stronger visual aspect than Logo, either in the sense that they enable designers to easily create graphical games or because they have a visual programming language, or both. This shifts the emphasis away from low-level programming, enabling learners to focus on the other roles as designers or writers. Thus, we investigate how GDFs are used in education through an experiment study and explore the evolution of the traditional lecture to be dynamic, collaborative, and attractive to the students under current technology-rich environment. However, this assertion needs to be further supported by relevant theory, application experiences, evaluation results, and empirical evidence. This is one motivation for sharing our experiences and empirical results in field of GDBL on using Android in a software architecture course.
The software architecture course at Norwegian University of Science and Technology (NTNU) (course code TDT4240) is taught in a different way than at most other universities, as the students also have to implement their designed architecture in a project. The motivation for doing so is to make the students understand the relationship between the architecture and the implementation and to be able to perform a real evaluation of whether the architecture and the resulting implementation fulfill the quality requirements specified for the application. The architecture project in the course has similarities with projects in other software engineering courses, but everything in the project is carried out from a software architecture perspective. Throughout the project, the students have to use software architecture techniques, methods, and tools to succeed according to the specified project.
The software architecture project consists of the following phases. COTS (Commercial Off-the-Shelf) exercise: learn the technology to be used through developing a simple game. Design pattern: learn how to use and apply design pattern by making changes in an existing system. Requirements and architecture: list functional and quality requirements and design the software architecture for a game. Architecture evaluation: use the Architecture Tradeoff Analysis Method (ATAM) [ Implementation: do a detailed design and implement the game based on the created architecture and on the changes from the evaluation. Project evaluation: evaluate the project as a whole using a Postmortem Analysis (PMA) method [
In the first two phases of the project, the students work on their own or in pairs. For phases 3–6, the students work in self-selected teams of 4-5 students. Meantime, students have one fixed primary assigned quality attribute to focus on during the project. For the secondary quality attribute, students can choose the quality attribute they like. The students spend most time in the implementation phase (six weeks), and they are also encouraged to start the implementation in earlier phases to test their architectural choices (incremental development). During the implementation phase, the students continually extend, refine, and evolve the software architecture through several iterations.
Previously, the goal of the project has been to develop a robot controller for the WSU Khepera robot simulator (Robot) in Java [
Integrating our experiences on running of game project in software architecture course in 2008, we conducted a new option to add one more COTS-Android in software architecture course project during 2010-2011. The students could now in addition to the Java Robot project and the XNA Game project choose to develop a social application or a game in Android. Independently of the COTS and the domain chosen, the students had to focus on the same software architecture issues during the project and follow the same templates. The introduction of game and social Android projects allowed us to compare how the domain the students work on in the project affects the learning and the project experiences independently of the COTS. A detailed description was in following sections.
This section describes the research method to get the relevant data for our experiment of using Android development in software architecture projects.
This paper focuses on using the same COTS but with different development domains to investigate whether the different domains produce different output. In our previous research, the effectiveness of GDBL conclusion was based on the different COTS-Robot and XNA. This paper excludes game developed in XNA and robot controller developed in Java and only focuses on the Android platform and development of social application versus game application. Our evaluation covers five topics: distribution of chosen domain, students’ perception of the project, project deliveries and code quality and complexity, students’ effort, and awarded project grades.
The comparison of the social and game project should help to discover the differences and reveal the effects of introducing a project on the Android platform. This evaluation is a quasiexperiment, not a controlled experiment. The research method used is based on the Goal, Question Metrics (GQM) approach [
GQM table.
Analyze | Software development project | |||
For the purpose of | Comparing social application versus game application domain on same COTS | |||
Goal | With respect to | Difference and effectiveness of two domains of the projects | ||
From the point of view of | Researcher and educator | |||
In context of | Students in software architecture course | |||
| ||||
Questions | Q1: Are there any differences in how the students perceive the project for students choosing an Android game project versus students choosing an Android social project? | Q2: Are there any differences in the software architectures designed by students doing an Android game project versus students doing an Android social project? | Q3: Are there any differences in the implementation effort in the project by students doing an Android game project versus students doing an Android social project? | Q4: Are there any differences in the performance of students doing an Android game project versus students doing an Android social project? |
| ||||
Metric | M1: Number of students choosing game project versus social project. | M3: Project reports | M4: Source code files | M6: Project score |
M2: Questionnaire survey with 5-Level Likert Scale: Strong disagree (1), Disagree (2), Neutral (3), Agree (4), Strong Agree (5) | M5: Time spent |
When students start the project and follow the projects phases, they should report the time they spend on each phase of the project. The first two phases allow the students individually or in pairs to get familiar with the COTS and architectural and design patterns. The main work of the project is carried out in the phases 3–5 and includes requirement specification, architectural design, architectural evaluation, implementation, and testing. The students produce a delivery for each phase, which is evaluated by the course staff, and feedback is given to improve before the final delivery. At the end of phase 5, the students will produce a final delivery, which is evaluated and graded by the course staff. After completing phase 5, the students have to answer a questionnaire that focuses on how the students perceive the project. In phases 6, the students must carry out a postmortem analysis of their project as a whole to reflect on their successes and their challenges.
In 2010 and 2011, the students could choose to do the project using three COTS: Robot (Java), XNA (C#), and Android (Java). The students’ selection of COTS is shown in Figure
Distribution of selection of type of software architecture projects.
The statistics of Figure
In the first phase of the project, the students were asked to fill in a questionnaire on the reasons to choose the COTS and domain. The top reasons list was: (1) programming reason (familiar with Java or C#) (70.7%), (2) to learn about the COTS (Robot, XNA, Android) (59.5%), (3) games motivation or amusement reasons (40.1%), (4) social application motivation (39.5%), (5) to learn about the domain (robot, game, social) (34.2%), (6) hardware motivation, running games on Android phone, Zuneplayer (33%), and (7) make games for Android Market or XNA club (24.5%). From above data, we found that the game domain has advantages in drawing students’ attention and its attractive peripherals, like hardware or software markets, and so does android social domain. This was not the case for the Robot domain.
The following subsections focus on the analysis of whether the domain game versus social causes any significant different output in the following four aspects: (1) students perception of the project, (2) the design complexity of software architectures, (3) students’ implementation effort in the project, and (4) students’ score in projects.
A project survey was conducted one week after the students completed their software architecture project. The goal of this survey was to reveal possible differences in the students’ perception of the project between teams working with social projects versus teams working with game projects on the same COTS—the Android platform. Statements in the survey made the students reflect on how the project helped them to learn software architecture.
The hypothesis defined for this survey was the following. : There is no difference in how students doing game project and social project on the same COTS-Android perceive the software architecture project.
To test hypothesis we used Kruskal-Wallis Test [
Wilcoxon Test of the statements PS1-PS11.
Statement | COTS | Average | Median | Standard deviation |
|
---|---|---|---|---|---|
PS1: I found it difficult to evaluate the other group’s architecture in the ATAM? | Game | 3.45 | 4 | 1.06 | 0.178 |
Social | 3.77 | 4 | 0.91 | ||
PS2: I found it difficult to focus on our assigned quality attributes | Game | 3.05 | 3 | 1.09 |
|
Social | 3.57 | 4 | 0.85 | ||
PS3: I found it easy to integrate known architectural or design patterns | Game | 3.21 | 3 | 0.93 | 0.332 |
Social | 2.94 | 3 | 1.03 | ||
PS4: I spent more time on technical matters than on architectural matters | Game | 3.71 | 4 | 1.20 | 0.175 |
Social | 4.06 | 4 | 1.03 | ||
PS5: I have learned a lot about software architecture during the project | Game | 3.50 | 4 | 0.86 | 0.552 |
Social | 3.31 | 4 | 0.99 | ||
PS6: I would have chosen another project if I could go back in time | Game | 1.13 | 1 | 0.34 | 0.289 |
Social | 1.20 | 1 | 0.41 |
From the test results, the lowest significant difference (
Further, even there is no significant difference for the two other low
Reponses to PS6: would you have chosen the same project if you could go back in time.
Figure
As an overall, the survey reveals one significant difference that students from game projects have a better focus on quality attributes. Statements got low
It is difficult to evaluate software architectures empirically, but we have chosen to do so by comparing the number of design patterns the students used, the number of main modules/classes identified in the logical view of the software architecture, and the number of hierarchical levels in the architecture. We admit that that there are many sources of errors in this comparison, as the two domains are so different. However, the emphasis in this course is on using software design patterns and presenting the different views of the software architecture in sufficient detail with emphasis on the logical view. The empirical data should highlight the differences between the two types of projects if any. The empirical data has been collected by reading through and analyzing the final project reports from 12 game project teams and 16 social project teams.
Table
Number of design patterns used.
Average | Standard deviation | Max | Min | ||
---|---|---|---|---|---|
Design Patterns | Game | 2.67 | 1.92 | 7 | 1 |
Social | 1.56 | 0.73 | 3 | 1 |
Table
Hypothesis tests on number of design patterns used.
Hypothesis | COTS |
|
Median |
|
---|---|---|---|---|
No difference in number of used design patterns | Game | 12 | 2 | 0.111 |
Social | 16 | 1 |
Table
Distribution of usage of design patterns for game and social projects.
Even there is no significant difference, but the low
Two metrics were chosen to indicate the complexity of the software architecture [
Measurement of software architecture complexity.
Numbers of main modules/classes | Number of levels in architecture | |||
---|---|---|---|---|
Game | Social | Game | Social | |
Average | 14 | 9.7 | 3 | 1.75 |
Standard deviation | 4.9 | 6.6 | 0.6 | 0.77 |
Max | 21 | 28 | 4 | 3 |
Min | 7 | 3 | 2 | 1 |
Table
Table
Hypothesis tests on architectural complexity.
Hypothesis | COTS |
|
Median |
|
---|---|---|---|---|
No difference in number of main modules/classes | Game | 12 | 14 | 0.021 |
Social | 16 | 7 | ||
| ||||
No difference in number of levels in architecture | Game | 12 | 3 | 0.000 |
Social | 16 | 2 |
To evaluate the effort of each project that students put into it, two indicators are used as the measurement criteria: (1) time spent on the project and (2) structure and size of project files and number of lines of code.
We have asked students to estimate on how many hours the project teams worked in the software architecture project during the phases 3–5 (core phases of the project). Table
Time spent on the project for each team.
Time per team (hours) | Game | Social |
---|---|---|
Average | 334 | 338 |
Standard deviation | 133.7 | 114.7 |
Max | 520 | 535 |
Min | 110 | 183 |
Based on each team’s time effort, we ran the Kruskal-Wallis Test on the difference on hours spending in the project for each team.
From previous results, there is no statistically significant difference on time spent on the project for game teams and social teams. On contrary, the time spending distribution in both projects is quite similar.
Further, we chose to look at metrics from the implementation to give an estimate on how much was produced during the project. It can give a good indication of the complexity of the software architecture and the resulting implementation of the application [
The following metrics were chosen to compute the effort of the student teams: (1) number of source Files (NoF); (2) number of comments in code (NoC); (3) lines of source code not counting empty lines or comments (LoC).
Table
Table
The results from the Kruskal-Wallis Test indicate that there is no statistically significant difference in LoC between the two types of project. But the low
From the Tables
Hypothesis on hours spending.
Hypothesis | COTS |
|
Median |
|
---|---|---|---|---|
No difference in time spending for each team | Game | 12 | 362 | 0.889 |
Social | 16 | 334 |
Implementation metrics from the architecture projects.
NoF | NoC | LoC | ||||
---|---|---|---|---|---|---|
Game | Social | Game | Social | Game | Social | |
Average | 37 | 24 | 1016 | 536 | 2585 | 1949 |
Standard deviation | 13 | 13 | 807 | 755 | 1172 | 1368 |
Max | 54 | 45 | 2571 | 2886 | 4173 | 5082 |
Min | 15 | 5 | 206 | 37 | 844 | 390 |
Hypothesis tests on project implementation codes.
Hypothesis |
|
Median |
| |
---|---|---|---|---|
No difference in number of lines of code | Game | 12 | 2672 | 0.114 |
Social | 16 | 1523 |
The project score is between 0 and 30 points and takes 30% of the final grade. The project grades intervals are classified as: A: score ≥90%; B: score ≥80% and score <90%; C: score ≥60% and score <80%; D: score ≥50% and score <60%; E: score ≥40% and score <50%; F: score <40% (fail).
In order to investigate if there were any differences in how the group scored (0–30 points) on the project for students that has chosen game and social projects on Android, the Kruskal-Wallis Test was used to test this hypothesis, as we cannot assume a normal population and the sample size of the two groups is different. Table
Kruskal-Wallis Test on different in project score.
Hypothesis | COTS |
|
Median |
|
---|---|---|---|---|
No difference in project score groups get from doing Game versus Social project | Game | 44 | 26 | 0.997 |
Social | 58 | 26 |
There is no significant difference in the project score using same COTS for development. We run the social project in 2010 and game project in 2011 separately. The project implementation requirements and templates are keeping the same from phase 3 to 6 in two years, and evaluation process and persons are the same; we can identify that students accomplished both projects under the same conditions. It reflects the difficulty could be similar. So, we only make a conclusion on the project score has no significant difference. In order to get an overview of the scores, Figure
Grades distribution on project.
We now turn to what are considered to be the most important threats to the validity of this evaluation.
The internal validity of an experiment concerns “the validity of inferences about whether observed covariation between A (the presumed treatment) and B (the presumed outcome) reflects a causal relationship from A to B as those variables were manipulated or measured” [
There are two main internal validity threats to this evaluation. The first internal threat is that the sample of two groups used in the evaluation is not randomized. The students were allowed to choose either an Android game or an Android social project. We do not believe that one specific type of student chose one project over the other, thus harming the evaluation results. The second internal threat is if there were any differences, how the students had to perform the project independently of the domain chosen. Independently of doing a social or a game project, the students had to go through exactly the same phases in the project and deliver exactly the same documents based on the same document templates in both 2010 and 2011. We have identified one difference in how the two types of projects were carried out. The 1-2 phases of the project phase were different for the game and social projects students. These two phases are not a part of inclusive data and material used to evaluate the project. We do not believe that these differences have had any major impact in the way the students did or performed in their projects since it is the preparation phases, that we noticed and excluded of them.
Construct validity concerns the degree to which inferences are warranted, from (1) the observed persons, settings, and cause- and effect-operations included in a study to (2) the constructs that these instances might represent. The question, therefore, is whether the sampling particulars of a study can be defended as measures of general constructs [
In the evaluation of using Android project in a software architecture course our research goal was to investigate the difference and similarity of game project and social project on Android platform. The GQM approach was chosen to detail this goal into four research questions with supporting metrics. In order to give answers to these four research questions the data sources and metrics available from our software architecture course were chosen. It cannot be claimed that the selected data sources and metrics in our evaluation give evidence for all the conclusions, but they are all strong indicators contributing to a picture that describes the differences between the two project types. Through the evaluation we have used various methods for comparing the results. The choice of methods is based on the best way of describing and visualizing the differences between the two groups using the available data.
The issue of external validity concerns whether a causal relationship holds (1) for variations in persons, settings, treatments, and outcomes that were in the experiment and (2) for persons, settings, treatments, and outcomes that were not in the experiment [
The results reported in this paper are most relevant for other teachers thinking of introducing game projects as a part of their software architecture course. Further, the results are also relevant for teachers that want to introduce game projects in SE and CS courses, as many of these courses have similar characteristics. A limitation of this study is that the subjects in the evaluation are CS or SE students who have completed their first three years. It is not evident that the results are valid for students without any or less than three-year background in CS or SE.
Based on our previous experiment of using XNA and current experiment of using Android in software architecture, we found game motivation and surrounding interesting peripherals are one of the most attractive factors. Besides the introduction of a new COTS-Android in a software architecture course, the goal of this paper is to identify the difference output of same COTS and get evaluation result to answer the four research questions.
The first research question asked is if there are any differences in how students choosing Android game versus Android social projects perceived the software architecture project (RQ1). The statistically significant finding is that social project students found it more difficult to focus on the assigned quality attributes than game project (
The second research question asked is if there are any differences in how students choosing Android game versus social projects designed their software architectures (RQ2). Even the analysis of the project reports concludes that no significant difference on the used design patterns, but the low
The third research question asked is if there were any differences in the effort the students put into the project when they worked with an Android game or an Android social project (RQ3). The results show that in similar time spending, teams working with game projects produced on average almost 133% as much code as teams working with Android social projects, and game project students had customs to make twice detailed comments on the codes and organized codes into more files than social projects students.
The fourth and final research question asked is if there are any differences in the performance of students doing a Game project versus students doing a Social project (RQ4). The comparison of the two types of projects showed that there was no statistically significant difference in the project.
According to the previous conclusion and compared with previous research on XNA and Robot project used in software architecture course [
Referring to Android COTS specifically, the main differences from Android game projects could be used as an interesting and effective tool in software architecture teaching aspect to motivate students on design of complex architecture with applying more patterns and more productive coding work than Android social projects. Further, compared to XNA and Robot simulator, Android is an attractive platform to the students from the students’ survey, that encourages us to conduct more practices on improvement of using Android as a development tool in software engineering practices and inspires us the possibility to bring more choices, like iPhone SDK into COTS domains.