ICSE-SEET '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering Education and Training

Full Citation in the ACM Digital Library

SESSION: Novel approaches in SE education 1

Using public and free platform-as-a-service (PaaS) based lightweight projects for software architecture education

Background: Software architecture is a crucial while significantly challenging topic in the computer science curriculum. Although "learning by doing" is widely advocated to address this topic's abstract and fuzzy nature, educators are still facing various difficulties in practice, especially including students' vicious circle of inexperience and the mental model dilemma in experiential learning.

Aims: To help break the aforementioned vicious circle and mental model dilemma, this work aims to investigate our educational strategy of using lightweight projects with public and free PaaS resources (1) to help students accumulate architectural experience from the early stage and (2) to facilitate strengthening students' fundamental architecture knowledge.

Method: To collect more empirical evidence, we conducted action research on our educational strategy across three undergraduate-curriculum courses for two years. In particular, we employed the Technology Acceptance Model (TAM) to validate how well students received such an educational strategy.

Results: The students involved in the relevant coursework generally gave positive feedback with respect to learning theoretical concepts and a set of architectural patterns. Meanwhile, we also observed diverse learning curves for utilizing PaaS resources.

Conclusions: Although there still exist PaaS-related limits, our educational strategy can foster students' experiential learning and collaborative learning in fundamental architecture courses. We believe that there are also potentials to extend our work to broader and advanced courses of software architecture.

An interactive learning method to engage students in modeling

Modeling is an important skill in software engineering. However, it is often not tangible for students and not appreciated. Students prefer coding because they receive immediate feedback from the compiler. Engaging students in modeling is difficult, especially in large introductory courses.

We have developed an interactive learning method for modeling which is based on an easy to use online editor. Students learn modeling in guided tutorials in the lecture right after the theory is introduced and deepen their modeling skills in group work and homework exercises. This learning method was applied in a large introductory course with more than 1000 students.

An empirical evaluation of the method demonstrated that the students' learning outcome in modeling improved significantly by up to 87 %. Students are motivated to use models in their future projects and understand how to approach problems with models. The use of interactive models in programming exercises improves their understanding of the taught concepts.

VeriSIM: a learning environment for comprehending class and sequence diagrams using design tracing

In this paper, we describe the design tracing strategy, which enables students to comprehend class and sequence diagrams by tracing different scenarios. In design tracing, for a given scenario, students identify relevant variables from the class diagram, relevant events from the sequence diagram and trace the flow of these data variables and events by constructing a state diagram. We have developed a web-based learning environment - VeriSIM, which trains students to apply the design tracing strategy. We conducted a study where 86 final-year undergraduates interacted with VeriSIM. Findings from the pre-test and post-test show that students are able to trace a given scenario by identifying relevant variables and events and are able to simulate change of state for these variables. A focus-group interview was also conducted with 13 participants in order to understand their perception of the usefulness of design tracing. A thematic analysis of the focus-group interview showed that students perceived design tracing helped them understand the relationship between different diagrams and identify different scenarios in the design. Interaction with VeriSIM also helped students understand the usefulness of creating class and sequence diagrams. These results show that design tracing can be a useful pedagogy to help learners form an integrated and correct understanding of class and sequence design diagrams.

Turn up the heat!: using heat maps to visualize suspicious code to help students successfully complete programming problems faster

Automated grading systems provide feedback to students in a variety of ways, but they typically focus on identifying incorrect program behaviors. Such systems provide notices of test case failures or runtime errors, but without debugging skills, students often become frustrated when they don't know where to start addressing these defects. Borrowing from work in software engineering research related to automated defect location, we leverage previous research describes using statistical fault localization (SFL) techniques to identify the probable locations of defects in student coding assignments. The goal is to use these SFL techniques to provide a scaffold for students, to direct their debugging efforts without giving too much guidance, and thus minimizing the learning associated with investigating the defects. After determining the "suspiciousness" for each line of code involved in the defect, we create a "heat map" visualization overlay onto their source code of the "suspiciousness" scores to visually guide a student's attention to parts of their code that are most likely to contain problems.

This paper describes the results of an analysis comparing two semesters of CS 1114: Introduction to Software Design (CS1) students, one which had access to the heat map feedback (Fall 2017: n~170), and one that did not (Fall 2015: n~270). The results show that when the heat maps were available, students found it easier to make improvements on their code from submission to submission, as well as spending less time overall achieving the maximum score on the instructor testing assessments. In fact, while we were optimistic in the impact of the heat map feedback, the results we observed were unexpectedly strong. To address this concern, we conducted an additional study, using student submissions from our Spring 2019 CS 1 students (n~230). The results of the second analysis confirmed the results of the first.

Teaching software engineering for AI-enabled systems

Software engineers have significant expertise to offer when building intelligent systems, drawing on decades of experience and methods for building systems that are scalable, responsive and robust, even when built on unreliable components. Systems with artificial-intelligence or machine-learning (ML) components raise new challenges and require careful engineering. We designed a new course to teach software-engineering skills to students with a background in ML. We specifically go beyond traditional ML courses that teach modeling techniques under artificial conditions and focus, in lecture and assignments, on realism with large and changing datasets, robust and evolvable infrastructure, and purposeful requirements engineering that considers ethics and fairness as well. We describe the course and our infrastructure and share experience and all material from teaching the course for the first time.

SESSION: Empirical studies of SE education

Presenting and evaluating the impact of experiential learning in computing accessibility education

Studies indicate that much of the software created today is not accessible to all users, indicating that developers don't see the need to devote sufficient resources to creating accessible software. Compounding this problem, there is a lack of robust, easily adoptable educational accessibility material available to instructors for inclusion in their curricula. To address these issues, we have created five Accessibility Learning Labs (ALL) using an experiential learning structure. The labs are designed to educate and create awareness of accessibility needs in computing. The labs enable easy classroom integration by providing instructors with complete educational materials including lecture slides, activities, and quizzes. The labs are hosted on our servers and require only a browser to be utilized.

To demonstrate the benefit of our material and the potential benefits of our experiential lab format with empathy-creating material, we conducted a study involving 276 students in ten sections of an introductory computing course. Our findings include: (I) The demonstrated potential of the proposed experiential learning format and labs are effective in motivating and educating students about the importance of accessibility (II) The labs are effective in informing students about foundational accessibility topics (III) Empathy-creating material is demonstrated to be a beneficial component in computing accessibility education, supporting students in placing a higher value on the importance of creating accessible software. Created labs and project materials are publicly available on the project website: http://all.rit.edu

An empirical study of teaching qualities of popular computer science and software engineering instructors using RateMyProfessor.com data

The employment opportunity for Computer Science (CS), Information Technology and Software Engineering and Development (SE) related occupations is projected to grow much faster than the average of all other occupations. Therefore, increase in student enrollment, retention and graduation rate is becoming very important, so is the need for effective teaching in these subjects. Many universities commonly use formal, institutional Student Evaluation of Teaching (SET) systems to measure the teaching effectiveness. After each semester, through SET, students provide feedback and comments for their courses and instructors. However, evaluations are private and only a handful people have access to these. Therefore, these evaluations cannot be utilized to create a common understanding of the students' expectations, perspective, desired characteristics of the courses and instructors. On the other hand, third party online platforms like RateMyProfessor.com (RMP) are public, solicit anonymous student feedback and host tremendous amount of data about the instructors and their courses. These platforms are also popular among students. We mined and analyzed the RMP data for some research questions, e.g.: What are the common characteristics of the popular CS instructors? How different are they for the SE instructors? Are there any examples of special characteristics, tools and techniques popular CS instructors use? We captured and analyzed more than 9,000 students' comments for over 300 CS instructors for the top 20 universities in the U.S. and Canada. The paper contributes by presenting the findings for the research questions and making the data and the scripts available for public use for future research.

Authentic individual assessment for team-based software engineering projects

In order to give students an authentic learning experience and better prepare them for the life-long learning required in contemporary workplaces, educational institutions increasingly use project-based learning in teams. However, this poses the challenge of developing authentic and equitable assessment criteria that reflect individual contributions in a team-work setting without jeopardizing project outcomes. We present a novel and innovative portfolio-based assessment framework that focuses on qualitative outcomes and ensures that the level of achievement reflects a student's competency across each of the defined learning dimensions, including professional behaviour, teamwork, process and relevant contributions towards project deliverables. In this paper, we present the main motivation behind devising and introducing the framework and also reflect on the educational outcomes and challenges of implementing the framework in the context of two final year Software Engineering project units.

Sidekicks and superheroes: a look into student reasoning about concurrency with threads versus actors

When computations consist of different processes or threads that may execute at the same time or in an interleaved manner, we say the computation is concurrent or that it exhibits concurrency. University students often struggle to reason about concurrency. Researchers have presented students with natural language descriptions of scenarios involving concurrency and have evaluated students' natural language responses to explore their prior knowledge and to determine characteristics of student reasoning about concurrency. However, natural language responses are necessarily abstract and pertain to the design stage of software development, while studies in which students are asked to implement solutions provide a more concrete view of student reasoning and reveal difficulties that arise when putting design into practice. In this work we build upon prior work and ask: What are common and problematic features in student approaches to solving concurrency-related problems? How do these features differ from the design phase to the implementation phase of the software development life cycle? In this study we asked students enrolled in a jointly offered upper division undergraduate / MS level course on "Programming with Concurrency" to implement a solution to the same problem (a sidekick/superhero version of the party matching problem) using two different approaches: Threads-based in Java, and Actors-based in Scala. We performed qualitative feature analysis of their implementations and their written reflections to better understand student reasoning when programming concurrent solutions. The feature of additional complexity was found in our earlier natural language study in which students described their designs. Additional complexity was also represented in student implementations in this study and adversely impacted student success as their programs became cumbersome and hard to trace. Student reflections provided insight into their resistance to modeling prior to implementation, and indicated that they found formal modeling techniques difficult. These findings suggest that lighter weight interventions that address student difficulties earlier in the development process may be more palatable to students and thus have substantial impact.

What prevents finnish women from applying to software engineering roles?: a preliminary analysis of survey data

Finland is considered a country with a good track record in gender equality. Whilst statistics support the notion that Finland is performing well compared to many other countries in terms of workplace equality, there are still many areas for improvement. This paper focuses on the problems that some women face in obtaining software engineering roles. We report a preliminary analysis of survey data from 252 respondents. These are mainly women who have shown an interest in gaining programming roles by joining the Mimmit koodaa initiative, which aims to increase equality and diversity within the software industry. The survey sought to understand what early experiences may influence later career choices and feelings of efficacy and confidence needed to pursue technology-related careers. These initial findings reveal that women's feelings of computing self-efficacy and attitudes towards software engineering are shaped by early experiences. More negative experiences decrease the likelihood of working in software engineering roles in the future, despite expressing an interest in the field.

SESSION: Industry-relevant teaching

We should teach our students what industry doesn't want

This experience report draws on over 20 years of teaching introductory ethics courses to software students. It describes a lightweight introductory quiz that has been administered to many cohorts and highlights some unexpected and persistent patterns in the quiz results. Lessons are drawn relating to the education of future software professionals, specifically on the differing professional obligations of educators and practitioners. It is concluded that software engineering education must go beyond industry needs and include topics and values that industry may not even want us to teach.

Understanding devops education with grounded theory

DevOps stands for Development-Operations. It arises from the IT industry as a movement aligning development and operations teams. DevOps is broadly recognized as an IT standard, and there is high demand for DevOps practitioners in industry. Since ACM & IEEE suggest that undergraduate computer science curricula "must adequately prepare [students] for the workforce", we studied whether undergraduates acquired adequate DevOps skills to fulfill the demand for DevOps practitioners in industry. We employed Grounded Theory (GT), a social science qualitative research methodology, to study DevOps education from academic and industrial perspectives. In academia, academics were not motivated to learn or adopt DevOps, and we did not find strong evidence of academics teaching DevOps. Academics need incentives to adopt DevOps, in order to stimulate interest in teaching DevOps. In industry, DevOps practitioners lack clearly defined roles and responsibilities, for the DevOps topic is diverse and growing too fast. Therefore, practitioners can only learn DevOps through hands-on working experience. As a result, academic institutions should provide fundamental DevOps education (in culture, procedure, and technology) to prepare students for their future DevOps advancement in industry. Based on our findings, we proposed five groups of future studies to advance DevOps education in academia.

Adopting industry agile practices in large-scale capstone education

This paper presents the practice and experience in adopting an agile organizational model for a final-year capstone program in Software Engineering. The model developed is motivated by having real (and developing) software artifacts with incrementally changing team members working on a product-line. This in turn results in more sophisticated capstone student-project outcomes. The model proposed supports student mentoring and promotes, through its internal organization, leadership and personal responsibility. The students are supported by professional software engineers, up-skilling workshops, and academic supervisors who act as a personalized reporting and grading point for the team. The academic supervisors are themselves supported by a tribe leader, a faculty member who assumes overall responsibility for a product-line, and who acts as a report to an external industry client/sponsor. This paper describes the motivation for the capstone model, its adoption, and some preliminary observations.

Addressing the double challenge of learning and teaching enterprise technologies through peer teaching

Students face difficulties when transitioning from introductory programming to using more complex enterprise technologies, such as libraries, software frameworks and development kits. Whilst much literature has been devoted to how to teach introductory programming effectively, less attention is devoted towards managing the transition towards more complex technologies. This work presents the design, experience and evaluation of a module that engages students with a range of enterprise technologies. The module uses peer teaching to transfer the responsibility to students for teaching each other about the technologies. As such, this reduces the need for the teacher to invest time in preparing materials, and it is feasible to cover more technologies depending upon the number of teaching teams. The evaluation, conducted on a cohort of 34 students studying six enterprise technologies over the course of one week, revealed overwhelmingly positive experiences with this approach. For the teacher, effort for preparation and delivery was minimal, and the feedback on the module was highly positive.

Characterizing the pedagogical benefits of adaptive feedback for compilation errors by novice programmers

Can automated adaptive feedback for correcting erroneous programs help novice programmers learn to code better? In a large-scale experiment, we compare student performance when tutored by human tutors, and when receiving automated adaptive feedback. The automated feedback was designed using one of two well-known instructional principles: (i) presenting the correct solution for the immediate problem, or (ii) presenting generated examples or analogies that guide towards the correct solution. We report empirical results from a large-scale (N = 480, 10,000 + person hour) experiment assessing the efficacy of these automated compilation-error feedback tools. Using the survival analysis on error rates of students measured over seven weeks, we found that automated feedback allows students to resolve errors in their code more efficiently than students receiving manual feedback. However, we also found that this advantage is primarily logistical and not conceptual; the performance benefit seen during lab assignments disappeared during exams wherein feedback of any kind was withdrawn. We further found that the performance advantage of automated feedback over human tutors increases with problem complexity, and that feedback via example and specific repair have distinct, non-overlapping relative advantages for different categories of programming errors. Our results offer a clear and granular delimitation of the pedagogical benefits of automated feedback in teaching programming to novices.

SESSION: Novel approaches in SE education 2

Innovation and active learning for training mobile app developers

The mobile market is growing every year, creating a demand for mission-critical and innovative enterprises, and personal applications. This demand implies the need for skills training necessary for students to enter the professional development market. The Innovation, Development and Teaching in Complex Software Environments approach (IDEAS) was developed for the training of mobile developers to meet this need, through an innovative process for student selection, training through the active methodology Challenge Based Learning, software production, and startups creation. The approach was created through action-research applied during two cycles of a real project in a university environment, where 398 students applied for the first cycle, of which 110 were selected, and 900 applied for the second cycle, of which 100 were selected. We undertook evaluations with students and teachers through periodic reflections, questionnaires, and interviews at the end of each cycle. Some of the positive results included the prominence of 125 professional-looking apps that were finalized and published, report of dropout reduction and disapproval in the programming courses and experience in entrepreneurship and pre-acceleration startup program.

Towards an open repository for teaching software modeling applying active learning strategies

Modeling is a core topic in Software Engineering Education. Nevertheless, students face difficulties while learning software modeling. To teach software modeling effectively in computing courses, instructors who usually employed traditional methods could use active learning strategies. However, instructors are reluctant to change their teaching approaches due to several barriers that hinder the application of active learning strategies. Besides, relatively little research addresses how to mitigate them. The objective of this research is to help instructors implementing active learning strategies when teaching software modeling with UML diagrams. To achieve this objective, we conducted a Design Science Research (DSR). We proposed an artifact called OpenSMALS, an Open Repository for teaching Software Modeling applying Active Learning Strategies. OpenSMALS provides specific guidelines on how instructors can apply these strategies and helping instructors to identify the active learning strategies best suit their teaching context. We performed four DSR Design Cycles---in four different universities---to evaluate and evolve OpenSMALS. Our results show that OpenSMALS satisfactorily reduced the barriers faced by instructors, and it achieved an appropriate maturity level to be adopted by other instructors.

Enriching programming student feedback with audio comments

Introductory programming is challenging for many students, requiring them to engage with a deep approach to learning concepts in order to succeed. These challenges compound for online students who do not have direct face-to-face interactions with teaching staff. With the growing demand for online education, we need to examine approaches that assist in building supportive learning environments for these students. A growing body of work from other education disciplines indicates that audio feedback provides an opportunity for developing stronger relationships with students. Further studies recommend an integrated implementation of audio recording into the virtual learning environment. To evaluate audio feedback for use in programming education, we developed an integrated, cross-browser audio feedback feature into the open-source Doubtfire learning management system. Doubtfire is used to support and scale a task-oriented teaching and learning system built upon the principles of constructive alignment and has been shown to help students engage with programming concepts in campus-only units. Our findings from experimental and observational activities indicate that programming tutors can use a blended approach of audio and text feedback via the learning management system to better support student learning. The blended approach provides more nuanced feedback, conveying personality and feelings of connectedness to students, while retaining the benefits of specificity for code-specific issues.

GidgetML: an adaptive serious game for enhancing first year programming labs

Serious games have become a popular alternative learning tool for computer programming education. Research has shown that serious games provide benefits including the development of problem solving skills and increased engagement in the learning process. Despite the benefits, a major challenge of developing serious games is their ability to accommodate students with different educational backgrounds and levels of competency. Learners with a high-level of competence may find a serious games to be too easy or boring, while learners with low-level competence may be frequently frustrated or find it difficult to progress through the game. One solution to this challenge is to use automated adaptation that can alter game content and adjust game tasks to a level appropriate for the learner. The use of adaptation has been successfully utilized in educational domains outside of Software Engineering, but has not been applied to serious programming games. This paper presents GidgetML, an adaptive version of the Gidget programming game, that uses machine learning to modify game tasks based on assessing and predicting learners' competencies. To assess the benefits of adaptation, we have conducted a study involving 100 students in a first-year university programming course. Our study compared the use of Gidget (non-adaptive) with GidgetML (adaptive) and found that students who played Gidget during lab sessions varied significantly in their performance while this variance was significantly reduced for students who played GidgetML.

Exploiting group shuffling dynamics to convey the importance of good software design

Analysis, modeling and design activities are integral parts of any modern software development project. However, educators often struggle in convincing undergraduate students of their value. Grasping the importance of good software design requires a certain maturity that is not yet attained in the first years of a computer science curriculum. In this study, we introduce group shuffling dynamics to a formal software design course to convey the importance of these concepts by exposing students to the experience of switching groups mid-semester. This grants a whole new perspective on the matter as the cost of managing technical debt is felt when the respective developers leave, and modeling artifacts become pivotal in aiding the incorporation of new team members. This short paper presents our first experience while implementing this methodology and shares our preliminary results derived from a first set of interviews with our students.