SIGSOFT Annual Report

Fiscal Year 2020, July 2019 - June 2020

Submitted by Thomas Zimmermann, SIGSOFT Chair

SIGSOFT seeks to improve our ability to engineer software by stimulating interaction among practitioners, researchers, and educators; by fostering the professional development of software engineers; and by representing software engineers to professional, legal, and political entities.

This report provides a summary of key SIGSOFT activities over the past year. The COVID-19 pandemic had a significant impact on the world and also the SE community. Our hearts go out to all those affected by COVID-19. Since the start of the pandemic, students and professors switched to online teaching, software engineers now work from home, and many conferences had to be cancelled or switched to virtual formats. It’s difficult to predict the long-term effects of the pandemic but SIGSOFT remains committed to support its members during this difficult time. On a positive note, despite the pandemic, the SIGSOFT community had a strong technical year with many significant contributions and SIGSOFT remains financially healthy.

AWARDS GIVEN OUT

The SIGSOFT awards program recognizes the many achievements of the software engineering community.

  • The ACM SIGSOFT Outstanding Research Award was presented to Michael Ernst for contributions to programmer productivity through software analysis, testing and verification.
  • The ACM SIGSOFT Influential Educator Award was presented to Greg Wilson for life-long contributions to software engineering education throughout the computing industry.
  • The ACM SIGSOFT Distinguished Service Award was presented to Nenad Medvidović for significant service to the software engineering community and for leading important community innovations.
  • The ACM SIGSOFT Early Career Research Award was presented to Claire Le Goues for groundbreaking work on automated program repair, impact on industrial practice, and service to the software engineering research community.
  • The ACM SIGSOFT Outstanding Doctoral Dissertation Award was presented to Rachel Tzoref-Brill, Tel Aviv University, for the thesis Comprehension and Evolution of Combinatorial Models and Test Plans.
  • The ACM SIGSOFT Impact Paper Award recognizes a paper published in a SIGSOFT conference at least 10 years earlier that has had exceptional impact on research or practice. This year, the award went to the paper Model Checking Programs, authored by Willem Visser, Klaus Havelund, Guillaume Brat and SeungJoon Park, published in the Proceedings of the 15th IEEE International Conference on Automated Software Engineering (ASE 2000, Grenoble, France, 11-15 Sep 2000).

In addition to the Impact Paper Award, many SIGSOFT conferences also have Most Influential Paper Awards, also known as "test of time awards", which are given to papers that have appeared at a particular conference. At ICSE the SIGSOFT co-sponsored conference with the longest track record of awarding Most Influential Papers, the award went to the ICSE 2010 paper "Oracle-Guided Component-Based Program Synthesis", by Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, Ashish Tiwari. The ESEC/FSE conference recognized two papers from ESEC/FSE 2009: "Fair and balanced?: Bias in Bug-fix Datasets", by Christian Bird, Adrian Bachmann, Eirik Aune, John Duffy, Abraham Bernstein, Vladimir Filkov, Premkumar Devanbu and "Cross-project Defect Prediction: A Large Scale Experiment on Data vs. Domain vs. Process", by Thomas Zimmermann, Nachiappan Nagappan, Harald Gall, Emanuel Giger, Brendan Murphy.

We recognized the new ACM Distinguished Members and Fellows from the SIGSOFT community. Valerie Barr, Andrew B. Begel, Eric Bodden, Yuriy Brun, David Lo, Shan Lu, Manu Sridharan have been elevated to Distinguished Member; and Matthew B. Dwyer has been elevated to ACM Fellow for contributions to the specification and analysis of software.

Many of SIGSOFT’s sponsored meetings also presented Distinguished Paper Awards. SIGSOFT allows up to 10% of the accepted papers to be selected for this award. The list of recognized papers is available at http://sigsoft.org/awards/distinguishedPaperAward.html.

SIGNIFICANT PAPERS ON NEW AREAS

Software plays a prominent role in different application domains as well as other research areas in computer science, such as human-computer interaction, mobile computing, artificial intelligence, distributed systems, and more recently big data and machine learning. Everything depends on software today. As examples for the breadth of the software engineering field, we highlight below distinguished papers from the ESEC/FSE 2019 and ICSE 2020, two of the main general software engineering conferences sponsored by SIGSOFT. The trend over the past few years to adopt machine learning (ML) and artificial intelligence (AI) in software has continued. This has led to research both on leveraging ML and AI to improve the way software is being built (AI4SE), but also to research that incorporate SE concepts into ML and AI processes and workflows (SE4AI). An increased emphasis has been placed on testing and debugging systems that are based on ML and AI. Automated program repair, test generation, and reproducing bugs continue to remain popular topics in the SE research community. In recent years a strong focus was placed on better understanding software developers. Two examples of papers in this area are on understanding how developers learn new languages and how cognitive biases influence decision making. Over the next few years it can be expected that the SE community will make further progress on being able to build more reliable, robust and fair software systems at a larger scale, while increasing the productivity of software developers at the same time.

Distinguished papers at ESEC/FSE 2019.

  • Thomas Durieux, Fernanda Madeiral, Matias Martinez, and Rui Abreu: Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2 141 Bugs and 23 551 Repair Attempts
  • Claudio Menghi, Shiva Nejati, Khouloud Gaaloul, and Lionel Briand: Generating Automated and Online Test Oracles for Simulink Models with Continuous and Uncertain Behaviors
  • Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman: The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities
  • Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng: Assessing the Quality of the Steps to Reproduce in Bug Reports
  • Pengyu Nie, Rishabh Rai, Junyi Jessy Li, Sarfraz Khurshid, Raymond J. Mooney, and Milos Gligoric: A Framework for Writing Trigger-Action Todo Comments in Executable Format
  • Sen He, Glenna Manns, John Saunders, Wei Wang, Lori Pollock, and Mary Lou Soffa: A Statistics-based Performance Testing Methodology for Cloud Applications

Distinguished papers at ICSE 2020.

  • Souti Chattopadhyay, Nicholas Nelson, Audrey Au, Natalia Morales, Christopher Sanchez, Rahul Pandita, Anita Sarma: A Tale from the Trenches: Cognitive Biases and Software Development
  • Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, Mao Yang: An Empirical Study on Program Failures of Deep Learning Jobs
  • Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, Andrea Janes: Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
  • Junjie Wang, Ye Yang, Song Wang, Yuanzhe Hu, Dandan Wang, Qing Wang: Context-aware In-process Crowdworker Recommendation
  • Nischal Shrestha, Colton Botta, Titus Barik, Chris Parnin: Here We Go Again: Why Is It Difficult for Developers to Learn Another Programming Language?
  • Zhen Dong, Marcel Böhme, Lucia Cojocaru, Abhik Roychoudhury: Time-travel Testing of Android Apps
  • Zishuo Ding, Jinfu Chen, Weiyi Shang: Towards the Use of the Readily Available Tests from the Release Pipeline as Performance Tests. Are We There Yet?
  • Carlos Bernal-Cárdenas, Nathan Cooper, Kevin Moran, Oscar Chaparro, Andrian Marcus, Denys Poshyvanyk: Translating Video Recordings of Mobile App Usages into Replayable Scenarios
  • Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, Jinshui Wang: Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning
  • Peixin Zhang, Jingyi Wang, Jun Sun, Guoliang Dong, Xinyu Wang, Xingen Wang, Jin Song Dong, Ting Dai: White-box Fairness Testing through Adversarial Sampling

SIGNIFICANT PROGRAMS

Over the past several years, SIGSOFT has introduced several programs to aid and expand our membership. Two signature programs are the Webinars and the CAPS funding.

  • The SIGSOFT Webinar series remains very popular; in the past year, SIGSOFT organized five webinars on topics such as joys and frustration of software engineering, automated debugging and profiling of AAA games, agile development, and continuous integration and delivery.
  • Through the Conference Aid Program for Students (CAPS), SIGSOFT provided travel support to conferences for dozens of graduate and undergraduate student-members as well as support to defray the costs of childcare for all members of our community (faculty qualify as well). CAPS has been extended to help defray travel costs of a certain number of professional SIGSOFT members as well. The new program launched in Fall 2019. Overall the CAPS support allocation was doubled in 2019-2020.

SIGSOFT continued the implementation of the Conference Surplus Reinvestment, which was launched in FY19. The surplus reinvestment for Year N of a conference is 25% of the Conference Net from Year N-2 and 25% of the Conference Net from Year N-3. A limitation of the program is that it currently requires a significant administrative overhead.

SIGSOFT also continued the special projects program for projects that benefit SIGSOFT members, future SIGSOFT members or members of the software engineering community. An example of a special project was the Summer School on Modelling and Programming, from January 19-24, 2020 in Stellenbosch, South Africa. The school had 45 students and 10 speakers and assistants to learn about several aspects of software modeling: What is software modelling? How can we bridge the gap between a model and the program? How are these tools and techniques used in industry? The school was organized by Judith Bishop, Richard Paige, Willem Visser, and Bruce Watson for running a wonderful school. (www.ssmodpro.org). Another special project was SE Garage, which goal is to provide a central archive of curated tools developed in SE research (www.segarage.org).

SIGSOFT also appointed Andreas Zeller as the Climate Change Liaison, who is in charge of increasing the sustainability of SIGSOFT conferences and to coordinate with SIGSOFT conferences, ACM, and the other SIGs to reduce the climate impact of our work.

Martin Robillard was appointed as Research Highlights Chair with the charter to define a process to select SIGSOFT Research Highlights and to increase the participation of SE papers in the Research Highlights section from CACM.

The Software Engineering Notes have a new Editor, Dietmar Pfahl, who has modernized the newsletter. SEN is now published on time with more columns than before. The April 2020 issue was the last printed Software Engineering Notes. Going forward the SEN newsletter will be distributed exclusively in electronic form.

INNOVATIVE PROGRAMS

To encourage more volunteer participation, SIGSOFT launched several strategic initiatives with an open call for volunteers in 2019. The initiatives included Diversity, and Inclusion; Open Science; Data-driven Introspection; Conference Operations; Paper and Review Quality; and many more. Over the past year the initiatives made significant progress:

  • Conference Operations (led by Laura Moreno, Alexander Serebrenik): Improve the experience for conference attendees. Initially the team has run a survey to document how different conferences recognize most influential papers. The report is currently being finalized. In addition, they co-organized a session on Diversity and Inclusion at SE Conferences.
  • Data-driven Introspection (led by Tim Menzies, Bogdan Vasilescu): Drive the understanding of the entire ecosystem of software engineering conferences. The initiative is currently working on a SE community survey.
  • Diversity and Inclusion (led by Jo Atlee, Kelly Blincoe, Byron J. Williams): Lead the efforts for increasing diversity, inclusion, and belonging. Drafts have been created for codes of conduct and guidelines for conference EDI, accessible submissions and presentations, inclusive conference activities, inclusive review practices, inclusive session chairing practices. These documents are under review by the community and will be made available shortly. The initiative has also prepared a EDI climate survey, which will be sent to the community soon.
  • Open Science (led by Jonathan Bell, Daniel Graziotin): Increase open science practices in the software engineering community. Help the community to increase the accessibility, reproducibility, and replicability of our research outcomes. The initiative drafted a blueprint for generating open science policies and instructions for reviewers for research venues. The current draft is at: https://github.com/acmsigsoft/open-science-policies The initiative also has a working group on artifact evaluation. The latest discussion can be found here https://github.com/acmsigsoft/artifact-evaluation
  • Paper and Review Quality (led by Paul Ralph and Romain Robbes): Increase the quality of our research papers and reviews, for example by developing guidelines on how to review certain types of papers. Make recommendations on how to improve review processes. The task force has put together an initial draft of a Empirical Standards and Reviewer Field Manual that will be released soon for comments by the community.

The software engineering community has many community-driven vehicles to promote open science such as dedicated tracks for replications and negative results at conferences, the ROSE festival https://github.com/researchart/rose, and pre-registered studies (e.g., MSR 2020, ICSME 2020 in partnership with the Empirical Software Engineering journal).

BROADENING PARTICIPATION

Several events or programs were focused on broadened participation either geographically, or among under-represented members of your community and

  • The ICSE conferences continued the Student Mentoring Workshop aimed at advanced undergraduates and first/second year graduate students. The goal of the workshop is to attract students to research careers in software engineering, to demystify the graduate school experience, and to offer first-hand perspectives on graduate study from recent Ph.D. graduates, young scholars, and senior researchers.
  • Several SIGSOFT software engineering conferences (for example, ASE 2019) have started to appoint Diversity and Inclusion chairs to organize D&I activities at meetings. Improving the diversity and inclusiveness of the community that we serve is and always will be a priority.
  • To broaden, reach and membership, SIGSOFT has established national chapters in India (iSoft) and China (cSoft). Each chapter has a liaison on the SIGSOFT EC, in addition to our long-standing International Liaison. The ISSTA 2019 conference was held in China, the ESEM 2019 conference in Brazil, and the ICSE 2020 conference was scheduled to be in South Korea but unfortunately had to be moved to a virtual format. SIGSOFT also supported a summer school in South Africa.

KEY ISSUES FACING SIGSOFT

While SIGSOFT is stable and strong, there are several challenges we continue to face.

  • Given the uncertainty of the COVID-19 pandemic, it is unknown if and when physical conferences will restart and what format they will have. This makes planning more complicated and adds extra work and anxiety to the conference organizers. SIGSOFT will do its best to support all conferences formats going forward.
  • While virtual conferences have largely worked so far, thanks to heroic efforts by the organizers and other volunteers (e.g., Jonathan Bell, Crista Lopes and Benjamin Pierce for the Clowdr tool, https://www.clowdr.org/), they come at a cost of missed social connections and a lack of feeling included, especially for newcomers to the SE community. Furthermore, hybrid conferences will only double the work it takes to organize a conference because in addition to a physical format a virtual format has to be supported.
  • Despite a large growth of software engineers, SIGSOFT’s membership numbers have not been growing. We will aim to increase membership by providing more benefits to SIGSOFT members, increased visibility of SIGSOFT at conferences, and a focus on practitioners with the help of the industry liaison.
  • We continue to work to increase the global reach of SIGSOFT by continuing to establish long-term working relationships with our Indian, Chinese, and South American colleagues, as well as expanding the reach of SIGSOFT into Africa. Over the past few years, China has grown to the second most active country in software engineering research. We will work towards further growing the community and interest in SIGSOFT in these regions.
  • Several members of our community have raised concerns over the lack of gold open access offerings at SIGSOFT conferences. This may become a problem for researchers who are subject to the Plan S – Open Access Mandate. We are looking into how the changing open access landscape will affect SIGSOFT and how ACM’s policies, practices, and initiatives can work for SIGSOFT members. To accelerate open access for SIGSOFT conferences we have appointed Arie van Deursen as Open Access Liaison.