About IWCT 2020

IWCT 2020 has been postponed. Please find further details in the ICST homepage.

IWCT 2020 is to be held in conjunction with ICST 2020, focusing on combinatorial testing. The workshop welcomes academic research submissions, as well as industrial experience reports.

Combinatorial Testing (CT), or Combinatorial Interaction Testing (CIT), is a widely applicable generic methodology and technology for software verification and validation, considered a testing best practice. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing.

CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. For example, while pairwise testing is a well recognized and popular test planning method, investigations of actual failures in a number of software and systems convincingly show that pairwise testing is usually not sufficient, so high strength CT (i.e., t-way for t > 2) may be needed.

In addition, the combinatorial test suites need to exclude invalid combinations of test values that cannot be executed, which limits the degrees of freedom the algorithms have, thus complicating the problem. Moreover, modeling languages and tools for easily capturing the input test space are also required for real-life applicability of CT. Other obstacles for wide acceptance of CT in industry are the gap between the generated test plans and executable tests, and the difficulty in determining expected results for the generated tests. Finally, empirical studies on CT, as well as thorough comparison with other methods are also required.

In this workshop, we plan to bring together researchers actively working on combinatorial testing, and create a productive and creative environment for sharing and collaboration. Since there is no other venue dedicated to CT, yet there are many researchers working in the field, we expect, like in previous years, to see high responsiveness to take part in the workshop. Researchers attending the workshop will have an opportunity to publish their work in a dedicated venue, create new collaborations and take active part in the growing community of researchers working in the field.

The workshop will also be a meeting place between academia and industry, thus uniting academic excellence and industrial experience and needs. This will allow participants from academia to learn about the industrial experience in practical application of CT to real-life testing problems, and together with the colleagues from industry identify the difficulties that are obstacle to wider application of CT, and should be addressed in future research. Industrial participants will have an opportunity to meet the leading scientists in the field, and hear about the latest advances and innovations.

Call for Papers

Full Papers

up to 10 pages

Short Papers

up to 4 pages

Poster

up to 2 pages

Full and Short Papers : We invite submissions of high-quality papers presenting original work on both theoretical and practical aspects of combinatorial testing. We accept both full papers (up to 10 pages) and short papers (up to 4 pages).

Poster Session: We will also have a poster session, for authors to present their work in an informal and interactive setting. We encourage submissions presenting research work in progress or important conclusions from practical experience.

In particular, we welcome posters focusing on:

  • experience reports from integration of CT by industrial organizations
  • lessons learned from using CT tools in industrial applications

An extended abstract of the poster (up to 2 pages) should be submitted for review by the submission deadline. Accepted poster abstracts will be included in the proceedings.

Topics

Topics of interest for full and short papers include, but are not limited to:

Combinatorial testing workflow

  • Modeling the input space for CT
  • Efficient algorithms to generate t-way test suites, especially involving support of constraints
  • Determination of expected system behavior for each test case
  • Executing CT test suites
  • Combinatorial testing based fault localization
  • Implementation of CT with existing testing infrastructures
  • Handling changes in test requirements

Real-world experience in deployment of CT

  • Empirical studies and feedback from practical applications of CT
  • Evaluation and return of investment metrics to assess the degree of usefulness of CT
  • Methodology used for test space modeling and determination of interaction coverage requirements
  • Discussion of challenges and open problems in the application of CT in industrial settings

Applicability of combinatorial testing

  • Comparison and combination of CT with other dynamic verification methods
  • Investigation of historical records of failures to determine the kind of CT which may have detected faults
  • Combinatorial testing for concurrent and real-time systems
  • CT for testing cloud computing systems and use of combinatorial methods in cloud architecture
  • Application of CT in other domains, e.g. information security, study of gene regulation and other biotechnology applications, mechanical engineering, etc.
  • Combinatorial testing of variability models for software product lines

Combinatorial and complementing methods

  • Combinatorial analysis of existing test suites
  • Test plan reduction and completeness
  • CT and coverage metrics - combining the two, and studying the relationship between them

Proceedings and Best Paper Awards

Accepted papers and posters will be presented at the workshop. All full and short papers, as well as extended abstracts, will be published in the IEEE Digital Library - provided that at least one author registers and presents at the workshop (standard IEEE policy).

This edition of IWCT will feature two best paper awards in the categories of "best foundation/modelling paper" and "best CT application paper".

Paper Submission

IWCT 2020 submissions should be made at the EasyChair submission site: https://easychair.org/conferences/?conf=iwct2020. The paper category type (full paper, short paper or poster extended abstract) can be specified upon submission.

Author Instructions

All submissions should be in PDF format, and will be peer-reviewed. IWCT uses single-blind reviewing, so please include all author names and affiliations. Papers must neither have been previously accepted for publication nor be under submission in another conference or journal.

Research papers must conform to the two-column IEEE conference publication format. Templates for Latex and Word are available here; use the letter format templates and the “conference” option (i.e., not the Computer Society format).

Extended abstracts must conform to the same two-column IEEE conference publication format as for research papers. Extended abstracts are also subject to peer-review from the workshop's program committee.

Important Dates

Submission Deadline: extended to January 10, 2020
December 22, 2019
December 15, 2019
AoE – Anywhere on Earth
Author Notification: January 26, 2020
Camera Ready Version: February 4, 2020
Workshop: October 24, 2020
March 23, 2020

Keynote

Franz Wotawa

Graz University of Technology, Austria

Verifying autonomous systems using ontology-based testing

With the increase use of artificial intelligence methodologies even in safety-critical systems for implementing automated and autonomous functionality, there is a need for coming up with verification and certification methodologies allowing to justify the degree of verification at least to a certain extent. Combinatorial testing makes use of the combinatorial strength for this purpose. In recent work, we reported on an extension of combinatorial testing, i.e., ontology-based testing, where we utilize ontologies for describing environmental models that can be mapped to combinatorial testing input models. In my talk, I introduce the basic ideas, discuss the foundations, and present results obtained when using combinatorial testing from various domains, including autonomous and automated driving functions and security testing.

Speaker Bio: Franz Wotawa received a M.Sc. in Computer Science (1994) and a PhD in 1996 both from the Vienna University of Technology. He is currently a professor of software engineering at the Graz University of Technology and the head of the Institute for Software Technology. His research interests include model-based and qualitative reasoning, theorem proving, mobile robots, verification and validation, and software testing and debugging. Since October 2017 Franz Wotawa has been the head of the Christian Doppler Laboratory for Quality Assurance Methodologies for Autonomous Cyber-Physical Systems. During his career, Franz Wotawa has written more than 380 peer-reviewed papers for journals, books, conferences, and workshops. He supervised 90 master and 36 PhD students. For his work on diagnosis, he received the Lifetime Achievement Award of the Intl. Diagnosis Community in 2016. Franz Wotawa has been member of a various number of program committees and organized several workshops and special issues of journals. He is a member of the Academia Europaea, the IEEE Computer Society, ACM, the Austrian Computer Society (OCG), and the Austrian Society for Artificial Intelligence and a Senior Member of the AAAI.

Program

You can access the complete program for IWCT 2020 at the link: https://conf.researchr.org/home/iwct-2020#program

Times are displayed in time zone: (GMT+01:00) Greenwich Mean Time
13:00 - 14:00: Opening Session
13:00 - 13:15 Welcome to IWCT [slides]
Dimitris Simos, Jeff Lei, and Changhai Nie
13:15 - 14:00 Keynote: Verifying autonomous systems using ontology-based testing
Franz Wotawa
14:00 - 14:15: Online Coffee Break
14:15 - 14:20: Poster Presentation Session (starting on coffee break)
14:15 - 14:20 Toward an Encoding Approach to Interaction-based Test Suite Minimization
Kohei Nishino, Takashi Kitamura, Tomoji Kishi, Cyrille Artho
14:20 - 15:10: Test Generation and Combinatorial Testing Applications Session
14:20 - 14:40 An Automata-Based Generation Method for Combinatorial Sequence Testing of Finite State Machines
Andrea Bombarda, Angelo Gargantini
14:40 - 14:50 Combinatorial Methods for Explainable AI
Rick Kuhn, Raghu Kacker, Jeff Lei, Dimitris Simos
14:50 - 15:10 Generation of Invalid Test Inputs from Over-Constrained Test Models for Combinatorial Robustness Testing
Konrad Fögen, Horst Lichter
15:10 - 15:30: Online Coffee Break
15:30 - 16:40: Combinatorial Testing Tools Session
15:30 - 15:40 On Using Ontologies for Testing Compilers
Yihao Li, Franz Wotawa
15:40 - 16:00 A Comparison Infrastructure for Fault Characterization Algorithms
Torben Friedrichs, Konrad Fögen, Horst Lichter
16:00 - 16:20 Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application
Jan Richter, Bestoun S. Ahmed, Miroslav Bures, Cleber R. Rosa Junior
16:20 - 16:40 CAGEN: A fast combinatorial test generation tool with support for constraints and higher-index arrays
Michael Wagner, Kristoffer Kleine, Dimitris Simos, Rick Kuhn, Raghu Kacker
16:40 - 16:55: Online Coffee Break
16:55 - 17:55: Wrap Up
16:55 - 17:55 Wrap up (feedback and planning)
This program is tentative and subject to change.

Organization

General Chair

Dimitris Simos

SBA Research, Austria

Program Co-chair

Yu Lei

University of Texas Arlington, USA

Program Co-chair

Changhai Nie

Nanjing University, China

Organization Committee

Jacek Czerwonka

Microsoft Research, USA

Angelo Gargantini

University of Bergamo, Italy

George Sherwood

Testcover.com

Rachel Tzoref-Brilla

IBM Research, Israel

Steering Committee

Raghu Kacker

NIST, USA

Richard Kuhn

NIST, USA

Itai Segall

Bell-labs, Israel

Program Committee

Keizo Tatsumi
Fujitsu LTD & ASTER, Japan
Charles Colbourn
Arizona State University, USA
Franz Wotawa
Graz University of Technology, Austria
Krishnan Rangarajan
Dayananda Sagar College of Engineering, India
Jim Lawrence
George Mason University, USA
Jun Yan
Chinese Academy of Sciences, China
Peter Zimmerer
Siemens AG, Germany
Ziyuan Wang
Nanjing University of Posts and Telecommunications, China
Akihisa Yamada
National Institute of Informatics, Japan
Takashi Kitamura
AIST, Japan
Peter M. Kruse
Berner & Mattner, Germany
Murat Ozcan
Siemens Industry Inc., USA
Tatsuhiro Tsuchiya
Osaka University, Japan
Huayao Wu
Nanjing University, China
Bestoun S. Ahmed
Karlstad University, Sweden
Yvan Labiche
Carleton University, Canada
Violet Syrotiuk
Arizona State University, USA
Xintao Niu
Nanjing University, China
Horst Lichter
RWTH Aachen University, Germany