Software Reliability

Module aims

To introduce students to techniques and tools for improving the reliability of software systems. The course will provide an overview of the main types of software reliability techniques and discuss their respective strengths and weaknesses. The course will present in detail a selection of these techniques and associated tools, focusing on recent directions in both research and practice.

More specifically students will:

• Gain familiarity with different types of program analysis techniques for improving software reliability
• Learn how to assess the trade-offs offered by different techniques and tools
• Experiment with a variety of open-source tools implementing popular software reliability techniques
• Read and analyse research papers in the area
• Critically evaluate and present a software reliability tool
• Implement a software reliability tool for a specific domain

Recommended: Compilers, logic, good C programming skills.

Learning outcomes

After the course, students should be able to:

1. Evaluate the trade-offs of different software reliability techniques
2. Explain the main types of generic software bugs, such as memory errors
3. Implement well-known program analysis techniques for software testing
4. Explain the main ways in which programs can be exploited and how program analysis can be used to defend against security attacks
5. Critically read and analyse a research paper
6. Apply popular open-source tools to test software applications

Module syllabus

The curriculum will be subject to changes across years, to reflect recent developments in the research and practice in the area of software reliability.
 

• Basic software reliability concepts
  • Limitations of manual testing
  • General vs. functional properties
  • Static vs. dynamic analysis
  • False positive vs. false negatives
• Software fuzzing
  • Generation-based vs. mutation-based fuzzing
  • Dumb vs. smart fuzzing
  • Black-box vs. grey-box vs. white-box fuzzing
  • Feedback-directed fuzzing
• Compiler fuzzing
  • Differential compiler testing
  • Equivalence modulo inputs compiler testing
  • Test case reduction
• Compiler sanitizers
  • Shadow memory
  • Address sanitization
  • Memory sanitization
• Undefined behaviour
  • Rationale for undefined behaviour
  • Compiler optimisations and unstable code
• Dynamic symbolic execution
  • Main concepts and design choices
  • Path exploration challenges
  • Constraint solving challenges
• Program analysis for security
  • Overview of basic exploits and defenses
  • Control-flow integrity
  • Data-flow integrity
  • Write-integrity testing

Teaching methods

7 weeks of lectures, tutorials, and student presentations.

An online service will be used as a discussion forum for the module.

Assessments

The coursework will consist of a significant implementation project in which students will have to build and evaluate software reliability techniques. The project will consist of two parts, accounting for 24% of the marks.

As an example of project from previous years, students were asked to build a fuzzer based on metamorphic testing to find bugs in SAT-solver implementations.

In addition to the project component, students will also be asked to evaluate and present a software reliability tool of their choice. This part will account for 6% of the marks.

The coursework component will be done in groups of 2 or 3 students. 70% of the marks will be assessed via a standard paper-based exam consisting of 2 questions.

For each implementation part, the students will need to submit the actual software implementation and a report describing the algorithms used and the overall design of the software.

The projects will be assessed via auto-testing, as well as by a TA who will evaluate and provide feedback on the report.

For the evaluation and presentation of the software reliability tool, the mark will be based on the quality of the presentation.

Module leaders

Professor Cristian Cadar