Mutation testing for Swift with Mull: how it could work. Looking for contributors.

TL;DR

Mull, mutation testing tool based on LLVM, can work with code written in Swift but the code to find mutations on AST level is missing. We are focusing more on C and C++ programming languages and therefore are looking for contributors who could help us to continue our work on Objective-C and Swift support.

What has been done so far and what is missing?

A good part of the work is done: we can run Swift with Objective-C interoperability enabled with LLVM JIT, see LLVM JIT, Objective-C and Swift on macOS: knowledge dump.

There is an extremely hacky branch on Mull with which we are able to run Mull on Swift/Objective-C code. The problem there is that Mull finds too many useless mutations in the LLVM IR code generated by Swift compiler and there is no good way to filter these mutations out. Such useless mutations are called "junk mutations" because they cannot be achieved by a mutation of a source code, so for a programmer it is not possible to write tests that would kill such mutations. Usually these mutations are found in the boilerplate code generated by a compiler, not in the code that is written by a human, so it does not make sense to highlight these mutations to a programmer who is working with a mutation testing tool.

The challenge

A finer-grained approach to find the right mutations in Swift code is needed.

Mull finds garbage mutations in C/C++ too, but for these languages there is a class CXXJunkDetector that uses libClang to check the AST information. The AST information helps Mull to find the right mutations that trace back to the source code and filter out everything else.

A similar approach is needed for Swift. But in contrast to C/C++, we believe that instead of filtering the junk mutations out of all of the mutations found on the LLVM IR level ("blacklisting"), the proper implementation should rather find the right mutations in the source code on the AST level and then pass the information about found mutations to the LLVM IR level where Mull can use it to perform mutations and execute the mutated versions of the program. In this scenario we see Mull as a backend for mutation testing which works with the mutations found by a mutation testing frontend for Swift code that finds mutations on the Swift AST level.

Why do you not do it yourself?

We have a number of reasons to put our work on Swift and Objective-C to a low-priority background thread.

The first and foremost reason is that we no longer work with Swift programming language. We started working on Mull when we were iOS developers but since then we have spent quite some time working with C/C++ languages and now we work with these languages at our day jobs.

The second reason is that we don't see enough interest in mutation testing from iOS and macOS communities. It looks like mutation testing coverage is not something that iOS/macOS testing experts are up to. We would be happy to learn that we are wrong about this though.

The third reason: we think that we have managed to get LLVM JIT with Swift working, but even if Mull works with Swift, it will only run on macOS. Support for iOS will be a whole another story.

What can I do if I want mutation testing for Swift?

You can contact us and we will provide you with pointers on how to proceed. You will most likely have to do some hacking on Swift AST and LLVM code to find the mutations in the Swift source code that Mull will execute. From our side we could help you to understand the design and implementation details of Mull - everything you might need to get the whole thing working.

The contact information can be found on the Mull's README page.