Beyond Unit Tests

Official Description: You’ve used pytest and you’ve used mypy, but bugs are still slipping through your code. What’s next? In this talk, we cover two simple but powerful tools for keeping your code problem-free. Property-based testing, provided by the Hypothesis library, lets you run hundreds of tests from a single template. Contracts, via dpcontracts, make your program test itself. You’ll learn how and why to use these tools and how to combine them with the rest of your testing suite.

Actual Description: I will complain about unit testing, bash type systems, and try to convince everyone that contracts are a good idea despite there being no good Python libraries for them. I ‘solve’ integration testing, just like the 1,000 other idiots who thought they ‘solved’ integration testing. I will conclude by rambling about formal methods and run out before anyone can ask for a real example.

Talk is here.

FAQ

These are the questions I got after the talk. I will add more as people ask them.

How do I come up with good properties for property tests?

It’s something you learn with practice, but the easiest way to do it is to add contracts to your codebase. Then you don’t need to write invariants in your tests at all; your invariant is that your contract isn’t broken!

What are good properties for Pandas?

Hypothesis has a set of Pandas strategies for generating data.

Can I property test databases? What contracts should I be using?

You can! Hypothesis has Django strategies and there are third-party plugins for SQLAlchemy. Re contracts, I think databases are a special case: you already have a solid contracts system in the form of your innate database constraints! These can be a lot more powerful than your ORM might lead you to believe. Your property invariant then becomes “none of my constraints are broken”, just as we can make it “none of my contracts are broken.”

Should I be running contracts in production?

The usual answer is “no”. I have mixed feelings on this. On one hand, if you have control over your production environment, it’s better to fail fast than let an error propagate. On the other hand, contracts are a runtime overhead. On the other other hand, if you’re concerned about runtime overheads you really should be profiling your code instead of just turning off consistency checks and hoping that’s where the slowdown is.

On the other other other hand, many of the “full specification”-style contracts involve coming up with a “definition” of the function that’s more “obviously correct” than whatever implementation you have. In practice this usually means brute forcing the solution, which probably would be a source of slowdown in prod. So my personal rule is “some contracts stay up, some get turned off.”

Do these tools exist in other languages?

The question was specifically about Java, but we can generalize it. The first property-based testing library was called Quickcheck, so most languages riff on that as a name. The two I found for Java were called QuickTheories and junit-quickcheck.

It’s harder to find a good contracts library, because contracts are an overloaded term. “Asserted pre/post/inconditions” was the first and most precise use of them term. We also have “contracts” as “testing what third party services return”, or “what a provider gives you”, or even “what your API is”. And, more recently, we have “smart contracts”, which have nothing to do with testing at all!

Usually you want to be looking for “design by contract” or “require/ensure” or “runtime contracts”. I don’t have any experience with Java, but I’ve been told to look into JML.

Someone else asked about Javascript. In that case, I know Babel-Contracts is pretty good.

Where can I learn more about Stateful Testing?

Here’s a general introduction I really like. There’s also a page in the Hypothesis manual.