Skip to Content

What We Know We Don't Know

Official Description: There are many things in software we believe are true but very little we know. Maybe testing reduces bugs, or maybe it’s just superstition. If we want to improve our craft, we need a way to distinguish fact from fallacy. We need to look for evidence, placing our trust in the hard data over our opinions.

Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, with a particular focus on software defects and productivity.

Actual Description: Nothing is real, we don’t understand what we’re doing, and the only way to write good software is to stop drinking coffee. Burn it all down. Burn it to the ground.

Talk is not up yet.


I referenced a bunch of papers in my talk. These are links so you can read them yourself:


Small Functions Have Problems

Effects of Clean Code on Understandability

Tech and GDP

Cyberstates 2018

Big Data is slower than laptops

Scalability! But at what COST?

Programmers make the same mistakes / N-version programming fails

An Experimental Evaluation of the Assumption of Independence in Multi-Version Programming

A Reply to the Criticisms of the Knight and Leveson Experiment


Abbreviated Code is readable
Fixing Faults in C and Java Source Code: Abbreviated vs. Full-Word Identifier Names (preprint)
Code Smells
On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation
Programming Language Effects

A Large Scale Study of Programming Languages and Code Quality in Github (original faulty study)

On the Impact of Programming Languages on Code Quality (replication study)

Software Defects

Parachute Studies
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials
Defect detection in code
“Beyond Lines of Code: Do we Need More Complexity Metrics?“, Harraiz & Hassan, Making Software (ch 8)
Conway’s Law
“Conway’s Corollary”, Making Software (ch 11)
Defect detection in organizations
The Influence of Organizational Structure On Software Quality: An Empirical Case Study
Worst bugs are design bugs
“Where Do Most Software Flaws Come From?” Dewayne Perry. Making Software (ch 25)
Simple Testing Can Prevent Most Critical Failures
Test-Driven Development

Realizing quality improvement through test driven development (positive results)

Analyzing the Effects of Test Driven Development in GitHub (negative results)

Test-Driven Development - Still a Promising Approach?

Code Review

Best Kept Secrets of Peer Code Review

An Empirical Study of the Impact of Modern Code Review Practices on Software Quality


Impact of a Night of Sleep Deprivation on Novice Developers Performance

Sleep deprivation: Impact on cognitive performance


Scheduled Overtime Effect On Construction Projects

Overtime and Extended Work Shifts


Crunch Makes Games Worse


Additional Sources

Recommended Reading

Making Software

Leprechauns of Software Engineering

Calling Bullshit

Free research

Making Software

It Will Never Work In Theory



These are questions I got after the talk. I will add more as people ask them.