Skip to Content

What We Know We Don't Know: Empirical Software Engineering

Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, with a particular focus on software defects and productivity.

Talk doesn’t seem to be online yet; in the meantime, you can see a recording of an older version of the talk here. Slides are here.


I referenced a bunch of papers in my talk. These are links so you can read them yourself:


Big Data is slower than laptops

Scalability! But at what COST?

Why we care

Section references

The German Tech Sector

The Pragmatics of TDD

TDD is dead. Long live testing.


Controlled Trials

Comparing syntax highlightings and their effects on code comprehension

Natural Experiments

Simple Testing Can Prevent Most Critical Failures

Observational Studies

Towards the Creation of Interspecies Digital Games: An Observational Study on Cats’ Interest in Interactive Technologies

Errata: In the talk I said it studied “cats on keyboards”, but the paper was actually on cats on tablets (digital projections).

Software Developers’ perspectives on productivity


Programming Language Effects

A Large Scale Study of Programming Languages and Code Quality in Github (original faulty study)

On the Impact of Programming Languages on Code Quality (replication study)

My 6,000 word writeup of the whole fiasco

Code Review

Best Kept Secrets of Peer Code Review

An Empirical Study of the Impact of Modern Code Review Practices on Software Quality

First Come First Served: The Impact of File Position on Code Review

Modern Code Review: A Case Study at Google

Human Factors

Crunch Makes Games Worse


Impact of a Night of Sleep Deprivation on Novice Developers Performance

Sleep deprivation: Impact on cognitive performance

Scheduled Overtime Effect On Construction Projects

Overtime and Extended Work Shifts

Additional Sources

Recommended Reading

Teaching tech together

Leprechauns of Software Engineering

The Programmer’s Brain

Free research

Making Software

It Will Never Work In Theory

ACM digital library