Software Entomology or Where Do Bugs Come From?

Noah Sussman and Michelle D’Nettoetsy

or, Where Do Bugs Come From?

Software Entomology

There will always be bugs in production

Most Bugs Are Non-Obvious

Debugging is hard

I can remember the exact instant when I realized that a large part of my life... was going to be spent in finding mistakes in my own programs

~Maurice Wilkes, 1949

Admiral Grace Hopper liked to tell a story in which a technician solved a glitch in the Harvard Mark II mainframe by pulling an actual insect out from between the contacts of one of its relays

~ The Jargon File

Complete testing is impossible.

Even the simplest Web site has more edge cases than you could test, even if you ran one test per second until the heat death of the universe.

assure that the probability of failure due to hibernating bugsis sufficiently low to be acceptable

~ Boris Beizer, 1983

What constitutes a “sufficiently low” number of problems?

Declaring a program “bug free,” is an entirely

subjective excercise

Engineers suck at finding bugs in their own code

too close to the software

Engineers + designers are

Computer programs cannot detect bugs in other computer programs

Debugging is twice as hard as writing the code in the first place.

~ Brian Kernighan

not the same as reporting a bug

Getting frustrated by software is

Non-source-aware users tend to report only surface symptoms; they take their environment for granted...

~ ESR, “How Many Eyeballs Tame Complexity”, 1997

Users never read manualsUsers would save time in the long term by learning more about the system. But that's not how people behave in the real world

~Jakob Nielsen

On two occasions I have been asked... if you put into the machine wrong figures, will the right answers come out?

~Charles Babbage, 1864

When you report a bug

you have done a hard thing

A bug report is a story about a problem and a good story has a beginning, middle and end

Description of problemSteps to reproduceExpected resolution

Provide context, and lots of it

Be Quantitative

Engineers respond well to hard evidence

Graph or it didn’t happenAt Etsy we monitor over 100,000 real-time metrics

A good editor ismeticulously attentive to detail,a stickler for technical correctness,aware of the work as a whole andsensitive to the expectations of the audience

Keep a journalTake screenshots (use Skitch)Log IRC, Skype and other chatsMake multiple redundant backups (use DropBox)Save everything— you won’t know you need it until you need it

You are doing science

How many times has this happened?How many users were affected?How much time has been lost?What was the overall cost?

Users

The Support Team spend the most time engaging and empathizing with users

You are a participant observer

When you talk to your users, andyou write down what you learn,and you share what you wrote with others, you’re doing science

digital computer: less than 70 years old,personal computer: less than 40,the Web: less than 20,mobile Web: less than 15

The Web is going to keep on growing and changing

80% of us aren’t even on the Web yet

And you are teaching them how to do it

Most users of a Web site are doing something that is entirely new to them