Eureka

I awoke at 4am. The cause of the software issue that I had been flummoxed by two days ago was now obvious. This has happened to me so many times in my career, I’m sure I can’t remember them all. Significant algorithmic inventions, mysterious behaviors diagnosed, and even fixes for bugs that have never manifested. (Think about that — there is a bug in the software and nobody has ever seen it, and I suddenly realize it is there, and I fix it before it ever causes a problem.) And every time, every single time, the 4am solution is exactly correct.

I’ve heard it is a common phenomenon. It happens to people in all kinds of work. You try and try to solve a problem. You finally give up. Move on to other things, and a day or two later, you have the answer. It just comes to you.

This is not my brain, although I think I may have dreamt about that chick once

The analogy we like to use is that our brain is a computer, and our subconscious has been quietly working away on the solution for all this time, and when it is ready, it goes ding, and out comes the paper tape with the answer.

Bullshit.

I think I understand what is happening here, and it definitely is not that.

The solutions that pop into my head are not the result of deeply crunching big data. They are small, profound insights. Small insights do not need processing. But they do need time. And I think I even understand why that is.

A tasteful example of parallel, not serial, wiring

Certainly you have seen “National Lampoon’s Christmas Vacation.” Poor Clark has strung all the lights, thousands of them, and they only work intermittently. So he dives in and starts checking every single bulb. Nevermind that the bulbs are clearly 120VAC lights that are wired in parallel. That would be like thinking that the lamp wasn’t working because another lamp plugged into the same outlet was loose. Electricity doesn’t work that way. But Clark doesn’t seem to understand that, and he painstakingly checks every single bulb. And his wife, from time to time, flips a switch in the mudroom causing the lights to go on or off.

Debugging software is just like that. You can get so focused on the minutiae that you neglect a piece of the big picture. We, the audience, see the connection between the light switch and the lights switching right away. But neither Clark nor his wife have a broad enough perspective to see it.

Another example happens from time to time in my office. The programmers are working away on some difficult problem. Spending God knows how long, but hours at least. In teams. And they can’t figure it out. Eventually, they call in the Alpha Geek. I glance at the screen. Ask a couple key questions. They answer. Dramatic pause for effect. Then I tell them the solution. Turn on my heel. And walk away. (I’m always right, there is no need to stick around to find out.)

It’s great fun.

Part of that is my experience and my knowledge base. I just know a lot more stuff than most other programmers. But a big part of the reason I can perform that magic trick is because I haven’t been involved in all the hours of investigation leading up to it. They tell me what it isn’t, and I can therefore easily see what it is.

My father was a psychologist, and he was fascinated by brain chemistry. Back in the 1960s, a lot of the mechanics of how the brain works were just being sorted out, and he was there keeping up on all the discoveries and figuring out how they tied to human behavior.

He was able to explain to me the mystery of practicing a musical instrument. If you’ve ever done that, you’ll know that your ability to play a piece degrades the longer you play it. You don’t get better while you practice, you get worse. When you are first starting out, you get a lot worse, really fast. This is very discouraging to young musicians. An athlete understands that the longer they train, the more tired they become. But brains shouldn’t get tired. They don’t have muscles. That doesn’t make intuitive sense.

More accurate model of the brain, although I miss that chick from the first picture

As my father explained it, there are neural pathways that encode the activity you are practicing. And each pathway can only fire once. Then it has to recover. So each time through the piece, you are using a different pathway in your brain. And eventually you run out of pathways, and you have to start making new ones. And this is the first time that pathway ever played this song, so cut it some slack for making so many mistakes, already. Sheesh.

So practice makes perfect, but not until tomorrow. During practice you will use up all your good wiring, and have to start building new wiring.

I see this everywhere. My first three holes on the golf course are always my best. Then all that intuition and “muscle memory” is used up, and I’m left with having to think. And thinking and golfing do not mix.

Is this actually how the brain works? Beats me. But it’s what my dad thought, so if it isn’t exactly right, it’s at least a good analogy.

Those programmers in the office have no pathways left to see the simple solution. I’ve been sitting in my office doing God knows what, and my pathways are ready for action. They are pushing through the end of a marathon, and I’m a sprinter who just finished stretching. Of course I can show them up.

And that takes us back to this morning. While I was working and working on a problem and not getting anywhere, I was using up all my pathways. And while I was busy testing every lightbulb, I used up all the really great pathways, so there was no way I was going to see the light switch part of the equation.

While I’m sleeping, my brain isn’t processing. It’s recovering. Those pathways are getting all fixed up, ready to solve some hard problems. And there was this pestering one sitting there in my hippocampus or wherever the to-do lists go, and one of these fresh and ready pathways picked it up and said, “Well, duh!”

And, for me at least, that happens at 4am. And then I wake up. And there’s the problem and the solution, fully formed. Ready to go. I don’t even need to write it down, because it’s so obvious.

I feel a little bit guilty sometimes. It seems like I’m cheating when I figure out the big answers to the big questions and I have no idea how that happened. The answers come suddenly, and without explanation. I know that they must have happened in my brain, so I guess I get the credit. But still, it doesn’t seem quite fair to everyone else.

Not a lot guilty. Just a little bit.