Monday, January 22, 2007

Getting Your Head On Straight

There is a fundamental problem in building software, and that is deciding what to build. It's fundamental because everything else rests on this one decision, and it's often overlooked, or made in the wrong way. Because everything else depends on it, you have to start at ground level, from first principles, if you want to make a sound decision.

Thinking from first principles is simple but hard. Very hard.

The need for a piece of software may come from several directions. It may arise because of a critical need of the business. Then again it may be only a perceived need. It could be an impulse, driven by one strong personality or the tide of politics internal to the business. This would not be good.

You may need software because of new laws to comply with, fear of competition, or just because you're feeling a little behind and want to keep up with trends by copying everyone else.

A good reason will arise from a true need, and if carried through it may result in a real benefit to your business. If you're working from faulty principles, though, you'll not only have a poor foundation but a poor result.

You need to know your business through and through. And your customers. One way or another you're in business to do something for someone else, something they can't do, or don't want to, so they come to you. If you can deliver they'll be glad to pay you what you need to get by. But you do have to keep their needs in sight.

Somewhere along the way you'll have to deal with staff, even if you're self-employed. The people you employ are the experts in running the business. They know how things work, and when, and all of them are acutely aware of exactly everything that doesn't work. They are your surface of contact with customers, the membrane your business breathes through, its nerve endings.

Staying close to your customer's needs is critical, but so is anticipating where they might be going, so you need to make your decisions match the long term interests of the business. With the customers, and your staff, and the long term in mind, go for the greatest gain possible from the minimum investment.

Plan to start small, with something functional, something you can use tomorrow, or the day after, but don't shoot for next year. You need to start small and see how it feels, then let the software evolve and accrete over time while it's being used.

Evolution is a wonderful thing, and it has a proven track record. If you build software that your business really needs, and start small, and keep adding to it, then you'll get to keep testing it every day, starting with the first day, when it's still very small and very simple.

Daily stressing of any living thing is the best way to have it grow up to be strong. The more you use the first parts you create, the more time you'll have to squeeze out all the gaps, and build more and more strength. As time goes by, your foundation will be more than strong enough to support the whole edifice, no matter how big it gets, and the whole structure will have been thoroughly tested by time.

By starting small you'll also be able to stay within your budget, and within your competence, and assess all risks. You can measure small things pretty easily, so when faced with small problems you'll be able to head them off, knowing that they're there, and exactly where they are, and that they're small. You can also measure progress a whole lot better on a small project. Because it's small enough to understand.

By starting small you can also keep an eye on resources. Not just money, but time and the energy of your staff as well. Burning through your money, wasting your time, and flaming out your staff are all bad, and much less likely to happen when your project starts small and grows organically.

Then, with a little success under your belt, and because you've learned to assess risks and measure resources, you can think about going outside your areas of maximum comfort and competence and branch into new areas.

But first, last and always you have to keep your head on straight. There are some deep pits to fall into. Maybe you don't need software at all. If you do, you'll likely be better off buying something rather than building it.

If you do decide to build it, there are two attitudes you especially want to avoid. One is dismissive, and it runs something like "Why should I have to get involved? You're the computer people. You should know what to do."

The other is arrogant, and follows the rule that we all have running around in our heads, no matter who we are, that if I don't understand something, then it can't be important, so I don't have to pay attention to it.

Considering that about 80% of software projects still fail after all these decades, you really want to avoid a losing attitude. If you are going down the route of building software, then you have to be real. You have to be serious about knowing what you want, why you want it, and what it will do.

You need a real plan, you need coordination, and you have to measure everything and make sure you're on track. You need to keep checking your sanity all the way through, and every day you need to wonder if this is really the right thing to be doing.

If you don't treat software development like a real job, it can eat you up and put you right out of business. It's happened.

Monday, January 08, 2007

Author Blames Programmers for Hard-To-Use Software

See http://www.foxnews.com/printer_friendly_story/0,3566,241578,00.html

"The problem, says consultant David Platt, lies not with the user but with the programmers, who just don't think like the people who use their products."

So what else is new under the sun, my friends? This is 2007, and the story started back at the beginning, 30, 40 or more years ago. No, more than that -- back to 1954, more than 50 years ago -- with the introduction of Fortran, which was to make programming so easy than anyone could do it.

Programming has never been a science, though taught in colleges under the title of "computer science". There is no science. It is a branch of applied mathematics, and mathematics is a branch of philosophy, of reason, of art for art's sake, for the beauty of it. Mathematics is speculation and is checked by whether the results are consistent with their own premises. I know science, and this is no science. So how can programming relate to products?

We think of computer programmers as being smart wire pullers, clever electricians, especially erudite carpenters who build things, but this is all imaginary. There is no science in computer science, or technology, no high technology in computer programming, unless you think that thinking with our nearly five billion year old wetware, the inherited stuff between our ears, is high technology. Not by the usual definitions. Computer programming is like writing a hit song on a guitar. Pick another instrument if you like, but it's still the same game.

First you need motivation. Maybe you want to be rich, or famous, or both, or you just want to do something difficult, but first you learn to play an instrument. A specific instrument, and to play it well. You learn to play it well enough so that you can play it and not think about it. You have to think inside the instrument, without thinking. It has to be part of you, and you have to be part of it, and the twain shall meet without seam, and the music shall be fine and sweet. Yea. So must it be.

And that's step one.

Once you're good enough that you can do anything you need to, and understand music, and understand the work of others, and sight read, then you can try doing something that's never been done before. You can try to write your own songs and play them on your own instrument.

If you decide to switch from guitar to soprano sax, or to a keyboard, or a violin, you'll be able to trail some expertise along behind you. It won't be a totally new game -- it's still music -- but you'll have to start over and work very hard at gaining new expertise. A new programming language, a new programming environment, both are like new worlds, like switching musical instruments and musical genres.

When the first English fop set foot in North America, sat on a fallen tree trunk, and thought about what to do next he was still on the same planet. The air was the same, the water still liquid and drinkable. He had hunger and needed sleep. The force of gravity did not vary, but this was indeed a new world in many ways, and so it is switching from C to Java or to Ada, from OS-360 to OSX or to XP or Solaris. From PowerPC to Sparc to AMD. It's all the same and it's all very, very different. You can't survive in the new world trying to use the skills and conventions of the old world. You have to start over and constantly eye the forest for wild beasts.

It is never easy. It is always art. These two things are true. "Ars longa vita brevis." Art is long, but life is short. Remember that. It's harder than it sounds. It always takes longer than it possibly can.

We try hard, we all do. It helps some, but software is a new realm. It is possibly the hardest thing than humans have ever invented. It takes layers and layers, one on top of another, all working together properly, to succeed. Electricity, the power grid, the quality of wiring in the building, functioning computer circuits, up to date operating systems, monitors, browsers, interprocess protocols, reliable internet connections, version control, requirements, analysis, design, coding, testing, configuration management, change control. Smart and committed people. Every single piece of everything in a hundred different universes has to operate perfectly, beginning with an idea in one person's head and ending with that idea in everyone's head. And all copies of that idea must match.

And over the history of software we've developed some very bad habits. The winner is the first one to market, and within a given company, the winner is the first one to finish coding. Throw a piece of crap at the wall, and if it sort of sticks you can claim to be done. You win.

You go home, it blows up. You come in a 2 a.m. and fix it, several times in fact, and finally it works most of the time. You're given awards for riding in on your white horse and saving the company. People have come to expect software to be crap. And that's what they get. And they applaud you for creating crap and then fixing it.

Software is a new field of endeavor among humans. We still don't understand what were up against. We keep trying to treat it like a product but it's a process. On the commercial level it's a service, but never a product. Software is never finished, and you can never turn your back on it. It is impossible to produce perfect software because no one can describe perfection in writing, or even conceive of it. There is no proof verifying perfection.

Perfection cannot be converted into a software specification, and so it is not achievable even if we could imagine it. Everyone wants software to do the basic job and also to cover all their own exceptions, but the only really successful software deals just with the general case, the basic case, the 60% or 80% or 90% that everyone needs to get done. We all want that last 0.00001% though, that last little bit that is absolutely unique only to ourselves, but we will never get it unless we write a million programs for every person. Not otherwise. Good luck with that.

But most of all, creating software is an act of the imagination. It is a fantasy trip. A software engineer is a dreamer, a designer, the creator of imaginary machines.

Do you think you have ever seen a software program in action? Well, you have not, and you never will. It is like being in someone else's dream. Programs are written down in code, and you can do that yourself, or look at someone else's, but code is only a score, it is not the performance. Code is translated into something only machines can understand, deep inside those machines, and brought to life there, translated once again from a machine's excuse for language into a stream of electrons, the push of electric fields and the pull of magnetic fields. A computer program runs in the dark, among those electrons, and resistors, through fields of capacitors and transistors, and is entirely magic and alien to us and untouchable and incomprehensible.

So software, fundamentally, is the outline of a dream, and if you deal with it as a user, then you have no real knowledge of it in fact, and only slightly more knowledge if you are its creator, or one of them, for any useful software is the product of at least dozens, or hundreds, or thousands of individuals all working together, each on his own piece. Even they don't know. They can't.

Can software be better. You bet. Will software be better? You bet. Just wait a bit. It will take a while. Several more decades, at least. Remember, this isn't an easy problem, like developing safe and universally-available air travel, or sending a can of humans to the moon and back. Those were easy. Software is the hard stuff.

Right now you've got some things that sort of work. Be glad about it. Be glad that you have some software, and that is sort of works. Be upset about the rest, the crap, the empty promises, business as usual. We can do a lot better. But don't worry too much right now. If it's a matter of markets at work, then support, through the market, those things you like the best, or if that isn't possible, then support those things you hate the least. I'm right there with you. But there are lots of people pushing back against us. A whole long history of recurring stupidity.

Eventually we'll start to catch on. Maybe this internet thing will do it. Maybe the development of the internet really was more important than the invention of the printing press, because of the way it brings people together and allows them to share.

"None of us is as dumb as all of us" may be true if we're inside a closed system with no air, but the internet isn't like that. Ideas migrate freely across continents and form unexpected flocks. This may be our hope. We may find new ways of working together so those bright software engineers no longer have to work alone inside cramped little walls like chickens confined to cages to lay our eggs, but can be part of a larger and more intelligent community.

Together I think we can get this right, but it will take all of us. Let us point our fingers not at "them" but back at ourselves and think about what each of us can do to help. It's a hard one.