Software Project Survival Guide: How to Be Sure Your First Important Project Isn't Your Last. Redmond, Wa.: Microsoft Press, 288 pages, 1997. Retail price: $24.99. ISBN: 1-57231-621-7. Available from Microsoft Press 1-800-MS-PRESS (1-800-677-7377). Buy SPSG from Amazon.com.

Computer Literacy Talk About SPSG, January 31, 1998 

McConnell: - Steve McConnell

Audience: - (Questions)

McConnell: It’s nice to be here to talk today about the Software Project Survival Guide. I always like talking about software development to software developers because I think software developers have a really unique way of looking at the world. As an example of what I mean, you might have heard about, the time a boy was taking a walk in the woods. He came across a frog. The frog said "Kiss me and I’ll turn into a beautiful princess." The boy picked up the frog and looked at it and smiled and put it into this pocket. The frog said "Kiss me, I’ll turn into a beautiful princess, and I’ll stay with you for a week." The boy pulled the frog out of his pocket and looked at it and put it back into his pocket. The frog said "Okay, I’ll stay with you for a week and I’ll do whatever you want." The boy patted the frog and smiled and finally the frog said, "Look, what’s it going to take? I’ve told you if you kiss me, I’ll turn into a beautiful princess, I’ll stay with you for a week, and I’ll do whatever you want. Why won’t you kiss me?" The boy said "Look, I’m a software developer, I don’t have time for a girlfriend, but a talking frog is cool."

Well... there are a lot of projects that I’ve seen lately that haven’t been so cool. I’ve been trying to figure out how to talk about the Survival Guide. What I’d like to do this afternoon is to describe a project that I was involved with as an observer over the last couple of years, and use that as a jumping off point for discussing effective software development practices and ineffective software development practices. You should have a handout that was on your chair when you sat down that contains a list of top do’s and don’ts. This is summarized from the last chapter of the Survival Guide. As I go through the case study, what I’d like you to do is look at that list of do’s and don’ts and bounce that off the case study and also bounce it off your own experiences. And after I describe this project that I observed, what I would like to do is open up the floor for discussion and talk about what could have been done on that project to make it run more smoothly.

The specific project I’m thinking of was Version 2 of a successful vertical market software application. Version 1 had sold quite well and the company that was developing it decided that they would like to go ahead and release Version 2. Unfortunately, they did not really have the resources they needed to develop Version 2 in house. So, they decided that they would like to put the project out to bid, get some proposals from vendors and then have a vendor do the development work for them. The first thing they did was to create a statement of requirements. That was a fairly detailed statement of requirements. It was about 50 pages long, I guess. And then they held a vendor meeting to answer vendors’ initial questions and 10 vendors showed up that meeting. After holding the vendor meeting, a few weeks went by and at the proposal submission deadline, five vendors submitted proposals for the project. Out of those five proposals, there was one proposal that came in at around $300,000...

I should back up and say one more thing about the proposal meeting. At the proposal meeting, the client did not mention what his budget for the project was, but they did say that they saw as a relatively minor upgrade to Version I or incremental upgrade to Version I. They said they’d really like to have the software delivered within five months, and that they had a strong preference for a fixed price bid. Although they would accept time and material bids if the vendors absolutely felt they had to provide a time and materials bid rather than fixed price.

So the proposal deadline came and five vendors ultimately submitted proposals. The low bid was $300,000. There were three bids in the $400,000 range, and then there was a fifth bid that was at $1.2 million. The client took a look at that set of bids and threw out the low bid and the high bid with the expectation that since those were on the extreme ends of the scale, they were statistical outlieroutliers and therefore weren’t valid. They took a look at the bids in the middle and selected the least expensive bid, which was right at $400,000, and then negotiated the vendor’s price down to $350,000, and they got started with the project.

I guess I should also mention that of those bids, the four lowest bids, all said that they could get the project done in five months. The fifth bid, the high bid said the project would take a year, and all five of the bids turned out to be time and materials bids, rather than the fixed price bid that the client had asked for.

So the client selected the vendor and got the project underway. Five months came up pretty quickly on this project and at that point the vendor was nowhere near finishing the software, and the client really was not all that concerned about it. They felt that they had a market window of about eight months. The vendor said they were quite certain that they could deliver the software within eight months, and so they rescheduled for eight months and worked toward that. Well, they got to eight months and the vendor still was not very close to shipping the software. So they took a six-week schedule slip, after that they took another six-week schedule slip, after that they took another six-week schedule slip, and so on. Finally, after 14 months, the vendor delivered about 25% of the originally planned functionality at a cost of 1.5 million dollars.

When you consider the combined effect of the cost of schedule overrun with the functionality under-run, in real terms this project overran its schedule by about 1000% and overran its budget by about 1400%.

So what I’d like to do now is ask you if anything jumped out at you about this project description, either from your own experience or maybe from that list of do’s and don’ts that was handed out at the beginning of the talk about what might have been done to make this project go better. And if there are details that you need to get a little bit more explanation about, you could certainly feel free to ask about those too.

So who noticed something?

Audience: Well, if they just used the cost and didn’t do any other homework, didn’t they kind of get what they deserved? Isn’t that kind of egregious?

McConnell: Yeah, I agree with that. I think, in this case, the vendor made the selection based on choosing--mostly on choosing--the second lowest bid after throwing out the first lowest bid.

What’s really crazy about that is that these were time and material bids. The difference in the bids was really accounted for by different expectations about how much work was involved in the project. It wasn’t based on the fact that one vendor’s rates were 25% lower than another vendor’s rates, it was based on the fact that they thought there was 25% less work there. So yeah, I think choosing the winning vendor based on the dollar value of a time and materials bid was not a very good idea.

Audience: Lack of internal milestones.

McConnell: Yes! "Do" number 5 is "Take periodic snapshots of project health and progress and re-plan when necessary." In this case, the client really didn’t get any kind of progress tracking other than the fact that the vendor missed its ultimate delivery of the software. There was really no tracking of the project on an ongoing basis.

I happen to know that for a fact, but even if I didn’t know it for a fact, I think that I would be willing to bet with almost 100% certainty that that was true for the following reasons: Suppose I’m a client and I have a vendor working on a project and I’m tracking it carefully and I get to the point in the project where I say, "Okay, it really looks like this vendor is having some problems. They’re going to deliver the software 100% over budget. 100% over budget -- that’s not very good, but we don’t really have any choice at this point. I guess we’re just going to have to live with it." I can conceive of a client going through that line of thinking. What I can’t conceive of is a client that says, "We’re tracking this project carefully and it really looks like it’s going to go 1400% over budget, I guess that’s the best we can do." I simply can’t conceive of a client going through that thought process, so the conclusion I would have to draw is that they simply weren’t tracking the project at all, or at least not in any meaningful way. And we can see obviously the damage that’s caused by that.

In this particular case, in the winning vendor’s proposal there were some initial tasks that were described and that were estimated at a fairly low level of detail for some of them. And the initial work that was done on those tasks within the first six to eight weeks of the project overran the estimates by roughly a comparable proportion to the overrun of the entire project. And so, if the client had been paying attention, they really could have had a pretty good idea six to eight weeks into this project of the ultimate project overrun magnitude.

If they had simply been paying attention ... I think with my background, I would have been very confident at eight weeks saying this project is way off the rails, way out of alignment with the initial estimates. And I think even someone who hasn’t gotten into this stuff as much as I have would have been very confident. Even if they weren’t confident at eight weeks, they would have at least put the project on probation, or on alert, after eight weeks. And I think by the end of 12 weeks, they shouldn’t have had any doubts about where the project was headed.

Audience: I think that the problem was when they described the high bid at the beginning as though it was an outlier, like there was some reason that that one vendor would estimate that the project would take 1.2 million dollars. They should have figured out what the doubts were that that person who submitted the high bid had, and then asked the other vendors whether they had addressed those doubts.

McConnell: Yeah, in this case, actually there were a couple things going on. One is outliers, when you throw out outliers, you typically do that because you’re looking at random events. It’s a statistical idea that applies to random events, and these proposals were not random events. So this approach of throwing out "outliers" was really not a valid.

In this case, the high bid actually had another interesting characteristic, which was that that particular vendor had hired one of the programmers who worked on Version 1 of this software, and that developer was involved in creating that high bid. None of the other vendors had any particular expertise with that software and so I think the client was really doubly to blame in this case. One for not really understanding what an outlier was and the other was for ignoring the fact that the best-informed vendor created the highest bid.

That doesn’t mean that they had to accept the highest bid, but I think what it does mean is, if they want to contain their risk on this project at all, then they really ought to take a hard look at why the vendor that knows the most about this software is bidding the highest amount? What is it seeing that the other vendors don’t see? And that should have created an opportunity for the client to go to the other vendors and say, Have you considered factors x, y and z, and how does that affect your bid? That wasn’t done with this project.

Audience: It is somewhat surprising that all of the vendors bids came in so close to each other except the really high one.

McConnell: I think it’s hard to look at that without being a little bit cynical about the bidding process. The client had not said, "We have a budget of $350,000," which was in fact their budget. However, they did say, "We want this done in five months, we see it as a relatively small upgrade to the product," and I think that there were enough clues given that the vendors came in with bids between $300,000 and $400,000. So they actually did end up zeroing in on the client’s budget pretty well. And I guess that I really am pretty cynical about that process.

I think that there almost had to be an attempt to guess the client’s budget and then reverse-engineer the proposals based on what they thought the budget was. If I wanted to be a little less cynical about it, if I wanted to give it the most charitable possible explanation, then I think what I would guess is that the vendors all said, "Well, the client knows this software a lot better than we do. They say it can be done in five months, who are we to second guess the client when we don’t know as much about it as they do?"

You can’t exactly start work on a new project, staff it up to 25 people instantly and work effectively. I think that given the description by the client, it was almost inevitable that the bids would come in that way.

Yeah, so there is some convergence among the estimates. I don’t think it was for technical reasons, I think it was more for sales and business reasons. And I think that is one of the reasons why the client ended up taking the bid it did. The client had a budget of $350,000. They hadn’t gone through any very careful estimation approach to come up with that budget number. They just kind of said, "Well, what do you think it will take to develop Version 2? Well, I think we could get $350,000 from the budget committee. Okay, well, let’s try to do it for that amount." And that was the thought process they went through. I think that when the vendor bids came in in that same ballpark, the psychology on the client part was kind of interesting. Which is the people on the client said "Hey, you know we’re pretty smart. We didn’t even work that hard on estimating this. It looks like we were right."

The client ended up confusing cause and effect.

Audience: How did the client pick the feature lists? Was there a lot of feature creep on this project?

McConnell: I don’t know that I would say the feature list was out of bounds. I would say the vendor, who was selected and started out working under the assumption that the project was a lot smaller than it was, worked very, very inefficiently. And so the client got only 25% of the functionality for 1 1/2 million dollars. I think they paid an incredibly high price for the functionality they got. I wouldn’t necessarily say the feature list was outlandish. I think the feature list might have been excessive, but I think probably the high vendor was in the ballpark. It might have been off by 25% or 50% for delivering the whole functionality, but I think there’s a big difference between starting out a project with a realistic idea of what the true scope of the project is versus having to adjust your approach midway through the project when you realize it’s bigger than you thought at first.

One of the key ideas I try to get across in the Survival Guide is the idea that software is a little unusual in that if you don’t do it right the first time, a lot of times you don’t get a chance to fix it later. I should make clear what I mean by that. If you start out developing relatively low quality requirements, low quality designs and so on, you can get a chance to fix the software. That is, you can rework the code and bring it up to some level of quality with testing and so on. Sometime you can get the defects out of the software. So in that sense, yeah you get a chance to fix it.

But what you don’t get a chance to fix is the project itself. We have this phenomenon in software where if you have a requirement defect, let’s say, that might simply be a defect in a one sentence statement of requirements. But if you can change that at requirements time, all you have to do is change that one sentence. But if you let that stay in the project, in the flow of the project, that can turn into a handful of design diagrams, it can turn into several hundred lines of code, dozens of test cases, training of end user support personnel, online documentation and other kinds of documentation, help screens, paper documentation, and so on.

Clearly, if you have a chance to correct the error when it’s a one sentence statement of requirements or a handful of design diagrams, that is an awful lot cheaper than if you have to rework hundreds of lines of code and dozens of test cases and dozens of help screens and so on.

So in a software project, one of the things that we find is if you don’t get it right the first time, that is requirements time or design time, you can fix it later on -- maybe. Maybe you can fix it later on, but you’re going to fix it only at very great cost. So you can fix the product later, but you can’t fix the project. It’s going to be very, very expensive, and you could have done it much more efficiently if you had applied that effort at the right time in the project.

Audience: Were the vendors fully aware of the whole feature list before they made their bids and sort of a corollary to that, was the customer really honest about what it was they were expecting? And if not, were they being actively misleading or merely ignorant?

McConnell: The customer put out a 50 page requirements document. I think there was actually blame in both camps in this case. There was a 50 page statement of requirements, and I have a hard time seeing how any of the vendors bid as low as they did based just on that statement of requirements. Having said that, I think that the client actually did in this case take many opportunities to try to persuade the vendors that the project was a small project, an incremental release or an incremental improvement over Version 1 -- that kind of thing. And I can’t tell you whether the client was being actively dishonest or whether they were just engaging in really active wishful thinking.

I think they wanted it to be a small project, they wanted it to cost $350,000, they wanted a vendor who would give them a fixed price bid for $350,000, and things just didn’t work out the way they wanted.

Audience: I’d be interested in knowing why the client didn’t take stronger action after 5 months, when they saw the vendor was having problems. Did they do any kind of major status review? Why did they accept the vendor’s revised estimate of 8 months?

McConnell: I think really the answer is the client’s evaluation of the vendor kind of went like this: "We like the programmers who are working for the vendor. They seem like they’re honest. They seem like they’re trying hard. They seem like nice people, and so if they say they can get the project done in eight months, they probably can."

I think that was the extent of the review on the client side. And I think all those things were true, at least my observation was that the people working on the vendor side were sincere, they were trying hard, they were being honest and they were nice -- and they were in way over their heads. That was the real problem.

Audience: Was there any background check, or references, with these vendors?

McConnell: I really don’t know the answer to that. Vendors were required to submit references, but I don’t know whether the client actually checked the references.

Audience: Did the vendor do any additional work to try to get a better idea of the scope of the project -- additional requirements work, or anything like that?

McConnell: One of the interesting thing about this project was that the winning vendor’s proposal had suggested an initial prototyping and further specification phase, and the client crossed that out of the proposal and said, "We don’t want to spend the extra money for that."

We can see how well that idea worked out.

Audience: Wasn’t this really some kind of a maintenance project? Did the vendor take that into account?

McConnell: The answer to this question goes back to the earlier question about what kind of expectations did the client try to set about this project. In the proposal materials in this case, the client actually said that there were certain sections of the old program that would simply be completely thrown away and rewritten, and the other sections of the program would not be changed at all.

To me, that sort of thing never happens. You can hope that it happens, but unless you’re on the eighth generation of your software and you know that the subsystem interfaces are extremely pure, that kind of approach is really a pie in the sky wishful thinking. And to the extent that the client said that, I think that that was, at best, an irresponsible statement and I think at best it was extremely wishful thinking. At worse, it was misleading the vendors.

To the extent that the vendor believed it, I think that there’s some blame to be had on the vendor’s side too because I think anybody who has been through this sort of activity a few times would have to be pretty skeptical about that kind of a claim. And one of the interesting facts about this, which goes to the same point, is that a couple of the vendors requested source code so that they could see what the quality of the code was, and the client refused to provide that. Which again, I think on the vendor side it should have been a pretty big warning sign. .

Audience: Are this client’s project management skills any better now than they were?

McConnell: Well, who do you think the client blamed for all the problems in this case?

Yeah, the client laid 100% of the blame at the feet of the vendor. This was the client’s second attempt to outsource a project. The first attempt had gone about the same as this one, although it was on a smaller scale. I think the client was very frustrated with the vendor’s poor performance in this case, including their poor management performance and their poor ability to track progress. In some sense, I think the client’s attitude there is justified, but I think this is one of those cases where the client’s attitude throughout the project was, "We didn’t have the technical resources to develop this software, that’s why we outsourced it. It doesn’t make sense for us to also practice really active management of this project. If we have to do that, why do we bother to outsource the project in the first place?"

From kind of a moralistic point of view, I can see that that line of reasoning makes sense. But from a practical point of view, you’re just asking for trouble if you take that line of reasoning. You can say who should do all this work as much as you want to, but if your main concern is getting software of a level of quality you want, when you want it, you have to take on that responsibility yourself, even if it isn’t "right" or it doesn’t seem fair that you should have to do that work.

And one of the points I make specifically about third party relationships like this is that it certainly may reduce the amount of management work involved, but I think it increases the degree of management sophistication required to effectively keep tabs on the project.

In this case, I think the client side management really needed to insist at proposal time on some very tangible project tracking mechanisms so that they didn’t have to just rely on this gut feeling about whether the vendors were good guys or not, but could actually have tangible evidence of progress all along throughout the project. And that just wasn’t so. So I’m not sure the client learned anything about project management on this project, and, interestingly enough, this project, among other reasons, basically caused this client to say, "We don’t want to be in the software business anymore." That wasn’t their main line of business, so they are in the process of completely getting out of it.

Audience: What kind of project management team was put together?

McConnell: What kind of project management team was put together? Was it experienced developers or was it just a bunch of developers who were told to go off and make this work?

On the do’s and don’ts list, do #9 is start the project with a small senior staff. One thing that’s kind of interesting about this project is that the two main developers on the project were ages 35 and 40. So you might think that that would constitute a small senior staff, but the 40 year old had gotten his Ph.D. in English and had kind of made a career transition to software, and this was the first project he actually got paid for writing software for. And the other guy was a more experienced software developer, but he’d been working mainly in scientific systems and so this was his first shrink wrap application. I don’t know whether he was a good scientific systems programmer or not, but in terms of the unique demands for mass distribution, end-user-intensive kinds of products, he was kind of a rooky. So they started the project with a small staff, but I think it really wasn’t a senior staff. And that went for the project manager too. He had managed some large projects, but they had been in-house client/server kinds of projects and so I think if you didn’t really appreciate some of the differences between in-house software and commercial quality software, you might think that the team looked okay. But if you understand that there is a pretty big difference in robustness and compatibility issues and that kind of thing, I think you’d see that this team was pretty green.

Audience: I have a question about what should the vendor have done at about five months, should they have really taken a hard look and tried to accurately reschedule or should they have done what they did because the client might not have accepted a really huge overrun?

McConnell: I think this is one of those cases where you get to see what the character of the vendor is. There’s a really odd, odd interaction in this case between dishonesty and incompetence I think.

There is active dishonesty where you get to five months and you say, "we know that this project is going to take nine months more, we’re just going to tell the client that it’s only going to take three months more." That’s active dishonesty.

And then there’s simple incompetence where people take a really hard look at it and they say after taking a really hard look at it, "we are 100% sure that we can deliver this in three months."

And there’s this kind of gray area where they just don’t really want to know that it’s going to take, they think they might not like the answer if they find out, so they just don’t try very hard to find out.

And so I think that this case was somewhere in that gray area. I think given the nature of the initial bid of five months, if somebody was basically competent to start with, they probably wouldn’t have bid that five months in the first place and so they wouldn’t know that they should announce a 200%, well 300% slip when they got to the five month mark. So I see it as a case where it’s hard to address the hypothetical really directly because it asks you to assume competence that’s not in evidence.

Audience: I mean if they were competent, what should they have done?

McConnell: I really don’t know how to answer that question because if they were competent, they wouldn’t have given the really low bid in the first place.

If I can restructure your question a little bit. What if they brought in some really expert consultant who took another look at their requirements and at that point, they got a really clear idea that they were looking at a really huge project and massively missing their deadline. Realize it’s not nine months more at that point, they only delivered 25% of the original functionality. So at that point to come up with a whole project estimate, I mean we can assume they’re talking maybe 15 to 24 months for 100% of the functionality.

So what should they have done at that point? I guess I draw a pretty hard line there. I think they should’ve had a meeting with the client and said, "We goofed up. We did a bad job in the proposal. We’re to blame, now we see that this project is a lot bigger." If they really felt the client had actively misled them, I think they could try to share some of that blame with the client.

I don’t see this as a gray area. I think they just need to lay it out for the client and then try to work together with that. I think in this case and in a lot of other cases too, the client would have recognized that there wasn’t perfect information at the beginning of the project to estimate the whole project. And of course the client is not going to like hearing that news, it’s going to be awful news, it’s not going to be well received. But I think in any case, you can convince the client or at least make the case that it’s better to get the bad news now than it is to get it later, and if the client blows up and is completely unreasonable, then you may be better off without the work. So I think that’s what they should have done assuming they had the expertise to do it.

Audience: What was the user involvement? What kind of beta test cycles were cycles were involved and were there any quality criteria?

McConnell: The user involvement I don’t think was a really big issue. This was Version 2. I think the client company had a pretty clear idea of what functionality users really wanted in Version 2, so the bad requirement stuff wasn’t really an issue in this case. As for Beta testing, they went through some kind of a thing with a handful of in-house users of the product. They didn’t do anything you would normally think of as beta testing. And as far as quality practices are concerned, as we see so often in cases like this, the system testing cycle got truncated because they needed to get the software out into the marketplace.

And as a result, I don’t know if anybody would be really surprised at this, but my understanding is that the software that was released was very buggy and unreliable, and they began working on a maintenance release almost immediately. User satisfaction with the functionality was pretty good. User satisfaction with the reliability was not good.

Audience: What are some of the things you think killed their productivity? It sounds like one problem is that if they try to go out with this ambitious list of features and then five months later, they can’t do all that and they throw away a lot of it, they’ve wasted a lot of time on stuff that is eventually going to be thrown away.

McConnell: What do I think killed their productivity? I think the staff was probably incapable of doing a good job on this project. Their two people with the most responsibility were very junior. They had very little Windows experience. They had very little or no shrink wrap experience. They had basically no experience in the domain area of this application. If I had been brought in at that early stage of the project, I would have said the aggregate risk level on this project is simply insurmountable. And so I think in some sense, the die was cast very early in that project. Personnel issues were a big, big deal on this project.

In terms of process issues, they didn’t really have much of a project plan there and the project plan that was stated in the proposal was so far out of alignment with the real scope of the project that they essentially abandoned the plan and let the project run free form. So they didn’t get the benefit of any kind of planning either, and so they had bad personnel, or ill suited personnel, bad process. There are lots and lots of reasons why there was basically 100% chance this project wasn’t going to come out anywhere near what they thought it would.

Audience: Companies are always dealing with make or buy decisions. I assume the client had their own in-house staff produce Version I. Why didn’t they just produce version 2 in house?

McConnell: The answer to that is an interesting case of a medium size mistake giving rise to a big mistake. On Version 1 of the software, the client had put together a team that included four contract programmers and one in-house programmer and after Version 1 was completed, the four contract programmers scattered to the winds and a few months later the one in-house programmer left the company. So the client company really didn’t have any in-house programmers who had any familiarity with Version 1. And I think that was a really fundamental mistake they made on Version 1.

This was a fairly key product for them and if you look at the fact that it’s a key product, it really doesn’t make much sense to have four contractors and one in-house employee developing all this knowledge about this product. I think they should have done either of a couple things. Either initially they should have staffed Version 1 with more of their in-house programmers, or at the end of the project, if they didn’t realize it ahead of time, when they realized, hey we’ve got this strategic technology, it’s been developed with four contract programmers and only one in-house programmer, this is a big risk for us. They should have tried really hard to convert the key contractors to employee status so they could keep that expertise in-house. Just from a risk point of view, there were some big risks that hadn’t been addressed in Version 1 that ended up giving rise to other kinds of problems in Version 2.

Any thing else jump off that list of do’s and don’ts at you?

Audience: Is there any evidence that the vendor actually tried to tell them that they were incompetent?

McConnell: That the vendor was incompetent?

Audience: Yeah, not capable of completing the work, once they got into the project and understood it better? Did they try to tell the client they were in over their heads?

McConnell: I’ve seen the same thing you have where the client wants so much for the vendor to be able to do the job at the price the vendor said that they will tell the vendor, in essence, "you are qualified to do the job even if you think you aren’t."

In this case, I think it was actually even worse than that because the vendor had actually proposed as its main proposal an implementation in Visual Basic, and they had said "Oh and by the way, if you really want to do it in Visual C ++, then for an extra $50,000, we can do it in that." And much to their surprise, the client said "Well we do really want to do it in Visual C ++, we’ll pay the extra $50,000 for it." Well the vendor had made its proposal based on the strength of some specific people in Visual Basic, and all of a sudden they found themselves scrambling to try to find some people who knew Visual C++.

And what’s really awful about this project, which is where I really became intensely aware of this project, is the vendor called me on Thursday afternoon and said, "We have our first meeting with the client on Monday, do you know where we can get three experienced C++ programmers?" And I said "No, I don’t know where to get that." The situation was really dicey. I actually ended up talking with my company’s attorney about that because it seemed to me at that point there were pretty good signs that this was a train wreck waiting to happen and I felt like somebody ought to say something, but my attorney said otherwise.

Audience: Did the requirements stay pretty stable or was there a lot of creep and moving around?

McConnell: The requirements were basically stable. If you were to say, "describe a typical project kind of like this one," I would probably include requirements creep as one of the typical characteristics of this kind of project. But in this particular case, they really got lucky on that one and there really weren’t any significant requirements changes.

Audience: 50-pages of requirement sounds like a lot for a minor revision.

McConnell: Yeah, I think that 50 page document included upwards of 400 individually enumerated requirements, and they weren’t specified in amazing detail either. One of my friends talks about "the squint test." The squint test is squint your eyes and look at something and say "does this even look like it is possible, it’s remotely right?" And I don’t think this passed the squint test.

Audience: Given that the vendors submitted time and materials bids, why did the client even look at the total prices? Isn’t that meaningless for time and materials contracts?

McConnell: Yeah, absolutely. Estimates are meaningless in a time and materials context because the difference in the price is the difference in the estimate of the amount of work involved.

There are a couple issues here. One is that the client indicated a strong preference for fixed price and all the bids came in time and materials. That should have been a warning sign that caused the client to take a look at whether they really understood the scope of the project. The vendors are saying in pretty clear language, "Our estimates involve enough risk that we’re not willing to take on that risk in the form of a fixed price contract. In fact, we’re not even willing to apply a big multiplier to how big we think the project is and do a fixed price." And that, to me, is pretty clear in contract-speak that says "We don’t have any idea how big this project is." So that was one warning sign.

Another warning sign was the spread in the estimates. There was a factor of four difference between the $300,000 estimate and the $1.2 million estimate. That’s another warning sign the client should have paid attention to and didn’t pay attention to. And there are all kinds of things the client could have done in that case.

One thing the client could do that applies as much to in-house projects as outsourced projects is to re-estimate system size effort and schedules periodically. The client essentially is in a position at that time to know that it doesn’t have a really clear estimate. And what it should have done is try to set up the project in such a way that it could re-estimate fairly quickly and get a better idea of how big the project was. It could’ve gone to something like a two phase acquisition where it gives out a fairly short project of one or two months, the purpose of which is to develop information on the real scope of the project. And then they estimate that second part of the project separately. That would have been one thing they could’ve done and I think there were really clear warning signs in this case that something like that was needed. I guess that’s what I wanted to say about that.

We have time for a couple more questions and then we’ll wrap things up.

Audience: Who wound up doing the maintenance version?

McConnell: At the end of the project, there were very unhappy people on both the client and vendor side, there were threats of litigation on both sides, both the client and vendor ended up walking away with just a bad taste in their mouths, and nothing really happened. To the best of my knowledge, in-house staff ended up doing the maintenance. I'm not really sure.

Audience: Did the client do anything to try to build a team spirit? I’m thinking in this context if I screwed up and I know this is strategic, I’m going to try to bring the external vendor in close and be real buddy, buddy with them. So that way even I don’t have a project plan, I at least have a relationship with the vendor I can turn to if things start to go sour.

McConnell: I think that’s an excellent point. That idea of team spirit can apply to just the developers; or to developers and QA; or developers, QA, and management; or to client and vendor relationship.

In this case, I think that that’s right. If the client lacks the technical expertise to track the project really carefully, they at least should’ve tried to stay in close communication with the vendor so that they could have subjective sense of how well the project was going. I don’t think that subjective sense is by any means the ideal, but as a fallback position, in this case that would have been a lot better than what they did. In this case, the client and the vendor were in cities about 60 miles apart so there was a geographic barrier between the client and the vendor that I think contributed to the problem somewhat. In that case, the client really should’ve been very active in making sure that they called the vendor frequently, scheduled interactive meetings and that kind of a thing.

Audience: What kind of staff did the client put on the project or did they just have the vendor go away and come back in five months?

McConnell: The client had somebody designated as the project manager, but as I said, the project manager really didn’t do much and even if I didn’t know for a fact that the project manager didn’t do much, just looking at the outcome of the project makes it clear that the project manager didn’t do much or didn’t have the remotest idea of how to do much on this particular project

Audience: Did the vendor try to educate the client at all about what the project was going to look like?

McConnell: Keep in mind that the people on the vendor side were pretty inexperienced as far as this kind of software project went. I guess I don’t know the exact answer to that question from my information about the project. My expectation would be that, given the experience level of the people on the vendor side, is that really didn’t happen. Number one, because they didn’t have the expertise to share and number two, because if they did, they probably wouldn’t have been far enough down the road with it to know that it was important to share it.

Audience: Some of the key features are probably defined by the code and if you can’t look at the code, what basis do you have for even knowing whether you should do the work?

McConnell: On the vendor’s side,  if I were bidding on that project, to me that would have been a deal breaker...basically when I saw hat answer, I would’ve said, "I can’t bid on this project." Either that or, I think that might have contributed to the time and materials bids rather than the fixed price bids because you’re right, you might end up deciding that the code is garbage and you have to throw all or some of it away. And so, yeah, I think that that has to play a big factor. Assuming that for some reason you have to bid on a project, I think at the very least you have to identify the fact that you don’t know anything about the code base as a big risk. And there are other things they didn’t know about too. Like, is there any high level design documentation? Somebody hands you 150,000 lines of code with no design documentation, it’s going to take quite a while to get to a point where you’re going to be able to work with that. So yeah, I would identify it as a very high level risk item.

Audience: What is the main problem? What is the one thing that they should have done differently?

McConnell: This gets to a point that I don’t mention in the Survival Guide, but I try to mention emphatically when I talk about Rapid Development, which  I talked about in the Classic Mistakes chapter. A lot of times when people are trying to develop software effectively, they’re looking for the one or two things that they have to get right. But there are so many things that can go wrong in a software project that I’m convinced that the key to success in software is not so much getting a few things right, but getting almost nothing wrong. And in this case, they had mistakes lined up. And if they had eliminated the first half dozen mistakes, they still would’ve had a project that was very much at risk. And the outcome shows that. They struggled to get 25% of the functionality delivered. So I can’t answer your question directly. I don’t think there were any one or two things that would’ve made a difference.

Audience: In that case, would it have been easier to start from scratch or evolve the existing code base? This is a common issue. Is there any general answer?

McConnell: Right, this is common issue, do you evolve the existing code base or do you start over? I have a couple of reactions to that. One is that there has been an interesting data point that was created by the NASA Software Engineering Lab. Their rule of thumb is that if you have to modify more than 25% of the code in an existing code base, that it’s more efficient to rewrite it from scratch than it is to modify the existing code base.

The other phenomenon that I see pretty often is that companies get naturally very attached to their existing code base and they’re very reluctant to throw it out and start over, and so what very often happens is that they always do one more major release on any given code base than they should have done. The last release on any given code base almost always involves very brittle code, code that’s hard to extend, code that ends up being very defect-prone, and people end up throwing it away one version too late. And so I think I would probably try to tilt the balance toward rewriting rather than keeping, unless you have code that you know to be very reliable.

Audience: After all is said and done, what has the client learned? Will they do it with their own staff next time? What will they do differently?

McConnell: The comment is what has the client really learned in this case? Is Version 3 going to be any different or are they just going to do something relatively superficial like say well we need to do it in-house next time, but they end up doing essentially the same thing in-house that they did with the vendor the last time?

Yeah, one of the really agonizing things about this kind of situation is that even if the client calls in somebody like me, odds are pretty good that they’re going to ignore what I tell them. A lot of times the changes need to be made at a pretty high level on the client side. That’s one of the reasons I wrote this Survival Guide. One of my big goals for that book was to write a short book that would be short enough for upper management to have the time to read. Upper management isn’t going to read any of my other books because they’re too long and they’re too detailed, and it doesn’t make sense really. So I had the manuscript for the book reviewed by upper managers at large organizations to make sure that I was considering the right issues and talking about them in the right way.

I guess I am here to promote the book, so in a way it is a sales pitch for the book, but I think that there is--regardless of whether you use that book or you use your discussions or you work with your manager to educate that manager’s manager--there is this education in the industry that needs to happen in upper management.

Software makes up an increasingly portion of many company’s budgets and upper management needs to get a lot more sophisticated about their software decisions, and that’s a sophistication they’re not going to attain throughout osmosis. It’s actually going to require some education from somewhere and if it has to come from the bottom up, then fine. It’s probably in your interest if you’re working for that company to get that started.

Audience: Given the amount of QA they had, were there red flags coming up from QA that they could’ve paid more attention to?

McConnell: The warning signs in this case were not that subtle. They didn’t need QA. After five months, the software simply wasn’t there. Nobody was even saying that it was there. It just wasn’t done yet. So there wasn’t anything to test or perform other quality assurance on. In some other projects, the warning signs might be a little harder to distinguish, but in this case, they weren’t. They were very, very blatant.

Audience: How did you find out so much about the project? What was your involvement?

McConnell: My involvement was that I knew people on both the vendor company and the client company, and had gotten this phone call that my lawyer advised me not to do anything about. And so my involvement was that I was very frustrated about not being able to go to the client company and say, "You’re really headed for trouble here and I have some specific reasons to think that." And I also couldn't go to the vendor company and say, "I think you have the wrong idea about how big this project is." So it was a painful experience for me because I felt like I should do something about it, but I wasn’t able to do anything about it. So I kept tabs on it pretty carefully.

Audience: Am I seeing project management get better, stay the same or get worse?

McConnell: I think there’s a huge amount of variation in the software industry right now. One of the issues that we’re dealing with is that the industry is growing more quickly than new people can be trained to work in the industry. There have been front page stories in the New York Times and the Wall Street Journal in the last couple months that have talked about the labor shortage for programmers. Estimates in the US range from 200,000 to 400,000 open programming jobs right now. We’re creating new programming jobs at a rate of about 100,000 a year in this country and we’re granting about 35,000 computer science and computer related degrees a year. So most of the people coming into the field are not getting even as much training as an undergraduate degree in computer science. And so I think that we’re basically growing this pyramid and the number of people who are educated and experienced on the top and the middle of the pyramid is relatively small compared to all the people coming in at the bottom.

That whole situation is compounded by the fact that we’re still a relatively young industry and I think there are really good management practices out there and I talked about some of them in Rapid Development and the Survival Guide, but I think the knowledge of those is not very widespread. So in a way, I think the situation is basically stable or the level of management may be basically stable, but the industry is getting bigger and so it appears maybe that the problem is getting worse. There are pockets where the management is very sophisticated, but I think at this point, I really couldn’t call it anything much more than a few pockets.

I’ll take two more comments.

Audience: Earlier you commented that if somebody brought you in as a consultant in a case like this, your advice might be ignored so what hope is there really?

McConnell: I might have been being a little too flip when I made that comment earlier. If the project manager of that project brought me in, I think that there’s a good chance that upper management would ignore whatever came about as a result of that. But if upper management brought me in,  there’s a very good chance that action would be taken. Basically, I think it’s a long, ongoing educational process and the hope has to be long term rather than short term. Bringing one guy in for a day or two to talk about some stuff is probably not going to make a difference unless it’s part of a long term education strategy. My answer is to think long term, not short term and things will get better over time.

Audience: What software might be new but there other things that are similar, what kind of parallels do we draw to other industries?

McConnell: I agree with that point. I think that it’s very constructive to draw parallels to other industries as long as you don’t take the parallels too seriously. And this is the key point: as long as you know enough about the other industry to draw a meaningful parallel.

For example I draw a lot of parallels to building construction based on the idea that you go through a steady progression of activities that range from not very expensive to quite expensive. You’re going to draw blueprints before you actually start ordering materials, sawing boards and pounding them together because it’s a lot easier to change lines on a blueprint than it is to actually move a physical wall when that’s been constructed. And I think that parallels to software from that kind of thing are pretty clear. That is it’s easier to change things in requirements time or design time than it is to change hundreds of lines of code even though it doesn’t look as physical as the materials in a building look. The main cost in both cases is actually people's time rather than the material in most cases, and so the parallel works pretty well.

There are other parallels that are interesting that are not quite as obvious. Like one thing you hear people say sometimes is, "Why can’t these software guys estimate their projects more accurately? Look at the building industry." Well, look at the building industry! For small projects, like if you’re going to build a $200,000 house, you can estimate that kind of project pretty accurately, but when you look at one-of-a-kind projects that are in the millions of dollars, cost overruns on those kinds of projects can be very, very high, for the same reason that they are in software--it’s a new thing. So I think those parallels need to be drawn carefully and with some knowledge about the other industry that you’re comparing them to.

Thanks very much for coming today.