Here’s what I think it comes down to. Say theres some new type of software product. Let’s say the set of all features that this product can have is F, with features F1…F10. In the commercial world, someone will sit down and think hard and decide which subset of F will result in a good tradeoff between development time and usefulness. That is product will be designed to minimize the time spent developing it, while maximizing its coverage of usecases.
That happens becuase with the commercial software product, you try to sell your wares to the most people possible, while spending the least amount of money to create it. There’s a natural incentive to not waste time on features that nobody uses, or on re-implementations of the same feature-set. There’s also a disincentive for other developers to come in and try to build the same thing, unless it’s really truly better, or unless it has such a different subset of F that it can useful to customers who find the original product lacking.
In the open source world, the incentives are backwards. The cost of developer time is small. Instead a developer has the incentive to have his product known, have his name known, which means that he must differentiate his product from others. So given an existing product that has a subset of F, many a developer will choose a slightly different subset of F and develop a product based on that. Maybe there’s some technical reason to do so — there are many standard ones, better design, better performance, old one is unmaintained, because we can — but in the end, the differences, while they may be significant to developers, are mostly insignificant to users.
So this leads to an explosion of many programs doing roughly the same thing. The justification for their coexistence is most often irrelevant to the user. The user would rather just have one product that has the union of the feature set of the similar products. Maybe it’s not possible to have one uber product, but I’m willing to be that most of the time you can take 20 and reduce it down to two or three.
What it really comes down to is shortsighted-ness. People are blinded by the pride and glory of saying you developed the framework instead of the plugin to someone else’s framework. And in developing your own framework, you’re isolating yourself from the benefits of other people’s work, and the ability to take advantage of other people’s time to do maintenance. This leads to piles and piles of slightly different, poorly maintained products.
Put another way, there’s also a missing disincentive for a developer to reinvent the wheel. His own time is essentially free, and costs like hardware and hosting are virtually negligible these days. So what if he implements the same feature set? There are only positive incentives for him to do so.
But then again, who am I to complain. I’m using it, and I’m not paying for any of this stuff, right? ;-P
I guess all I really want to say is if you’re looking to make a big impact for a lot of users, try to improve something that has some traction, instead of reinventing your own, unless you can really justify to a user other than yourself why they should use yours instead of what’s available already. It seems like you should be able to do this before you write a single line of code.
But hey, I don’t want to be responsible for holding back the next great implementation, so code on!
Then again, maybe this is the only way it can work. Given the lack of proper incentives, the only way solid open source programs can evolve is by first having an explosion of competing implementations that explore the feature space and distill the requirements, and then reducing down to a common implementation (or a small number of them with standard interfaces) that can now serve as a building block for other people to build on. It’d just be nice if it was easier to tell when that has happened. There are too many people plugging some half-baked idea for all the wrong reasons.