This seems to be a question that is very relevant to my work, although we don’t quite exactly sell an open source product.
I think pretty much what it comes down to, is that there are so far two (plus one) kinds of open source business models.

• A service-based company (supposedly like redhat) which is not selling the software, but selling a service. The software is only a means to provide their service. This kind of company’s main revenue comes from the selling of the services (obviously) and can pay developers to develop software that is immediately open sourced. This is common in cases like Redhat hiring many gnome developers, or Linus getting paid to do kernel work full time.
• Companies that build closed source software, but build upon open source code bases. There are things like google, or Oracle, as well as VMware (the ESX product strongly benefits from the existence Linux). Maybe not for Oracle, but its arguable that Google or ESX would have never really been practical if it wasn’t for the availability of Open source components.
• This isn’t really a business model per se, but in the sense that money goes in and out, I think it’s still worth mentioning, and that is the non-profit entity. Government organizations (the NSA developing SELinux, an example that I have become very familiar with) are a good example of people still getting paid (and paid pretty well even) to develop software that becomes open source.

So how does open source foster innovation? The key is seeing that although open source products themselves may not quite be so innovative, their existence fosters, and at times forces innovation. Below are some common mechanisms by which this happens.

• The existence of commoditized, freely available, modifiable software allows companies to avoid spending resources (including cash) to solve problems that have been solved. It also allows new companies to be formed, which can build upon existing technologies and provide solutions that would have been otherwise impossible.
• On the flip side, this also means that a company which produces no innovation beyond existing open/free solutions has no reason to exist. In this way, open source can be brutally efficient. VMware is seeing the effect of this (as is most other software companies). Xen, the open source virtualization software, although still in its infancy, is commoditizing the core virtualization aspect of VMware products. As such VMware will need to continue to develop its software offerings that go beyond what commodity software provides. If one assumes that companies like VMware want to stay alive (a pretty reasonable assumption) then the existence and development of Open source technologies forces those companies to stay ahead of the game.
• A sub point to this is that assuming this model, software companies (not service companies) are probably going to be closed source in the future as well. It may not even be that whether the source is open or not is what really matters — the point is that there has to exist a mechanism by which other companies can’t just outright steal the value of the product). As long as the closed source, uncopyable products justify their value beyond what open source products provide, then the market will still be there to purchase them, as those that use the software will realize real advantage ahead of their competitors.
• A third, more indirect way that open source fosters innovation is the fact that it can lead to cooperation and resiliency in manners that closed source products cannot acheive. For example, think of RSS — the standard itself is nowhere near a technological miracle, rather it seems quite obvious. But its openness and freeness have enabled it to become a very widely used and useful technology.
• The combination of the internet and the open source idea has shown that even a bunch of volunteer developers who contribute code on their spare time can combine to create products that are fairly useable and provide real value. But sometimes they can just fix bugs and make incremental obvious improvements. This fact provides incentive to software companies to open source software parts that are not central to their business (Apple Darwin, for example, or the various journaling filesystems available for linux (JFS, XFS)). As software companies open up source, things get better for everyone, and the earlier points in this list are re-inforced.
• A trend I have observed is that open source also seems to continually increase the productivity of a single developer. There are probably several reasons for this.. that the developer doesn’t have to solve all the problems, but rather just one he cares about, and also that improvement in developer tools are greatly valued by the community. Things that are a means to end often end up as open source (programming languages, scripting language engines, or a webserver, which nowadays is pretty much a development/deployment environment) and do very well there. This has the positive effect of allowing volunteers to make greater and greater contributions, which is inherently valuable to all developers. It also has the side effect of greatly improving the productivity of developers at commercial companies (if harnessed correctly). Better developer productivity leads to more innovative products. And commercial software companies will always be able to develop better products because they actually pay people to develop them.
• This last point assumes something fundamental that I think has held true since the beginning of time. Given equal talent, a group of paid full-time employees will always acheive more than a single person. That seems like simple math. Obviously, given the wrong set of employees, this does not necessarily hold in the real world, but you have to assume it does most of the time.