Open Source Infrastructure at Web 2.0 Conference: the Marc Cantor Show

The open-source infrastructure workshop was fascinating. Open-source software development is (by now) a fairly well-understood social process. But how do you create infrastructure (servers and bandwidth) that costs real money to run, for the purpose of making a more open and remixable web? This is a business and social problem, not just a technology problem.

Throughout this workshot, Marc Cantor was constantly interrupting, doing lots of public arm-twisting to make various organizations agree to support other organizations formats. This was a)totally hilarious, and b)a very productive way of getting thing done. It’s not an approach that will work if you’re afraid to look like an asshole. But it plays well on other people’s fear of looking like an asshole to get business done. A choice quote: “MusicBrains is a non-profit. Yahoo, if you pay taxes, you’re paying for war and killing people. Give it to MusicBrainz instead, and use them instead of gracenote!”
This is actually a pretty valid point. A lot of the infrastructure that big players run on can be comodified by open source infrastructure. As long as it’s not a point of competition between the big players, the economics of tax-deductible donations to a shared infrastructure are probably reasonably compelling. The problem is that organizations like Yahoo! might benefit from keeping the costs of this type of infrastructure high (since they can afford the costs, and presumably upstart competitors can’t). However, this game gets dangerous if open and free alternatives start to become available: all of a sudden, your costs are higher than the competition!
The first piece on open-source infrastructure that was presented was EVDB, the event API & infrastructure used by (an online event calendaring application
Next, Matt Mullinwey (from WordPress / Ping-o-matic) talked about scaling Pin-o-matic. He described how Ping-o-matic went through the process of rolling out an open source infrastructure that gets corporate support.
Problem: new ping servers coming up every day. Sending pings slowed down posting process for WordPress.
Solution: ping one server (Ping-o-matic), that is responsible for spreading the ping.
Problem: The solution got overloaded, and the resource exploited (ping-spam).
Revised Solution: bring in outside help from businesses who benefit from the pings. Technorati contributing a server / bandwidth (as well as some technical help).
This kind of community investment seems like a key part of the Web 2.0 boom. Unlike the first generation of dot-com companies, in Web 2.0 the appearance of being a good neighbor who works and plays well with others is very important. This makes sense, since Web 2.0 is all about remixing, which only works if there are open standards and infrastructure that everyone can rely on. Interestingly, this means that this new boom will involve a lot of non-profits as well as for-profit companies.
Next Tantak from Technorati presented the site. It seems like a great way of dealing with the problem of standards taking too long to develop, being poorly integrated, etc. He presented several microformat design principals, namely:
Solve a specific problem
Keep as simple as possible
Humans First, Machines second (make format human readable, learnable)
Reuse widely adopted standards (duh!)
Make standard modular / embeddable (make sure it works inside rss/atom/html)
Decentralized design / development / infrastructure
One thing that comes through is that the minor search players (like PubSub, Technorati, etc) get the biggest advantage out of the internet being more structured. Since they are so close to the standards development process, and so focused on blog search, they can stay ahead of google if they don’t have to worry about the central problem google solves (searching unstructured data).
Finally, the head of MusicBrainz (an open and non-profit competitor to CDDB/gracenote) spoke. MusicBrainz is run as a non-profit service for music identification. Apparently, the process of founding a non-profit is MUCH harder than founding an LLC: one place where big players could surely do some good would be in providing umbrella non-profits that could host multiple pieces of open-source infrastructure.
The challenges of building a communal infrastructure that requires millions of dollars of yearly funding are challenging. But both top-down (corporate) and bottom-up (individual) models seem to be having success. It will be fascinating to see how this all plays out.