Tuesday, March 14, 2006

Battles with Proprietary Software Support

It always amazes me how much pain reliance on proprietary IT solutions can cause, yet people go back for more time and time again. Having worked in the past with a large, managed security vendor, I've seen first-hand the kind of trouble relying on proprietary solutions can cause. Here are just a couple of examples:

  • This vendor's managed security service relies on some closed components at its core, including backend databases, ticketing, and alerting. Over the past year, the security industry has seen a drastic change in what people expect from managed security vendors (which is now a lot more than it used to be) - this vendor has been unable to change quickly enough for the demands of the market, and is now feeling competitive pressures from those firms that are using Open Source components and can quickly adapt to what the market demands. Internal development resources are underfunded to begin with (the thought process was "Oh, good, an all-in-one solution, I don't need developers, let's lay them all off". They then regretted it when a solution had to be developed, because the old bunch of proprietary components no longer fit with, well, anything).
  • Providing support for proprietary software is a losing battle. The aforementioned vendor provides some of the best Check Point firewall support around, and has in previous years taken direct support business away from Check Point themselves. Where the support falls over, though, is the inevitable problem caused by software bugs (there seem to be about 30 of those nasty Check Point bugs that impact this vendor's clients each year), which they can only get resolved by going to Check Point for support. As you might imagine, since Check Point is the only company in the position of fixing software bugs in their closed-source product, they have no impetus to provide good support. Delays of a month or more are common, regardless of the severity. Non-critical bugs sometimes never get fixed. Nokia's IPSO platform is the same way - while based on FreeBSD originally, it is completely proprietary, and support vendors and clients are unable to get bugfix support elsewhere. I have personally been involved with two IPSO kernel bug-fixes (simple ones, at that) that have taken over a year each to resolve.

Now, you might think "So what, don't buy it if you don't like the support lock-in". But at some point this is like sticking your head in the sand. If you believe Check Point, every single Fortune-100 firm uses their firewall software. How long will it be before a serious bug or security vulnerability affecting Check Point impacts not just the firm using their software, but that firm's clients and customers? At what point do we look at the risks imposed on users of technology, and give the potential impact of those risks more weight than rights of the technology implementers? This question is at the heart of most hot-button topics centered around Free/Open Source software (FOSS), including the BSD/GPL debate. The BSD license favors the rights of the developers, while the GPL favors the rights of the users. Keep in mind that "developer" and "user" might be third or fourth generation, so a BSD-licensed piece of software that stayed that way through three vendors can suddenly become proprietary - the users have just lost most rights they had prior to that, if they use that software. A GPL-licensed piece of software allows it's end-users the same rights, no matter how many vendors have touched it (assuming they each honor the license).

I had personal experience with some incidents that made me really see the direct value that FOSS can bring to a business (as opposed to just reading about such ideas). No really new ideas here, but experiencing the problems with proprietary software first- or second-hand, while being in the middle of a support nightmare as a product support provider - is, I think, much more enlightening than presenting theories.

In one incident, a large financial institution had many Nokia/Check Point firewalls, configured in active-passive failover. In one of their many two-firewall sites, they use dynamic routing (OSPF) to direct traffic through a primary firewall, while a secondary firewall sits idle, waiting to take over for the first, should it fail.

Check Point's state synchronization is commonly used in these scenarios to share connection table information between two or more firewalls, so if one fails, no users experience dropped connections.

At 1 AM one morning, the state sync between one such pair of firewalls failed, causing the firewall daemon's log files on the active firewall to be flooded with errors. The logging pushed the firewall's CPU over the edge, and it was running at 90-99% CPU, with 1-3 second packet latencies.

Needless to say, the client was not happy, as it was impacting their business.

Still nothing too unusual there, except that this company, like many others, pays quite a bit of money to keep their hardware and software support "up-to-date". In this particular case, the Check Point version had hit "end-of-life" during the time they were trying to resolve the incident. Despite the support ending in June, and this event occurring in May, the company was told that they would not be able to get code support, should it rise to that level. So, if a bug was found, Check Point refused to even entertain the idea of providing a patch for their software.

The solution, of course, as presented by the vendor, was to upgrade to the newest, supported version of Check Point, while any bug found would not even be guaranteed to be fixed. The client has to accept the software vendor's word that the latest version of their software is "much more reliable", and wouldn't cause the firewall to fall over.

This particular company had no choice, they had to accept these terms, regardless of the dire straits they found themselves in. Sure, legal action could be possible, but they need help now, and any such action would take months or years. Besides, we all know how EULAs are written - certainly suing for defective software is not an option.

Now, why is it significant that this is all proprietary software? Because if it were truly FOSS, the client would have several options (outside of an upgrade):

  • They could fix the bug themselves, assuming they had skilled people on staff.
  • They could hire someone to fix the bug, and do so in a cost-effective manner, since everyone has access to the application source, and presumably more than one vendor could provide a fix.

(Note that this applies to using software provided by large vendors of FOSS, like Red Hat, even though they might follow similar "end-of-life" practices. The source code is still free to obtain, modify and distribute.)

The second point is important, as the availability of source code, along with the rights to change and distribute it (for money, or not), actually results in competition. In this scenario, the vendors providing the best support make the most money. Sounds pretty sensible, right? In today's current software market, software vendors who write and sell proprietary software hinge their entire business model on the software itself, the marketing hype that surrounds it, and the assumption that people will pay to upgrade when the time comes. Support is always an afterthought, a "necessary evil", and getting good support for such software is almost impossible. There is absolutely no incentive to provide good support once the software is sold and the customer is locked-in to one vendor for support. It is important to note at this point that I'm referring to software support, which is different from product support. The latter can be quite competitive, unlike the former. As an example, plenty of people provide Check Point product support, some better than others, but only one company can provide Check Point software support (i.e. bug-fixes).

It always amazes me that many people see FOSS as "anti-capitalist", when the correct term should be "anti-monopolist". No one could ever convince me that the vendor lock-in that many companies subject themselves to (as in this case) was somehow "the capitalist way". Why else would MS be fighting the corporate adoption of GNU/Linux servers so strongly - such adoption threatens their monopolistic business model. I would say that the current software model is harming consumers and the overall world economy, and this can only get worse as we become more and more dependent on proprietary software.

Why do companies put up with this? My belief is that it's a lack of awareness, coupled with the usual corporate stigma against anything that is "free" being poor quality - really just an artifact of the term "free" in the English language having more than one meaning, as has been noted by others. While the term is not perfect, I've seen firsthand that "Open Source" seems to be received without too much explanation much more readily than "Free Software" in most businesses.

Technorati Tags: , , ,

1 comment:

Kyle Gonzales said...

Spot on. This is an excellent commentary.

I see this all the time, and becomes even more problematic in situations were people are using multiple proprietary binaries (say, a kernel driver and a database) who have differing kernel requirements, and neither vendor will budge. One such way to work around the problem is to get rid of one of the proprietary binaries. One way to ensure they don't come up again is to get rid of ALL the proprietary binaries. Not realistic in many enterprise settings, but the closer you get to pure open source, the fewer interoperability issues you face.