Log in

No account? Create an account
17 May 2008 @ 02:04 pm
Some lessons to learn  

There are obviously some things we need to remind if we don’t want something like the OpenSSL debacle to happen again. It doesn’t mean we need to throw stones nor to rush into changing our processes without thinking. However, there are already some things that should be obvious but unfortunately are not.

  1. Shipping a giant diff.gz that contains all changes in one, putting security fixes, policy fixes, bug fixes, cosmetic changes and autotools files at the same level, is not something we should accept anymore. Improvements in the dpkg-source format are much welcome in this direction, but they are useless if maintainers don’t use them. Neither a VCS nor a build tool will be able to know which line of the changes is related to which bug. Only the maintainer can.
  2. Core packages should all have co-maintainers. This is pretty much stating the obvious, and is much easier said than done. The OpenSSL case is one of the best examples here: Kurt is not one of those who refuses help, but frankly, would you want to maintain that package? Having already maintained packages with messy code, upstreams not understanding at all the needs of a distributor, avalanches of security alerts and randomly-changing ABIs, I can tell you this is no fun like it can be to hack on a desktop environment or a device driver. The only sane reason to do this is that you need the package to work. The only visible result you get from your work is that programs are not randomly crashing.
    I have no magic recipe to propose so that more people help with such packages, and that’s where we need to be really innovative. Cross-distribution teams, mandatory co-maintainership on a core package for each DD… these (and all ideas I have not heard of) are the experiments we should start now.
  3. Patching bad code leads to unpredictable results. What maintainer of a complex package has not introduced a new bug while trying to fix another one? Even when a piece of code is maintained by uncooperative developers, is not commented, uses arcane variable names or is impossible to understand without having contributed 3 winning entries to the IOCCC, it needs to evolve. And in these cases, it is only a matter of time until such things happen.
    Don’t get me wrong: I’m not trying to put the blame on upstream here. They have contributed very valuable code to the community and their work helped in the considerable widespread of cryptography. It’s just that their code is not enough for our needs. If we can’t patch it safely (and I’m now convinced we can’t), maybe we need to focus on alternatives and help them getting used by crypto-related packages. The code in GnuTLS and NSS is not necessarily better, but most (if not all) patches Debian needs to apply to them are build and portability fixes.
  4. Unless Debian-specific, 1 patch = 1 bug in the upstream tracker. This should be obvious, but given the number of patches that are never forwarded, it doesn’t seem so. You should not only give a chance for upstreams to review the patch, but you need them to track it, and you must give them the chance to review it anytime someone else stomps on a similar issue. If upstream does not have a bug tracker, they probably think their software has no bugs. Which means they are not trustworthy, and we go back to point 3.
  5. We need to give more priority to security. Issues in the security team seem now fixed for good and they have been doing an awesome work. There isn’t much left to do so that packages are all built with security-hardening features, but it still needs to be done. And there is much more to do so that we can provide out of the box a decent SELinux setup, or, if it turns out unrealistic to do, a decent system hardening setup using another framework. I know the SELinux zealots will jump on their high horses to explain that their framework is better, but the current situation where it is impossible for the average system engineer to setup a Debian-based MAC system is much worse than having a suboptimal setup that already works.

All in all, this incident has a great impact on Debian’s image. If we don’t react accordingly, adapting our processes and our system to match what our users expect from us – and they expect the best – they will turn away from us. With very good reasons to do so.

Update : It seems OpenSSL does have a bug tracker. Thanks Kurt for pointing me to it.

q_ on May 17th, 2008 12:34 pm (UTC)
openssl does have a bugtracker at rt.openssl.org

np237np237 on May 17th, 2008 12:44 pm (UTC)
Re: bugtracker
Oh, that’s good to hear. I had never heard of it and couldn’t see it on their web site.

BTW, is it private? If so, that defeats the point of a bug tracker.
q_ on May 17th, 2008 12:51 pm (UTC)
Re: bugtracker
It's linked on their website. First on the left you click on support, then at the top there is a "request tracker" link. This gets you to:

I have to admit that I didn't see it the first time either.

(Anonymous) on May 17th, 2008 06:18 pm (UTC)
I don't know much about the bug in detail.

What about testing? Develop tests that, as much as possible, verify that the software functions as it should. It was possible to foresee that a test could have caught this bug, because it affected the output of the program in an obvious way that could have been tested for.

OpenSSL has a test suite. Is it applied to all patched SSL software before being released by Debian, so that software that fails the suite isn't mistakenly released? Would the current test suite have caught this bug?

Whenever a security vulnerability is patched, should it be a requirement to open a second related bug report to determine whether a test could be developed that would catch the very error, or a similar one, and if so, create and include such a test in the suite?