Fighting Cargo Cult – The Incomplete SSL/TLS Bookmark Collection

Engage Padlock!Throughout the recent months (and particularly: weeks), people have asked me how to properly secure their SSL/TLS communication, particularly on web servers. At the same time I’ve started to look for good literature on SSL/TLS. I noticed that many of the “guides” on how to do a good SSL/TLS setup are actually cargo cult. Cargo cult is a really dangerous thing for two reasons: First of all, security is never a one-size-fits-all solution. Your setup needs to work in your environment, taking into account possible limitation imposed by hardware or software in your infrastructure. And secondly, some of those guides are outdated, e.g. they do neglect the clear need for Perfect Forward Secrecy, or use now-insecure ciphers. At the worst case, they are simply wrong. So I won’t be providing yet another soon-outdated tutorial that leaves you non-the-wiser. Instead, I’ll share my collection of free and for-pay documents, books and resources on the topic which I found particularly useful in the hope that they may help you in gaining some insight.

Introduction to SSL/TLS

If you’re unfamiliar with SSL/TLS, you definitely should take half an hour to read the Crypto primer, and bookmark SSL/TLS Strong Encryption: An Introduction for reference.

Deploying SSL/TLS

So you want to get your hands dirty? Check your server setup with Qualys SSL Labs’ server test. Make sure you fix the most important issues. You should at least be able to get an “A-” grading. If you find yourself in trouble (and are the administrator of an Apache or nginx setup), you should read the OpenSSL cookbook. Professional system administrators should have Bulletproof SSL/TLS and PKI on the shelf/eBook reader.1)

If you find yourself with too little time on your hands, you can skip through to Mozilla’s awesome config tool which will help you with setting up your SSL vhost for Apache, nginx and HAproxy. However, some background may still be needed. You will find it on Mozillla’s Cipher recommendation page and the OpenSSL cookbook.

The SSL, the TLS and the Ugly

If you are a dedicated IT professional, you should not miss the next section. Although it’s not crucial for those wishing to “simply secure their server”, it provides those who are responsible for data security with a clear understanding of the numerous theoretical and practical limitations of SSL/TLS.

Tools and Utilities for Debugging SSL/TLS

Sometimes you need to debug errors during the SSL handshake. While a bit primitive, OpenSSL’s s_client tool is the weapon of choice. When it comes to monitoring SSL/TLS encrypted communications, use mitmproxy or Charles. They need to be added as proxies, but can also intercept PFS connections, due to their active MITM position.

This list is not exhaustive and if you have more suggestions, please go ahead and post them in the comments. I’ll be happy to add them. Finally, just like with system administration in general, you’re never “done” with security. SSL/TLS is a swiftly moving target, and you need to be aware of what is going on. If you are an IT professional, subscribe to security mailing lists and the announcement lists of your vendor. Finally, while I’m aiming to update this page, there’s never a guarantee of up-to-dateness for this list either.

Update (22.04.2014): Don’t miss the discussion on this article over at Hacker News.

Article History

  • 21.04.2014 – Initial version
  • 21.04.2014 – Added “The Case for OCSP-Must-Staple”, Mozilla Cipher suite recommendation
  • 22.04.2014 – Updated to add sslyze and cipherscan, added HN link, fixed typos
  • 02.05.2014 – Add “Analyzing Forged SSL Certificate” paper
  • 19.12.2014 – Add Mozilla SSL Generator, updated text on book availability

1) I do realize that I am courting Ivan a lot in this section and that relying on only an a single external web service that can go away any day is not a good thing. At the same time I think that the handshake simulation and the simple rating process are priceless, as such assessment cannot be trivially done by people whom’s life does not revolve around crypto and security 24/7. At the same time, I’m happy for any pointers towards other, user friendly tools.

2) While blindly following the rating can easily lead to the establishment of cargo cult, ssllabs.com is continuously updated to only give those a good grading that follow the best pactices. Again: Avoid Cargo Cult, make sure you have a good idea of what you are doing.

ownCloud Client 1.6: The Tour

Now that ownCloud 1.6.0 beta1 is out, it’s time to explain the story behind it:

owncloud-icon-256This release was developed under the promise that it would improve performance 1), and we have made tremendous improvements: Using a new Qt-based propagator implementation, we can now perform multiple simultaneous up- and downloads. We still provide the old propagator for certain situation where it’s more suitable, such as for situations where bandwidth limitation is needed.

Furthermore, the sync journal access code has been significantly optimized. It paid tribute to most of the high CPU load during the mandatory interval checks. CPU usage should be much lower now, and the client should be usable with more files at the same time.

Windows users should also find update times improved as the time spent in file stat operations has been reduced. Mac OS X users will enjoy the benefits of a much improved file watcher. To be able to use the more efficient API, 1.6 drops support for Mac OS Snow Leopard (10.6) and now requires Mac OS 10.7 or better.

At the same time, production releases are now using Qt 5 rather than Qt 4 on Windows and Mac OS X2). This fixes a lot of visual bugs in Mac OS X, especially for Mavericks users, and allows us to profit from improvements in the SSL handling, especially on the Mac.

We also implemented an item that was on many peoples wish list: a concise sync log. Next to the database, the sync folder now holds a hidden file called .owncloudsync.log. It will store all sync processes in a minimal CSV file. Contrary to previous logging facilities, it always logs and only collects information relevant to the actual sync algorithm decisions.

Because this tour was not as colorful as the previous one, let’s close this blog post with a feature contributed by Denis Dzyubenko: The settings dialog on Mac OS X now has a native look & feel:

Get ownCloud Client 1.6.0 beta1 now and provide feedback!

1) Now that while the client is multi-threaded, you may find that the transfer time still doesn’t improve as much as you would expect. This is due locking issues on the server which prevent efficient parallel transfers. This has been improved in 1.7, and could potentilly improved even further by implementing support for X-Sendfile/X-Accel-Redirect in SabreDAV, the DAV framework used by ownCloud server.

2) We can’t do the switch even on modern Linux distributions mostly due of the poor support for modern and divergent Systray/Notification area support in Qt5: Even in Qt 4 we could only use it because Canonical had patched their Qt to make QSystemTrayIcon work with Unity, which they have not ported to Qt 5 yet. Gnome 3 also hides away traditional Systray icons way to well, not to speak of Plasma. Any leads would be helpful.

PS: Martin’s blog on the subject indicates that Qt 5.3 might solve the problem.

On Practical Qt Security

At 30C3, Ilja van Sprundel gave a talk on X Security. In this talk, he also discussed Qt security matters, specifically how running a setuid binary which links against Qt is unsafe due to exploitable bugs in the Qt code base (citing the infamous setuid practice in KPPP). While his points are valid, he misses the greater picture: Qt was not designed for use in setuid applications! Consequently there are a lot of ways the security of a Qt application can be compromised when it runs as root. So I went on to discuss this issue with QtNetwork maintainer Richard Moore, and we both agree that in contrary to Iljas claim, we do need to dictate policy. So here it goes:

Do not ship Qt applications that require setuid. While the same is probably true for any other toolkit, we have only discussed this for Qt in more depth. Actually, Rich has prepared a patch for Qt 5.3 that will quit if you try to run an application setuid unless you ask it to. This should make it harder to shoot yourself into the foot.

While making QtCore and QtNetwork safe for setuid use is possible, they currently are not. If you absolutely have to (and you really shouldn’t), at least unset QT_PLUGIN_PATH and LD_LIBRARY_PATH in main(). The latter is required because even though LD_LIBRARY_PATH is ignored by the linker for setuid binaries, it is used internally by QtNetwork unconditionally to look for OpenSSL. Of course, you also need to follow all the other best practices (note that even this list is incomplete, e.g. it doesn’t mention to close FDs).

However, there are also situations where a Qt application running as user can be unsafe, so to those who ship their own Qt build to their customers, there are even more policies:

  • Never build Qt so its prefix is a publicly writable directory, such as /tmp: Suppose you build a in-source (developer) build in /tmp/qt, then Qt will go ahead and look for plugins in /tmp/qt/plugins. A malicious user could simply provide a fake style there that next to calling the style which the user would expect (e.g. via QProxyStyle) executes arbitrary malicious code. The same goes for Image IO plugins, which are handled in QtCore.
  • Never build Qt so its prefix is a home directory: This one is more tricky and a lot harder/unlikely to exploit, but it’s a valid attack vector nonetheless: Suppose Joe Coder compiles Qt in-source on /home/joe/dev/qt. Now every customer needs to make sure that a local user by the same name is a really nice person.

So in conclusion, a better summary of the above would be:

Never distribute binaries built with a prefix that is a non-system directory!

If you already have this setup, but need a hotfix, there is hope: libQtCore.so contains strings starting in qt_plugpath= and qt_libspath=. Both are padded to 1024 bytes. Adding a binary null after the first / keeps Qt from looking for loadable code in user accessible locations.

TL;DR: The bugs Ilja points out are valid, but only affect applications that don’t follow good practice. We will attempt to make it harder for developers to make these mistakes, but writing suid applications isn’t something that will ever be recommended, or easy to do safely. Apart from the suid issue however, there are more traps lingering if you provide your own Qt and build it in an unsafe way.

Further reading: Google+ discussion on the topic.
Acknowledgements: Richard Moore for contributing vital information to this document, Thiago Macieira for proof-reading.

Update: Clarified the wording to ensure it’s clear that a prefix is meant. Thanks, Ian.

Update 2: As Rich and David Faure pointed out, KPPP is dropping permissions before calling Qt code, and KApplication already has a setuid safeguard in place.

Update 3: Richs setuid check has been merged.

FrOSCon 2013: Call for Projects, Papers

frosconThe Free and Open Source Software Conference (FrOSCon) will take place on the 24th and 25th of August this year, kindly hosted by the Bonn-Rhein-Sieg University of Applied Sciences, Sankt Augustin.

We are looking for projects that would like to present in the exhibition area, as well as interesting talks or workshops. Additionally, you can request a project room if you want to sit together to do some hacking, or have your own, topic specific course of lectures. Sign up now!

But even more so, we are looking for speakers. The focus this year is on Seamless Computing, How Free Software should deal with closed ecosystems and How to let the computer do your chores. Even if your topic is not in one of those domains: Submit your proposal.

Going MirrorBrain, seeking Mirrors

Once upon a Time…

When the Qt sources were first released on ftp.troll.no, the Troll Tech (mind the spelling, it’s that long ago!) FTP server, many FTP mirror admins quickly started mirroring the software, because it was cool, and useful to them or their company/university. And quite right so, because this meant that everyone in the world could use his local mirror to get their files real fast, i.e. high speed and low latency. However, over time, traditional mirrors were replaced by Content Delivery Networks (CDNs). They were deemed more convenient because, well, the user doesn’t need to look for the his closest mirror. The CDN is a one-stop-shop which will automatically find a fast download location. This can either be done by employing GeoIP, or by performing a distance calculation based on routing information (the users’ ASN). There are even more ways of doing this, but let’s not digress:

During the time at Nokia, the Qt packages were also moved to a Commercial CDN. It was good, but it was also expensive. As an Open Source project, we should use our resources cost-effectively and have an infrastructure that is decentralized to a certain degree. Fortunately there is a software that helps us doing that, by combining the ease of CDN-like user-experience with the old system of FTP and HTTP mirrors, and it’s called MirrorBrain. Happy users include LibreOffice, openSUSE and the KDE Project. MirrorBrain presents itself to the user as one page, redirecting the actual downloads to mirrors. While doing that it makes sure that only those mirrors that contain the latest, up-to-date version of the files will be used.

A Call for Mirrors

Since a few days, we have a MirrorBrain infrastructure in place. If you run a mirror or have a reliable, well-connected machine (>= 50 MBit/s) that can help us delivering Qt (currently a total of 120GB), please read the mirroring instructions on the Qt Project page, send us a mail and join the mirrors mailing list.

Credit where Credit is due

The KDE Sysadmin team has helped us a lot in getting MirrorBrain started while setting up the evaluation system. Peter Poeml, the uberbrain behind MirrorBrain, has been of great assistance. Finally, Digia has been sponsoring the man power to set up the production system.

Badge and Progress support in Qt Creator

For quite a long time now, Qt Creator has been using the native features of Mac OS X and Windows 7 to display build errors and progress as badges and progress bars on the icon in the Dock and Task Bar respectively. Unfortunately, the X11 variant has been sorely empty. After realizing that a few applications, among them Chrome, could display certain information such as the number of ongoing downloads in Unity, I was wondering how this was implemented.

It turns out that libunity provides all the features required. Applications, identified by their .desktop name, can add progress and a badge. So I implemented it for Qt Creator, here is unity showing two compile errors in a demo application:

qtc_unity_badge

Now the gory details: libunity seems to be binary incompatible. After researching for a while, I noticed that it is best to follow the Chromium implementation and only try to open known-to-work versions of libunity. This it not really optimal, but it works, at least until the next unity release :(.

*Magnum narrator voice on*: Now I know what you’re thinking 1), and you are right: “X11” does not equal “Unity”, but I have no idea if there are equivalents in GNOME 3 or KDE/Plasma. Probably the terminology is just different. If so, I’d happy to learn about them and implement these as well. A plus if there a FD.O library (and I’m afraid there isn’t). If not, I’d like to toss in the question whether KDE should have such a feature, or how else applications can usefully reflect certain statuses while in the background.

1) especially if you are reading this through Planet KDE!

New Planet, new Domain

It took a bit longer than anticipated (sorry about that), but now it’s finally done: Planet Qt got a face lift, and is now hosted with the Qt Foundation. Just like Planet KDE which it was greatly inspired by (thanks to Jonathan Riddell!), it’s now based on RawDog, rather than on the aging PlanetPlanet. But how does that affect you, the inclined reader or contributor to the planet:

If you are a reader, nothing much has changed, except that the URL for the RSS feed has changed and we now get FOAF and OPML for free. You might need to remove duplicate elements from your feed reader, sorry about that. We also decided to redirect planetqt.org to planet.qt-project.org for consistency with the other foundation’s activities.

If you are a contributor, and want your blog aggregated, you can do so by simply following the guide lines on the side bar:

All who blog in English about Qt can either add their complete RSS feed or a feed based on a specific category or tag for those blogging about multiple topics. All you need to do is clone the Planet Qt repository, add your feed’s address, and put your change up for review.

That’s right, adding feeds now works via the usual review process. If you are a contributor, and a Reviewer can be convinced that your blog is relevant for PlanetQt, it will be added. For now this involves a manual update on the server still, but if we see no abuse we might just as well automate this process.

The same is true if you want to improve the layout: Just checkout the repository (you will need Python, though) and test it yourself, tweaking it to your liking and then submit the result as a patch.

Finally, big hands to Ben Meyer, who has been hosting this web site for so long!

And now: Enjoy the new PlanetQt!

Gran Canaria, here I come!

Wow! The last few days have been eventful. Only four Days after LinuxTag and the KDE Wiki Meeting I am sitting in the check-in area of the Berlin-Tegel Airport heading for Madrid. If everything works out as expected, I will then transfer to a flight to Las Palmas. I swore myself not to blog before I have checked in successfully, so the time for this entry is now, and to make it even more obvious:

The weather is awesome in Berlin already so I am looking forward how Gran Canaria will beat this (probably less thunderstorms in the evening, although they are really refreshing).

At GDCS, I will present Qt Creator, the scalable C++ IDE from Qt Software (I even brought the leaflets I printed LinuxTag, my bag I short of over baggage). I am looking forward to meet everyone again tonight at the welcome party!