I came across a really interesting bug that was plaguing me for about a day and a half and I finally tracked it down today, with JIT assistance from the superhuman @brianm.

I have a fairly simple app that uses Netty for its underlying event handling. Just a Web service, no great shakes. But, there was this weird bug that I found during load testing whereby it would just stop all network activity after a few minutes for no discernable reason. No more connections could be established, existing connections did not finish, etc. At first, I thought it was maybe a bug in Tsung, my load testing tool, but that wasn’t the case.

I upped the logging and found this strange error that was happening right around the time of the network shutdown:

I googled around for this error and found only a few references to it, mostly having to do with a partially-uploaded JAR. My app is packaged in a fat JAR, so I found this unlikely, but I reuploaded it and verified the upload with the MD5 hash and still the error occurred and the network shut down.

I was pretty much out of other options at this point, since there were no other anomalies and YourKit was unable to find any deadlocks or such. I took a stab and sent the gist above to @brianm to see if he had seen it before. He immediately noted that the call to java.security.AccessController.doPrivileged was bad news. He then suggested I try the “bazooka-squirrel approach”: if you can’t hit the squirrel with your .22 because the tree’s a mess, take out the whole tree with a bazooka ;-) In this case, the “tree” was the logging attempt in the backtrace.

However, I wasn’t using java.util.logging in the app. I was using logback. I checked out the source code of NioServerSocketPipelineSink and found that it was Netty using java.util.logging. Oh, really? At this point, java.util.logging was not configured. I configured it with a simple ConsoleLogger and all of a sudden *BLAM*… problem solved.

Netty is using java.util.logging which, if left unconfigured, was causing it to reach out into the fat JAR for the logging configuration which wasn’t there. When it did so, it caused an InternalError due to some kind of JVM security issue which blew up the server socket. Netty catches that and shuts down the boss thread. *sigh*

tl;dr You must configure java.util.logging to go somewhere or if there’s an exception in your boss thread it will decapitate Netty rather effectively.

This morning, on a conference mailing list, I made some disparaging remarks about Node.js (the title of this post, in fact). A couple people asked me why I felt that way. Rather than respond individually, I’ll just list my reasons here:

V8 is not server-class

At my current place, we have a Web crawler where some portions are leveraging Chrome/V8. Take a guess which component absolutely dominates the bug count and issue list? Not to mention the fact that its balls-ass slower than some straight un-optimized Scala. We’re looking to get rid of it completely ASAFP.

On a larger note, using JavaScript on the server-side seems kind of ridiculous. If Linden Labs came out with a server-side framework in LindenScript, would you use that? How about if Apple came out with a framework based on AppleScript?

Callback spaghetti is about the last pattern with which you’d ever want to write anything

I’ve written servers in just about every kind of server framework pattern there is. Node’s is the absolute worst. It provides the least amount of aid and comfort to the programmer and its nigh-on-impossible to follow the code 6 months later. The idea that the broken-ass concurrency model forced on Javascript by old browser implementations is actually a *good thing* gives me a fucking aneurism.

Non-blocking != fast != scalable

This is probably the most annoying point for me. First of all, scalability has very little to do with raw speed. Just because you’re fast does not mean you’re scalable. You know what’s fast? MySQL. You know what’s not scalable by itself? MySQL. The hype around Node.js on this issue makes me want to punch faces. Furthermore, Node.js isn’t even that fast. You can do much better with Scala and its a much nicer language, to boot. Oh, and nevermind about those extra CPUs you bought: you won’t be needing those. Events and non-blocking are instantaneous, so you can do everything on a single core, right?

JavaScript

The Ruby and Python communities are just now, many years after the hype has faded, learning that stuff like dependency injection and proper modularization are actually good things that help you maintain code over time. JavaScript has very little support for any of those nice things: it doesn’t even have namespaces, for chrissakes. Why would I want to repeat these same mistakes over and over again in a new language? Knowing a language is not the same as being able to maintain services written in that language over the long haul.

It seems to me that people who are really crazy about Node.js are people who only knew JavaScript to begin with and to whom none of the above would ever even occur. Perhaps, this is a case of Worse Is Better and I’ll eventually have to eat my words on this one. But the kind of misunderstanding going on in this video clip seems to pervade the Node.js hype and gives me rageface.