I will be discontinuing this blog with a change in the scope of my research and interests. If there is anything of interest please print now because it will be out of print shortly.

At the Velocity Conf this week in Berlin (which had a very good turnout in terms of audience and speakers) I was stunned to hear NewRelic during their 5 minute lighting presentation claim that other tools to be “crappy”. Yes from the product/engineering team that gave us Wily Introscope which in itself completely redefined what crappy means in the enterprise space and who then went on to show that Crappy 2.0 (Lew Cirne’s second attempt at trying to code) can repeat its success with those with very little in the way of performance management knowledge, expertise and awareness – in the cloud. Then on the second day we had the “DevOps” team from http://guardian.co.uk advocate the use of logging (excessively that is with log4j) as a means to monitor and manage Java applications. Apparently no amount of logging could impact application response times which says a whole lot about the performance itself. Two huge WTF moments in two days. It seems we can’t seem to get over logging and we can’t seem to calibrate our own tools.

Allow me to kill two birds with one benchmark test. Here is the clean version.

Using JXInsight/OpenCore’s agent the code was instrumented dynamically at load time. No code changes needed. I ran with two different configurations. One that simply meters in memory and the second one that along with metering performs binary logging of the begin and end metering events, the meter readings as well as the name of the probe and thread context.

Here is the manual instrumentation needed to have a comparable test with log4j logging approach.

With NewRelic‘s Java agent there is no means to transparently instrument code that falls outside the few frameworks they support. You are forced to use a NewRelic specific @Trace annotation.

Here is a comparison of the average clock time cost reported after executing each test 100 million to 1 billion times.

You might initially think that NewRelic does not fair so bad compared to the log4j approach but bear in mind that NewRelic does not actually do any file IO. Instead it aggregates the data (losing all trace history in our case) and then dispatches it as a small packet to their web site service in 1 minute intervals.

None of our customers could tolerate the overhead incurred by the log4j and NewRelic with transaction latency as low as 200-300 microseconds and with 10 or more measurement points within such.

OpenCore without the recorder probes provider enabled and using the following configuration had an average clock time overhead of 130 ns with more than 50% of that attributed to the two clock time access reads.

j.s.p.strategy.enabled=false
j.s.p.meter.clock.time.include=false
j.s.p.meter.jvm.clock.time.include=true
j.s.p.aggregates.enabled=false
j.s.p.global.enabled=false

With the recorder enabled as follows the overhead increased to 710 ns and thats for two metering event IO writes. That’s 10x better than log4j but considering this is mostly IO on a slow hard disk the efficiency differences are hugely bigger. I can’t imagine what the other 7 microseconds is being spent on by log4j.

j.s.p.recorder.enabled=true

Below is a slice of the CPU usage monitoring during each of the benchmark test runs. OpenCore is not only so much faster it also uses far much less system resources.

There is a reason why cloud platform vendors like Heroku (Salesforce) and CloudBees offer NewRelic subscriptions but don’t actually use it themselves internally to monitor and manage their own critical systems and services. They are not in the business of conserving your resource consumption (or cost).

Relatively speaking log4j and NewRelic are pretty darn expensive and for the value they offer it puzzles me why anyone would ever consider such for Java application monitoring. Granted not everyone or application operates at such speeds and resolutions but bear in mind that today we advise our customers that the recorder be used mainly in development and focused performance testing, yet it is 10 times faster than these solutions promoted/advised as production solutions.

Yes you can find low hanging fruit with whatever “crappy” approach and agent you choose but if you are still using Crappy 2.0 after resolving such issues then you must question whether you are truly trying hard enough to be good if not better (than the rest).

Clearly what I consider “crappy” and what Brain Doll over at NewRelic considers “crappy” are light years apart and in different time/space dimensions.

Apart from these two blips the Velocity EU conference was a resounding success. Lots of great discussions in the corridors and in the speakers room and not forgetting some really thought provoking talks from John Allspaw, Theo Schlossnagle, Jeff Veen, Johannes Mainusch, and the WTF’er of the year Artur Bergman.

Hardware
Model Name: iMac
Model Identifier: iMac11,1
Processor Name: Intel Core i7
Processor Speed: 2.8 GHz
Number Of Processors: 1
Total Number Of Cores: 4
L2 Cache (per core): 256 KB
L3 Cache: 8 MB
Memory: 8 GB
Processor Interconnect Speed: 4.8 GT/s

log4j.properties
log4j.rootLogger=warn, root
log4j.appender.root=org.apache.log4j.FileAppender
log4j.appender.root.File=log4j.log
log4j.appender.root.layout=org.apache.log4j.PatternLayout
log4j.appender.root.layout.ConversionPattern=%p %t %c – %m%n

Velocity, the Web Performance and Operations conference from O’Reilly Media, is coming to Europe!  I will be attending the conference for the 2 days including (if travel plans permit) the Unconference on the Monday. I expect my session on QoS for Web Applications to be challenging, entertaining and inspiring especially as I frame Quality of Service within the context of self adaptive software (self aware, self regulated, self healing,…) and sketch out how the future of application management involves us managing by proxy by way of controllers, models, plans, goals and policies built directly in applications, containers and runtimes.

I hope to see you there. If you have not registered please consider doing so now. It looks like a great line-up and to be an important conference in the European calendar for performance and operations of web and cloud services.

And if you see me wandering around don’t be shy I promise not to bite .