It was another Monday in the office, testing the next release of our user experience monitoringsoftware. It had been tailwinds towards the release for weeks – everything was progressing nicely. Until the moment I found myself waiting for our Java Agent startup for minutes instead of seconds.
The problem was reproducible – attaching our Java Agent even to a tiny test application increased its startup from two seconds to 30 seconds. My first guess in the hunt for the root cause of the issue was suspecting any of the recent changes in Agent. This guess was quickly dismissed as deployments of previous Agent versions resulted in the very same behavior, which was definitely not present during the previous weeks.
Now I was confused. The Logback library stuck at java.net.InetAddress.getLocalHost() has been performing the lookups for localhost for ages. The confusion increased when discovering the issue was not reproducing in our test matrix – throughout the dozens of machines in the matrix, everything seemed to work well.
And then it struck – over the weekend, the good old Mac OS X in my laptop had upgraded itself to MacOS Sierra. Due to yet unknown reasons, this started influencing DNS lookups for localhost which now started to take 30+ seconds instead of milliseconds as before.
Thanks to the right keywords now in place, the solution was just one google query away. Apparently, we were not the first ones to be hit by the issue and with the help of StackOverflow, the solution was now at our fingertips.
I do hope this saves some of our Mac-using readers hours of debugging time down the road. And if anyone out there happens to know anyone from the MacOS Sierra team responsible for filesystem / csrutils, let them know that there are a lot of confused Sierra converts who struggle with simple DNS lookups.