DNS Caching in Performance Tests on the JVM

Background

When conducting performance testing with tools like JMeter/JMeter-DSL, it's crucial to ensure that your test environment accurately reflects the production environment. One common pitfall arises when dealing with round-robin DNS load balancing. If DNS lookups are cached, all virtual users may end up hitting the same backend server, causing an overload on that server while the others remain underutilised. This blog post will walk you through a solution to this problem using a custom InetAddressResolverProvider, available from Java 18 and onwards.

The Problem with DNS Caching

Round-robin DNS is a load balancing technique where multiple IP addresses are associated with a single DNS hostname. Each time a DNS query is made, the DNS server rotates the IP addresses in a round-robin fashion, distributing the load among multiple servers.

However, there are two levels of caching that can interfere with this load distribution:

• JVM Level DNS Caching: Java's default DNS resolver caches the DNS lookup results. In a performance test scenario, this means all your virtual users could end up using the same IP address from the cache, thus hitting the same server.
• OS Level DNS Caching: Even if the JVM DNS caching is disabled, the operating system itself caches the DNS lookup results. When the OS cache is hit, it presents the list of IP addresses in the same order every time, leading to the same issue of unbalanced load distribution.

The Solution

To address this, we need to disable DNS caching in the JVM and implement a custom InetAddressResolverProvider that shuffles the addresses returned from lookups at the OS level. This ensures that different IP addresses are returned in a random order for each lookup, even when the results come from the OS cache. Below is the implementation of such a class:

 

package com.radovation.testutils;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.net.spi.InetAddressResolver;
import java.net.spi.InetAddressResolverProvider;
import java.security.Security;
import java.util.Collections;
import java.util.List;
import java.util.Random;
import java.util.stream.Collectors;
import java.util.stream.Stream;

/**
 * InetAddressResolverProvider that will shuffle the addresses returned from DNS lookups when there are multiple records
 * for a lookup. Useful in performance tests when target servers are load balanced with round-robin DNS.
 */
public class ShuffleInetAddressResolverProvider extends InetAddressResolverProvider {
  static {
    // the shuffling only makes sense if we disable the JVM DNS caching
    Security.setProperty("networkaddress.cache.ttl", "0");
  }

  private final Random random = new Random();

  @Override
  public InetAddressResolver get(Configuration configuration) {
    return new InetAddressResolver() {
      @Override
      public Stream lookupByName(String host, LookupPolicy lookupPolicy) throws UnknownHostException {
        List addresses =
            configuration.builtinResolver().lookupByName(host, lookupPolicy).collect(Collectors.toList());
        Collections.shuffle(addresses, random);
        return addresses.stream();
      }

      @Override
      public String lookupByAddress(byte[] addr) throws UnknownHostException {
        return configuration.builtinResolver().lookupByAddress(addr);
      }
    };
  }

  @Override
  public String name() {
    return "ShuffleInetAddressResolverProvider";
  }
}

Explanation

• Disable JVM DNS Caching:
  The static block at the beginning disables JVM DNS caching by setting the property networkaddress.cache.ttl to 0. This ensures that each lookup results in a fresh query to the OS.
• Shuffling DNS Results:
  In the lookupByName method, we collect the list of InetAddress objects returned by the default DNS resolver (OS level DNS cache). We then shuffle this list to ensure that different IP addresses are returned in a random order for each lookup, addressing both JVM and OS-level caching.

How to Use This Class

To use this custom resolver in your JMeter/JMeter-DSL performance tests, you need to register it as a service provider in the META-INF/services directory of your project. Create a file named java.net.spi.InetAddressResolverProvider in this directory and include the fully qualified name of your class:

 

com.radovation.testutils.ShuffleInetAddressResolverProvider

Conclusion

By implementing this custom InetAddressResolverProvider, you can ensure that your performance tests distribute load more accurately across your backend servers when using round-robin DNS. This approach prevents the skewing of results due to both JVM and OS-level DNS caching and provides a more realistic simulation of production traffic.

Feel free to use and adapt this solution in your own projects. Happy testing!