Unable to instrument apache httpclient using javaagent for spring boot uber jar application

Question

I'm trying to write a javaagent with Bytebuddy to intercept apache httpclient requests and I want to use this agent for spring boot application. The agent works fine when I start my test spring boot application from Idea (run the main method directly). However, when I package the application into a spring boot uber jar and run it using java -javaagent:myagent.jar -jar myapplication.jar, it throws the following exception.

onError:org.apache.http.impl.client.AbstractHttpClient
java.lang.NoClassDefFoundError: org/apache/http/HttpHost
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethods(Class.java:1975)
    at net.bytebuddy.description.method.MethodList$ForLoadedType.<init>(MethodList.java:106)
    at net.bytebuddy.description.type.TypeDescription$ForLoadedType.getDeclaredMethods(TypeDescription.java:985)
    at net.bytebuddy.implementation.MethodDelegation$MethodContainer$ForExplicitMethods.ofStatic(MethodDelegation.java:1037)
    at net.bytebuddy.implementation.MethodDelegation.to(MethodDelegation.java:247)
    at net.bytebuddy.implementation.MethodDelegation.to(MethodDelegation.java:226)
    at com.yiji.dtrace.agent.httpclient4.interceptor.HttpClient4Interceptors$1.transform(HttpClient4Interceptors.java:48)
    at net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:457)
    at net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:2791)
    at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:3081)
    at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:170)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:170)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:170)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.yjf.common.net.HttpUtil.<init>(HttpUtil.java:118)
    at com.yjf.common.net.HttpUtil.<init>(HttpUtil.java:81)
    at com.yjf.common.net.HttpUtil.<clinit>(HttpUtil.java:78)
    at com.daidai.dtrace.agent.test.Main.main(Main.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.http.HttpHost
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 60 more

and here is my agent related code.

public class DTraceAgent {

    public static TypeDescription abstractHttpClientDescription() {
        return new TypeDescription.Latent("org.apache.http.impl.client.AbstractHttpClient",
                Modifier.PUBLIC|Modifier.ABSTRACT,
                TypeDescription.OBJECT,
                Arrays.asList(httpClientDescription()));
    }

    public static TypeDescription httpHostDescription() {
        return new TypeDescription.Latent("org.apache.http.HttpHost",
                Modifier.PUBLIC|Modifier.FINAL,
                TypeDescription.OBJECT,
                Arrays.asList(new TypeDescription.ForLoadedType(Cloneable.class),
                        new TypeDescription.ForLoadedType(Serializable.class)));
    }

    public static TypeDescription httpContextDescription() {
        return new TypeDescription.Latent("org.apache.http.protocol.HttpContext",
                getInterfaceModifiers(),
                TypeDescription.OBJECT,
                null);
    }

    public static TypeDescription httpRequestDescription() {
        return new TypeDescription.Latent("org.apache.http.HttpRequest",
                getInterfaceModifiers(),
                httpMessageDescription(),
                null);
    }

    public static void premain(String arguments, Instrumentation instrumentation) {
        new AgentBuilder.Default()
                //.withBinaryLocator(binaryLocatorFor(instrumentation))
                .withListener(DebugListener.getListener())
                .type(is(abstractHttpClientDescription()))
                .transform(new AgentBuilder.Transformer() {
                    public DynamicType.Builder transform(DynamicType.Builder builder,
                                                         TypeDescription typeDescription) {
                        return builder.method(named("execute")
                                .and(takesArguments(httpHostDescription(), httpRequestDescription(), httpContextDescription()))
                                .and(returns(named("org.apache.http.HttpResponse"))))
                                .intercept(MethodDelegation.to(HttpClientInterceptor4dot3Plus.class));
                    }
                }).installOn(instrumentation);
    }
}

public class HttpClientInterceptor4dot3Plus {

    public static CloseableHttpResponse doExecute(
            @SuperCall Callable<CloseableHttpResponse> client, @Argument(1)HttpRequest request
            ) throws Exception {
        StringBuilder builder = new StringBuilder(1024);
        if (request != null && request.getRequestLine() != null) {
            RequestLine requestLine = request.getRequestLine();
            builder.append(requestLine.getMethod()).append(" ").append(requestLine.getUri());
        }
        try (TraceScope scope = Trace.startSpanForEntry(builder.toString())) {
            Trace.spanType(Span.SPAN_TYPE_HTTP);
            try {
                return client.call();
            } catch (Exception e) {
                Trace.exception(e);
                throw e;
            }
        }
    }
}

public class DebugListener {
    public static AgentBuilder.Listener getListener() {
        return new AgentBuilder.Listener() {
            @Override
            public void onTransformation(TypeDescription typeDescription, DynamicType dynamicType) {
                System.err.println("onTransformation:" + typeDescription.getCanonicalName());
                try {
                    dynamicType.saveIn(new File("generated_classes"));
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }

            @Override
            public void onIgnored(TypeDescription typeDescription) {
                //System.err.println("onIgored:" + typeDescription.getCanonicalName());
            }

            @Override
            public void onError(String typeName, Throwable throwable) {
                System.err.println("onError:" + typeName);
                throwable.printStackTrace();
            }

            @Override
            public void onComplete(String typeName) {
                //System.err.println("onComplete:" + typeName);
            }
        };
    }
}

I think this problem is caused by the way spring boot uber jar bootstraps an application. Spring boot provides a dedicated class loader named LaunchedURLClassLoader to load application related classes from the uber jar, while javaagent jar is loaded by default system classloader (if my understanding is correct). So the apache httpclient lib (included in the uber jar) is not visible to the system classloader.

I tried to provider a BinaryLocator to the AgentBuilder, but it didn't work. Maybe the BinaryLocator was not correctly constructed. Anyway, a proper BinaryLocator maybe a possible solution.

Thanks a lot for any solutions or suggestions.

Other infomation may be helpful:
spring-boot version 1.3.1.RELEASE
byte-buddy 0.7.7, packaged into the agent using maven-assembly-plugin's jar-with-dependencies descriptorRef
apache httpclient 4.3.2

Answer 1

I solved this problem through two steps:

1. Use Spring Boot's dedicated ClassLoader to deligate to an unloaded type:

   public static void premain(String arguments, Instrumentation instrumentation) {
     ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
     ClassFileLocator.Compound compound = new ClassFileLocator.Compound(ClassFileLocator.ForClassLoader.of(classLoader), ClassFileLocator.ForClassLoader.ofClassPath());
     TypeDescription delegator = TypePool.Default.of(compound).describe(delegatorClass).resolve();
     new AgentBuilder.Default()
       //.withBinaryLocator(binaryLocatorFor(instrumentation))
       .withListener(DebugListener.getListener())
       .type(is(abstractHttpClientDescription()))
       .transform(new AgentBuilder.Transformer() {
         @Override
         public DynamicType.Builder transform(DynamicType.Builder builder,
             TypeDescription typeDescription) {
           return builder.method(named("execute")
             .and(takesArguments(httpHostDescription(), httpRequestDescription(), httpContextDescription()))
             .and(returns(named("org.apache.http.HttpResponse"))))
             .intercept(MethodDelegation.to(delegator));
         }
     }).installOn(instrumentation);
   } 
   

2. Package the javaagent jar into Spring Boot's uber-jar such that the delegator class can reference classes related to the intecepted classes.

This issue is discussed in greater detail in Byte Buddy's issue tracker.