gRPC transport and channel providers

Let’s discuss some details around that how gRPC manages transport providers.

Remember, that we have two closely related concepts in gRPC:

  • NameResolvers
  • ChannelProviders

If you didn’t familiar with NameResolvers, I strongly encourage you to peruse previous part of our narrative here: https://mchesnavsky.tech/name-resolution-providers. This article will be about the latter.

Alright, we should elaborate, that a gRPC channel has a transport for connections. ManagedChannelProvider is an abstract class responsible for creating and managing channels. These channels handle the underlying communication between clients and servers, ensuring efficient and reliable data exchange. We have at least these implementations of ManagedChannelProvider:

  • io.grpc.netty.NettyChannelProvider (and its shaded version io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider)
  • io.grpc.netty.UdsNettyChannelProvider (shaded version io.grpc.netty.shaded.io.grpc.netty.UdsNettyChannelProvider).

These guys represent transport. If you want to double check, which one you have operating, you should put a breakpoint to this call: ManagedChannelRegistry.getDefaultRegistry().provider() in the class io.grpc.ManagedChannelProvider.

ChannelProviders are also being loaded the same way as NameResolvers. Look at io.grpc.ManagedChannelRegistry:

List<ManagedChannelProvider> providerList = ServiceProviders.loadAll(..)

And going further to ServiceProviders.loadAll() -> getCandidatesViaServiceLoader():

Iterable<T> i = ServiceLoader.load(klass, cl);

Again, we have META-INF/services/io.grpc.ManagedChannelProvider file with following content:

io.grpc.netty.NettyChannelProvider
io.grpc.netty.UdsNettyChannelProvider

As long as we have multiple transport providers registered in ManagedChannelRegistry, gRPC has to have the way to choose the default provider. Therefore, we have a concept of priority again (did you read the article about NameResolvers?). For instance, NettyChannelProvider has a priority of 5:

public int priority() {
    return 5;
}

And UdsNettyChannelProvider has a priority of 3. It means that NettyChannelProvider always win:

System.out.println(new NettyChannelProvider().priority()); // priority: 5
System.out.println(new UdsNettyChannelProvider().priority()); // priority: 3

Sorting algorithm is located under ManagedChannelRegistry here:

List<ManagedChannelProvider> providerList = ServiceProviders.loadAll(
          ManagedChannelProvider.class,
          getHardCodedClasses(),
          ManagedChannelProvider.class.getClassLoader(),
          new ManagedChannelPriorityAccessor());

ManagedChannelPriorityAccessor acts soft of comparator.

Important difference between ManagedChannelRegistry and NameResolverRegistry (which is a registry for name resolvers, obviously), that you can’t force ManagedChannelRegistry to use certain Channel provider. Also, you can’t deregister any provider you have registered by default (since deregister takes an instance of provider to delete, and you don’t have that instance; you are able just to create a new one). I saw some folks making a decorator/proxy of channel provider they want to have higher priority, overriding priority() method: https://stackoverflow.com/a/75962678. Why not.

This is how you can register new provider manually. Use it whenever you need to register your custom one:

// Can be used to register proxy to default channel provider, with increased priority
ManagedChannelRegistry.getDefaultRegistry().register(new YourCustomChannelProvider());

If you have your desired channel provider registered, but it is still being unused by gRPC, you should double check, that isAvailable() method returns true for this provider. Example with UdsNettyChannelProvider:

System.out.println(new UdsNettyChannelProvider().isAvailable());

If you check, how isAvailable() works in this case, you will find out that if creates an instances for these classes:

  • io.netty.channel.epoll.EpollDomainSocketChannel;
  • io.netty.channel.epoll.Epoll

If you don’t have them in your classpath, well… it’s time to add new dependency:

<dependency>
  <groupId>io.netty</groupId>
  <artifactId>netty-transport-native-epoll</artifactId>
  <version>4.1.94.Final</version>
</dependency>

More straightforward approach for UDS channel provider (Remember that Unix Domain Sockets (UDS) provide a way for processes on the same machine to communicate with each other using socket addresses that are based on file path? See article on NameResolvers):

System.out.println(io.netty.channel.epoll.Epoll.isAvailable());

You can run into this exception:

java.lang.IllegalArgumentException: isAvailable() returned false
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at io.grpc.ManagedChannelRegistry.addProvider(ManagedChannelRegistry.java:69)
	at io.grpc.ManagedChannelRegistry.register(ManagedChannelRegistry.java:64)
	at org.example.grpcbase.client.GrpcClient.main(GrpcClient.java:17)

If you go deeper and investigate wilds of Epoll, eventually you came to this guy:

io.netty.channel.epoll.Epoll.ensureAvailability();

And this is exactly what we wanted for so long. This method throws exception with the cause, explaining why this transport is not available (if it is actually not available).

➤ Expand to see the full possible stacktrace 
Exception in thread "main" java.lang.UnsatisfiedLinkError: failed to load the required native library
	at io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)
	at org.example.grpcbase.client.GrpcClient.main(GrpcClient.java:17)
Caused by: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll_x86_64
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:239)
	at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:323)
	at io.netty.channel.epoll.Native.<clinit>(Native.java:85)
	at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40)
	at org.example.grpcbase.client.GrpcClient.main(GrpcClient.java:16)
	Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:239)
		at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:326)
		... 3 more
	Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:186)
		... 4 more
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
			at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
			at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
			at java.base/java.lang.System.loadLibrary(System.java:2059)
			at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
			at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:396)
			at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
			... 4 more
			Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
				at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
				at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
				at java.base/java.lang.System.loadLibrary(System.java:2059)
				at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
				at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
				at java.base/java.lang.reflect.Method.invoke(Method.java:580)
				at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:430)
				at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
				at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:422)
				at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:388)
				... 5 more
Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll_x86_64.so
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:186)
	... 4 more
	Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
		at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
		at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
		at java.base/java.lang.System.loadLibrary(System.java:2059)
		at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
		at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:396)
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
		... 4 more
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
			at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
			at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
			at java.base/java.lang.System.loadLibrary(System.java:2059)
			at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
			at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
			at java.base/java.lang.reflect.Method.invoke(Method.java:580)
			at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:430)
			at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
			at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:422)
			at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:388)
			... 5 more

Btw, in my case, the reason was the absence of netty_transport_native_epoll for my architecture in classpath. I tested this inside this Docker base image: FROM --platform=linux/amd64 ubuntu:23.10 on Mac M1:

java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path

Simplest (but not most correct) solution here is to add netty-all dependency:

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-all</artifactId>
    <version>4.1.94.Final</version>
</dependency>

For sure, it is worth to know, which one you need to include exactly:

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-transport-native-epoll</artifactId>
    <version>${project.version}</version>
</dependency>

Double check your architecture and the need to add classifier:

<classifier>linux-x86_64</classifier>

Remember, since you have gRPC configured, you probably have os-maven-plugin also. Then you can use it here:

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-transport-native-epoll</artifactId>
    <version>${project.version}</version>
    <classifier>${os.detected.classifier}</classifier>
</dependency>

Pay attention to the environment when you want to use your app, since you may compile on Windows (and ${os.detected.classifier} gave you windows classifier), but your intention was to run your application on Windows.

Telegram channel

If you still have any questions, feel free to ask me in the comments under this article or write me at promark33@gmail.com.

If I saved your day, you can support me 🤝

​

Leave a Reply

Your email address will not be published. Required fields are marked *

                   Mark Chesnavsky. Email me or connect via LinkedIn.