gRPC cannot assign requested address

Suppose you just run into this exception, saying that gRPC cannot assign requested address:

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: connect(..) failed: Cannot assign requested address
Caused by: java.net.ConnectException: connect(..) failed: Cannot assign requested address
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
	at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
	at io.grpc.netty.shaded.io.netty.channel.unix.Socket.connect(Socket.java:270)
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773)

TL;DR: most likely you opened too many connections, and you run out of ephemeral ports. Consider channel/stub reuse: https://mchesnavsky.tech/grpc-channel-stub-reuse.

Details: first, we should distinguish between fixed ports, and ephemeral ports.

Ephemeral ports are typically used by client devices initiating communication with a server. The server, on the other hand, uses a well-known port number (fixed) that clients can easily identify.

Unlike fixed ports assigned to specific services (like port 80 for web traffic), ephemeral ports are temporary and only last for the length of the current connection.

Total number of possible ephemeral ports in OS may equal 16384: from 49152 to 65535.

Suppose you have 100 threads on your client, each of whose is doing something in a loop, each time creating and closing a new connection/gRPC channel. You may first think, that you have to have 100 ephemeral ports available in your system.

But it may not be the case: OS needs to have some time to free up the port even after you closed the channel and waited until it closure:

ManagedChannel channel = ((ManagedChannel) stub.getChannel());
channel.shutdownNow();
channel.awaitTermination(timeout, TimeUnit.MILLISECONDS);

Consider thread number 5 of our hundred threads. It just created a channel, occupied port number 50001, did some work, and closed it. Then, it went to another iteration and created the channel again. However, OS hasn’t freed port 50001 yet. Eventually, all these operations cascade, and there are no ports left.

What you should really do is to reuse gRPC channels, or at least impose timeouts between thread iterations.

Consider using thread pools of gRPC stubs or their analogue – kind of a shared thread pool, where you can store your stubs that can be in use by different threads. Find more on this here: https://mchesnavsky.tech/grpc-channel-stub-reuse.

That’s all.

If you still have any questions, feel free to ask me in the comments under this article or write me at promark33@gmail.com.

​