Suppose, that we need to read the file from the Spring Boot Yarn application. We seem to have done everything right: placed the file next to the Yarn-client, Yarn-container and Yarn-appmaster jars; have used the File class, Files class, FileInputStream class and others. But for some reason we see exceptions like below:
java.nio.file.NoSuchFileException: <filename>
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at java.nio.file.Files.newBufferedReader(Files.java:2784)
at java.nio.file.Files.lines(Files.java:3744)
at java.nio.file.Files.lines(Files.java:3785)ava.io.FileNotFoundException: <filename> (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)The problem is that the file, that we need is simply, is not available on the hosts where AppMaster and Containers are launched. Yarn needs to move the files we need to these hosts. This process is called resource localization. More details can be found here: https://blog.cloudera.com/management-of-application-dependencies-in-yarn.
To set up localization, you need to add the files and localizer sections in the application.yml configuration of the Yarn client:
spring:
...
yarn:
...
appmaster:
...
files:
- "*.txt"
- "*.<your_file_extension>"
- "*.<another_extension>"
localizer:
patterns:
- "*.txt"
- "*.<your_file_extension>"
- "*.<another_extension>"
Now, Yarn will transfer files with specified extensions from the folder with the Yarn-client, Yarn-appmaster and Yarn-container jars to the hosts on which the client and appmaster will run.
Telegram channel
If you still have any questions, feel free to ask me in the comments under this article or write me at promark33@gmail.com.
If I saved your day, you can support me 🤝