Invalid signature file digest

Full error text: This error can be encountered if there are signed libraries among the dependencies of your project. To solve the problem, you need to exclude files with the extensions .SF, .DSA, .RSA from the build. Of course, we will assume that you are building uber jar (fat jar). We need to add a […]

READ MORE

No main manifest attribute Gradle Kotlin DSL

You can get this error if you try to run jar without specifying the name of the class that contains fun main() and which you want to run. To do this, you need to register the full class name in the Main-Class key of your manifest. Plain jar without dependencies You need to replace the […]

READ MORE

Uber JAR with Gradle Kotlin DSL

In this article, we will find a solution to the question – how to make an uber jar Gradle Kotlin DSL or how to make a fat jar Gradle Kotlin DSL. Let’s add this to build.gradle.kts file: After that, you can build the jar with dependencies: That’s all. Possible errors Error: no main manifest attribute, […]

READ MORE

Spark’s User Defined Functions in Java

In this article we will find the answer for questions: How to change the column in Spark? How to modify column in Spark? In other words: how to create a user defined function (UDF) and apply it. For example, let’s have a look to UDF, that takes a String and returns a String. For Spark […]

READ MORE

Spark failed to connect to the MetaStore Server

The problem You may encounter errors like this when running a Spark script / application: Solutions If you do not need the MetaStore server, there are two ways to disable it. Please note that Spark version >= 2.x is required. The first way (via spark2-submit parameters) The second way (via SparkConf object) Java example: Scala […]

READ MORE

Spark custom parquet OutputCommiter

If you need to make your own implementation of OutputCommiter for spark parquet-output tasks, then first of all you need to make a class that extends from org.apache.hadoop.mapreduce.OutputCommiter: Further, regardless of implementation of the OutputCommiter, you need to register the full class name in the Hadoop (!) configuration of Spark like this: Example for Scala: […]

READ MORE

Spark concurrent write to same HDFS path

The problem Sometimes you need to run such a scenario when several Spark tasks write data along the same path to HDFS. During the execution of tasks, you may encounter some errors: Suppose we have one Spark task, that writes to the hdfs://data/test directory. At runtime, Spark will make a temporary directory: hdfs://data/test/_temporary/0. There is […]

READ MORE

Spring TCP server

Let’s suppose that you need configure a TCP server in Spring / Spring Boot application. We will make it that the server prints the client’s request on the screen, then sends a response to the client. First, let’s include the spring-integration-ip dependency. In Maven it will look like this: Like gRPC ideology, in this case […]

READ MORE

Terracotta Ehcache notes

Java has a caching standard – JCache. This standard is described in the JSR-107 specification. This standardized approach makes it easier to integrate with different tools that perform the same function. Thus, Spring Boot supports JCache integration. Therefore, whatever implementation of JCache we would use, the code remains the same. Some of the common JCache […]

READ MORE