How to run Scala script with Object on Spark

Suppose you have code like this in Scala: And you run it like this: And you want to put it in a Scala Object so that IntelliJ IDEA doesn’t highlight the code in red. Firstly, you are going to do smth like that: But when you run the script, nothing happens and spark2-shell just opens. […]

READ MORE

ZooKeeper recursive watcher

If you need to set up a recursive watchers (watch on all nodes), the standard ZooKeeper’s Watcher class will not help much – it is installed on only 1 node (or one-level-forward when you calling getChildren()), and is also a one-time event. This means that after each watch trigger, you need to install a new […]

READ MORE

How ZooKeeper ACL works

In this post I will describe the basic principles of how ACL works in ZooKeeper. ACL is not set recursively and is not inherited by the child nodes. If we have a read-only ACL for /path1/path2 or /path1/path2/path3, then deleting /path1 will fail, regardless of AСL of /path1. Several ACL records can be set on […]

READ MORE

Apache Atlas – Building & Installing

Let’s say we want to got working Apache Atlas instance with embedded Hbase & Solr on our machine. Notice, that you need to install JDK 8 before start. Go to the Apache Atlas GitHub page, and download the zip file with the source code of the latest stable release from here: https://github.com/apache/atlas/releases Important notice: DO […]

READ MORE

How to immediately terminate the Spring Boot Yarn container with an error

Imagine an error or exception occurs while running the Spring Boot Yarn container, and we need to kill container from itself and return an error code. You can use @OnContainerStart annotation as mentioned in this article: https://mchesnavsky.tech/how-to-set-up-exit-code-on-spring-boot-yarn-container. But if we need to stop the container immediately, we just need to call: – where parameter is […]

READ MORE

IgniteException: Work directory does not exist

Suppose that when starting the application, you encounter the following exception: It is assumed, that the /tmp/ignite/work directory is located on the local file system. If the application is running on a cluster, then Ignite will try to create such a folder on each node of the cluster. Exception reasons: The account under which the […]

READ MORE