Nifi-Docker

NiFi docker container customization from the official Apache Nifi Docker Image

1. Build Docker

Pull the latest version of Apache Nifi Docker, build and run

$ docker pull apache/nifi:1.27.0

Option 1: Run without authentication

$ docker run --name noharm-nifi -e NIFI_WEB_HTTP_PORT='8080' -p 8080:8080 -d apache/nifi:1.18.0 --restart=always

Open the URL http://localhost:8080/nifi.

Option 2: Run with authentication

docker run --name noharm-nifi -e NIFI_WEB_HTTPS_PORT='8443' -p 8443:8443 -d -e SINGLE_USER_CREDENTIALS_USERNAME=nifi_user -e SINGLE_USER_CREDENTIALS_PASSWORD=nifi_pass   apache/nifi:1.18.0 --restart=always

Watch until the container is ready. You are looking for the "NiFi has started. The UI is available" message.

docker logs -f noharm-nifi

Open the URL https://localhost:8443/nifi

2. Download Connectors and Add Timezone

Dive into container shell, add timezone and download JARs & NARs

$ docker exec --user="root" -it nifi /bin/bash
nifi@container_id:/opt/nifi/nifi-current$ echo "java.arg.8=-Duser.timezone=America/Sao_Paulo" >> conf/bootstrap.conf
nifi@container_id:/opt/nifi/nifi-current$ wget https://github.com/noharm-ai/nifi-composer/blob/main/genkeypair.sh
nifi@container_id:/opt/nifi/nifi-current$ ./genkeypair.sh
nifi@container_id:/opt/nifi/nifi-current$ cd lib

Copy & Paste it

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar
wget https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/23.2.0.0/ojdbc8-23.2.0.0.jar
wget https://jdbc.postgresql.org/download/postgresql-42.7.3.jar
wget https://repo1.maven.org/maven2/org/apache/nifi/nifi-kite-nar/1.15.3/nifi-kite-nar-1.15.3.nar
wget https://truststore.pki.rds.amazonaws.com/sa-east-1/sa-east-1-bundle.pem

• Updated JDBC PostgreSQL Driver at https://jdbc.postgresql.org/download
• Updated JDBC Oracle Driver at https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html
• Updated JDBC MySQL Driver at https://dev.mysql.com/downloads/connector/j/

3. Restart Nifi Web Service

Inside container, stop the service. It will kick you out of the container.

nifi@container_id:/opt/nifi/nifi-current$ ./bin/nifi.sh stop

Restart you container

$ docker start nifi

Update the restart policy

$ docker update --restart=always nifi

Read current restart policy

docker inspect -f "{{ .HostConfig.RestartPolicy }}" nifi

4. Access Nifi Web Service

Now you can access your Nifi web service at http://localhost:8080/nifi/

If you are in a VPN environment remember to tunnel 8080 port

$ ssh -f user@ip_address -L 8080:localhost:8080 -N

How to tunnel in Putty (for Windows users)

5. Basic Nifi Usage

This wiki page will cover the main proccess of NoHarm integration: Basic Nifi Usage

Troubleshooting

a. Testing JDBC Connection

• Download tool from http://jdbcsql.sourceforge.net/
• Run: java -jar jdbcsql-1.0.zip -h hostname -p 1521 -U user -P pass -d service -m oracle "SELECT table_name FROM all_tables"

b. Install SQL Plus

• Install:

sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-basic-21.6.0.0.0-1.el8.x86_64.rpm
sudo yum install -y yum install https://download.oracle.com/otn_software/linux/instantclient/216000/oracle-instantclient-sqlplus-21.6.0.0.0-1.el8.x86_64.rpm

• Run:

sqlplus user/pass@localhost:1521/service

SQLPlus for Debian: https://askubuntu.com/questions/159939/how-to-install-sqlplus

c. Repair "no space left" issue

Prune useless container and images:

$ docker system prune -a

Check fuller folders (if possible):

$ docker exec -t nifi sh -c "du -sh *"

Empty target folder (usually content_repository):

$docker exec -t nifi sh -c "rm content_repository/* -rf"

Restart nifi to repair empty folder:

docker exec -t nifi sh -c "./bin/nifi.sh restart"

You can do the same inside the container:

$ docker exec --user="root" -it nifi /bin/bash

d. Fix "docker.sock permission denied" issue

sudo chown $USER /var/run/docker.sock

e. Restarting Docker service if Docker is down:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

sudo service docker start

e. Changing Properties from HTTPS to HTTP:

• https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L53-L69