Import NiFi pipelines

ZWE nifi process extracts information from the moodle instance to later create analytic data and have further understanding of the progress of the students and how the content is being consumed by them.

This page contains the following information:

Technologies used

How to install

You can find all the files related to this process in the following link: https://github.com/KnowTechTure/ZWE_NiFi

1. Create/download NiFi template

First, make sure to have a template that includes all the definition of the NiFi pipeline. This template comes in a .xml format. To create a template, make sure that you are in the desired pipeline. This means that if your definition is inside a process group, you must be inside the process group before creating the template.

Click on create template on the left panel of the screen:

This will ask you for a name and description, provide the information to create the template:

Once you already have the template, this template will appear in the templates created so you can download it from there.

Click on the hamburger menu on the top right side of the screen and click on templates:

A new window will be displayed listing all the templates created. At the end of the row of the desired templated, two options are available: download and delete. Click on download button to download the .xml file (template definition):

2. Upload the template in a new NiFi instance

Once the first step is already completed https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#1.-Create%2Fdownload-NiFi-template, it is time to import the template into the desired NiFi instance.

To upload a a template into NiFi, click on upload template button on the left panel of the NiFi window:

This will open a window enabling to upload the template from a local storage.

3. Create context parameters

Once the step 1 https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#1.-Create%2Fdownload-NiFi-template and step 2 https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#2.-Upload-the-template-in-a-new-NiFi-instance are completed, you will likely have the pipeline imported in your NiFi instance. The pipelines require you to create a parameter context (that act as environment variables). To create these, first go to your NiFi instance and click on the hamburguer icon on the top right corner of the NiFi screen. Then, select parameter context:

Here, a table with a list of the parameter context will be displayed, to create a new parameter context, click on the plus button on the right side of the window:

A new window will be displayed for you to add a name and description to the parameter context:

Then, to add parameters to the parameter context, click on the parameters tab and click on the plus button on the right corner. A new window will be displayed for you to add one by one the parameters:

The following parameters must be added in order to make the flow work properly:

After creating the parameter context, it is vital to add this parameter context to each process. To add the parameter context to each process, it must be added to the process group by selecting the process group and then clicking on the configuration button on the left panel:

This will open a new window to configure the process group. To select the desired parameter context, look for the second option that is a dropdown and select the parameter context created.

4. Adding controllers

The NiFi flow contains processes that requires to interact with a database. This interaction requires the creation of a controller service called DBCPConnectionPool. The NiFi flow creates this controller services automatically when the user import the process.

To access this controller services, click on the configuration buttton on the left panel:

A new window will open. Click on the controller services tab:

Here, two controller services will be disabled: DBCPConnectionPool and JsonTreeReader. The JsonTreeReader does not require any additional configuration so it can be enabled right away. The DBPCConnectionPool does require some fields to be updated so before enabling, click on the configuration button for this controller service:

And then open click on the properties tab:

Here, there are four fields that the user must set up before enabling the controller service:

• Database Connection URL.

• Database Driver Class Name.

• Database Driver Location(s).

• Database User.

• Database Password.

This fields are already set up for a postgresql database connection, in case there is a need to change the database instance, this fields may change accordingly.

Also, it is important to have a look on the field called Database Driver Location(s), this field allows NiFi to search for a .jar file that is required for the database connection to work. This driver does not come with the NiFi instance, so the file is required to be downloaded and placed in the location defined.

Once the fields are already set up, click on Apply button at the right bottom of the window and now the controller service can enabled:

When clicking on the enable button, a new window will show listing all the components using this controller service. Every component will have a warning icon, once the service is enabled, all the listing components will have a stoped icon referencing that the service is now available for the components: