Import NiFi pipelines

ZWE nifi process extracts information from the moodle instance to later create analytic data and have further understanding of the progress of the students and how the content is being consumed by them.

This page contains the following information:

Technologies used

How to install

You can find all the files related to this process in the following link: https://github.com/KnowTechTure/ZWE_NiFi

1. Create/download NiFi template

First, make sure to have a template that includes all the definition of the NiFi pipeline. This template comes in a .xml format. To create a template, make sure that you are in the desired pipeline. This means that if your definition is inside a process group, you must be inside the process group before creating the template.

Click on create template on the left panel of the screen:

This will ask you for a name and description, provide the information to create the template:

Once you already have the template, this template will appear in the templates created so you can download it from there.

Click on the hamburger menu on the top right side of the screen and click on templates:

A new window will be displayed listing all the templates created. At the end of the row of the desired templated, two options are available: download and delete. Click on download button to download the .xml file (template definition):

2. Upload the template in a new NiFi instance

Once the first step is already completed https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#1.-Create%2Fdownload-NiFi-template, it is time to import the template into the desired NiFi instance.

To upload a a template into NiFi, click on upload template button on the left panel of the NiFi window:

This will open a window enabling to upload the template from a local storage.

3. Create context parameters

Once the step 1 https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#1.-Create%2Fdownload-NiFi-template and step 2 https://psi.atlassian.net/wiki/spaces/ZMO/pages/edit-v2/583630850?draftShareId=0ab54fe1-d538-4710-9283-5b5e5436cdb7#2.-Upload-the-template-in-a-new-NiFi-instance are completed, you will likely have the pipeline imported in your NiFi instance. The pipelines require you to create a parameter context (that act as environment variables). To create these, first go to your NiFi instance and click on the hamburguer icon on the top right corner of the NiFi screen. Then, select parameter context:

Here, a table with a list of the parameter context will be displayed, to create a new parameter context, click on the plus button on the right side of the window:

A new window will be displayed for you to add a name and description to the parameter context:

Then, to add parameters to the parameter context, click on the parameters tab and click on the plus button on the right corner. A new window will be displayed for you to add one by one the parameters:

The following parameters must be added in order to make the flow work properly:

After creating the parameter context, it is vital to add this parameter context to each process. To add the parameter context to each process, it must be added to the process group by selecting the process group and then clicking on the configuration button on the left panel:

This will open a new window to configure the process group. To select the desired parameter context, look for the second option that is a dropdown and select the parameter context created.

4. Adding controllers

The NiFi flow contains processes that requires to interact with a database. This interaction requires the creation of a controller service called DBCPConnectionPool. The NiFi flow creates this controller services automatically when the user import the process.

To access this controller services, click on the configuration buttton on the left panel:

A new window will open. Click on the controller services tab:

(create service link)

## How to use

There are a couple of steps to follow in order the make everything to work:

1. Download the .xml file, this file contains all the definitions for NiFi to create the proper flow.

2. Log in into your nifi instance and upload the .xml as a template.

3. Once you have the template uploaded in NiFi, it is recommended to create a process group to place the recently uploaded .xml file.

4. Inside the process group, place the template. This will generate the whole flow.

5. Once the flow is generated, is important to create the a parameter context with the parameters as detailed in the following table:

| name | value |
|-----|-------|
| baseurl | your moodle host url |
| errorLogsDirectory | name of the folder to put the error logs|
| moodleanalyticsplugintoken | token to access the functions for the moodle analytics custom plugin | 
| moodletoken | token to access the core functions of moodle | 
| useKeyCloak | no |

6. When you create the parameter context, you need to specify, for each process, the parameter context to use. In case the process is indeed a process group, the sub processes will inherit the parameter context set up in the process group.

7. After configuring every process of the flow, you must create/setup two controller services:
    * **DBCPConnectionPool**: database connection url, driver class name path (driver .jar file must exist in the path provided), databse username and password must be provided.
    * **JsonTreeReader**: default configuration works well.

8. Finally, you must ensure that the database with the tables exist. Use the **tables.sql** file to create the core tables so the process can work properly.