Import NiFi pipelines
- Ricardo Gomez
ZWE nifi process extracts information from the moodle instance to later create analytic data and have further understanding of the progress of the students and how the content is being consumed by them.
This page contains the following information:
Technologies used
Tool | Version |
|---|---|
NiFi | 1.27.0 |
Postgres | 17 |
Moodle | 4.1 |
Superset | 4.0.2 |
How to install
You can find all the files related to this process in the following link: https://github.com/KnowTechTure/ZWE_NiFi
1. Create/download NiFi template
First, make sure to have a template that includes all the definition of the NiFi pipeline. This template comes in a .xml format. To create a template, make sure that you are in the desired pipeline. This means that if your definition is inside a process group, you must be inside the process group before creating the template.
Click on create template on the left panel of the screen:
This will ask you for a name and description, provide the information to create the template:
Once you already have the template, this template will appear in the templates created so you can download it from there.
Click on the hamburger menu on the top right side of the screen and click on templates:
A new window will be displayed listing all the templates created. At the end of the row of the desired templated, two options are available: download and delete. Click on download button to download the .xml file (template definition):
2. Upload the template in a new NiFi instance
Once the first step is already completed Import NiFi pipelines | 1. Create/download NiFi template, it is time to import the template into the desired NiFi instance.
To upload a a template into NiFi, click on upload template button on the left panel of the NiFi window:
This will open a window enabling to upload the template from a local storage.
It is recommended that before uploading the template, add a process group and name it accordingly to avoid having a mixed flows in the main NiFi screen.
3. Create context parameters
Once the step 1 Import NiFi pipelines | 1. Create/download NiFi template and step 2 Import NiFi pipelines | 2. Upload the template in a new NiFi instance are completed, you will likely have the pipeline imported in your NiFi instance. The pipelines require you to create a parameter context (that act as environment variables). To create these, first go to your NiFi instance and click on the hamburguer icon on the top right corner of the NiFi screen. Then, select parameter context:
Here, a table with a list of the parameter context will be displayed, to create a new parameter context, click on the plus button on the right side of the window:
A new window will be displayed for you to add a name and description to the parameter context:
Then, to add parameters to the parameter context, click on the parameters tab and click on the plus button on the right corner. A new window will be displayed for you to add one by one the parameters:
The following parameters must be added in order to make the flow work properly:
Parameter name | Parameter value |
|---|---|
baseurl | your moodle host url |
errorLogsDirectory | name of the folder to add the error logs files |
moodleanalyticsplugintoken | token to access the functions for the moodle analytics custom plugin |
moodletoken | token to access the core functions of the moodle instance |
useKeyCloak | no |
After creating the parameter context, it is vital to add this parameter context to each process. To add the parameter context to each process, it must be added to the process group by selecting the process group and then clicking on the configuration button on the left panel:
This will open a new window to configure the process group. To select the desired parameter context, look for the second option that is a dropdown and select the parameter context created.
Parameter context is applied to the processes inside a process group. In case there are more process groups inside the main process group, it is needed to add the parameter context for each process group.
4. Adding controllers
The NiFi flow contains processes that requires to interact with a database. This interaction requires the creation of a controller service called DBCPConnectionPool. The NiFi flow creates this controller services automatically when the user import the process.
To access this controller services, click on the configuration buttton on the left panel:
A new window will open. Click on the controller services tab:
Here, two controller services will be disabled: DBCPConnectionPool and JsonTreeReader. The JsonTreeReader does not require any additional configuration so it can be enabled right away. The DBPCConnectionPool does require some fields to be updated so before enabling, click on the configuration button for this controller service:
And then open click on the properties tab:
Here, there are four fields that the user must set up before enabling the controller service:
Input name | Value |
|---|---|
Database Connection URL | Connection URI for the database. Example: jdbc:postgresql://host:port/database_name |
Database Driver Class Name | Database driver according to the database manager to use. |
Database Driver Location(s) | NiFi path where the database driver is located. |
Database User | Database user. |
Database Password | Database password. |
This fields are already set up for a postgresql database connection, in case there is a need to change the database instance, this fields may change accordingly.
Also, it is important to have a look on the field called Database Driver Location(s), this field allows NiFi to search for a .jar file that is required for the database connection to work. This driver does not come with the NiFi instance, so the file is required to be downloaded and placed in the location defined.
Database connections made via SSH are not supported.
Once the fields are already set up, click on Apply button at the right bottom of the window and now the controller service can enabled:
When clicking on the enable button, a new window will show listing all the components using this controller service. Every component will have a warning icon, once the service is enabled, all the listing components will have a stoped icon referencing that the service is now available for the components:
5. Creating database and views
After going through all the previous steps to configure the NiFi flow in Import NiFi pipelines, the final step is now to create the database and its views. This will allow NiFi to refresh the data properly.
Check the following documentation for the creation of the database, tables and views required for this NiFi flow: DataWareHouse | DataWareHouse Creation.