Versions Compared

Version Old Version 1 New Version Current
Changes made by

Rodolfo Melia

Gaspar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The following page details the process and files stored for ZWE MOH machines:

Table of Contents
minLevel1
maxLevel6
outlinefalse
stylenone
typelist
printabletrue

...

Backup Files and Process

The different databases/files for the e-Learning, VMMC and Analytic systems are backed up every 24hrs, at 1am local time. Resulting files are stored in the folder root/backups. Details about what is backed up is detailed on the table below.

Each server contains their own backups stored in the following folder:

Code Block
/opt/backups/

The process to backup files are triggered by multiple bash files inside the /opt/backups/ path of each app/service. The following table details the order that each script file is triggered in order to be backed up:

System

What is backup

Backup procedure

VMMC DB

Mongo DB

Mongo dump → compress compressed fileMoodle

eLearning

Resource files

Status
colourRed
titleMISSING

Postgres SQL database

SQL dump → xxx.sql

files folder

Superset Configuraiton (User, Dashboards, Connections, etc)

Dashbaord definitons

./dasbharods/xxxxx.yaml

API ?

SQL backup ?

Víctor Enrique Mendoza Yanes
Ricardo Gomez

to update what file are we going to backup?

  1. Dump postgres database.

  2. Extract database dumped from docker container.

  3. Transfer dump file to backup machine.

Superset

Config files

  1. Extract configuration file from docker container.

  2. Transfer configuration file to backup machine.

Dashboard

  1. Rename existing backup files

  2. Remove previous backup files.

  3. Create new dashboards backup.

  4. Extract dashboards backup.

  5. Transfer dashboard to backup machine.

Database

  1. Dump postgres database.

  2. Extract database dumped from docker container.

  3. Transfer dump file to backup machine.

NiFi

Configuration folder (includes nifi flows, user credentials, etc)

  1. Compress configuration directory.

  2. Extract configuration directory from docker container.

  3. Transfer configuration directory to backup machine.

Datawarehouse

Postgres SQL Database

  1. Dump postgres database.

  2. Extract database dumped from docker container.

  3. Transfer dump file to backup machine.

Docker analytics volume

Volumes folder

  1. Compress volume directory

  2. Transfer compressed directory to backup machine.

Docker elearning volume

Volumes folder

  1. Compress volume directory

  2. Transfer compressed directory to backup machine.

...

File Naming Conventions

We use the following convention:

  • YYYY-MM-DD - the date that the backup was completed, local time

  • HH-MM: the time that the backup was completed, local time

  • [dwm]: one of these characters, depending if the backup is daily, weekly or monthly

  • “MOH_ZWE”+system: the name of the system backed up (MONGO, ELEARNING…)

  • databaseName: in the vmmc case, we also need to add the database name, we have to: db1 and dbOthers.

  • .X.X - The version of the associated product, to ensure “restorability”

  • .XXX the file extension

Resulting file names for MOH -ZWE-MOODLEZWE:

  • Moodle db: yyyy-mm-dd-hh-mm-moh-zwe-moodle-d-4.5.

...

  • sql.

...

YYYY-MM-DD-HH-MM-MOH-ZWE-MONGO.7.ZIP

...

  • giz

  • Moodle files: yyyy-mm-dd-hh-mm-moh-zwe-moodle—m-files.giz

  • VMMC db: yyyy-mm-dd-hh-mm-moh-zwe-d-mongo-dbOthers.gz

  • Superset: yyyy-mm-dd-hh-mm-moh-zwe-d-superset.3.giz

...

Retention Policies

In the absence of any mandate on backup frequency and retention, the following backup procedure was implemented for the various systems' databases, as of December 2024:

  • Daily backups, kept for 31 days

  • Weekly backups (Sunday), kept for 1 year

  • Monthly backups (last day of the month), kept for 7 years

For the Moodle folder, we implemented the following backup policy:

  • Daily backups, 31 days

  • Weekly backups, kept for 6 months

  • Monthly backups, kept for 7 years

The moodle files folder contains:

  • Language customizations

  • Files & images uploaded to Moodle

  • Cache storage

  • User session’s data

...

Storage needs

Average backup size & transmisison time

  • Mooodle DB

  • Moodle files

Backup

Year-to-date

Year 2-7

Retention for databases
remote disk (PSH)
local disk (MOH)

Last 30 days

Last 52 weeks (Sunday)

Last 12 month (Last day of month)

= 94 backups

Last day of the month Year 2-7

= 12 backups per year

= 72 backups

Retention for Moodle files
Local disk (MOH)

Last 7 days

n/a

Retention for Moodle files
Remote disk

Daily backups, 14 days

Weekly backups, 6 months

Monthly backup for 6 month

Monthly backups, kept for 7 years

Moodle/ SQL

avg backup size: 1.0 Mb

Transmission time: 5s

+ 20% annual increase

Moodle files

VMMC Mongo

avg backup

5 .0 GB

Superset backup

avg backup

1.0 Mb