Data Ecosystem Infrastructure LA4LD Status Update

In March I wrote a post about my BSc graduate assignment on developing a data ecosystem infrastructure. This Free and Open-Source Software (FOSS) collects, cleans, and structures the learning analytics data gathered by the different systems in the ecosystem. At the same time the software makes the structured and unstructured (raw) data available to other systems for further processing or presentation. This project is part of Marcel Schmitz’s PhD research project.

Recap

To give a quick recap of the project I’m working on:

  • I’m using the Design Based Research Process as defined by Peffers
  • I have done a Systematic Mapping Review on literature about Data Ecosystems
  • I’m using Agile and Scrum to manage the project
  • The software and scrum-board is hosted on GitHub.com
  • The software is being developed using Test-Driven Development (TDD)
  • The language and framework being used are Python and Flask

Status

Currently I’m in the design & development phase of the Design Based Research Process, which is the third phase (problem definition & motivation being the first and objectives of solution the second). So for about the last 6 weeks or so I have been working through the iterative process of designing a little, then building a little.

The core functionalities currently realized are student results tracking, attendance tracking, a xAPI endpoint, and part of the admin panel. The only core feature missing at this point is the storage of questionnaire data (like data from the MSLQ). Right now the xAPI data is just the raw xAPI calls without any cleaning or indexing, because at the moment, no test data, which contains examples of xAPI data generated by the new Moodle setup, is available.

Since the goal is still to run this software at the start of September, deployment of the application also needs to be considered.

Overall the project is going very well. I’m hitting my deadlines for the core features while at the same time being able to implement some extra features (like extended logging and internationalization) and having time available to go back and clean or improve the existing code.

Reflection So Far

As I’ve said in the previous section, the project overall is going well. Still, there are some point I want to reflect upon.

Test-Driven Development

This is the first time I’m actually using Test-Driven Development (TDD) in a larger scale project. Up until now I had only worked through the book Test-Driven Development with Python by H. Percival (which is great f.y.i.) and some small scale projects. Turns out, TDD is quite difficult to do correctly and consistently.

Thinking through the tests is doable, the difficult part is actually developing the unit tests and getting them to interact as intended with the Flask framework. This is not due to any fault of Flask, it is just me learning a new methodology with a new testing toolkit.

While I’m getting the hang of both the methodology and the tooling, I’m still convinced attempting to use TDD was the right choice. Even though I’m not (yet) able to fully implement it as intended, it still results in a more complete and better test suite for the application.

Back-end Front-end Separation

This is a relatively minor problem, due to the system being intended as mostly a back-end application with a rudimentary GUI. But if I had to start again I would create a bigger separation between back-end and front-end. I would have probably chosen to use a JavaScript framework like React or Angular to build the user-interface. The Flask application itself would have been only an API with which the front-end communicates. For now though, the Flask templating engine is good enough.

Planning

During the next 3 to 4 weeks, I’ll be doing my lasts sprints. During these sprints I will add support for the MSLQ questionnaire, index xAPI partially (if example data become available), and make the application ready for deployment. During this period the documentation and extended abstract will be written to complete my graduation portfolio.

After these last sprints, and thus completing the design & development phase of the project, I will mainly be helping Marcel with the actual deployment of the software in an experimental setting and prepare for my final presentation.

So, Once Again, What Now?

This is my 2nd of 4 posts I will write about my assignment. After the design & development phase, I’ll write a post presenting the results of my project. And finally, after completing my graduation project, I’ll write a post to reflect and look back on my experiences.