Building capacity and ensuring competence in FSD's IT Services
To become a partner in international projects, FSD must have the ability to participate in the development of shared technical solutions and to implement the solutions in its own systems. FSD must be able to operate according to the principles of modern software development, flexibly and efficiently. This will ensure that the organisation can carry out in its main tasks effectively and be a competitive partner for various projects. Many tasks related to capacity and competence building take place 'under the hood'. The outcomes are practically invisible to outsiders, and often also to FSD staff but yet in many ways necessary for carrying out the daily data curation work.
Over the last few years, FSD's IT service manager Matti Heinonen has overseen two such tasks. FSD’s software development methodology has been updated under the task “Aligning FSD's IT to CESSDA service development practices". This meant introducing modern development methods and tools so that application development would be consistent with CESSDA's development environment. This work has greatly improved FSD's ability to compete especially for CESSDA's internal project funding and Horizon Europe project funding. It also improved the way processes are managed in FSD's technical services and contributed towards keeping the developers satisfied with their work environment. Having a modern development environment is also a competitive advantage when trying to hire skilled professionals or professionals with a great competence potential.
Another task was related to "Building resilience and interoperability to FSD's IT systems". Resilience refers to improving the fault tolerance of FSD's technical platform. Much effort was also made to use human resources more efficiently. This is described in more detail in the chapter ‘Observability metrics’.
CI/CD environment and testing
CI stands for Continuous Integration. In this environment changes to program code are made available to other developers as they are implemented. Any code changes made must be compatible with other frequent changes made by multiple developers. Therefore, changes must be submitted to unit and integration testing when made available. Since there can be several changes per day, the testing process must be automatic so that testing is possible without slowing down the development activity.
CD means continuous delivery. Changes to the code are made ready for release automatically if they pass integration tests as well as functional tests. Both CI and CD require software that supports them. In this project, the focus was on continuous integration. A set of applications that specifically support this practice were included in FSD’s software stack.
—Any selected software alone does not make IT operations follow Continuous Integration practice. What is also required is the adoption of new ways of working and training how to work according to his method, Matti Heinonen explains.
Gitlab is the version control tool for FSD's CI/CD environment. When new code is pushed to Gitlab, it is automatically subjected to integration tests, which are run using an automation server Jenkins. Jenkins uses Docker containers, which ensures that tests can always be performed in a clean software environment, so that, for example, there is no old code or dependencies that could distort the test results. Jenkins tests the execution of the software and in addition initiates a static analysis of the code. This is carried out by SonarQube, which focuses on a systematic review of the quality and cleanliness of the code. The feedback provided by Jenkins and SonarQube is passed back to the developers, who then fix errors and improve the code based on the results. The cycle continues until the developer sees the change is complete. The last step is a peer review of the code and approval of the change by another FSD developer. Gitlab provides tools to do this efficiently. Only the changes that have been reviewed in this way are added to the actual codebase.
Observability metrics
Observability is divided into two parts: metrics and log monitoring. FSD has more than a dozen different services and more than 30 external or internal virtual servers in production. Anticipating server maintenance requires information that provides a warning of future needs in advance. Matti Heinonen provides an example:
—A simple case is disk space, which should be increased in a controlled manner and not when the service has already crashed after the disk space is full. For this, you need metrics about the disc usage and its projected development.
Collecting information by hand would be slow and inefficient because information is considered relevant only when certain boundary conditions are met. At that point, system administrators need to get the information to be able to process it. To monitor the status of the servers, FSD uses Zabbix software.
Log Monitoring
Another important source of information for activity monitoring are the logs produced by different services. While metrics help to predict future problems, log data helps to react to errors and find out the root causes. Various servers generate a huge amount of log data and in normal conditions, most of it is just noise. The volume is too large to handle manually. Instead, we needed a system that centralizes the logs in one place, provides a user interface for processing them, and gives alerts about relevant errors. During the project, FSD implemented Elasticstack for this purpose.
—In practice, only a few employees work with the log data, but the system is necessary to guarantee trouble-fee working days for everyone and a quick recovery from error situations, Heinonen points out.
Lessons learnt
The needs and challenges described above cannot be solved just by purchasing software. In addition, the operational processes supporting the selected software must be assessed and implemented. Training and various upskilling activities and peer support are important. New operational methods in software development and IT maintenance also require a new operational culture and additional hands for development and implementation work. The project funding has made it possible to grow enough expertise so that the real benefit of the new, more modern environment can be brought out both in IT and in FSD’s main functions: ingest, curation and reuse of research data.
Text: Tuomas J. Alaterä based on Matti Heinonen’s project presentation. Image: Matti Heinonen (Logos used in accordance with branding guidelines, Jenkins logo under the CC BY-SA 3.0 (Opens in a new tab) License)
