(group picture from 5th EasyBuild User Meeting in Barcelona (Jan'20)

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

The EasyBuild User Meeting is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild enthusiasts, discuss related topics and learn about new aspects of the tool. It is intended for people that are already familiar with EasyBuild, ranging from occasional users to EasyBuild core developers and experts, and topics will be less introductory in nature than during EasyBuild hackathons/workshops that have been organized in the past. The program includes presentations by both EasyBuild users and developers, as well as talks about open source projects relevant to the EasyBuild community.

Practical info - Registration - Organisation - Agenda - CernVM-FS tutorial - ReFrame tutorial - Contact

Practical information

The 6th EasyBuild User Meeting will be held online, during the week of Mon-Fri Jan 25-29 2021.

Attendance is free of cost. This is an open meeting, anybody interested is welcome to join.

Zoom & YouTube

All presentations will be given live at the time listed in the agenda via Zoom sessions, and live streaming will be available via the EasyBuild YouTube channel. We intend to record all sessions, and will make the recordings available shortly after the live presentations.

A YouTube playlist collecting all live streams for EUM'21 is available here.

Attendees will be able to join Zoom sessions for interactive discussions with the speakers.
Note that only registered attendees will have access to the Zoom sessions!

Q&A via #eum channel in EasyBuild Slack

Next to raising questions or comments in the Zoom session, you can also submit questions via the #eum channel in the EasyBuild Slack. Comments in YouTube have been disabled for the live streaming events.

If you are not logged in to the EasyBuild Slack yet, you can create an account via https://easybuild.io/join-slack.

Registration

Registration for EUM'21, and the CernVM-FS and ReFrame tutorials, is now closed!

Only registered attendees are able to join the Zoom sessions and engage with the speakers and other attendees during live Q&A and discussions, and get access to the cloud resources provided for the CernVM-FS and ReFrame tutorials.

If you would still like to join one of the Zoom sessions, please contact eum@lists.ugent.be.

Organisation

• Simon Branford (University of Birgminham, UK)
• Kenneth Hoste (HPC-UGent, Belgium)
• Alan O'Cais (Jülich Supercomputing Centre, Germany)
• Åke Sandgren (Umeå University, Sweden)

Agenda

The 6th EasyBuild User Meeting consists of several sessions spread over the week of Jan 25-29 2021. Please note that all times are in Universal Coordinated Time (UTC)!

We intentionally left ample time in between talks to allow for Q&A, interactive discussions, switching between speakers and breaks.

Overview

• Mon Jan 25th 2021
• [15:45-15:55 UTC] Welcome to EUM'21
• [16:00-16:30 UTC] Writing powerful HPC regression tests with ReFrame
• [17:00-17:30 UTC] CernVM-FS: Overview and Roadmap
• [18:00-19:30 UTC] EasyBuild: State of the Union
• Tue Jan 26th 2021
• [09:00-09:45 UTC] CernVM-FS tutorial (part 1/4)
• [10:15-10:45 UTC] Easy-Build Singularity Containers (presentation)
• [11:00-11:30 UTC] Gentoo Prefix: a unified experience on top of another distribution
• [12:00-12:45 UTC] European Environment for Scientific Software Installations (EESSI)
• [13:15-13:45 UTC] ReFrame tutorial (part 1/3)
• Wed Jan 27th 2021
• [09:00-10:00 UTC] CernVM-FS tutorial (part 2/4)
• [14:00-14:15 UTC] EasyBuild site presentation: CSCS
• [14:30-14:45 UTC] EasyBuild site presentation: ITER
• [15:00-15:30 UTC] PRACE Best Practice Guides
• [16:00-16:40 UTC] Lmod and XALT: Updates
• [17:00-17:30 UTC] Magic Castle
• [18:00-18:45 UTC] BLIS: A Framework for Rapidly Instantiating BLAS Functionality
• [19:15-19:45 UTC] EasyBuild site presentation: AstraZeneca
• Thu Jan 28th 2021
• [09:00-09:45 UTC] CernVM-FS tutorial (part 3/4)
• [10:15-10:45 UTC] ReFrame tutorial (part 2/3)
• [11:00-11:15 UTC] LearnHPC
• [11:25-11:40 UTC] EasyBuild site presentation: Jülich Supercomputing Centre
• [12:00-12:45 UTC] Cluster-in-the-Cloud
• [13:15-13:30 UTC] EasyBuild site presentation: LuxProvide
• [13:45-14:00 UTC] EasyBuild site presentation: University of Birmingham
• [14:15-14:30 UTC] EasyBuild site presentation: Compute Canada
• Fri Jan 29th 2021
• [09:00-10:00 UTC] CernVM-FS tutorial (part 4/4)
• [13:00-13:30 UTC] ReFrame tutorial (part 3/3)
• [14:00-14:30 UTC] Easy-Build Singularity Containers (tutorial)
• [15:00-15:30 UTC] buildtest: Testing Framework for HPC systems
• [15:40-15:55 UTC] Automate Module Testing with Lmodule
• [16:15-17:00 UTC] Spack update

Mon Jan 25th 2021

• 
  

  [15:45-15:55 UTC] Welcome to EUM'21
  (recording @ YouTube) - (slides, PDF)

  by Kenneth Hoste (HPC-UGent, Belgium)

  Quick overview of agenda + practical guidelines for attending EUM'21.

  
  
• 
  

  [16:00-16:30 UTC] Writing powerful HPC regression tests with ReFrame
  (YouTube live stream) - (slides, PDF)

  by Vasileios Karakasis (CSCS, Switzerland)

  In this talk series we will present ReFrame, a framework for writing regression tests for HPC systems. The goal of the ReFrame is to abstract away the complexity of the interactions with the system, separating the logic of a regression test from the low-level details, which pertain to the system configuration and setup. This allows users to write easily portable regression tests, focusing only on the functionality. We will start with an introductory talk about the framework and its capabilities, which we will gradually explore in more depth in the subsequent series of tutorial talks.

  
  

• [17:00-17:30 UTC] CernVM-FS: Overview and Roadmap
  (recording @ YouTube) - (slides, PDF)

  by Jakob Blomer (CERN, Switzerland)

  Delivering complex software across a worldwide distributed infrastructure of data centers and supercomputers is a major challenge in scientific computing. Copying the entire software stack everywhere it’s needed is often impractical and inefficient -- software stacks can grow very large, new versions are produced frequently, while any given job only needs a small fraction of the total software. To address application delivery at scale, CernVM-FS distributes software to hundreds of thousands of machines around the world in the form of a global, purpose-built shared software area. This presentation gives a technology overview and discusses use cases, experience from operations, and future plans.

  
  

• [18:00-19:30 UTC] EasyBuild: State of the Union
  (recording @ YouTube) - (slides, PDF)

  by Kenneth Hoste (HPC-UGent, Belgium)

  Let's look back at what was changed in EasyBuild in the last year, how we are doing right now, what we are currently working on, which challenges are ahead, and the enhancements and changes in EasyBuild we envision for the future. In addition, the highlights of the last EasyBuild User Survey will be covered in this talk.

  
  

Tue Jan 26th 2021

• 
  

  [09:00-09:45 UTC] CernVM-FS tutorial (part 1/4)
  (recording @ YouTube)

  by Kenneth Hoste (HPC-UGent) and Bob Dröge (Univ. of Groningen, The Netherlands)

  (see below for more information)

  
  

• [10:15-10:45 UTC] Easy-Build Singularity Containers (presentation)
  (recording @ YouTube) - (slides, PDF)

  by Jörg Saßmannshausen (Guy's and St Thomas' NHS Foundation Trust and King's College London, UK)

  Reproducibility in research and science is an important aspect. Without this, any result obtained is more or less useless. Whereas in the real world we are conducting an experiment a few times, like 3 analysis of a compound, large field studies for vaccinations etc, in the virtual world we are not always able to afford this. This is particularly true for larger HPC calculations where the resources are simply often not there. Thus, we need to make sure that we are not generating random numbers, but we are generating meaningful results which are relevant to the real world.

  For this reason we need to benchmark our computational studies and we need to be able to reproduce the installation. One way forward here is of course EasyBuild, which is installing the software in a reproducible way.

  However, not all HPC facilities have such a sotware management tool installed. On top of that, in particular in the Bioinformatic Community, users are often running long pipelines or have very special requirements on the way the software is installed. This can make it a bit more difficult to reproduce results between different HPC facilities. One way forward here is for example a container solution like Singularity.

  In this presentation we are showing a way how to combine the best of the two worlds: using EasyBuild for the software installation and using Singularity for the platform-independant way to run the calculations. We show how a set of simple bash-scripts enables users like for example clinitians, who are not that Linux savvy, to create such containers easily.

  
  

• [11:00-11:30 UTC] Gentoo Prefix: a unified experience on top of another distribution
  (recording @ YouTube) - (slides, PDF)

  by Fabian Groffen (Gentoo)

  People who frequently use different systems know: each and every vendor or distribution typically maintains it own setup of software packages. While tools like sed and awk may give small annoyances on which flags to use based on whether the tools are GNU or BSD-based, something like a compiler may be a completely different version, causing considerable problems for development purposes.

  Gentoo Prefix aims to build the exact same software, with the exact same patches and configuration on a multitude of distributions. The major targets are macOS, Linux, and Solaris-based, aiming at ppc, x86 and sparc based architectures.

  In this talk we will walk through a bit of the history of the project, how it works, using Gentoo tooling, and its current state.

  speaker bio:
  Fabian Groffen is a Gentoo Developer since 2006, working mainly on the Prefix project. At that time, Fabian worked on his PhD which he received from the University of Amsterdam, The Netherlands in 2009. He keeps a bunch of (old) machines in his basement, which allows him to test Gentoo Prefix on different architectures. Next to his voluntary work for Gentoo, Fabian works as software engineer for a large multinational, specialising on applications written in the C language.

• [12:00-12:45 UTC] European Environment for Scientific Software Installations (EESSI)
  (recording @ YouTube) - (slides, PDF)

  by Bob Dröge (Univ. of Groningen, The Netherlands)

  The European Environment for Scientific Software Installations (EESSI, pronounced as "easy") is a collaboration between different HPC sites and industry partners, with the common goal to set up a shared repository of scientific software installations that can be used on a variety of systems, regardless of which flavor/version of Linux distribution or processor architecture is used, or whether it is a full-size HPC cluster, a cloud environment or a personal workstation.

  The concept of the EESSI project was inspired by the Compute Canada software stack,
  and consists of three main layers:

  • a filesystem layer leveraging the established CernVM-FS technology, to globally distribute the EESSI software stack;
  • a compatibility layer using Gentoo Prefix, to ensure compatibility with different client operating systems (different Linux distributions, macOS, Windows Subsystem for Linux);
  • a software layer, hosting optimized installations of scientific software along with required dependencies, which were built for different processor architectures, and where archspec, EasyBuild and Lmod are leveraged;

  In this talk, we will present how the EESSI project grew out of a need for more collaboration to tackle the challenges in the changing landscape of scientific software and HPC system architectures. The project structure will be explained in more detail, covering the motivation for the layered approach and the choice of tools, as well as the lessons learned from the work done by Compute Canada. The goals we have in mind and how we plan to achieve them going forward will be outlined.

  Finally, we will demonstrate the current pilot version of the project, and give you a feeling of the potential impact.

  
  

• [13:15-13:45 UTC] ReFrame tutorial (part 1/3)
  (recording @ YouTube)

  by Vasileios Karakasis & Victor Holanda Rusu (CSCS, Switzerland)

  (see below for more information)

  
  

Wed Jan 27th 2021

• 
  

  [09:00-10:00 UTC] CernVM-FS tutorial (part 2/4)
  (YouTube recording)

  by Kenneth Hoste (HPC-UGent) and Bob Dröge (Univ. of Groningen, The Netherlands)

  (see below for more information)

  
  

• [14:00-14:15 UTC] EasyBuild site presentation: CSCS
  (YouTube recording) - (slides, PDF)

  by Luca Marsella (CSCS)

  
  

• [14:30-14:45 UTC] EasyBuild site presentation: ITER
  (YouTube recording) - (slides, PDF)

  by Simon Pinches (ITER)

  
  

• [15:00-15:30 UTC] PRACE Best Practice Guides
  (YouTube recording) - (slides, PDF)

  by Ole Saastad (Univ. of Oslo, Norway)

  
  

• [16:00-16:40 UTC] Lmod and XALT: Updates
  (YouTube recording) - (slides Lmod, PDF) - (slides XALT, PDF)

  by Robert McLay (TACC, US)

  Lmod is the modern Environment Module Tool. Sysadmins define packages and let users choice which package and which version of that package. This part of the talk will include a brief overview of Lmod. This will be followed up with the new features such as extensions for connecting modules to packages. (e.g. the python module and packages such as numpy and scipy). This is also include a discussion of the changes to Lmod 8+.

  XALT is a tool to take the census of what programs and libraries are run on your cluster. This talk will briefly cover what XALT does and how it does it. It will include some "war" stories of XALT usage. Finally I'll cover XALT move to version 3 namely tracking usage in containers and package tracking from R, Python and MATLAB.

  
  

• [17:00-17:30 UTC] Magic Castle
  (YouTube recording) - (slides, PDF)

  by Félix-Antoine Fortin (Université Laval, Compute Canada)

  
  

• [18:00-18:45 UTC] BLIS: A Framework for Rapidly Instantiating BLAS Functionality
  (YouTube recording) - (slides, PDF)

  by Field G. Van Zee (University of Texas at Austin, US)

  BLIS is a portable software framework for instantiating high-performance BLAS-like dense linear algebra libraries. The framework was designed to isolate essential kernels of computation that, when optimized, immediately enable optimized implementations of most of its commonly used and computationally-intensive operations. In this talk, I'll take the audience on wide-ranging tour of BLIS, including its history, its design considerations, the APIs it exports, its implementations and how they promote productivity and maintainability, performance on modern hardware, its community organization, its impact on the HPC ecosystem, and our plans for the future.

  
  

• [19:15-19:45 UTC] EasyBuild site presentation: AstraZeneca
  (YouTube recording) - (slides, PDF)

  by Jon Jackson and Ben Carr (AstraZeneca)

  
  

Thu Jan 28th 2021

• 
  

  [09:00-09:45 UTC] CernVM-FS tutorial (part 3/4)
  (YouTube live stream)

  by Kenneth Hoste (HPC-UGent) and Bob Dröge (Univ. of Groningen, The Netherlands)

  (see below for more information)

  
  

• [10:15-10:45 UTC] ReFrame tutorial (part 2/3)
  (YouTube live stream)

  by Vasileios Karakasis & Victor Holanda Rusu (CSCS, Switzerland)

  (see below for more information)

  
  

• [11:00-11:15 UTC] LearnHPC
  (YouTube recording) - (slides, PDF)

  by Alan O'Cais (JSC, Germany)

  In a newly successful PRACE-ICEI proposal, E-CAM, FocusCoE, HPC Carpentry and EESSI join forces to bring HPC resources to the classroom in a simple, secure and scalable way. Our plan is to reproduce the model developed by the Canadian open-source software project Magic Castle. The proposed solution creates virtual HPC infrastructure(s) in a public cloud, in this case on the Fenix Research Infrastructure, and generates temporary event-specific HPC clusters for training purposes, including a complete scientific software stack. The scientific software stack is fully optimised for the available hardware and will be provided by the European Environment for Scientific Software Installations (EESSI).

  
  

• [11:25-11:40 UTC] EasyBuild site presentation: Jülich Supercomputing Centre
  (YouTube recording) - (slides, PDF)

  by Anke Kreuzer (JSC, Germany)

  
  

• [12:00-12:45 UTC] Cluster-in-the-Cloud
  (YouTube recording) - (slides, PDF)

  by Matt Williams (Univ. of Bristol, UK)

  [Cluster in the Cloud](https://cluster-in-the-cloud.readthedocs.io) is a tool to easily create scaleable and heterogeneous compute clusters on public cloud resources. This talk will introduce the tool, explain its history and detail its strengths and weaknesses. Time and resource-availability permitting, there will be a live demonstration.

  
  

• [13:15-13:30 UTC] EasyBuild site presentation: LuxProvide
  (YouTube recording) - (slides, PDF)

  by Valentin Plugaru and Robert Mijakovic (LuxProvide, Luxembourg)

  
  

• [13:45-14:00 UTC] EasyBuild site presentation: University of Birmingham
  (YouTube recording) - (slides, PDF)

  by Simon Branford (Univ. of Birmingham, UK)

  
  

• [14:15-14:30 UTC] EasyBuild site presentation: Compute Canada
  (YouTube recording) - (slides, PDF)

  by Bart Oldeman (Compute Canada, Calcul Québec, McGill University)

  
  

Fri Jan 29th 2021

• 
  

  [09:00-10:00 UTC] CernVM-FS tutorial (part 4/4)
  (YouTube live stream)

  by Kenneth Hoste (HPC-UGent) and Bob Dröge (Univ. of Groningen, The Netherlands)

  (see below for more information)

  
  

• [13:00-13:30 UTC] ReFrame tutorial (part 3/3)
  (YouTube live stream)

  by Vasileios Karakasis & Victor Holanda Rusu (CSCS, Switzerland)

  (see below for more information)

  
  

• [14:00-14:30 UTC] Easy-Build Singularity Containers: tutorial
  (YouTube live stream) - (tutorial, PDF)

  by Jörg Saßmannshausen (Guy's and St Thomas' NHS Foundation Trust and King's College London, UK)

  In this hands on tutorial we will build 2 containers: a simple one to give users an understanding of how the set of scripts are working and how we can build for example Debian or CentOS containers without much knowledge of the background mechanisms. In the second, more advanced part, we show how to open up the containers and give users are chance to install their bespoken pipelines and use the power of the containers to run that on virtually any suitable platform.

  
  

• [15:00-15:30 UTC] buildtest: Testing Framework for HPC systems
  (YouTube live stream) - (slides, PDF)

  by Shahzeb Siddiqui (NERSC, US)

  Buildtest is an HPC testing framework to aid HPC facilities to perform acceptance testing for their system. HPC systems are growing in complexity, with a tightly coupled software and system stack that requires a degree of automation and continuous testing. In the past decade, two build frameworks (Spack, EasyBuild) have emerged and widely used in HPC community for automating build & installation process for scientific software. On the contrary, testing frameworks for HPC systems are limited to a few handful (ReFrame, Pavilion2, buildtest) that are in active development. In buildtest, users will write test recipes in YAML called buildspecs that is the primary interface for writing tests. buildtest utilizes a JSON Schema for validating buildspecs. In this talk we will present an overview of buildtest and how one can write buildspecs. Furthermore, we will discuss Cori Testsuite in buildtest with several real examples on testing various components for Cori system at NERSC. Finally, we will present an overview of the E4S Test Suite, which is an ECP project to help test the Spack E4S software stack, and how buildtest integrates with E4S Testsuite.

  
  

• [15:40-15:55 UTC] Automate Module Testing with Lmodule
  (YouTube live stream) - (slides, PDF)

  by Shahzeb Siddiqui (NERSC, US)

  HPC software stacks have become increasingly complex and significantly grown in size with some sites having up to 5,000+ modulefiles. It's imperative that all modules are tested because it is the primary interface for users to interact with the software stack. LModule is a Python API for Lmod, with a focus on automating module testing. Lmodule is able to query a software stack via the 'spider' Lmod tool, and individually test each module using "module load". The API consist of three Python classes: Module, Spider, and ModuleLoadTest. Lmodule works with Environment Modules. In this talk we will present an overview of the Lmodule API at Cori (environment-modules) and OLCF (Lmod), to demonstrate how we can automate module testing of software stacks.

  
  

• [16:15-17:00 UTC] Spack update
  (YouTube live stream) - (slides, PDF)

  by Todd Gamblin (LLNL, US)

  
  

CernVM-FS tutorial

https://cernvm.cern.ch/fs/

Tutorial materials are available at https://cvmfs-contrib.github.io/cvmfs-tutorial-2021.

Practical info

• The tutorial is split in 5 short sessions: one introductory presentation + 4 tutorial sessions.


• Hands-on exercises will be provided after the first 3 tutorial sessions. We expect that tutorial attendees will spend about 1 hour (per session) to work through these exercises.


• Support will be provided through a dedicated Slack channel througout the day.


• Dedicated cloud resources in Microsoft Azure (sponsored by Microsoft) will be available (only) to registered tutorial attendees for working on the hands-on exercises.
  

Sessions

• [Mon Jan 25 2021, 17:00-17:30 UTC] Introductory presentation (by Jakob Blomer, CERN)


• [Tue Jan 26 2021, 09:00-10:00 UTC] CernVM-FS tutorial (part 1/4)
• Practical information
• Introduction to CernVM-FS concepts and terminology
• Getting started: Stratum-0 and client access
• hands-on exercises


• [Wed Jan 27 2021, 09:00-10:00 UTC] CernVM-FS tutorial (part 2/4)
• Stratum-1 + proxies
• hands-on exercises


• [Thu Jan 28 2021, 09:00-10:00 UTC] CernVM-FS tutorial (part 3/4)
• Publishing
• hands-on exercises


• [Fri Jan 29 2021, 09:00-10:00 UTC] CernVM-FS tutorial (part 4/4)
• Advanced topics

Organisation

• Kenneth Hoste (HPC-UGent)
• Bob Dröge (Univ. of Groningen)
• Jakob Blomer (CERN)

ReFrame tutorial

https://reframe-hpc.readthedocs.io

Tutorial materials are available at
https://reframe-hpc.readthedocs.io/en/stable/tutorials.html.

Sessions

• [Mon Jan 25 2021, 16:00-16:30 UTC] Introductory presentation


• [Tue Jan 26 2021, 13:15-13:45 UTC] ReFrame tutorial (part 1/3)


• [Thu Jan 28 2021, 10:15-10:45 UTC] ReFrame tutorial (part 2/3)


• [Fri Jan 29 2021, 13:00-13:30 UTC] ReFrame tutorial (part 3/3)

Organisation

• Vasileios Karakasis (CSCS)
• Victor Holanda Rusu (CSCS)

Contact

In case of questions, please contact eum@lists.ugent.be.