RSEConUK 2019

Case Study of Porting a Bioinformatics Pipeline into Clouds

Cloud providers have different UIs, architectures and APIs. For the research community, it is extremely important to be cloud-agnostic while enjoying advantages of different clouds. It is also extremely important to make the cloud technologies easily accessible for lab scientists with little to no training in clouds. Kubernetes and Docker as a de facto standard is making such goals closer to reality.

We ported a legacy pipeline from IBM Load Sharing Facility, a bare-metal HPC stack to Kubernetes on OpenStack. With minimal changes to the pipeline itself, but by using cloud features intelligently we have made major improvements, changing the pipeline from a single-user local application to a shared multi-user application accessible over the Internet. To investigate and to confirm the cloud-agnostic nature of our solution, we created a CI/CD toolchain to deploy the pipeline onto Kubernetes clusters created on all four major clouds: Google, Amazon, Microsoft and OpenStack. The pipeline can run consistently on GKE, EKS, AKS and EHK, where EHK is a Kubernetes service at European Bioinformatics Institute for research teams to request clusters from.

The general solution that we have developed provides a common set of programming interfaces to support major cloud providers in a consistent, agnostic manner. This reduces the learning curve and skill requirements to port and deploy pipelines in the clouds. This talk presents the methods and the lessons learned during the exercise. It demonstrates the feasibility to rejuvenate legacy pipelines in the clouds with minimum effort.