RSEConUK 2019 has ended
The Fourth Conference of Research Software Engineering was held at the University of Birmingham.

Content from all sessions is licensed under a Creative Commons Attribution 2.0 UK: England & Wales License.
Back To Schedule
Wednesday, September 18 • 13:30 - 13:55
#4D1 - Cloud Technologies and Case Studies - Case study of porting a pipeline to EMBL-EBI Cloud Portal

Log in to save this to your schedule, view media, leave feedback and see who's attending!

Case Study of Porting a Bioinformatics Pipeline into Clouds

Cloud providers have different UIs, architectures and APIs. For the research community, it is extremely important to be cloud-agnostic while enjoying advantages of different clouds. It is also extremely important to make the cloud technologies easily accessible for lab scientists with little to no training in clouds. Kubernetes and Docker as a de facto standard is making such goals closer to reality.

We ported a legacy pipeline from IBM Load Sharing Facility, a bare-metal HPC stack to Kubernetes on OpenStack. With minimal changes to the pipeline itself, but by using cloud features intelligently we have made major improvements, changing the pipeline from a single-user local application to a shared multi-user application accessible over the Internet. To investigate and to confirm the cloud-agnostic nature of our solution, we created a CI/CD toolchain to deploy the pipeline onto Kubernetes clusters created on all four major clouds: Google, Amazon, Microsoft and OpenStack. The pipeline can run consistently on GKE, EKS, AKS and EHK, where EHK is a Kubernetes service at European Bioinformatics Institute for research teams to request clusters from.

The general solution that we have developed provides a common set of programming interfaces to support major cloud providers in a consistent, agnostic manner. This reduces the learning curve and skill requirements to port and deploy pipelines in the clouds. This talk presents the methods and the lessons learned during the exercise. It demonstrates the feasibility to rejuvenate legacy pipelines in the clouds with minimum effort.

avatar for David Yuan

David Yuan

Cloud Bioinformatics Application Architect, European Bioinformatics Institute
David Yuan is a Cloud Bioinformatics Application Architect working at European Bioinformatics Institute (EBI), European Molecular Biology Laboratory (EMBL). He is driving cloud-adoption onto both private cloud (OpenStack) and public clouds (Google Cloud Platform, Amazon Web Services... Read More →

Wednesday September 18, 2019 13:30 - 13:55 BST
4. Aston Webb, Room WG12 Aston Webb Building