How to Achieve Workflow Interoperability ER-flow & SHIWA project Gabor Terstyanszky, University of Westminster, UK 18 January 2013 ER-flow is supported by the FP7 Capacities Programme under contract No. RI-261585 WF Ecosystem 2 2 What does a WF developer need? Using a portal/desktop to parameterize and run these applications, and to further develop them accessing a large set of various DCIs to make these WF applications run WF App. Repository Portal Access to a large set of ready-to-run scientific WF applications Supercomputer based SGs (DEISA, TeraGrid) Cluster based service grids (SGs) (EGEE, OSG, etc.) Local clusters Clouds Supercomputers E-science infrastructure Desktop grids (DGs) (BOINC, Condor, etc.) Grid systems 3 SHIWA: Enabling Workflow Interoperability • Publish WF applications in the repository to be continued by other appl. developers • Application developers use the portal/ desktop to develop complex applications (executable on various DCIs) for various end-user communities SHIWA App. Repository Application developers SSP Portal Supercomputer based SGs (DEISA, TeraGrid) Cluster based service grids (SGs) (EGEE, OSG, etc.) Local clusters Clouds Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Grid systems 4 Coarse-Grained Interoperability: submitting non-native workflow Workflow Engine A Workflow Engine A Workflow of Workflow Engine B WF submissio client Submission Service Workflow Engine B DCI • non-native workflow: WF - non-native workflows are black boxes which are managed as legacy code applications 5 Coarse-Grained Interoperability: meta-workflow submission Workflows of Workflow Engine A Workflow of Workflow Engine B J1 WF2 WF4 Submission client Submission Service Workflow Engine B Workflow Engine A J3 DCI • native workflows: J1, J3 and WF2 • non-native workflows: WF4 - black boxes which are managed as legacy code applications SSP: CGI Infrastructure DCIs SHIWA Science Gateway SHIWA Portal GEMLCA admin ARC DCI SHIWA Repository WF1 WFn gLite DCI Unicore DCI Globus DCI GEMLCA Repository WS-PGRADE Workflow editor WS-PGRADE Workflow engine SHIWA Science Gateway WF1 WFm WE1 WEp ASKALON WE Galaxy WE GWES WE Kepler WE GEMLCA Service Pegasus WE MOTEUR WE Proxy Server ProActive WE PGRADE WE GEMLCA with GIB SHIWA Proxy Server na5ve WE portal repository WS-‐PGRADE WS-‐PGRADE v3.5 GEMLCA + SHIWA repo submiKer proxy management GEMLCA with GIB SHIWA Proxy Server Resources Triana WE Taverna WE Workflow Engines local resources: invoca5on of locally deployed WEs WE submission to local cluster remote resources: through remotely pre-‐deployed WEs to ARC, gLite, Globus Unicore DCIs 7 SHIWA Repository: Browse View 8 SHIWA Repository: Table View 9 SHIWA Portal: Workflow Editor 10 SHIWA Portal: Configuring Workflow 11 SHIWA Portal: Executing Workflow 12 CGI Developer Scenario: Specifying Workflow Engine step 1 specify WE data SHIWA Science Gateway SHIWA Portal workflow engine developer GEMLCA admin SHIWA Repository WF1 WFn GEMLCA Repository step 2 upload WE binary, dependencies WS-PGRADE Workflow editor WS-PGRADE Workflow engine WF1 WFm WE1 WEp step 3 deploy WE GEMLCA with GIB GEMLCA Service Proxy Server SHIWA Proxy Server 13 CGI Developer Scenario: Specifying Workflows step 1 specify WF data SHIWA Science Gateway SHIWA Repository WF1 WFn SHIWA Portal WF1 WFm WS-PGRADE Workflow editor WE1 WEp workflow developer step 3 deploy WF WS-PGRADE Workflow engine GEMLCA Repository step 2 upload WF GEMLCA with GIB GEMLCA Service Proxy Server SHIWA Proxy Server 14 14 CGI User Scenario: PGRADE as Native WE DCIs ARC DCI SHIWA Science Gateway gLite DCI Globus DCI Unicore DCI SHIWA Repository WF1 WFn step 1 search WF step 3 retrieve WF data ASKALON WE Galaxy WE GWES WE Kepler WE Pegasus WE MOTEUR WE ProActive WE PGRADE WE Triana WE Taverna WE GEMLCA Repository SHIWA Portal WS-PGRADE Workflow editor WF list WF1 WFm WE1 WEp WE + WF WS-PGRADE Workflow engine step 5 retrieve WF Workflow Engines e-scientists GEMLCA with GIB GEMLCA Service step 4 submit WF step 2 edit WF step 6 retrieve proxy step 7 run WF Proxy Server SHIWA Proxy Server 15 ER-flow Project Partners: University of Westminster Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete Centre National de la Recherche Scientifique Stichting European Grid Initiative Academic Medical Center of the University of Amsterdam Technische Universität Dresden Ludwig-Maximilians-Universität München University College London Trinity College Daudublin Istituto Nazionale di Astrofisica UoW MTA-SZTAKI United Kingdom Hungary CNRS EGI.eu AMC France The Netherlands The Netherlands TUD LMU UCL TCD INAF Germany Germany United KIngdom Ireland Italy Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-Physics Computational Chemistry Helio-Physics Life Science INAF LMU + TUD TCD + UCL AMC Duration: September 2012 – August 2014 16 Project Aim and Services Aim: • • To provide a simulation platform for research communities to enable seamless execution of workflows of different workflow systems through workflow interoperability To investigate data interoperability issues in the workflow domain and propose solutions Services: • • • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse-grained workflow interoperability solution To provide GUIs to manage workflows Key actors: • researchers workflow engine developers workflow developers 17 17 User Communities and Simulation Platform Research communities supported by the project • Astro-Physics • Computational Chemistry • HelioPhysics • Life Science Further research communities • at least four further research communities will be supported candidate communities § Hydrometeorology § Seizmology § further communities considered 18
© Copyright 2024