scheduler extension for pywps

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

scheduler extension for pywps

Carsten Ehbrecht-3
Hi *,

in our copernicus project we want to provide processing functionality
using PyWPS next to a climate data store. To handle the workload we need
to attach batch job scheduler systems like slurm
(https://slurm.schedmd.com/), grid-engine and torque. We are also
interested in using docker/container to run WPS processes ... but our
compute-providers are not ready for it yet.

I've prepared a demo to show how such a PyWPS/extension might look like.
When this extension is ready we would like to contribute it to PyWPS.

Would be nice to get some comments from you :)

I've added a "processing" package to PyWPS which by default uses
multiprocessing but can also delegate processing jobs to a scheduler:

https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing

It uses the drmaa python library to talk to a scheduling service:
https://pypi.python.org/pypi/drmaa

You can try the docker demo with slurm:
https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo

Maybe the "processing" interface could also be used as an
extension-point for processing with docker/containers.

Cheers,
Carsten
--
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: [hidden email]
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: scheduler extension for pywps

Jachym Cepicky
Carsten,

this is just cool! Could you add description and some user documentation to docs? Can we put this into `development` branch of PyWPS?

Thanks a lot!

Jachym

po 29. 5. 2017 v 13:46 odesílatel Carsten Ehbrecht <[hidden email]> napsal:
Hi *,

in our copernicus project we want to provide processing functionality
using PyWPS next to a climate data store. To handle the workload we need
to attach batch job scheduler systems like slurm
(https://slurm.schedmd.com/), grid-engine and torque. We are also
interested in using docker/container to run WPS processes ... but our
compute-providers are not ready for it yet.

I've prepared a demo to show how such a PyWPS/extension might look like.
When this extension is ready we would like to contribute it to PyWPS.

Would be nice to get some comments from you :)

I've added a "processing" package to PyWPS which by default uses
multiprocessing but can also delegate processing jobs to a scheduler:

https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing

It uses the drmaa python library to talk to a scheduling service:
https://pypi.python.org/pypi/drmaa

You can try the docker demo with slurm:
https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo

Maybe the "processing" interface could also be used as an
extension-point for processing with docker/containers.

Cheers,
Carsten
--
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: <a href="tel:+49%2040%20460094148" value="+4940460094148" target="_blank">+49 40 460094-148
FAX:   <a href="tel:+49%2040%20460094270" value="+4940460094270" target="_blank">+49 40 460094-270
Email: [hidden email]
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: scheduler extension for pywps

Carsten Ehbrecht-3
Hi Jachym,

i've made a pull request:
https://github.com/geopython/pywps/pull/271

I've added an "extensions" section to the documentation describing this
scheduler extension. A diagram shows the interactions between PyWPS and
the scheduler system.

It should be save to merge ... you need to decide if the
"pywps.processing" interface fits to your demands. You probably need to
adapt and change it when using celery.

Cheers,
Carsten

On 06/04/2017 10:30 PM, Jachym Cepicky wrote:

> Carsten,
>
> this is just cool! Could you add description and some user documentation
> to docs? Can we put this into `development` branch of PyWPS?
>
> Thanks a lot!
>
> Jachym
>
> po 29. 5. 2017 v 13:46 odesílatel Carsten Ehbrecht <[hidden email]
> <mailto:[hidden email]>> napsal:
>
>     Hi *,
>
>     in our copernicus project we want to provide processing functionality
>     using PyWPS next to a climate data store. To handle the workload we need
>     to attach batch job scheduler systems like slurm
>     (https://slurm.schedmd.com/), grid-engine and torque. We are also
>     interested in using docker/container to run WPS processes ... but our
>     compute-providers are not ready for it yet.
>
>     I've prepared a demo to show how such a PyWPS/extension might look like.
>     When this extension is ready we would like to contribute it to PyWPS.
>
>     Would be nice to get some comments from you :)
>
>     I've added a "processing" package to PyWPS which by default uses
>     multiprocessing but can also delegate processing jobs to a scheduler:
>
>     https://github.com/bird-house/pywps/tree/dev-scheduler/pywps/processing
>
>     It uses the drmaa python library to talk to a scheduling service:
>     https://pypi.python.org/pypi/drmaa
>
>     You can try the docker demo with slurm:
>     https://github.com/bird-house/birdhouse-playground/tree/master/docker/pywps-scheduler-demo
>
>     Maybe the "processing" interface could also be used as an
>     extension-point for processing with docker/containers.
>
>     Cheers,
>     Carsten
>     --
>     Carsten Ehbrecht
>     Abteilung Datenmanagement
>
>     Deutsches Klimarechenzentrum GmbH (DKRZ)
>     Bundesstraße 45 a • D-20146 Hamburg • Germany
>
>     Phone: +49 40 460094-148 <tel:+49%2040%20460094148>
>     FAX:   +49 40 460094-270 <tel:+49%2040%20460094270>
>     Email: [hidden email] <mailto:[hidden email]>
>     URL:   www.dkrz.de <http://www.dkrz.de>
>
>     Geschäftsführer: Prof. Dr. Thomas Ludwig
>     Sitz der Gesellschaft: Hamburg
>     Amtsgericht Hamburg HRB 39784
>     _______________________________________________
>     pywps-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.osgeo.org/mailman/listinfo/pywps-dev
>

--
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: [hidden email]
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: scheduler extension for pywps

Carsten Ehbrecht-3
In reply to this post by Carsten Ehbrecht-3
Hi Tom,

Stephan and i would be interested in taken part at the OGC telco ... at
least from time to time ...

Could you send us the connection details?

I won't make it this monday ... but next week.

Cheers,
Carsten

On 05/29/2017 01:53 PM, Tom Landry wrote:
> For your information, every monday 11am (Eastern), we have a conference
> call with OGC, NRCan, ESA and other participants from testbed 13 - EO
> Clouds. Maybe you or Stephan could be observer for the testbed. This
> would make it easier to share material (architecture documents, wiki
> entries, engineering reports, etc.) with you, from CRIM or other.
>
>

--
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: [hidden email]
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784
_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev