Docker extension

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Docker extension

Adam Laža
Hi devs,

during work on my thesis[1] I've added Docker extension to pywps. At my git there is a demo server (pywps-demo with extended config file) [2]. To run it with Docker extension it's necessary to use pywps[3] and OWSLib[4] from my git. Also you need Docker engine to build image and create container (the extension was tested only on Linux).

I will be glad for any comment. Recently, I faced a problem with zombie processes when using multiprocessing python package for deamon that manages created docker containers. Eventually, I used threading[5] instead but not really sure if it's the right solution. So any opinion will be appreciated.

Cheers,
Adam

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: Docker extension

Jorge Samuel Mendes de Jesus
Hi Adam
Stilll didnt had the problem  to look what you did on on your fork,

We added your email to the issue concerning containers, so that we keep track

If the problems with threads are solved when you can request a PR and we will merge it

Cheers
Jorge


On Mon, Mar 12, 2018 at 8:58 PM, Adam Laža <[hidden email]> wrote:
Hi devs,

during work on my thesis[1] I've added Docker extension to pywps. At my git there is a demo server (pywps-demo with extended config file) [2]. To run it with Docker extension it's necessary to use pywps[3] and OWSLib[4] from my git. Also you need Docker engine to build image and create container (the extension was tested only on Linux).

I will be glad for any comment. Recently, I faced a problem with zombie processes when using multiprocessing python package for deamon that manages created docker containers. Eventually, I used threading[5] instead but not really sure if it's the right solution. So any opinion will be appreciated.

Cheers,
Adam


_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: Docker extension

Adam Laža
Hi Jorge,

original idea was to use multiprocessing package for the deamon that manages containers removal. However zombie processes accumulated on the server when using multiprocessing. Using threads solved the problem - no zombie processes remain on the server. Nevertheless I wanted to point on it to know if it's acceptable solution for you.

Also before the PR, I would appreciate to get rid of dependency on my fork of OWSLib where I use async/sync switch. We already discussed it at ML[1] with Carsten. Can you guys now when you're all together merge Carsten's solution into master of OWSLib?

Cheers,
A.

[1] https://lists.osgeo.org/pipermail/owslib-devel/2018-January/000304.html

2018-03-22 16:22 GMT+01:00 Jorge Mendes de Jesus <[hidden email]>:
Hi Adam
Stilll didnt had the problem  to look what you did on on your fork,

We added your email to the issue concerning containers, so that we keep track

If the problems with threads are solved when you can request a PR and we will merge it

Cheers
Jorge


On Mon, Mar 12, 2018 at 8:58 PM, Adam Laža <[hidden email]> wrote:
Hi devs,

during work on my thesis[1] I've added Docker extension to pywps. At my git there is a demo server (pywps-demo with extended config file) [2]. To run it with Docker extension it's necessary to use pywps[3] and OWSLib[4] from my git. Also you need Docker engine to build image and create container (the extension was tested only on Linux).

I will be glad for any comment. Recently, I faced a problem with zombie processes when using multiprocessing python package for deamon that manages created docker containers. Eventually, I used threading[5] instead but not really sure if it's the right solution. So any opinion will be appreciated.

Cheers,
Adam



_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: Docker extension

Jorge Samuel Mendes de Jesus
Hi Adam

The idea of multiprocessing is to we run things in multiple cores out of the GIL, if your process is running in a docker container it should be run in what ever core it is available. Therefore having a Python GIL thread using checking processes that run in different systems make sense, since you experience problems (IMHO).

For the record, in another project I had bad experience using multiprocess on python3.5 on ubuntu server, then I put all the mutiprocessing code in a docker container running alpine in  docker-machine running KVM and everyything worked 100% . It maybe that multiprocess lib has some problems

Jorge

On Thu, Mar 22, 2018 at 4:51 PM, Adam Laža <[hidden email]> wrote:
Hi Jorge,

original idea was to use multiprocessing package for the deamon that manages containers removal. However zombie processes accumulated on the server when using multiprocessing. Using threads solved the problem - no zombie processes remain on the server. Nevertheless I wanted to point on it to know if it's acceptable solution for you.

Also before the PR, I would appreciate to get rid of dependency on my fork of OWSLib where I use async/sync switch. We already discussed it at ML[1] with Carsten. Can you guys now when you're all together merge Carsten's solution into master of OWSLib?

Cheers,
A.

[1] https://lists.osgeo.org/pipermail/owslib-devel/2018-January/000304.html

2018-03-22 16:22 GMT+01:00 Jorge Mendes de Jesus <[hidden email]>:
Hi Adam
Stilll didnt had the problem  to look what you did on on your fork,

We added your email to the issue concerning containers, so that we keep track

If the problems with threads are solved when you can request a PR and we will merge it

Cheers
Jorge


On Mon, Mar 12, 2018 at 8:58 PM, Adam Laža <[hidden email]> wrote:
Hi devs,

during work on my thesis[1] I've added Docker extension to pywps. At my git there is a demo server (pywps-demo with extended config file) [2]. To run it with Docker extension it's necessary to use pywps[3] and OWSLib[4] from my git. Also you need Docker engine to build image and create container (the extension was tested only on Linux).

I will be glad for any comment. Recently, I faced a problem with zombie processes when using multiprocessing python package for deamon that manages created docker containers. Eventually, I used threading[5] instead but not really sure if it's the right solution. So any opinion will be appreciated.

Cheers,
Adam




_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: Docker extension

Carsten Ehbrecht-3
In reply to this post by Adam Laža
Hi Adam,

I will merge my owslib.wps changes (including async/sync switch) in the next days into master.

> On 22. Mar 2018, at 16:51, Adam Laža <[hidden email]> wrote:
>
> Also before the PR, I would appreciate to get rid of dependency on my fork of OWSLib where I use async/sync switch. We already discussed it at ML[1] with Carsten. Can you guys now when you're all together merge Carsten's solution into master of OWSLib?

Cheers,
Carsten
--
Carsten Ehbrecht
Abteilung Datenmanagement

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a • D-20146 Hamburg • Germany

Phone: +49 40 460094-148
FAX:   +49 40 460094-270
Email: [hidden email]
URL:   www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784



_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev

smime.p7s (6K) Download Attachment