[PyWPS-dev] A question about PyWPS and celery demo

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[PyWPS-dev] A question about PyWPS and celery demo

Suzana Barreto
Hi Everyone,

Seasons greatings ;-). I am currently trying to find out if it is possible to use python Celery with PyWPS.  I have had a look at the following docker demo project https://github.com/fderue/pywps_celery_demo.  This works but is unfortunately running under python 2.7.  My PyWPS server instance is running under 3.6, having run the celery example under 3.6, the server fails because it cannot serialize the job.  The errors I get are "TypeError: write() argument must be str, not bytes" from the job.dump function, and if I change the function so that it opens the file in binary mode, then I get "io.UnsupportedOperation: underlying stream is not seekable" . 
Does anyone know why this code is not 3.6 compatible?  Does anyone know how I might be able to fix this?

I have also tried to serialise the wps_response and wps_request objects separately but these are not serializable although the Process is.  Any help would be greatly appreciated.

Thanks in advance,
Suzana
--

Dr Suzana Barreto
Software Analyst
Environment Systems

Tel: <a style="color:rgb(17,85,204)" href="tel:%2B44%20%280%29%201970%20626688" value="+441970626688" target="_blank">+44 (0) 1970 626688
www.envsys.co.uk

The information contained in this e-mail is intended only for the use of the person(s) to whom it is addressed and may contain confidential or privileged information. If you have received this e-mail in error please contact the sender and delete the material without copying, distributing or disclosing the content. All reasonable precautions have been taken to ensure that this e-mail is free from any virus, however, the recipient should verify this to be the case. Please 'think before you print'


_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: A question about PyWPS and celery demo

Carsten Ehbrecht-3
Hi Suzana,

unfortunately the developer of this Celery demo has left the project. I haven’t checked the demo/code … I just have seen it is using a PyWPS fork:
https://github.com/fderue/pywps.git

I know that this Celery demo is based on the scheduler extension which is part of pywps:
https://pywps.readthedocs.io/en/master/extensions.html#job-scheduler-extension

This extension is by myself … and I don’t promise that it still works with python 3.x. Our test-suite does not cover enough and I will need to check this by myself again.

Cheers,
Carsten

> On 12 Dec 2018, at 12:36, Suzana Barreto <[hidden email]> wrote:
>
> Hi Everyone,
>
> Seasons greatings ;-). I am currently trying to find out if it is possible to use python Celery with PyWPS.  I have had a look at the following docker demo project https://github.com/fderue/pywps_celery_demo.  This works but is unfortunately running under python 2.7.  My PyWPS server instance is running under 3.6, having run the celery example under 3.6, the server fails because it cannot serialize the job.  The errors I get are "TypeError: write() argument must be str, not bytes" from the job.dump function, and if I change the function so that it opens the file in binary mode, then I get "io.UnsupportedOperation: underlying stream is not seekable" .
> Does anyone know why this code is not 3.6 compatible?  Does anyone know how I might be able to fix this?
>
> I have also tried to serialise the wps_response and wps_request objects separately but these are not serializable although the Process is.  Any help would be greatly appreciated.
>
> Thanks in advance,
> Suzana
> --
> Dr Suzana Barreto
> Software Analyst
> Environment Systems
> Tel: +44 (0) 1970 626688
> www.envsys.co.uk
>
> The information contained in this e-mail is intended only for the use of the person(s) to whom it is addressed and may contain confidential or privileged information. If you have received this e-mail in error please contact the sender and delete the material without copying, distributing or disclosing the content. All reasonable precautions have been taken to ensure that this e-mail is free from any virus, however, the recipient should verify this to be the case. Please 'think before you print'
>
> _______________________________________________
> pywps-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/pywps-dev

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev
Reply | Threaded
Open this post in threaded view
|

Re: A question about PyWPS and celery demo

Carsten Ehbrecht-3
@Suzana
There is a ticket on PyWPS concerning celery:


On 14 Dec 2018, at 8:37 pm, Carsten Ehbrecht <[hidden email]> wrote:

Hi Suzana,

unfortunately the developer of this Celery demo has left the project. I haven’t checked the demo/code … I just have seen it is using a PyWPS fork:
https://github.com/fderue/pywps.git

I know that this Celery demo is based on the scheduler extension which is part of pywps:
https://pywps.readthedocs.io/en/master/extensions.html#job-scheduler-extension

This extension is by myself … and I don’t promise that it still works with python 3.x. Our test-suite does not cover enough and I will need to check this by myself again.

Cheers,
Carsten

On 12 Dec 2018, at 12:36, Suzana Barreto <[hidden email]> wrote:

Hi Everyone,

Seasons greatings ;-). I am currently trying to find out if it is possible to use python Celery with PyWPS.  I have had a look at the following docker demo project https://github.com/fderue/pywps_celery_demo.  This works but is unfortunately running under python 2.7.  My PyWPS server instance is running under 3.6, having run the celery example under 3.6, the server fails because it cannot serialize the job.  The errors I get are "TypeError: write() argument must be str, not bytes" from the job.dump function, and if I change the function so that it opens the file in binary mode, then I get "io.UnsupportedOperation: underlying stream is not seekable" .
Does anyone know why this code is not 3.6 compatible?  Does anyone know how I might be able to fix this?

I have also tried to serialise the wps_response and wps_request objects separately but these are not serializable although the Process is.  Any help would be greatly appreciated.

Thanks in advance,
Suzana
--
Dr Suzana Barreto
Software Analyst
Environment Systems
Tel: +44 (0) 1970 626688
www.envsys.co.uk

The information contained in this e-mail is intended only for the use of the person(s) to whom it is addressed and may contain confidential or privileged information. If you have received this e-mail in error please contact the sender and delete the material without copying, distributing or disclosing the content. All reasonable precautions have been taken to ensure that this e-mail is free from any virus, however, the recipient should verify this to be the case. Please 'think before you print'

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev


_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: A question about PyWPS and celery demo

Suzana Barreto
Hi Carsten,

Thanks for getting back to me regarding this, you are correct that the celery scheduler is based on your existing scheduler, it makes use of the Job and JobLauncher classes to launch the serialized job.  Unfortunately it is a pickling / dill issue, so I suppose it puts the problem out of the PyWPS scope.  I will keep investigating and keep you posted.

Thanks again,
Suzana

On Mon, 17 Dec 2018 at 12:43, Carsten Ehbrecht <[hidden email]> wrote:
@Suzana
There is a ticket on PyWPS concerning celery:


On 14 Dec 2018, at 8:37 pm, Carsten Ehbrecht <[hidden email]> wrote:

Hi Suzana,

unfortunately the developer of this Celery demo has left the project. I haven’t checked the demo/code … I just have seen it is using a PyWPS fork:
https://github.com/fderue/pywps.git

I know that this Celery demo is based on the scheduler extension which is part of pywps:
https://pywps.readthedocs.io/en/master/extensions.html#job-scheduler-extension

This extension is by myself … and I don’t promise that it still works with python 3.x. Our test-suite does not cover enough and I will need to check this by myself again.

Cheers,
Carsten

On 12 Dec 2018, at 12:36, Suzana Barreto <[hidden email]> wrote:

Hi Everyone,

Seasons greatings ;-). I am currently trying to find out if it is possible to use python Celery with PyWPS.  I have had a look at the following docker demo project https://github.com/fderue/pywps_celery_demo.  This works but is unfortunately running under python 2.7.  My PyWPS server instance is running under 3.6, having run the celery example under 3.6, the server fails because it cannot serialize the job.  The errors I get are "TypeError: write() argument must be str, not bytes" from the job.dump function, and if I change the function so that it opens the file in binary mode, then I get "io.UnsupportedOperation: underlying stream is not seekable" .
Does anyone know why this code is not 3.6 compatible?  Does anyone know how I might be able to fix this?

I have also tried to serialise the wps_response and wps_request objects separately but these are not serializable although the Process is.  Any help would be greatly appreciated.

Thanks in advance,
Suzana
--
Dr Suzana Barreto
Software Analyst
Environment Systems
Tel: +44 (0) 1970 626688
www.envsys.co.uk

The information contained in this e-mail is intended only for the use of the person(s) to whom it is addressed and may contain confidential or privileged information. If you have received this e-mail in error please contact the sender and delete the material without copying, distributing or disclosing the content. All reasonable precautions have been taken to ensure that this e-mail is free from any virus, however, the recipient should verify this to be the case. Please 'think before you print'

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev

_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev



--

Dr Suzana Barreto
Software Analyst
Environment Systems

Tel: <a style="color:rgb(17,85,204)" href="tel:%2B44%20%280%29%201970%20626688" value="+441970626688" target="_blank">+44 (0) 1970 626688
www.envsys.co.uk

The information contained in this e-mail is intended only for the use of the person(s) to whom it is addressed and may contain confidential or privileged information. If you have received this e-mail in error please contact the sender and delete the material without copying, distributing or disclosing the content. All reasonable precautions have been taken to ensure that this e-mail is free from any virus, however, the recipient should verify this to be the case. Please 'think before you print'


_______________________________________________
pywps-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pywps-dev