Question about *.mapcalc_start in Python scripting library

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about *.mapcalc_start in Python scripting library

Michael Barton
I've been using *.mapcalc_start to do some simple parallel processing in the i.pansharpen module and in r.landscape.evol. My understanding previously has been that *.mapcalc_start returns a Popen object that can be directed to wait to finish (like *.start_command), allowing multiple mapcalc operations to be initiated on multiple cores. 

But when running r.landscape.evol on another computer, it is acting like mapcalc is not waiting to finish in spite of the *.wait() commands. 

The description in the programmer's manual is not clear to me on this (see below). In one place, it says it does not wait to finish and in the example it says it does wait. 

Can anyone clarify this?

Thanks
Michael

==== from programmer's manual ====

script.raster.mapcalc_start(exp, quiet=False, verbose=False, overwrite=False, seed=None, env=None, **kwargs)[source]
Interface to r.mapcalc, doesn’t wait for it to finish, returns Popen object.

>>> output = 'newele'
>>> input = 'elevation'
>>> expr1 = '"%s" = "%s" * 10' % (output, input)
>>> expr2 = '...'   # etc.
>>> # launch the jobs:
>>> p1 = mapcalc_start(expr1)
>>> p2 = mapcalc_start(expr2)
...
>>> # wait for them to finish:
>>> p1.wait()
0
>>> p2.wait()
1
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity 
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice:  480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC),  480-727-0709 (CSDC)
















_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about *.mapcalc_start in Python scripting library

Moritz Lennert
Le Fri, 15 Mar 2019 13:15:16 +0000,
Michael Barton <[hidden email]> a écrit :

> I've been using *.mapcalc_start to do some simple parallel processing
> in the i.pansharpen module and in r.landscape.evol. My understanding
> previously has been that *.mapcalc_start returns a Popen object that
> can be directed to wait to finish (like *.start_command), allowing
> multiple mapcalc operations to be initiated on multiple cores.
>
> But when running r.landscape.evol on another computer, it is acting
> like mapcalc is not waiting to finish in spite of the *.wait()
> commands.
>
> The description in the programmer's manual is not clear to me on this
> (see below). In one place, it says it does not wait to finish and in
> the example it says it does wait.


AFAIU, the function mapcalc_start() does not wait, i.e. you can go on
with your program while the mapcalc call is still ongoing. If you want
to wait, you have to explicitely call the function wait() of the Popen
object as is done in the example.

Moritz

>
> Can anyone clarify this?
>
> Thanks
> Michael
>
> ==== from programmer's manual ====
>
> script.raster.mapcalc_start(exp, quiet=False, verbose=False,
> overwrite=False, seed=None, env=None, **kwargs)[source] Interface to
> r.mapcalc, doesn’t wait for it to finish, returns Popen object.
>
> >>> output = 'newele'
> >>> input = 'elevation'
> >>> expr1 = '"%s" = "%s" * 10' % (output, input)
> >>> expr2 = '...'   # etc.
> >>> # launch the jobs:
> >>> p1 = mapcalc_start(expr1)
> >>> p2 = mapcalc_start(expr2)  
> ...
> >>> # wait for them to finish:
> >>> p1.wait()  
> 0
> >>> p2.wait()  
> 1
> ____________________
> C. Michael Barton
> Director, Center for Social Dynamics & Complexity
> Professor of Anthropology, School of Human Evolution & Social Change
> Head, Graduate Faculty in Complex Adaptive Systems Science
> Arizona State University
>
> voice:  480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
> fax: 480-965-7671 (SHESC),  480-727-0709 (CSDC)
> www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about *.mapcalc_start in Python scripting library

Michael Barton
OK. That is how I've been using it. Wanted to make sure.

Thanks
Michael
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity 
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice:  480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC),  480-727-0709 (CSDC)















On Mar 16, 2019, at 11:58 AM, Moritz Lennert <[hidden email]> wrote:

Le Fri, 15 Mar 2019 13:15:16 +0000,
Michael Barton <[hidden email]> a écrit :

I've been using *.mapcalc_start to do some simple parallel processing
in the i.pansharpen module and in r.landscape.evol. My understanding
previously has been that *.mapcalc_start returns a Popen object that
can be directed to wait to finish (like *.start_command), allowing
multiple mapcalc operations to be initiated on multiple cores.

But when running r.landscape.evol on another computer, it is acting
like mapcalc is not waiting to finish in spite of the *.wait()
commands.

The description in the programmer's manual is not clear to me on this
(see below). In one place, it says it does not wait to finish and in
the example it says it does wait.


AFAIU, the function mapcalc_start() does not wait, i.e. you can go on
with your program while the mapcalc call is still ongoing. If you want
to wait, you have to explicitely call the function wait() of the Popen
object as is done in the example.

Moritz


Can anyone clarify this?

Thanks
Michael

==== from programmer's manual ====

script.raster.mapcalc_start(exp, quiet=False, verbose=False,
overwrite=False, seed=None, env=None, **kwargs)[source] Interface to
r.mapcalc, doesn’t wait for it to finish, returns Popen object.

output = 'newele'
input = 'elevation'
expr1 = '"%s" = "%s" * 10' % (output, input)
expr2 = '...'   # etc.
# launch the jobs:
p1 = mapcalc_start(expr1)
p2 = mapcalc_start(expr2)  
...
# wait for them to finish:
p1.wait()  
0
p2.wait()  
1
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice:  480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC),  480-727-0709 (CSDC)
www: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.public.asu.edu_-7Ecmbarton&d=DwIFaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=lk-7X7CEOMDN8GaGVhiDsuO6gEp1wbG6nfT1XEEEtR0&m=SMauS2VJMEiccsq53CfPMfOU38B8lsDVCAd7oYqvLoc&s=eKrRsY0uJUc5RU5WldLfYR1EFjS1oSvI46U21M3qS4U&e=, https://urldefense.proofpoint.com/v2/url?u=http-3A__csdc.asu.edu&d=DwIFaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=lk-7X7CEOMDN8GaGVhiDsuO6gEp1wbG6nfT1XEEEtR0&m=SMauS2VJMEiccsq53CfPMfOU38B8lsDVCAd7oYqvLoc&s=lOB9PM_7bO9sn7VRqX3ZH1gFKM7lufgKzh50Yo2au7M&e=


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev