Crashing during long raster processing loops

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Crashing during long raster processing loops

Moody, Alex

Hi all,

 

I consistently have an issue where I am running long loops for raster processing that crash after 100 iterations or so. Typically the error is a ValueError where “ValueError: the environment variable is longer than 32767 bytes”

 

This has happened when running scripts in the GUI and when using the grass_session package. A sketch of my loop is as follows:

 

 

def doSomething(x):

     user = Session()

     user.open(gisdb=mygisdb, location=mylocation, mapset=mymapset,

                   create_opts='')

 

     v.surf_idw(….)

     r.out_gdal(…)

 

     user.close()

 

def main():

     monthlist = list of dates

     for month in monthlist:

           doSomething(month)

 

if __name__ == "__main__":

    sys.exit(main())

 

So I’m opening and closing a session in each loop, which  I hoped would take care of memory Issues I may run into.

 

I’m using 7.6.1 on a 64-Bit Windows 7. The full error is

 

Traceback (most recent call last):

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 82, in <module>

    sys.exit(main())

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 79, in main

    makeETrSurface(ts)

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 30, in doSomething

    user = Session()

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 244, in __init__

    self.env = set_grass_path_env(gisbase=self.gisbase, env=self.env)

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 102, in set_grass_path_env

    env['PATH'] += os.pathsep + os.path.join(gisbase, 'extrabin')

  File "D:\OSGEO4~1\apps\Python27\lib\os.py", line 422, in __setitem__

    putenv(key, item)

ValueError: the environment variable is longer than 32767 bytes

 

Any ideas of how to prevent the ValueError? Perhaps some arguments when calling Session?

 

Thanks as always,

Alex


_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: Crashing during long raster processing loops

Micha Silver-2

Just a guess...

You are repeatedly adding the path to GRASS to your environment variable "PATH" until it becomes longer than allowed.

I would say, open *one* session (in main()), then run your processing loop.


On 22/08/2019 01:49, Moody, Alex wrote:

Hi all,

 

I consistently have an issue where I am running long loops for raster processing that crash after 100 iterations or so. Typically the error is a ValueError where “ValueError: the environment variable is longer than 32767 bytes”

 

This has happened when running scripts in the GUI and when using the grass_session package. A sketch of my loop is as follows:

 

 

def doSomething(x):

     user = Session()

     user.open(gisdb=mygisdb, location=mylocation, mapset=mymapset,

                   create_opts='')

 

     v.surf_idw(….)

     r.out_gdal(…)

 

     user.close()

 

def main():

     monthlist = list of dates

     for month in monthlist:

           doSomething(month)

 

if __name__ == "__main__":

    sys.exit(main())

 

So I’m opening and closing a session in each loop, which  I hoped would take care of memory Issues I may run into.

 

I’m using 7.6.1 on a 64-Bit Windows 7. The full error is

 

Traceback (most recent call last):

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 82, in <module>

    sys.exit(main())

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 79, in main

    makeETrSurface(ts)

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 30, in doSomething

    user = Session()

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 244, in __init__

    self.env = set_grass_path_env(gisbase=self.gisbase, env=self.env)

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 102, in set_grass_path_env

    env['PATH'] += os.pathsep + os.path.join(gisbase, 'extrabin')

  File "D:\OSGEO4~1\apps\Python27\lib\os.py", line 422, in __setitem__

    putenv(key, item)

ValueError: the environment variable is longer than 32767 bytes

 

Any ideas of how to prevent the ValueError? Perhaps some arguments when calling Session?

 

Thanks as always,

Alex


_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
-- 
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
+972-523-665918

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: Crashing during long raster processing loops

Moody, Alex

Hi Micah,

 

Thanks and good catch. I guess I assumed that the grass_session.close() function would clear the path variable, but like you said, that doesn’t appear to be the case. The number of entries increases by 5 each time in this case.

 

This doesn’t need addressing, but before I reformatted this to use grass_session I would crash GRASS when running a similar script within the GUI python console. Unfortunately I didn’t save the error from that, but maybe it could have been due to similar reasons? I remember it being a memory error.

 

Alex

 

From: Micha Silver [mailto:[hidden email]]
Sent: Thursday, August 22, 2019 12:07 AM
To: Moody, Alex <[hidden email]>; [hidden email]
Subject: Re: [GRASS-user] Crashing during long raster processing loops

 

Just a guess...

You are repeatedly adding the path to GRASS to your environment variable "PATH" until it becomes longer than allowed.

I would say, open *one* session (in main()), then run your processing loop.

 

On 22/08/2019 01:49, Moody, Alex wrote:

Hi all,

 

I consistently have an issue where I am running long loops for raster processing that crash after 100 iterations or so. Typically the error is a ValueError where “ValueError: the environment variable is longer than 32767 bytes”

 

This has happened when running scripts in the GUI and when using the grass_session package. A sketch of my loop is as follows:

 

 

def doSomething(x):

     user = Session()

     user.open(gisdb=mygisdb, location=mylocation, mapset=mymapset,

                   create_opts='')

 

     v.surf_idw(….)

     r.out_gdal(…)

 

     user.close()

 

def main():

     monthlist = list of dates

     for month in monthlist:

           doSomething(month)

 

if __name__ == "__main__":

    sys.exit(main())

 

So I’m opening and closing a session in each loop, which  I hoped would take care of memory Issues I may run into.

 

I’m using 7.6.1 on a 64-Bit Windows 7. The full error is

 

Traceback (most recent call last):

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 82, in <module>

    sys.exit(main())

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 79, in main

    makeETrSurface(ts)

  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 30, in doSomething

    user = Session()

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 244, in __init__

    self.env = set_grass_path_env(gisbase=self.gisbase, env=self.env)

  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 102, in set_grass_path_env

    env['PATH'] += os.pathsep + os.path.join(gisbase, 'extrabin')

  File "D:\OSGEO4~1\apps\Python27\lib\os.py", line 422, in __setitem__

    putenv(key, item)

ValueError: the environment variable is longer than 32767 bytes

 

Any ideas of how to prevent the ValueError? Perhaps some arguments when calling Session?

 

Thanks as always,

Alex



_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
-- 
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
+972-523-665918

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user