[GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

GRASS GIS
#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
-----------------------------------+-------------------------
 Reporter:  sbl                    |      Owner:  grass-dev@…
     Type:  enhancement            |     Status:  new
 Priority:  normal                 |  Milestone:  7.8.0
Component:  Default                |    Version:  svn-trunk
 Keywords:  environment variables  |        CPU:  Unspecified
 Platform:  All                    |
-----------------------------------+-------------------------
 Some Network Attached Storage (NAS) filers with snapshot backup on volume
 level do not allow to exclude specific file types or directories to be
 excluded from backup.
 If GRASS data DB is stored on such a storage system, temporary files can
 grow the snapshot file significantly until a point where available space
 is used up by backup files (how fast that happens depends on the snapshot
 frequency of course, and the time they are preserved).


 Currently, it is not possible to redirect all temporary data to other
 storage locations. Would be great to have an environment variable that
 allows to specify a path (directory) where all temporary data will be
 stored.


 Might be necessary to make a subdirectory per mapset within that
 directory... (Maybe an 8.0 thing?)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

GRASS GIS
#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter:  sbl          |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.8.0
 Component:  Default      |    Version:  svn-trunk
Resolution:               |   Keywords:  environment variables
       CPU:  Unspecified  |   Platform:  All
--------------------------+-----------------------------------

Comment (by neteler):

 As a workaround you could simply replace the location/mapset/.tmp/
 with a link to a directory on SSD.

 However, from the planning document:

 https://trac.osgeo.org/grass/wiki/Grass8Planning#Rasterlibrary

  * "implement GRASS_RASTER_TMPDIR_MAPSET like it exists for vector data
 (GRASS_VECTOR_TMPDIR_MAPSET), i.e. change all .tmp/ to variable in source
 code in init/grass.py, gis/open.c, gis/file_name.c, raster library"

 Maybe to be worked out in more detail but for sure very important (we
 frequently face the same issue).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:1>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
SBL
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

SBL
Hi Markus,

And thanks for the info/feedback.
Although the redirection of the .tmp directory (in future through an environment variable hopefully) already helps quite a bit and is a rather simple remedy (from the user perspective), intermediate maps can still cause similar issues.

Inspired by g.cloud, I tend to setup a local GRASS DB on SSD, where I link / copy mapsets from network storage as needed for the process. That way, (script based) processing does not affect snapshot backup at all. So, my follow up question is if a Python function to create a local grass DB from a central GRASS DB on network storage might be useful.

I am thinking along the following lines:

def setup_local_db(local_db_path, network_db_path, link_locations_mapsets, copy_locations_mapsets):
    """creates a local GRASS DB by linking or copying a user defined list of mapsets from a GRASS DB on network storage
        :param local_db_path: path where the local GRASS DB should be created
        :type local_db_path: str
        :param network_db_path: path to the GRASS DB on network storage from which mapsest should be linked / copied
        :type network_db_path: str
        :param link_locations_mapsets: tuple with combinations of (location, mapset) to be linked to local GRASS DB
        :type link_locations_mapsets: tuple
        :param copy_locations_mapsets: tuple with combinations of (location, mapset) to be copied to local GRASS DB
        :type copy_locations_mapsets: tuple
    """

Would that be suitable for the GRASS code base? Such a function seems little helpful to run from within an existing GRASS session, so an addon does not seem to make very much sense either, right? Maybe if Panos work on making grass modules importable without existing session is successful, one could setup local db and then start a session. But maybe this would be more suitable for grass-session?

Any comments recommendations, suggestions are most welcome...

Cheers,
Stefan

-----Original Message-----
From: GRASS GIS <[hidden email]>
Sent: torsdag 2. mai 2019 15:21
Cc: [hidden email]
Subject: Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

#3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter:  sbl          |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.8.0
 Component:  Default      |    Version:  svn-trunk
Resolution:               |   Keywords:  environment variables
       CPU:  Unspecified  |   Platform:  All
--------------------------+-----------------------------------

Comment (by neteler):

 As a workaround you could simply replace the location/mapset/.tmp/  with a link to a directory on SSD.

 However, from the planning document:

 https://trac.osgeo.org/grass/wiki/Grass8Planning#Rasterlibrary

  * "implement GRASS_RASTER_TMPDIR_MAPSET like it exists for vector data  (GRASS_VECTOR_TMPDIR_MAPSET), i.e. change all .tmp/ to variable in source  code in init/grass.py, gis/open.c, gis/file_name.c, raster library"

 Maybe to be worked out in more detail but for sure very important (we  frequently face the same issue).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:1>
GRASS GIS <https://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

GRASS GIS
In reply to this post by GRASS GIS
#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter:  sbl          |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.8.0
 Component:  Default      |    Version:  svn-trunk
Resolution:               |   Keywords:  environment variables
       CPU:  Unspecified  |   Platform:  All
--------------------------+-----------------------------------

Comment (by sbl):

 Consider also:
 https://trac.osgeo.org/grass/ticket/3857#comment:4

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:2>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

GRASS GIS
In reply to this post by GRASS GIS
#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter:  sbl          |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.8.0
 Component:  Default      |    Version:  svn-trunk
Resolution:               |   Keywords:  environment variables
       CPU:  Unspecified  |   Platform:  All
--------------------------+-----------------------------------

Comment (by martinl):

 Such variable is currently implemented only for vector library
 (GRASS_VECTOR_TMPDIR_MAPSET), see also related discussion,
 https://lists.osgeo.org/pipermail/grass-dev/2015-June/075220.html

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:3>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev