v.vol.rst - not enough disk space to write temp file

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

v.vol.rst - not enough disk space to write temp file

Francois Chartier
Hi,

trying to run v.vol.rst with a larger dataset but using a smaller footprint for the interpolation, using the same domain than  my previous dataset.  
The interpolation does not reach the end of the simulation as there is not enough disk space.  Any way around this?

(Wed Dec 19 20:24:48 2018)                                                      
v.vol.rst --overwrite input=GeomodDec16@Toronto wcolumn=dbl_2 tension=100 elevation=trialdec16
280957 records selected from table
WARNING: Some points outside of region -- will ignore...
WARNING: Strip exists with insufficient data
Processing all selected output files will require 1830391004 bytes of disk space for temp files
WARNING: There are points outside specified 2D/3D region--ignored 222354 points (total points: 280957)
WARNING: Points are more dense than specified 'DMIN'--ignored 1946 points (remain 279011)
WARNING: Unable to create <(null)> raster map without cross_input raster map being specified
ERROR: Not enough disk space - cannot write temp files
(Wed Dec 19 20:29:56 2018) Command finished (5 min 8 sec) 

g.region -3p  [i have set a number of rows and columns to 100 for each, and res3.value as 1 (not sure if this contradict the number of column and rows)
projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4862000
south:      4846000
west:       609200
east:       651700
top:        250.00000000
bottom:     50.00000000
nsres:      160
nsres3:     0.11358232
ewres:      425
ewres3:     0.44870509
tbres:      2
rows:       100
rows3:      140867
cols:       100
cols3:      94717
depths:     100
cells:      10000
cells3:     1334249963900

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Markus Neteler
On Thu, Dec 20, 2018 at 2:42 AM Francois Chartier
<[hidden email]> wrote:
> Hi,
>
> trying to run v.vol.rst with a larger dataset but using a smaller footprint for the interpolation, using the same domain than  my previous dataset.
> The interpolation does not reach the end of the simulation as there is not enough disk space.  Any way around this?

... get more disk space :-)

> (Wed Dec 19 20:24:48 2018)
> v.vol.rst --overwrite input=GeomodDec16@Toronto wcolumn=dbl_2 tension=100 elevation=trialdec16
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> WARNING: Strip exists with insufficient data
> Processing all selected output files will require 1830391004 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 222354 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 1946 points (remain 279011)
> WARNING: Unable to create <(null)> raster map without cross_input raster map being specified

(note: this <(null)> in that message I have fixed now but it is
unrelated to the problem).

> ERROR: Not enough disk space - cannot write temp files

When running the command keep an eye on the disk space: perhaps it
writes the temp files to a very small partition?

> (Wed Dec 19 20:29:56 2018) Command finished (5 min 8 sec)
>
> g.region -3p  [i have set a number of rows and columns to 100 for each, and res3.value as 1 (not sure if this contradict the number of column and rows)
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4862000
> south:      4846000
> west:       609200
> east:       651700
> top:        250.00000000
> bottom:     50.00000000
> nsres:      160
> nsres3:     0.11358232
> ewres:      425
> ewres3:     0.44870509

nsres3 and ewres3 are not equal, is that intended?

> tbres:      2
> rows:       100
> rows3:      140867
> cols:       100
> cols3:      94717
> depths:     100
> cells:      10000
> cells3:     1334249963900

This is a fairly large amount of voxels: 1.33425e+12

Likely the need for much temp space stems from that.

Markus
_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
Hi, 

I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.
So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space.... 
the memory situation is much better during the process.
for some reason it does not seem to increase the size of the temp file on the hard drive...
see error message below.
(Sat Jan 12 17:28:02 2019)                                                      
v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
280957 records selected from table
WARNING: Some points outside of region -- will ignore...
Processing all selected output files will require 2134724556 bytes of disk space for temp files
WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
ERROR: Not enough disk space - cannot write temp files
(Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)

below is the g.region -p3 printed, as you can see this is a large domain and high resolution. 

projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4924870
south:      4784003
west:       585000
east:       679717
top:        453.00000000
bottom:     2.16080000
nsres:      50.00603479
nsres3:     5.00007099
ewres:      50.00897571
ewres3:     5.00010558
tbres:      0.999643458980044
rows:       2817
rows3:      28173
cols:       1894
cols3:      18943
depths:     451
cells:      5335398
cells3:     240690193689



Le jeu. 27 déc. 2018 à 07:24, Markus Neteler <[hidden email]> a écrit :
On Thu, Dec 20, 2018 at 2:42 AM Francois Chartier
<[hidden email]> wrote:
> Hi,
>
> trying to run v.vol.rst with a larger dataset but using a smaller footprint for the interpolation, using the same domain than  my previous dataset.
> The interpolation does not reach the end of the simulation as there is not enough disk space.  Any way around this?

... get more disk space :-)

> (Wed Dec 19 20:24:48 2018)
> v.vol.rst --overwrite input=GeomodDec16@Toronto wcolumn=dbl_2 tension=100 elevation=trialdec16
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> WARNING: Strip exists with insufficient data
> Processing all selected output files will require 1830391004 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 222354 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 1946 points (remain 279011)
> WARNING: Unable to create <(null)> raster map without cross_input raster map being specified

(note: this <(null)> in that message I have fixed now but it is
unrelated to the problem).

> ERROR: Not enough disk space - cannot write temp files

When running the command keep an eye on the disk space: perhaps it
writes the temp files to a very small partition?

> (Wed Dec 19 20:29:56 2018) Command finished (5 min 8 sec)
>
> g.region -3p  [i have set a number of rows and columns to 100 for each, and res3.value as 1 (not sure if this contradict the number of column and rows)
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4862000
> south:      4846000
> west:       609200
> east:       651700
> top:        250.00000000
> bottom:     50.00000000
> nsres:      160
> nsres3:     0.11358232
> ewres:      425
> ewres3:     0.44870509

nsres3 and ewres3 are not equal, is that intended?

> tbres:      2
> rows:       100
> rows3:      140867
> cols:       100
> cols3:      94717
> depths:     100
> cells:      10000
> cells3:     1334249963900

This is a fairly large amount of voxels: 1.33425e+12

Likely the need for much temp space stems from that.

Markus

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Markus Neteler
Hi,

On Sat, Jan 12, 2019 at 11:56 PM Francois Chartier
<[hidden email]> wrote:
>
> Hi,
>
> I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.

I start to wonder if those tmp files created are in any way compressed
(probably not?). This may be worth a bug ticket at
https://trac.osgeo.org/grass/newticket

> So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is
> saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space....
> the memory situation is much better during the process.
> for some reason it does not seem to increase the size of the temp file on the hard drive...

So the grassdata directory with the location and mapset in question is
on the terabyte disk?

> see error message below.
> (Sat Jan 12 17:28:02 2019)
> v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> Processing all selected output files will require 2134724556 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
> WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
> ERROR: Not enough disk space - cannot write temp files

I think we should try to improve the error message, telling on which
device this occurs.

The code in question is:
vector/v.vol.rst/main.c, line 633

            /* filling temp file with zeroes */
            for (i = 0; i < n_rows; i++) {
                if (!
                    (fwrite
                     (zero_array_cell, sizeof(FCELL), n_cols, Tmp_fd_cell))) {
                    clean();
                    G_fatal_error(_("Not enough disk space - cannot
write temp files"));
                }
            }

Does anyone know how to get a better error message?
Would this help?
https://linux.die.net/man/3/explain_fwrite

> (Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)
>
> below is the g.region -p3 printed, as you can see this is a large domain and high resolution.
>
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4924870
> south:      4784003
> west:       585000
> east:       679717
> top:        453.00000000
> bottom:     2.16080000
> nsres:      50.00603479
> nsres3:     5.00007099
> ewres:      50.00897571
> ewres3:     5.00010558
> tbres:      0.999643458980044

Are the pixels/voxels non-square on purpose? This could be adjusted
with g.region.

> rows:       2817
> rows3:      28173
> cols:       1894
> cols3:      18943
> depths:     451
> cells:      5335398
> cells3:     240690193689

That is still a lot (just guessing around):
240,690,193,689 raster3d_cells * 64 bit (but it will be larger than
32bit) = 1.540417e+13 bit

So, in the first place we need to know where GRASS GIS really writes
to in your system.

Any devs having a suggestion here?

Markus
_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
grass data is on the external drive, see screenshot attached.
freespace on external drive 1.77TB
what is interesting with the external drive is the temp file does not really explode in size like if files are on laptop:
it has been running for about 50 min now.

I changed the region, made the domain slightly smaller and used a lower resolution:100x100x5
g.region
(Mon Jan 14 19:26:20 2019)                                                      
g.region n=4875000 s=4820000 e=680000 w=585000 t=350 b=50 nsres=100 ewres=100 tbres=5
(Mon Jan 14 19:26:21 2019) Command finished (0 sec) 

for some reason it seems that i have 2 nsres and ewres
C:\Users\Francois Chartier>g.region -p3
projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4875000
south:      4820000
west:       585000
east:       680000
top:        350.00000000
bottom:     50.00000000
nsres:      100
nsres3:     5
ewres:      100
ewres3:     5
tbres:      5
rows:       550
rows3:      11000
cols:       950
cols3:      19000
depths:     60
cells:      522500
cells3:     12540000000

Virus-free. www.avast.com

Le dim. 13 janv. 2019 à 08:52, Markus Neteler <[hidden email]> a écrit :
Hi,

On Sat, Jan 12, 2019 at 11:56 PM Francois Chartier
<[hidden email]> wrote:
>
> Hi,
>
> I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.

I start to wonder if those tmp files created are in any way compressed
(probably not?). This may be worth a bug ticket at
https://trac.osgeo.org/grass/newticket

> So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is
> saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space....
> the memory situation is much better during the process.
> for some reason it does not seem to increase the size of the temp file on the hard drive...

So the grassdata directory with the location and mapset in question is
on the terabyte disk?

> see error message below.
> (Sat Jan 12 17:28:02 2019)
> v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> Processing all selected output files will require 2134724556 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
> WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
> ERROR: Not enough disk space - cannot write temp files

I think we should try to improve the error message, telling on which
device this occurs.

The code in question is:
vector/v.vol.rst/main.c, line 633

            /* filling temp file with zeroes */
            for (i = 0; i < n_rows; i++) {
                if (!
                    (fwrite
                     (zero_array_cell, sizeof(FCELL), n_cols, Tmp_fd_cell))) {
                    clean();
                    G_fatal_error(_("Not enough disk space - cannot
write temp files"));
                }
            }

Does anyone know how to get a better error message?
Would this help?
https://linux.die.net/man/3/explain_fwrite

> (Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)
>
> below is the g.region -p3 printed, as you can see this is a large domain and high resolution.
>
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4924870
> south:      4784003
> west:       585000
> east:       679717
> top:        453.00000000
> bottom:     2.16080000
> nsres:      50.00603479
> nsres3:     5.00007099
> ewres:      50.00897571
> ewres3:     5.00010558
> tbres:      0.999643458980044

Are the pixels/voxels non-square on purpose? This could be adjusted
with g.region.

> rows:       2817
> rows3:      28173
> cols:       1894
> cols3:      18943
> depths:     451
> cells:      5335398
> cells3:     240690193689

That is still a lot (just guessing around):
240,690,193,689 raster3d_cells * 64 bit (but it will be larger than
32bit) = 1.540417e+13 bit

So, in the first place we need to know where GRASS GIS really writes
to in your system.

Any devs having a suggestion here?

Markus

Virus-free. www.avast.com

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user

external drive.jpg (213K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
it finally finished the process.  the files size that was created on the external disk was 50GB as you can see on the attached screenshot.
the interpolation took 49 min to complete.


(Mon Jan 14 19:28:05 2019)                                                      
v.vol.rst input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=2 elevation=dec16jan14
280957 records selected from table
WARNING: Some points outside of region -- will ignore...
WARNING: Strip exists with insufficient data
Processing all selected output files will require 836000000 bytes of disk space for temp files
WARNING: There are points outside specified 2D/3D region--ignored 55094 points (total points: 280957)
WARNING: Points are more dense than specified 'DMIN'--ignored 156188 points (remain 124769)
WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
(Mon Jan 14 20:17:49 2019) Command finished (49 min 43 sec)     

Virus-free. www.avast.com

Le lun. 14 janv. 2019 à 20:13, Francois Chartier <[hidden email]> a écrit :
grass data is on the external drive, see screenshot attached.
freespace on external drive 1.77TB
what is interesting with the external drive is the temp file does not really explode in size like if files are on laptop:
it has been running for about 50 min now.

I changed the region, made the domain slightly smaller and used a lower resolution:100x100x5
g.region
(Mon Jan 14 19:26:20 2019)                                                      
g.region n=4875000 s=4820000 e=680000 w=585000 t=350 b=50 nsres=100 ewres=100 tbres=5
(Mon Jan 14 19:26:21 2019) Command finished (0 sec) 

for some reason it seems that i have 2 nsres and ewres
C:\Users\Francois Chartier>g.region -p3
projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4875000
south:      4820000
west:       585000
east:       680000
top:        350.00000000
bottom:     50.00000000
nsres:      100
nsres3:     5
ewres:      100
ewres3:     5
tbres:      5
rows:       550
rows3:      11000
cols:       950
cols3:      19000
depths:     60
cells:      522500
cells3:     12540000000

Virus-free. www.avast.com

Le dim. 13 janv. 2019 à 08:52, Markus Neteler <[hidden email]> a écrit :
Hi,

On Sat, Jan 12, 2019 at 11:56 PM Francois Chartier
<[hidden email]> wrote:
>
> Hi,
>
> I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.

I start to wonder if those tmp files created are in any way compressed
(probably not?). This may be worth a bug ticket at
https://trac.osgeo.org/grass/newticket

> So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is
> saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space....
> the memory situation is much better during the process.
> for some reason it does not seem to increase the size of the temp file on the hard drive...

So the grassdata directory with the location and mapset in question is
on the terabyte disk?

> see error message below.
> (Sat Jan 12 17:28:02 2019)
> v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> Processing all selected output files will require 2134724556 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
> WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
> ERROR: Not enough disk space - cannot write temp files

I think we should try to improve the error message, telling on which
device this occurs.

The code in question is:
vector/v.vol.rst/main.c, line 633

            /* filling temp file with zeroes */
            for (i = 0; i < n_rows; i++) {
                if (!
                    (fwrite
                     (zero_array_cell, sizeof(FCELL), n_cols, Tmp_fd_cell))) {
                    clean();
                    G_fatal_error(_("Not enough disk space - cannot
write temp files"));
                }
            }

Does anyone know how to get a better error message?
Would this help?
https://linux.die.net/man/3/explain_fwrite

> (Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)
>
> below is the g.region -p3 printed, as you can see this is a large domain and high resolution.
>
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4924870
> south:      4784003
> west:       585000
> east:       679717
> top:        453.00000000
> bottom:     2.16080000
> nsres:      50.00603479
> nsres3:     5.00007099
> ewres:      50.00897571
> ewres3:     5.00010558
> tbres:      0.999643458980044

Are the pixels/voxels non-square on purpose? This could be adjusted
with g.region.

> rows:       2817
> rows3:      28173
> cols:       1894
> cols3:      18943
> depths:     451
> cells:      5335398
> cells3:     240690193689

That is still a lot (just guessing around):
240,690,193,689 raster3d_cells * 64 bit (but it will be larger than
32bit) = 1.540417e+13 bit

So, in the first place we need to know where GRASS GIS really writes
to in your system.

Any devs having a suggestion here?

Markus

Virus-free. www.avast.com

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user

tmp file.jpg (417K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
i will try with a smaller model domain to see if the size of the domain the source of the problem. 

On Mon, Jan 14, 2019, 20:19 Francois Chartier <[hidden email] wrote:
it finally finished the process.  the files size that was created on the external disk was 50GB as you can see on the attached screenshot.
the interpolation took 49 min to complete.


(Mon Jan 14 19:28:05 2019)                                                      
v.vol.rst input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=2 elevation=dec16jan14
280957 records selected from table
WARNING: Some points outside of region -- will ignore...
WARNING: Strip exists with insufficient data
Processing all selected output files will require 836000000 bytes of disk space for temp files
WARNING: There are points outside specified 2D/3D region--ignored 55094 points (total points: 280957)
WARNING: Points are more dense than specified 'DMIN'--ignored 156188 points (remain 124769)
WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
(Mon Jan 14 20:17:49 2019) Command finished (49 min 43 sec)     

Virus-free. www.avast.com

Le lun. 14 janv. 2019 à 20:13, Francois Chartier <[hidden email]> a écrit :
grass data is on the external drive, see screenshot attached.
freespace on external drive 1.77TB
what is interesting with the external drive is the temp file does not really explode in size like if files are on laptop:
it has been running for about 50 min now.

I changed the region, made the domain slightly smaller and used a lower resolution:100x100x5
g.region
(Mon Jan 14 19:26:20 2019)                                                      
g.region n=4875000 s=4820000 e=680000 w=585000 t=350 b=50 nsres=100 ewres=100 tbres=5
(Mon Jan 14 19:26:21 2019) Command finished (0 sec) 

for some reason it seems that i have 2 nsres and ewres
C:\Users\Francois Chartier>g.region -p3
projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4875000
south:      4820000
west:       585000
east:       680000
top:        350.00000000
bottom:     50.00000000
nsres:      100
nsres3:     5
ewres:      100
ewres3:     5
tbres:      5
rows:       550
rows3:      11000
cols:       950
cols3:      19000
depths:     60
cells:      522500
cells3:     12540000000

Virus-free. www.avast.com

Le dim. 13 janv. 2019 à 08:52, Markus Neteler <[hidden email]> a écrit :
Hi,

On Sat, Jan 12, 2019 at 11:56 PM Francois Chartier
<[hidden email]> wrote:
>
> Hi,
>
> I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.

I start to wonder if those tmp files created are in any way compressed
(probably not?). This may be worth a bug ticket at
https://trac.osgeo.org/grass/newticket

> So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is
> saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space....
> the memory situation is much better during the process.
> for some reason it does not seem to increase the size of the temp file on the hard drive...

So the grassdata directory with the location and mapset in question is
on the terabyte disk?

> see error message below.
> (Sat Jan 12 17:28:02 2019)
> v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> Processing all selected output files will require 2134724556 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
> WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
> ERROR: Not enough disk space - cannot write temp files

I think we should try to improve the error message, telling on which
device this occurs.

The code in question is:
vector/v.vol.rst/main.c, line 633

            /* filling temp file with zeroes */
            for (i = 0; i < n_rows; i++) {
                if (!
                    (fwrite
                     (zero_array_cell, sizeof(FCELL), n_cols, Tmp_fd_cell))) {
                    clean();
                    G_fatal_error(_("Not enough disk space - cannot
write temp files"));
                }
            }

Does anyone know how to get a better error message?
Would this help?
https://linux.die.net/man/3/explain_fwrite

> (Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)
>
> below is the g.region -p3 printed, as you can see this is a large domain and high resolution.
>
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4924870
> south:      4784003
> west:       585000
> east:       679717
> top:        453.00000000
> bottom:     2.16080000
> nsres:      50.00603479
> nsres3:     5.00007099
> ewres:      50.00897571
> ewres3:     5.00010558
> tbres:      0.999643458980044

Are the pixels/voxels non-square on purpose? This could be adjusted
with g.region.

> rows:       2817
> rows3:      28173
> cols:       1894
> cols3:      18943
> depths:     451
> cells:      5335398
> cells3:     240690193689

That is still a lot (just guessing around):
240,690,193,689 raster3d_cells * 64 bit (but it will be larger than
32bit) = 1.540417e+13 bit

So, in the first place we need to know where GRASS GIS really writes
to in your system.

Any devs having a suggestion here?

Markus

Virus-free. www.avast.com

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
My solution is to use a smaller model domain with 'only' 16 million cells, with grid resolution of 10 x 10 m horizontally, and 1 m vertically.

is there anyway to remove the shade in the 3D view ?

Le mar. 15 janv. 2019 à 08:31, Francois Chartier <[hidden email]> a écrit :
i will try with a smaller model domain to see if the size of the domain the source of the problem. 

On Mon, Jan 14, 2019, 20:19 Francois Chartier <[hidden email] wrote:
it finally finished the process.  the files size that was created on the external disk was 50GB as you can see on the attached screenshot.
the interpolation took 49 min to complete.


(Mon Jan 14 19:28:05 2019)                                                      
v.vol.rst input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=2 elevation=dec16jan14
280957 records selected from table
WARNING: Some points outside of region -- will ignore...
WARNING: Strip exists with insufficient data
Processing all selected output files will require 836000000 bytes of disk space for temp files
WARNING: There are points outside specified 2D/3D region--ignored 55094 points (total points: 280957)
WARNING: Points are more dense than specified 'DMIN'--ignored 156188 points (remain 124769)
WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
(Mon Jan 14 20:17:49 2019) Command finished (49 min 43 sec)     

Virus-free. www.avast.com

Le lun. 14 janv. 2019 à 20:13, Francois Chartier <[hidden email]> a écrit :
grass data is on the external drive, see screenshot attached.
freespace on external drive 1.77TB
what is interesting with the external drive is the temp file does not really explode in size like if files are on laptop:
it has been running for about 50 min now.

I changed the region, made the domain slightly smaller and used a lower resolution:100x100x5
g.region
(Mon Jan 14 19:26:20 2019)                                                      
g.region n=4875000 s=4820000 e=680000 w=585000 t=350 b=50 nsres=100 ewres=100 tbres=5
(Mon Jan 14 19:26:21 2019) Command finished (0 sec) 

for some reason it seems that i have 2 nsres and ewres
C:\Users\Francois Chartier>g.region -p3
projection: 1 (UTM)
zone:       17
datum:      nad83
ellipsoid:  grs80
north:      4875000
south:      4820000
west:       585000
east:       680000
top:        350.00000000
bottom:     50.00000000
nsres:      100
nsres3:     5
ewres:      100
ewres3:     5
tbres:      5
rows:       550
rows3:      11000
cols:       950
cols3:      19000
depths:     60
cells:      522500
cells3:     12540000000

Virus-free. www.avast.com

Le dim. 13 janv. 2019 à 08:52, Markus Neteler <[hidden email]> a écrit :
Hi,

On Sat, Jan 12, 2019 at 11:56 PM Francois Chartier
<[hidden email]> wrote:
>
> Hi,
>
> I checked the temp file that is saved during the v.vol.rst interpolation process, and the file went up to over 50 GB and more until the space on my laptop (free space of 100GB) was exausted.

I start to wonder if those tmp files created are in any way compressed
(probably not?). This may be worth a bug ticket at
https://trac.osgeo.org/grass/newticket

> So I copied all the grass data file onto an external hard drive with 1 Terabyte of space. started grass and browsed to the data that is
> saved on the external hard drive.  when running vvolrst, the process also crashed due to not enough space....
> the memory situation is much better during the process.
> for some reason it does not seem to increase the size of the temp file on the hard drive...

So the grassdata directory with the location and mapset in question is
on the terabyte disk?

> see error message below.
> (Sat Jan 12 17:28:02 2019)
> v.vol.rst --overwrite input=Dec16CleanedJjan12@Toronto wcolumn=dbl_2 dmin=1 elevation=dec16jan12
> 280957 records selected from table
> WARNING: Some points outside of region -- will ignore...
> Processing all selected output files will require 2134724556 bytes of disk space for temp files
> WARNING: There are points outside specified 2D/3D region--ignored 1 points (total points: 280957)
> WARNING: Points are more dense than specified 'DMIN'--ignored 158199 points (remain 122758)
> WARNING: Unable to create 'cross_output' raster map without 'cross_input' raster map being specified
> ERROR: Not enough disk space - cannot write temp files

I think we should try to improve the error message, telling on which
device this occurs.

The code in question is:
vector/v.vol.rst/main.c, line 633

            /* filling temp file with zeroes */
            for (i = 0; i < n_rows; i++) {
                if (!
                    (fwrite
                     (zero_array_cell, sizeof(FCELL), n_cols, Tmp_fd_cell))) {
                    clean();
                    G_fatal_error(_("Not enough disk space - cannot
write temp files"));
                }
            }

Does anyone know how to get a better error message?
Would this help?
https://linux.die.net/man/3/explain_fwrite

> (Sat Jan 12 17:54:14 2019) Command finished (26 min 12 sec)
>
> below is the g.region -p3 printed, as you can see this is a large domain and high resolution.
>
> projection: 1 (UTM)
> zone:       17
> datum:      nad83
> ellipsoid:  grs80
> north:      4924870
> south:      4784003
> west:       585000
> east:       679717
> top:        453.00000000
> bottom:     2.16080000
> nsres:      50.00603479
> nsres3:     5.00007099
> ewres:      50.00897571
> ewres3:     5.00010558
> tbres:      0.999643458980044

Are the pixels/voxels non-square on purpose? This could be adjusted
with g.region.

> rows:       2817
> rows3:      28173
> cols:       1894
> cols3:      18943
> depths:     451
> cells:      5335398
> cells3:     240690193689

That is still a lot (just guessing around):
240,690,193,689 raster3d_cells * 64 bit (but it will be larger than
32bit) = 1.540417e+13 bit

So, in the first place we need to know where GRASS GIS really writes
to in your system.

Any devs having a suggestion here?

Markus

Virus-free. www.avast.com

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user

smallerdomain.jpg (499K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Markus Neteler
On Thu, Jan 17, 2019 at 4:00 AM Francois Chartier
<[hidden email]> wrote:
>
> My solution is to use a smaller model domain with 'only' 16 million cells, with grid resolution of 10 x 10 m horizontally, and 1 m vertically.

Glad you got it running.

> is there anyway to remove the shade in the 3D view ?

Under the "Appearance" tab you find
Lighting for adjusting light source

Markus
_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.vol.rst - not enough disk space to write temp file

Francois Chartier
I think the problem came from my g.region setting.  under the resolution tab i was only filling  for example EW, NS and vertical. then i noticed that the 3d resolution was different when printing g.region and i was creating for ex in the order of 100 million cells. i solved that by setting every cell under the resolution tab, even if that means repeating the same info (perhaps i havent understood something yet), and it works. 

for the lighting, i have checked this tab earlier and i recall that u can onlh change the sun position which then results always in a shaded area which impacts 3d isosurfaces.  
thx

On Tue, Jan 29, 2019, 15:29 Markus Neteler <[hidden email] wrote:
On Thu, Jan 17, 2019 at 4:00 AM Francois Chartier
<[hidden email]> wrote:
>
> My solution is to use a smaller model domain with 'only' 16 million cells, with grid resolution of 10 x 10 m horizontally, and 1 m vertically.

Glad you got it running.

> is there anyway to remove the shade in the 3D view ?

Under the "Appearance" tab you find
Lighting for adjusting light source

Markus

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user