help with r.hants

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

help with r.hants

Veronica Andreo
Dear all,

I'm using the following command to run r.hants on a series of 506 maps of chlorophyll. It seems to start thinking and a few seconds after i get an error:

r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46

ERROR: no temp files available

As stated before, the total number of images is 506 (11 years) but the base period is 1 year, hence base_period=46 images, and i'm passing a plain text file with the map names (one per line) as stated here: http://grass.osgeo.org/grass70/manuals/addons/r.hants.html

The files are registered in a spatio-temporal raster data set, if that means something, dunno. 

Is the command correct? What am I missing? Do you know what that error mean? 
Please, help! :)

Thanks much in advance! And Happy Labor day for everybody!

Vero

_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Markus Neteler
hi Veronica,

On Thu, May 1, 2014 at 3:17 PM, Veronica Andreo <[hidden email]> wrote:
> Dear all,
>
> I'm using the following command to run r.hants on a series of 506 maps of
> chlorophyll. It seems to start thinking and a few seconds after i get an
> error:
>
> r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
>
> ERROR: no temp files available

probably it runs out of memory? How many pixels does one map have?

> As stated before, the total number of images is 506 (11 years) but the base
> period is 1 year, hence base_period=46 images, and i'm passing a plain text
> file with the map names (one per line) as stated here:
> http://grass.osgeo.org/grass70/manuals/addons/r.hants.html

You can simply use g.mlist for that, much easier (see below).

> The files are registered in a spatio-temporal raster data set, if that means
> something, dunno.

I guess that this will be simply ignored, so no problem.

-------------

I have tried myself, using the new dataset prepared by Anna:

Climate dataset for North Carolina from publicly available data, with
monthly temperature average and monthly precipitation from 2000 to 2012:
http://courses.ncsu.edu/mea592/common/media/02/nc_climate_spm_2000_2012.zip

#########

GRASS 7.1.svn (nc_climate_spm_2000_2012):~ >

# set computational region
g.region rast=2004_03_tempmean -p

# check for the maps, especially how many
g.mlist rast pattern=20??_??_tempmean  | wc -l
raster map(s) available in mapset <climate_1970_2012>:
156

# try out at lower resolution for quick assessment:
g.region -p res=5000

# HANTS
# just wildly guessing the parameters for a test run:
r.hants input=`g.mlist rast pattern=20??_??_tempmean sep=comma` \
      nf=6 fet=0.1 dod=5 delta=0.1 base_period=12
Harmonic analysis of 156 input maps...
 100%

# assign reasonable color tables
for map in `g.mlist rast pattern="*tempmean_hants"` ; do
      r.colors $map color=celsius
done

# verify with one of the 156 results:
# remember that we had reduced the resolution
# by factor 10 to this quick assessment
r.mapcalc "diff = 2000_06_tempmean - 2000_06_tempmean_hants"

r.colors diff color=differences
d.mon wx0
d.rast diff

r.univar diff -g
n=5066
null_cells=5040
cells=10106
min=-0.0899336115228095
max=0.359362050140941
range=0.449295661663751
mean=0.188579838052468
...

# see original as animation (still at 5km resolution)
# note: color table is different from standard "celsius"
g.gui.animation rast=`g.mlist rast pattern="*tempmean" sep=comma`

# see HANTS time series as animation
g.gui.animation rast=`g.mlist rast pattern="*tempmean_hants" sep=comma`


Looks all fine!

Best
markusN

_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Veronica Andreo
Hi Markus, all

Thanks much for answering! and for the example!


> I'm using the following command to run r.hants on a series of 506 maps of
> chlorophyll. It seems to start thinking and a few seconds after i get an
> error:
>
> r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
>
> ERROR: no temp files available

probably it runs out of memory? How many pixels does one map have?

I have 506 maps, 146880 pixels each... maybe that's too much for my laptop? :P
 
I've tried the example and works fine, also tried without changing resolution and it starts loading fine, too (i stopped it before it finished)... but when I try with my maps using g.mlist instead of file:
  
GRASS 7.1.svn (latlong_wgs84):~ > r.hants input=`g.mlist rast pat=*_arg sep=comma` nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
raster map(s) available in mapset <clorofila>:
ERROR: no temp files available

same error... seems it doesn't read the maps... g.mlist works just fine though... So, dunno... not sure about memory limit since it seems to work (slowly, of course) with Anna's data set at original resolution... is there a number of characters for map names? They have an awfully long name: A20133612013365.L3m_8D_CHL_chlor_a_4km_arg

well, no idea, this data is bewitched :P

Thanks again!

Best, 
Vero

_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Markus Neteler
On Fri, May 2, 2014 at 6:20 PM, Veronica Andreo <[hidden email]> wrote:

> Hi Markus, all
>
> Thanks much for answering! and for the example!
>
>>
>> > I'm using the following command to run r.hants on a series of 506 maps
>> > of
>> > chlorophyll. It seems to start thinking and a few seconds after i get an
>> > error:
>> >
>> > r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
>> >
>> > ERROR: no temp files available
>>
>> probably it runs out of memory? How many pixels does one map have?
>
>
> I have 506 maps, 146880 pixels each... maybe that's too much for my laptop?
> :P

Please run your calculation in DEBUG mode:

g.gisenv set=DEBUG=1
r.hants ...

The relevant part for us is around this error message (it will print a
lot more of course).
You may dump the interesting part here
http://pastebin.com/
and send the resulting URL only.

We'll figure out! :)

ciao
Markus
_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Veronica Andreo

2014-05-02 14:06 GMT-03:00 Markus Neteler <[hidden email]>:
On Fri, May 2, 2014 at 6:20 PM, Veronica Andreo <[hidden email]> wrote:
> Hi Markus, all
>
> Thanks much for answering! and for the example!
>
>> > I'm using the following command to run r.hants on a series of 506 maps
>> > of
>> > chlorophyll. It seems to start thinking and a few seconds after i get an
>> > error:
>> >
>> > r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
>> >
>> > ERROR: no temp files available
>>
>> probably it runs out of memory? How many pixels does one map have?
>
> I have 506 maps, 146880 pixels each... maybe that's too much for my laptop?
> :P

Please run your calculation in DEBUG mode:

g.gisenv set=DEBUG=1
r.hants ...

done 

The relevant part for us is around this error message (it will print a
lot more of course).
You may dump the interesting part here
http://pastebin.com/
and send the resulting URL only.

 
We'll figure out! :)

:) thanks much, again!  

Vero

_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Veronica Andreo
Here's also the last part of the output when

g.gisenv set=DEBUG=3 
r.hants ...


Thanks!
Vero

2014-05-02 15:27 GMT-03:00 Veronica Andreo <[hidden email]>:

2014-05-02 14:06 GMT-03:00 Markus Neteler <[hidden email]>:

On Fri, May 2, 2014 at 6:20 PM, Veronica Andreo <[hidden email]> wrote:
> Hi Markus, all
>
> Thanks much for answering! and for the example!
>
>> > I'm using the following command to run r.hants on a series of 506 maps
>> > of
>> > chlorophyll. It seems to start thinking and a few seconds after i get an
>> > error:
>> >
>> > r.hants file=lista_hants nf=6 fet=0.1 dod=5 delta=0.1 base_period=46
>> >
>> > ERROR: no temp files available
>>
>> probably it runs out of memory? How many pixels does one map have?
>
> I have 506 maps, 146880 pixels each... maybe that's too much for my laptop?
> :P

Please run your calculation in DEBUG mode:

g.gisenv set=DEBUG=1
r.hants ...

done 

The relevant part for us is around this error message (it will print a
lot more of course).
You may dump the interesting part here
http://pastebin.com/
and send the resulting URL only.

 
We'll figure out! :)

:) thanks much, again!  

Vero


_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Markus Neteler
On Sat, May 3, 2014 at 12:51 AM, Veronica Andreo <[hidden email]> wrote:
> Here's also the last part of the output when
>
> g.gisenv set=DEBUG=3
> r.hants ...
>
> http://pastebin.com/cp139prq

OK I think I know where a modification is needed (and I think that too
many files are open for your current settings, see "ulimit" stuff in
the manual page of r.series):

...
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/GDAL
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/.tmp/localhost.localdomain/12772.6
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/.tmp/localhost.localdomain/12772.7
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/GDAL
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/.tmp/localhost.localdomain/12772.8
D2/3: G_file_name(): path =
/home/veroandreo/grassdata//latlong_wgs84/clorofila/.tmp/localhost.localdomain/12772.9
ERROR: no temp files available
GRASS 7.1.svn (latlong_wgs84):~ >

in lib/gis/tempfile.c

char *G__tempfile(int pid)
{

[...]
    do {
    int uniq = G_counter_next(&unique);
    sprintf(name, "%d.%d", pid, uniq);
    G_file_name(path, element, name, G_mapset());
    }
    while (access(path, F_OK) == 0);
[...]


We need to decipher the error with perror() as done in

in lib/gis/mapset.c

    if (access(mapset, F_OK) != 0) {
    perror("access");
    G_fatal_error(_("MAPSET <%s> not available"), mapset);
    }

Anyone able to modify in lib/gis/tempfile.c properly?

thanks
Markus
_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Markus Neteler
On Sat, May 3, 2014 at 3:09 PM, Markus Neteler <[hidden email]> wrote:
> On Sat, May 3, 2014 at 12:51 AM, Veronica Andreo <[hidden email]> wrote:
>> Here's also the last part of the output when
>>
>> g.gisenv set=DEBUG=3
>> r.hants ...
>>
>> http://pastebin.com/cp139prq
...
> Anyone able to modify in lib/gis/tempfile.c properly?

I have opened a ticket:
http://trac.osgeo.org/grass/ticket/2278

Markus
_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: help with r.hants

Veronica Andreo
Hi Markus and all

Thanks much for all the help! Hope someone is willing to address the task in the ticket. It would be helpful to have a more informative error message. 

I've changed the limit of open files with ulimit to 2000 but it wasn't working anyway... so, i tried with -z flag, and it is doing the job now, though (as expected) it is taking quite some time... I should have tried that from the beginning :P

All the best! And thanks again!
Vero




2014-05-04 5:33 GMT-03:00 Markus Neteler <[hidden email]>:
On Sat, May 3, 2014 at 3:09 PM, Markus Neteler <[hidden email]> wrote:
> On Sat, May 3, 2014 at 12:51 AM, Veronica Andreo <[hidden email]> wrote:
>> Here's also the last part of the output when
>>
>> g.gisenv set=DEBUG=3
>> r.hants ...
>>
>> http://pastebin.com/cp139prq
...
> Anyone able to modify in lib/gis/tempfile.c properly?

I have opened a ticket:
http://trac.osgeo.org/grass/ticket/2278

Markus


_______________________________________________
grass-user mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-user