r.in.gdal -r and r.import extent=region not working properly?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

r.in.gdal -r and r.import extent=region not working properly?

Veronica Andreo
Hi devs,

I just found something odd. I have a CHIRPS tif for Africa [0], but I would only need to import Ghana, so I used r.in.gdal -r. However when I visualize the data, most of it is nodata, while the full map of Africa has data all over the continent. Then, I tried also with r.import extent=region. Same result. The only thing providing what I expected to be the right result (a subset from the full map) is importing the whole map and then using r.mapcalc to clip (i need the space in my hard disk)

These are the commands and r.univar output and I attach a screenshot with the visual results:

# import ghana boundaries from GADM
v.import input=GHA_adm0.shp output=GHA_adm0
g.region vector=GHA_adm0

# import full map
r.in.gdal in=Desktop/chirps-v2.0.2003.01.01.tif out=chirps_full

# import subsets
r.in.gdal -r in=chirps-v2.0.2003.01.01.tif out=chirps_ghana1
r.import in=chirps-v2.0.2003.01.01.tif out=chirps_ghana2 extent=region

# subset imported full map
r.mapcalc expression="chirps_ghana3 = chirps_full"

### r.univar outputs

r.univar chirps_ghana1

n: 12052
minimum: -9999
maximum: 0
range: 9999
mean: -9984.07
mean of absolute values: 9984.07
standard deviation: 386.135
variance: 149100
variation coefficient: -3.86751 %
sum: -120327966

r.univar chirps_ghana2

n: 12052
minimum: -9999
maximum: 0
range: 9999
mean: -9984.07
mean of absolute values: 9984.07
standard deviation: 386.135
variance: 149100
variation coefficient: -3.86751 %
sum: -120327966

## full map (region set to ghana)
r.univar chirps_full

n: 12052
minimum: -9999
maximum: 0
range: 9999
mean: -979.822
mean of absolute values: 979.822
standard deviation: 2972.74
variance: 8.83719e+06
variation coefficient: -303.396 %
sum: -11808819

## subset obtained with r.mapcalc
r.univar chirps_ghana3

n: 12052
minimum: -9999
maximum: 0
range: 9999
mean: -979.822
mean of absolute values: 979.822
standard deviation: 2972.74
variance: 8.83719e+06
variation coefficient: -303.396 %
sum: -11808819

What am I doing wrong here? I appreciate any hint :)

I use grass trunk r72618

Cheers,
Vero

[0] ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0/africa_daily/tifs/p05/

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev

Screenshot_2018-04-15_20-34-23.png (188K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Neteler
On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:

> Hi devs,
>
> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
> the data, most of it is nodata, while the full map of Africa has data all
> over the continent. Then, I tried also with r.import extent=region. Same
> result. The only thing providing what I expected to be the right result (a
> subset from the full map) is importing the whole map and then using
> r.mapcalc to clip (i need the space in my hard disk)
>
> These are the commands and r.univar output and I attach a screenshot with
> the visual results:
>
> # import ghana boundaries from GADM
> v.import input=GHA_adm0.shp output=GHA_adm0
> g.region vector=GHA_adm0
>
> # import full map
> r.in.gdal in=Desktop/chirps-v2.0.2003.01.01.tif out=chirps_full
>
> # import subsets
> r.in.gdal -r in=chirps-v2.0.2003.01.01.tif out=chirps_ghana1
> r.import in=chirps-v2.0.2003.01.01.tif out=chirps_ghana2 extent=region

... here r.null is missing, like

r.null chirps_whatever setnull=-9999

# check with
r.univar -e chirps_whatever


> # subset imported full map
> r.mapcalc expression="chirps_ghana3 = chirps_full"
>
> ### r.univar outputs
>
> r.univar chirps_ghana1
>
> n: 12052
> minimum: -9999
                     ^-- this is the hint that r.null is needed

In my test I then got

r.univar -e chirps_v2_0_2017_01_09
 100%
total null and non-null cells: 2400000
total null cells: 1181279

Of the non-null cells:
----------------------
n: 1218721
minimum: 0
maximum: 195.447
range: 195.447
mean: 1.41408
mean of absolute values: 1.41408
standard deviation: 5.044
variance: 25.4419
variation coefficient: 356.698 %
sum: 1723369.67863694
1st quartile: 0
median (odd number of cells): 0
3rd quartile: 0
90th percentile: 3.99494

which looks AFAIK reasonable (also graphically)

HTH
markusN
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Neteler
On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]> wrote:

> On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:
>> Hi devs,
>>
>> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
>> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
>> the data, most of it is nodata, while the full map of Africa has data all
>> over the continent. Then, I tried also with r.import extent=region. Same
>> result. The only thing providing what I expected to be the right result (a
>> subset from the full map) is importing the whole map and then using
>> r.mapcalc to clip (i need the space in my hard disk)

trying harder I can reproduce the issue (of course not NULL related).

So, I have no clue right now why it behaves differently.

markusN
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Metz-3


On Sun, Apr 15, 2018 at 10:49 PM, Markus Neteler <[hidden email]> wrote:

>
> On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]> wrote:
> > On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:
> >> Hi devs,
> >>
> >> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
> >> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
> >> the data, most of it is nodata, while the full map of Africa has data all
> >> over the continent. Then, I tried also with r.import extent=region. Same
> >> result. The only thing providing what I expected to be the right result (a
> >> subset from the full map) is importing the whole map and then using
> >> r.mapcalc to clip (i need the space in my hard disk)
>
> trying harder I can reproduce the issue (of course not NULL related).
>
> So, I have no clue right now why it behaves differently.

fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug in r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.

Markus M

>
> markusN
> _______________________________________________
> grass-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/grass-dev


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Martin Landa
Hi,

2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
> fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug in
> r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.

good timing, we are close to RC1 [1]. Ma

[1] https://trac.osgeo.org/grass/milestone/7.4.1

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Metz-3


On Mon, Apr 16, 2018 at 11:35 AM, Martin Landa <[hidden email]> wrote:

>
> Hi,
>
> 2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
> > fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug in
> > r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.
>
> good timing, we are close to RC1 [1]. Ma
>
> [1] https://trac.osgeo.org/grass/milestone/7.4.1

Mabe a new bugfix release for 7.2 should also go out.

Markus M

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

NikosAlexandris
In reply to this post by Markus Neteler
* Markus Neteler <[hidden email]> [2018-04-15 22:49:09 +0200]:

>On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]> wrote:
>> On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:
>>> Hi devs,
>>>
>>> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
>>> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
>>> the data, most of it is nodata, while the full map of Africa has data all
>>> over the continent. Then, I tried also with r.import extent=region. Same
>>> result. The only thing providing what I expected to be the right result (a
>>> subset from the full map) is importing the whole map and then using
>>> r.mapcalc to clip (i need the space in my hard disk)
>
>trying harder I can reproduce the issue (of course not NULL related).
>
>So, I have no clue right now why it behaves differently.
Hi,

no intention to add noise. There is something weird, here too.

# first things first
grass -c /geo/geodata/examples/chirps-v2.0.2018.01.01.tif /geo/grassdb/onsight/chirps

# importing the whole map looks visually ok
r.in.gdal input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif output=chirps_v2

# then, using the following box coordinates for Ghana from
# https://en.wikipedia.org/wiki/User%3aThe_Anome/country_bounding_boxes
g.region w=-4.000 e=4.733 n=20.000 s=11.167 -p

# to draw the box on-display
v.in.region output=ghana_box

# and doing a partial import
r.in.gdal -r input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif output=chirps_v2_subset

Regardless whether the box figures are not correct, while the imported
portion is, indeed, a box fraction of the whole map, it isn't, visually
the "right" part. It appears as a part from the coast has been imported.

I am double-checking the steps I followed, I always get the same.

?

Nikos

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev

chirps-v2.0.2018.01.01_r.in.gdal.png (42K) Download Attachment
chirps-v2.0.2018.01.01_r.in.gdal_minus_r_over_ghana.png (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Metz-3


On Mon, Apr 16, 2018 at 12:28 PM, Nikos Alexandris <[hidden email]> wrote:

>
> * Markus Neteler <[hidden email]> [2018-04-15 22:49:09 +0200]:
>
>> On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]> wrote:
>>>
>>> On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:
>>>>
>>>> Hi devs,
>>>>
>>>> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
>>>> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
>>>> the data, most of it is nodata, while the full map of Africa has data all
>>>> over the continent. Then, I tried also with r.import extent=region. Same
>>>> result. The only thing providing what I expected to be the right result (a
>>>> subset from the full map) is importing the whole map and then using
>>>> r.mapcalc to clip (i need the space in my hard disk)
>>
>>
>> trying harder I can reproduce the issue (of course not NULL related).
>>
>> So, I have no clue right now why it behaves differently.
>
>
> Hi,
>
> no intention to add noise. There is something weird, here too.
>
> # first things first
> grass -c /geo/geodata/examples/chirps-v2.0.2018.01.01.tif /geo/grassdb/onsight/chirps
>
> # importing the whole map looks visually ok
> r.in.gdal input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif output=chirps_v2
>
> # then, using the following box coordinates for Ghana from
> # https://en.wikipedia.org/wiki/User%3aThe_Anome/country_bounding_boxes
> g.region w=-4.000 e=4.733 n=20.000 s=11.167 -p

This bbox is north of Ghana, not Ghana.

>
> # to draw the box on-display
> v.in.region output=ghana_box
>
> # and doing a partial import
> r.in.gdal -r input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif output=chirps_v2_subset
>
> Regardless whether the box figures are not correct, while the imported
> portion is, indeed, a box fraction of the whole map, it isn't, visually
> the "right" part. It appears as a part from the coast has been imported.

Apparently you did not update your local copy of trunk or relbr74, see my previous post.

Markus M


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

NikosAlexandris
* Markus Metz <[hidden email]> [2018-04-16 14:00:24 +0200]:

>On Mon, Apr 16, 2018 at 12:28 PM, Nikos Alexandris <[hidden email]>
>wrote:
>>
>> * Markus Neteler <[hidden email]> [2018-04-15 22:49:09 +0200]:
>>
>>> On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]>
>wrote:
>>>>
>>>> On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]>
>wrote:
>>>>>
>>>>> Hi devs,
>>>>>
>>>>> I just found something odd. I have a CHIRPS tif for Africa [0], but I
>would
>>>>> only need to import Ghana, so I used r.in.gdal -r. However when I
>visualize
>>>>> the data, most of it is nodata, while the full map of Africa has data
>all
>>>>> over the continent. Then, I tried also with r.import extent=region.
>Same
>>>>> result. The only thing providing what I expected to be the right
>result (a
>>>>> subset from the full map) is importing the whole map and then using
>>>>> r.mapcalc to clip (i need the space in my hard disk)
>>>
>>>
>>> trying harder I can reproduce the issue (of course not NULL related).
>>>
>>> So, I have no clue right now why it behaves differently.
>>
>>
>> Hi,
>>
>> no intention to add noise. There is something weird, here too.
>>
>> # first things first
>> grass -c /geo/geodata/examples/chirps-v2.0.2018.01.01.tif
>/geo/grassdb/onsight/chirps
>>
>> # importing the whole map looks visually ok
>> r.in.gdal input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif
>output=chirps_v2
>>
>> # then, using the following box coordinates for Ghana from
>> # https://en.wikipedia.org/wiki/User%3aThe_Anome/country_bounding_boxes
>> g.region w=-4.000 e=4.733 n=20.000 s=11.167 -p
>
>This bbox is north of Ghana, not Ghana.
>
>>
>> # to draw the box on-display
>> v.in.region output=ghana_box
>>
>> # and doing a partial import
>> r.in.gdal -r input=/geo/geodata/examples/chirps-v2.0.2018.01.01.tif
>output=chirps_v2_subset
>>
>> Regardless whether the box figures are not correct, while the imported
>> portion is, indeed, a box fraction of the whole map, it isn't, visually
>> the "right" part. It appears as a part from the coast has been imported.
>
>Apparently you did not update your local copy of trunk or relbr74, see my
>previous post.
>
>Markus M

Early test performed with GRASS 7.5.svn (2018), libgis Revision: 72327,
libgis Date: 2018-03-06 12:12:44 +0100 (Tue, 06 Mar 2018).

Comfirmed, buggy behaviour is away using latest trunk.

Nikos

_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Neteler
In reply to this post by Markus Metz-3
On Mon, Apr 16, 2018 at 11:39 AM, Markus Metz
<[hidden email]> wrote:

> On Mon, Apr 16, 2018 at 11:35 AM, Martin Landa <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> 2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
>> > fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug
>> > in
>> > r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.
>>
>> good timing, we are close to RC1 [1]. Ma
>>
>> [1] https://trac.osgeo.org/grass/milestone/7.4.1
>
> Mabe a new bugfix release for 7.2 should also go out.

Does the bug(fix) affect all imports made with -r or only some under
"special" (whatever that is) conditions?

markusN
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Metz-3


On Mon, Apr 16, 2018 at 5:23 PM, Markus Neteler <[hidden email]> wrote:

>
> On Mon, Apr 16, 2018 at 11:39 AM, Markus Metz
> <[hidden email]> wrote:
> > On Mon, Apr 16, 2018 at 11:35 AM, Martin Landa <[hidden email]>
> > wrote:
> >>
> >> Hi,
> >>
> >> 2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
> >> > fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug
> >> > in
> >> > r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.
> >>
> >> good timing, we are close to RC1 [1]. Ma
> >>
> >> [1] https://trac.osgeo.org/grass/milestone/7.4.1
> >
> > Mabe a new bugfix release for 7.2 should also go out.
>
> Does the bug(fix) affect all imports made with -r or only some under
> "special" (whatever that is) conditions?

this affects only r.in.gdal, and only in G7.4+. G7.2 is not affected because there is no -r flag for r.in.gdal.

Markus M
>
> markusN


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Neteler
In reply to this post by Markus Metz-3
On Mon, Apr 16, 2018 at 11:39 AM, Markus Metz
<[hidden email]> wrote:

> On Mon, Apr 16, 2018 at 11:35 AM, Martin Landa <[hidden email]>
> wrote:
>> 2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
>> > fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug
>> > in
>> > r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.
>>
>> good timing, we are close to RC1 [1]. Ma
>>
>> [1] https://trac.osgeo.org/grass/milestone/7.4.1
>
> Mabe a new bugfix release for 7.2 should also go out.

Why 7.2? I thought is is not affected...?

markusN
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Markus Metz-3


On Mon, Apr 16, 2018 at 6:46 PM, Markus Neteler <[hidden email]> wrote:

>
> On Mon, Apr 16, 2018 at 11:39 AM, Markus Metz
> <[hidden email]> wrote:
> > On Mon, Apr 16, 2018 at 11:35 AM, Martin Landa <[hidden email]>
> > wrote:
> >> 2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:
> >> > fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug
> >> > in
> >> > r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.
> >>
> >> good timing, we are close to RC1 [1]. Ma
> >>
> >> [1] https://trac.osgeo.org/grass/milestone/7.4.1
> >
> > Mabe a new bugfix release for 7.2 should also go out.
>
> Why 7.2? I thought is is not affected...?

You are right, this affects only r.in.gdal, and only in G7.4+. G7.2 is not affected because there is no -r flag for r.in.gdal (see my previous post).

Markus M
>
> markusN


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: r.in.gdal -r and r.import extent=region not working properly?

Veronica Andreo
In reply to this post by Markus Metz-3
Dear Markus, 

Thanks much for such a quick fix!!! Awesome! :) 

Cheers,
Vero

2018-04-16 11:32 GMT+02:00 Markus Metz <[hidden email]>:


On Sun, Apr 15, 2018 at 10:49 PM, Markus Neteler <[hidden email]> wrote:

>
> On Sun, Apr 15, 2018 at 9:46 PM, Markus Neteler <[hidden email]> wrote:
> > On Sun, Apr 15, 2018 at 8:48 PM, Veronica Andreo <[hidden email]> wrote:
> >> Hi devs,
> >>
> >> I just found something odd. I have a CHIRPS tif for Africa [0], but I would
> >> only need to import Ghana, so I used r.in.gdal -r. However when I visualize
> >> the data, most of it is nodata, while the full map of Africa has data all
> >> over the continent. Then, I tried also with r.import extent=region. Same
> >> result. The only thing providing what I expected to be the right result (a
> >> subset from the full map) is importing the whole map and then using
> >> r.mapcalc to clip (i need the space in my hard disk)
>
> trying harder I can reproduce the issue (of course not NULL related).
>
> So, I have no clue right now why it behaves differently.

fixed in trunk r72620 and relbr74 r72621. That was a rather serious bug in r.in.gdal -r, therefore IMHO a new GRASS 7.4 release should go out asap.

Markus M

>
> markusN
> _______________________________________________
> grass-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/grass-dev



_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev