[gdal-dev] GDALCreateWarpedVRT and .msk files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] GDALCreateWarpedVRT and .msk files

Sean Gillies-3

Hi all,

I've noticed that while GDALCreateWarpedVRT does acquire nodata values, and thereby nodata-derived masks, from the source dataset, it does not pick up on a source's .msk file and does not acquire a source mask derived from that sidecar file.

Is this an oversight or intended behavior? Is there a best practice for adding a .msk derived mask to the VRT?

Thanks,

--

Sean Gillies


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Even Rouault-2

Hi Sean

 

>

> I've noticed that while GDALCreateWarpedVRT does acquire nodata values, and

> thereby nodata-derived masks,

 

I see provision for that in GDALAutoCreateWarpedVRT(), not in GDALCreateWarpedVRT()

 

> from the source dataset, it does not pick up

> on a source's .msk file and does not acquire a source mask derived from

> that sidecar file.

 

I haven't tested in the warped VRT case, but I believe that should normally be automatically done by

https://github.com/OSGeo/gdal/blob/trunk/gdal/alg/gdalwarpoperation.cpp#L2022

 

Are you sure the .msk is properly recognized as a GDAL mask dataset by GDAL ? For example when ou do gdalinfo on the source dataset. There is a compulsory metadata item that must be set on the .msk. See

"For an external .msk file to be recognized by GDAL," in https://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Sean Gillies-3
Hi Even,

On Mon, Feb 19, 2018 at 5:42 AM, Even Rouault <[hidden email]> wrote:

Hi Sean

 

>

> I've noticed that while GDALCreateWarpedVRT does acquire nodata values, and

> thereby nodata-derived masks,

 

I see provision for that in GDALAutoCreateWarpedVRT(), not in GDALCreateWarpedVRT()


Yes, sorry, I meant to write GDALAutoCreateWarpedVRT().
 

 

> from the source dataset, it does not pick up

> on a source's .msk file and does not acquire a source mask derived from

> that sidecar file.

 

I haven't tested in the warped VRT case, but I believe that should normally be automatically done by

https://github.com/OSGeo/gdal/blob/trunk/gdal/alg/gdalwarpoperation.cpp#L2022

 

Are you sure the .msk is properly recognized as a GDAL mask dataset by GDAL ? For example when ou do gdalinfo on the source dataset. There is a compulsory metadata item that must be set on the .msk. See

"For an external .msk file to be recognized by GDAL," in https://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask


Yes, gdalinfo detects the mask:

Driver: GTiff/GeoTIFF
Size is 48948, 15632
...
Band 1 Block=512x512 Type=Byte, ColorInterp=Red
  Overviews: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Mask Flags: PER_DATASET
  Overviews of mask band: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Unit Type: metre
Band 2 Block=512x512 Type=Byte, ColorInterp=Green
  Overviews: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Mask Flags: PER_DATASET
  Overviews of mask band: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Unit Type: metre
Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
  Overviews: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Mask Flags: PER_DATASET
  Overviews of mask band: 24474x7816, 12237x3908, 6119x1954, 3060x977, 1530x489, 765x245, 383x123
  Unit Type: metre

I'm using GDAL 2.2.2.

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Even Rouault-2

Sean,

 

I've just tested with GDAL trunk and latest state of 2.2 branch (but I think that 2.2.2 should behave the same), and when using gdal.AutoCreateWarpedVRT() on your below mentionned dataset, the .msk file is read.

 

$ CPL_CURL_VERBOSE=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif,msk FILE=/vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif python

[...]

>>> from osgeo import gdal

>> import os

>>> file = os.getenv('FILE')

>>> src_ds = gdal.Open(file)

[...]

>>> ds = gdal.AutoCreateWarpedVRT(src_ds)

[...]

* Connected to oin-hotosm.s3.amazonaws.com (52.216.130.219) port 80 (#2)

> GET /5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif.msk HTTP/1.1

Host: oin-hotosm.s3.amazonaws.com

Range: bytes=49152-114687

Accept: */*

 

< HTTP/1.1 206 Partial Content

< x-amz-id-2: oXRmqP1A9HzNTn9soSOh9vQJzGUa+lA0GbsAKa0m51H8wvZCgS31y85NAXu60r3MMjRTz1TssYk=

< x-amz-request-id: BC46544B595F1D30

< Date: Tue, 20 Feb 2018 11:52:54 GMT

< Last-Modified: Thu, 14 Dec 2017 09:58:29 GMT

< ETag: "97b96b954a22be9d7bae0bfd1f33922b"

< Accept-Ranges: bytes

< Content-Range: bytes 49152-114687/1730078

< Content-Type: binary/octet-stream

< Content-Length: 65536

< Server: AmazonS3

 

[...]

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Sean Gillies-3
Hi Even,

I got waylaid in following up, but am finally doing so now. Yes, I can also confirm that the .msk file is being read over the internet. I'm grateful for the CURL logging in GDAL, it's very handy.

However, I'm perplexed by not being able to get the mask I expect from the warped VRT I'm creating using GDALAutoCreateWarpedVRT. The source file has a mask, but no mask materializes in the warped VRT. I have a WarpedVRT class in Rasterio that calls GDALAutoCreateWarpedVRT and is equivalent to the Python GDAL code you showed. It has a method that calls GDALGetMaskFlags and here's what that reports.

with rasterio.open('https://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif') as src, WarpedVRT(src, dst_crs=src.crs) as vrt:
    print(vrt.mask_flag_enums)

# Output:
# ([<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>])

The source has a per-dataset mask, but this doesn't surface in the VRT. It reports that all the pixels are valid.

Is this the intended behavior for VRTs? Are we required to materialize the VRT and create an alpha band to contain the derived mask?

Thanks,

On Tue, Feb 20, 2018 at 4:57 AM, Even Rouault <[hidden email]> wrote:

Sean,

 

I've just tested with GDAL trunk and latest state of 2.2 branch (but I think that 2.2.2 should behave the same), and when using gdal.AutoCreateWarpedVRT() on your below mentionned dataset, the .msk file is read.

 

$ CPL_CURL_VERBOSE=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif,msk FILE=/vsicurl/http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif python

[...]

>>> from osgeo import gdal

>> import os

>>> file = os.getenv('FILE')

>>> src_ds = gdal.Open(file)

[...]

>>> ds = gdal.AutoCreateWarpedVRT(src_ds)

[...]

* Connected to oin-hotosm.s3.amazonaws.com (52.216.130.219) port 80 (#2)

> GET /5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif.msk HTTP/1.1

Host: oin-hotosm.s3.amazonaws.com

Range: bytes=49152-114687

Accept: */*

 

< HTTP/1.1 206 Partial Content

< x-amz-id-2: oXRmqP1A9HzNTn9soSOh9vQJzGUa+lA0GbsAKa0m51H8wvZCgS31y85NAXu60r3MMjRTz1TssYk=

< x-amz-request-id: BC46544B595F1D30

< Date: Tue, 20 Feb 2018 11:52:54 GMT

< Last-Modified: Thu, 14 Dec 2017 09:58:29 GMT

< ETag: "97b96b954a22be9d7bae0bfd1f33922b"

< Accept-Ranges: bytes

< Content-Range: bytes 49152-114687/1730078

< Content-Type: binary/octet-stream

< Content-Length: 65536

< Server: AmazonS3

 

[...]

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Even Rouault-2
On vendredi 1 juin 2018 18:25:27 CEST Sean Gillies wrote:

> Hi Even,
>
> I got waylaid in following up, but am finally doing so now. Yes, I can also
> confirm that the .msk file is being read over the internet. I'm grateful
> for the CURL logging in GDAL, it's very handy.
>
> However, I'm perplexed by not being able to get the mask I expect from the
> warped VRT I'm creating using GDALAutoCreateWarpedVRT. The source file has
> a mask, but no mask materializes in the warped VRT. I have a WarpedVRT
> class in Rasterio that calls GDALAutoCreateWarpedVRT and is equivalent to
> the Python GDAL code you showed. It has a method that calls
> GDALGetMaskFlags and here's what that reports.
>
> with rasterio.open('https://oin-hotosm.s3.amazonaws.com/
> 5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif') as
> src, WarpedVRT(src, dst_crs=src.crs) as vrt:
>     print(vrt.mask_flag_enums)
>
> # Output:
> # ([<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>],
> [<MaskFlags.all_valid: 1>])
>
> The source has a per-dataset mask, but this doesn't surface in the VRT. It
> reports that all the pixels are valid.

Hi Sean,

OK I now understand what you meant. The warper in general in GDAL (so not
specific to the VRT warping) can use the mask of the input dataset, but
doesn't generate one in output. However it should be able to generate an alpha
band if you call GDALCreateWarpedVRT() with psOptions->nDstAlphaBand = 4
(couldn't test easily, but hopefully should work)

Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Sean Gillies-3
On Fri, Jun 1, 2018 at 7:14 PM, Even Rouault <[hidden email]> wrote:
On vendredi 1 juin 2018 18:25:27 CEST Sean Gillies wrote:
> Hi Even,
>
> I got waylaid in following up, but am finally doing so now. Yes, I can also
> confirm that the .msk file is being read over the internet. I'm grateful
> for the CURL logging in GDAL, it's very handy.
>
> However, I'm perplexed by not being able to get the mask I expect from the
> warped VRT I'm creating using GDALAutoCreateWarpedVRT. The source file has
> a mask, but no mask materializes in the warped VRT. I have a WarpedVRT
> class in Rasterio that calls GDALAutoCreateWarpedVRT and is equivalent to
> the Python GDAL code you showed. It has a method that calls
> GDALGetMaskFlags and here's what that reports.
>
> with rasterio.open('https://oin-hotosm.s3.amazonaws.com/
> 5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif') as
> src, WarpedVRT(src, dst_crs=src.crs) as vrt:
>     print(vrt.mask_flag_enums)
>
> # Output:
> # ([<MaskFlags.all_valid: 1>], [<MaskFlags.all_valid: 1>],
> [<MaskFlags.all_valid: 1>])
>
> The source has a per-dataset mask, but this doesn't surface in the VRT. It
> reports that all the pixels are valid.

Hi Sean,

OK I now understand what you meant. The warper in general in GDAL (so not
specific to the VRT warping) can use the mask of the input dataset, but
doesn't generate one in output. However it should be able to generate an alpha
band if you call GDALCreateWarpedVRT() with psOptions->nDstAlphaBand = 4
(couldn't test easily, but hopefully should work)

Even

After setting that warp option and setting the color interpretation of the resulting band to GCI_AlphaBand, I'm in business!

Thanks, Even!

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDALCreateWarpedVRT and .msk files

Even Rouault-2
> After setting that warp option and setting the color interpretation of the
> resulting band to GCI_AlphaBand, I'm in business!

Cool, I've pushed a small change to automatically set the GCI_AlphaBand color
interpretation, and added a hint how to create the alpha band in the doc of
the function.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev