[gdal-dev] How to use RPC Metadata from NITF Files in a MapServer TileIndex

[Smith, Michael ERDC-RDE-CRREL-NH]

I have a large number of NITF JP2K files. These files have IGEOLO metadata to the second accuracy so reprojecting these images leads to poor results. The images also have rpc metadata and using the -rpc option in gdalwarp reprojects them quite well. These files are source epsg:4326 and need to be reprojected to epsg:3857 for display. 

I had orginally thought to just create a vrt for each ntf file with the rpc information but a tileindex of the vrts is dramatically slower to display in MapServer (for a 1:8000 scale area, from ~ 1 sec to over 10 min). It does reproject correctly though. Obviously not practical.

I then thought to reproject the vrts to an actual format (geotiff). The problem is that the black bounds (which should be nodata but are just near black areas) get expanded and aren't handled even when using the nearblack utility on the resulting geotiffs. 

The best option (if possible) would seem to use the source nitf files but force gdal to use the rpc information to reproject (without having to go through the vrt process). Perhaps a CPL config option that could be set inside mapserver?

Other suggestions are welcome.

Mike

-- 

Michael Smith

[hidden email]

US Army Corps

Remote Sensing GIS/Center

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Even Rouault]

Le dimanche 17 juin 2012 18:24:29, Smith, Michael ERDC-RDE-CRREL-NH a écrit :

> I have a large number of NITF JP2K files. These files have IGEOLO metadata
> to the second accuracy so reprojecting these images leads to poor results.
> The images also have rpc metadata and using the -rpc option in gdalwarp
> reprojects them quite well. These files are source epsg:4326 and need to
> be reprojected to epsg:3857 for display.
>
> I had orginally thought to just create a vrt for each ntf file with the rpc
> information but a tileindex of the vrts is dramatically slower to display
> in MapServer (for a 1:8000 scale area, from ~ 1 sec to over 10 min). It
> does reproject correctly though. Obviously not practical.
>
> I then thought to reproject the vrts to an actual format (geotiff). The
> problem is that the black bounds (which should be nodata but are just near
> black areas) get expanded and aren't handled even when using the nearblack
> utility on the resulting geotiffs.
>
> The best option (if possible) would seem to use the source nitf files but
> force gdal to use the rpc information to reproject (without having to go
> through the vrt process). Perhaps a CPL config option that could be set
> inside mapserver?

Did you use the -dstalpha option of gdalwarp, so that the areas in the
reprojected image that are not part of the source image are marked with a 0
alpha value ? -dstnodata could also do it.

>
> Other suggestions are welcome.
>
> Mike
>
> --
> Michael Smith
> [hidden email]
> US Army Corps
> Remote Sensing GIS/Center

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Smith, Michael ERDC-RDE-CRREL-NH]

I did originally use -dstalpha. I had dropped it when creating the
geotiffs as I used PHOTOMETRIC=YCBCR as a creation option.

I'll try with -dstalpha and dropping the YCBCR.

Is there a known issue with using the vrts inside a tileindex that makes
it so slow?

--

This is what I used to convert to tif

gdalwarp -rpc -of vrt -overwrite  -t_srs epsg:3857 -r cubic --config
GDAL_CACHEMAX 500 -wm 500 $1 $1.vrt
   gdal_translate -of GTiff -co "TILED=YES" -co "JPEG_QUALITY=95" -co
"COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" $1.vrt $1.tif
   gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
YCBCR --config INTERLEAVE_OVERVIEW PIXEL $1.tif 2 4 8 16 32


Mike

On 6/17/12 12:38 PM, "Even Rouault" <[hidden email]> wrote:

>Le dimanche 17 juin 2012 18:24:29, Smith, Michael ERDC-RDE-CRREL-NH a
>écrit :
>> I have a large number of NITF JP2K files. These files have IGEOLO
>>metadata
>> to the second accuracy so reprojecting these images leads to poor
>>results.
>> The images also have rpc metadata and using the -rpc option in gdalwarp
>> reprojects them quite well. These files are source epsg:4326 and need to
>> be reprojected to epsg:3857 for display.
>>
>> I had orginally thought to just create a vrt for each ntf file with the
>>rpc
>> information but a tileindex of the vrts is dramatically slower to
>>display
>> in MapServer (for a 1:8000 scale area, from ~ 1 sec to over 10 min). It
>> does reproject correctly though. Obviously not practical.
>>
>> I then thought to reproject the vrts to an actual format (geotiff). The
>> problem is that the black bounds (which should be nodata but are just
>>near
>> black areas) get expanded and aren't handled even when using the
>>nearblack
>> utility on the resulting geotiffs.
>>
>> The best option (if possible) would seem to use the source nitf files
>>but
>> force gdal to use the rpc information to reproject (without having to go
>> through the vrt process). Perhaps a CPL config option that could be set
>> inside mapserver?
>
>Did you use the -dstalpha option of gdalwarp, so that the areas in the
>reprojected image that are not part of the source image are marked with a
>0
>alpha value ? -dstnodata could also do it.
>
>>
>> Other suggestions are welcome.
>>
>> Mike
>>
>> --
>> Michael Smith
>> [hidden email]
>> US Army Corps
>> Remote Sensing GIS/Center

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Even Rouault]

Le dimanche 17 juin 2012 18:45:01, Smith, Michael ERDC-RDE-CRREL-NH a écrit :
> I did originally use -dstalpha. I had dropped it when creating the
> geotiffs as I used PHOTOMETRIC=YCBCR as a creation option.
>
> I'll try with -dstalpha and dropping the YCBCR.

Ah, ok, JPEG compression is indeed incompatible with alpha band. But you could
however use a mask band, which recent MapServer should handle.

gdalwarp rpc -of vrt -overwrite  -t_srs epsg:3857 -r cubic --config
GDAL_CACHEMAX 500 -wm 500 $1 $1.vrt -dstalpha

gdal_translate -b 1 -b 2 -b 3 -mask 4  -co "TILED=YES" -co "JPEG_QUALITY=95" -
co "COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" $1.vrt $1.tif

gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
YCBCR --config INTERLEAVE_OVERVIEW PIXEL $1.tif 2 4 8 16 32

>
> Is there a known issue with using the vrts inside a tileindex that makes
> it so slow?

RPC reprojection is probably slow, and VRT can add some overhead, especially
if the tiling scheme it adopts (128x128 blocks) is very different from the
source raster. Furthermore, if you make a request that is a zoom out, the VRT
cannot benefit from overviews of the source raster and will make reprojection
at full scale, before zooming out.
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Even Rouault]

Actually, you'll need to add "--config GDAL_TIFF_INTERNAL_MASK YES" to the 2
following commands :

gdal_translate -b 1 -b 2 -b 3 -mask 4  -co "TILED=YES" -co
 "JPEG_QUALITY=95" - co "COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" $1.vrt
$1.tif --config GDAL_TIFF_INTERNAL_MASK YES

gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
YCBCR --config INTERLEAVE_OVERVIEW PIXEL $1.tif 2 4 8 16 32 --config
GDAL_TIFF_INTERNAL_MASK YES

Otherwise, an external .msk file will be produced. And you'll need to run
gdaladdo on it too.
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Smith, Michael ERDC-RDE-CRREL-NH]

Thanks Even. This is working well.

Mike

--
Michael Smith

US Army Corps
Remote Sensing GIS/Center



On 6/17/12 1:10 PM, "Even Rouault" <[hidden email]> wrote:

>Actually, you'll need to add "--config GDAL_TIFF_INTERNAL_MASK YES" to
>the 2
>following commands :
>
>gdal_translate -b 1 -b 2 -b 3 -mask 4  -co "TILED=YES" -co
> "JPEG_QUALITY=95" - co "COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" $1.vrt
>$1.tif --config GDAL_TIFF_INTERNAL_MASK YES
>
>gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
>YCBCR --config INTERLEAVE_OVERVIEW PIXEL $1.tif 2 4 8 16 32 --config
>GDAL_TIFF_INTERNAL_MASK YES
>
>Otherwise, an external .msk file will be produced. And you'll need to run
>gdaladdo on it too.

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev