[gdal-dev] Need some help understanding why this raster is so big

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Need some help understanding why this raster is so big

Brian-2
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Cainã K. Campos
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Brian-2
Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? 

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <[hidden email]> wrote:
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Michael Sumner-2
What is the source (and target) projection and extent? Some projections have expansive limits, but you can specify the target extent with -te

HTH

On Fri., 13 Mar. 2020, 23:43 Brian, <[hidden email]> wrote:
Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? 

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <[hidden email]> wrote:
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Brian-2
Here is the gdalinfo output I am just re-projecting to 4326

Driver: VRT/Virtual Raster
Files: Depth (Max).vrt
       Depth (Max).Terrainredacted.tif
       Depth (Max). redacted .tif
Size is 309153, 277451
Coordinate System is:
PROJCS["USA_Contiguous_Albers_Equal_Area_Conic_USGS_version",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.2572221010042,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Albers_Conic_Equal_Area"],
    PARAMETER["standard_parallel_1",29.5],
    PARAMETER["standard_parallel_2",45.5],
    PARAMETER["latitude_of_center",23],
    PARAMETER["longitude_of_center",-96],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["US survey foot",0.3048006096012192,
        AUTHORITY["EPSG","9003"]]]
Origin = (2771031.965285023674369,6437561.791629494167864)
Pixel Size = (3.000000000000000,-3.000000000000000)
Corner Coordinates:
Upper Left  ( 2771031.965, 6437561.792) ( 85d57'44.50"W, 40d16'21.58"N)
Lower Left  ( 2771031.965, 5605208.792) ( 86d16'11.82"W, 38d 1'16.13"N)
Upper Right ( 3698490.965, 6437561.792) ( 82d38'29.14"W, 39d57'46.13"N)
Lower Right ( 3698490.965, 5605208.792) ( 83d 2'54.73"W, 37d43'14.99"N)
Center      ( 3234761.465, 6021385.292) ( 84d28'47.64"W, 39d 0'18.59"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  Min=0.001 Max=96.054
  Minimum=0.001, Maximum=96.054, Mean=6.209, StdDev=10.756
  NoData Value=-9999
  Metadata:
    STATISTICS_MAXIMUM=96.0537109375
    STATISTICS_MEAN=6.2090953579112
    STATISTICS_MINIMUM=0.0009765625
    STATISTICS_STDDEV=10.756049212573

On Fri, Mar 13, 2020 at 8:53 AM Michael Sumner <[hidden email]> wrote:
What is the source (and target) projection and extent? Some projections have expansive limits, but you can specify the target extent with -te

HTH

On Fri., 13 Mar. 2020, 23:43 Brian, <[hidden email]> wrote:
Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? 

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <[hidden email]> wrote:
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Cainã K. Campos
In reply to this post by Brian-2
if you run without the switch it will create the result without compression, so you will be back to a 400Gb file. Instead run everything in one single command.
With compression it is likely to take some extra time, as some calculation has to be done to achieve that.
If you want speed, tile the raster and also there are a few other switches to add so gdalwarp can use more than one core to make this calcs, and you can also too give more memory to it like:
--config GDAL_SWATH_SIZE 2000000000 --config GDAL_CACHEMAX 2000 -wm 2000 -co "tiled=yes" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -wo "NUM_THREADS=ALL_CPUS" -multi -co "NUM_THREADS=ALL_CPUS" -co "bigtiff=yes"


On Fri, Mar 13, 2020 at 9:43 AM Brian <[hidden email]> wrote:
Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? 

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <[hidden email]> wrote:
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

Daniel Evans-2

Given the filesize of the original raster, I suspect it’s mostly empty, and so I’d suggest also specifying:

 

-co “SPARSE_OK=TRUE”

 

That instructs GDAL to avoid writing any data at all where the entirety of a tile is nodata, rather than writing a (compressed) block of nodata values. However, the GDAL documentation does warn that this can cause trouble with external software.

 

 

From: gdal-dev <[hidden email]> On Behalf Of Cainã K. Campos
Sent: 13 March 2020 13:08
To: Brian <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] Need some help understanding why this raster is so big

 

if you run without the switch it will create the result without compression, so you will be back to a 400Gb file. Instead run everything in one single command.

With compression it is likely to take some extra time, as some calculation has to be done to achieve that.

If you want speed, tile the raster and also there are a few other switches to add so gdalwarp can use more than one core to make this calcs, and you can also too give more memory to it like:

--config GDAL_SWATH_SIZE 2000000000 --config GDAL_CACHEMAX 2000 -wm 2000 -co "tiled=yes" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -wo "NUM_THREADS=ALL_CPUS" -multi -co "NUM_THREADS=ALL_CPUS" -co "bigtiff=yes"

 

 

On Fri, Mar 13, 2020 at 9:43 AM Brian <[hidden email]> wrote:

Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? 

 

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <[hidden email]> wrote:

Hello Brian,

 

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

 

On Fri, Mar 13, 2020 at 9:07 AM Brian <[hidden email]> wrote:

So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

T +44 (0) 1756 799919
www.jbarisk.com

Visit our website  Follow us on Twitter

Our postal address and registered office is JBA Risk Management Limited, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire BD23 3FD.

Find out more about us here: www.jbarisk.com and follow us on Twitter @JBARisk and LinkedIn

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.
 

JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919

 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Need some help understanding why this raster is so big

jratike80
In reply to this post by Brian-2
Hi,

You asked help for understanding, so

Check the size of the original image:
309153 x 277451
Check how many bands:
1
Check how many bits per pixel for each band:
32 bits (makes 4 bytes)

Multiply:
309153 x 277451 x 4 = number of bytes as uncompressed = 3.43099E+11 = 343 GB

Compare with the 120 MB that you have and make a conclusion that the
original is well compressed.

-Jukka Rahkonen-



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev