Serving SRTM1 data with GeoServer

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Serving SRTM1 data with GeoServer

Barry Carter
I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and data
using GeoServer, but can't figure out how to serve such a large dataset
efficiently. Here's what I've done so far:

   - Downloaded the 1 degree squared HGT files. The Earth is 360*180 = 64800
     square degrees, but only 14295 HGT files since the other square degrees
     are 100% water. The HGT files are uncompressed and 25,934,402 bytes each.

   - I then used commands like:

gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt N58W122.hgt.tif

to convert these HGT files into TIFFs. The TIFFs vary in size, but are much
smaller.

   - I then imported the directory with the TIFFs as a store using
     ImageMosaic. This took a while, but appears to have worked, since "srtm1"
     (the name I gave it) now appears in my list of stores.

   - I then created a layer using this source, but when I use "OpenLayers"
     preview, the server hangs for a while and ultimately dies with "too many
     open files" (or similar). This seems odd, since my Linux OS had
     fs.file-max set to 13063650. I even tried doubling it to 26127300, but
     that didn't help.

At this point, I suspect I need to create an image pyramid or something so
GeoServer can handle the data efficiently, but I'm not sure exactly what to
do. I've used 'gdal_retile.py' previously to create pyramids from one large
image into tiles at various zoom levels, but I think I want to do the opposite
here (merge smaller images for lower zoom levels). The gdal_retile.py option
-pyramidsonly seems to say it does that, but I can't get it work (it just
hangs and creates no files)



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Serving SRTM1 data with GeoServer

PeterSm
Hi Barry

I have recently done something similar and had a slightly different problem, but still related to too many files, with GeoServer performing very slowly when zoomed out.  Our solution, which works very well, was to create a layer group with the original (TIFF) ImageMosaic store from scale 50000 - 2600000 and another top layer ImageMosaic store from scale 2600000 - 400000000. 

This top layer was created by downsampling each DEM image to 6.25% with gdal_translate and then combining them into 16x16 degree tiles with gdalwarp.  The same gdaldem hillshade command can then be run on these top layer tiles.

By splitting the layer like this, the number of files being accessed was reduced to a level that GeoServer can very easily manage and it works perfectly for us.  The top layer does not consume too much extra disk space.

I'm not sure if this is the best way to do it, so I'm always willing to hear from others if there's a better solution.

Regards

Peter
AfriGIS

On Tue, 21 May 2019 at 04:18, Barry Carter <[hidden email]> wrote:
I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and data
using GeoServer, but can't figure out how to serve such a large dataset
efficiently. Here's what I've done so far:

   - Downloaded the 1 degree squared HGT files. The Earth is 360*180 = 64800
     square degrees, but only 14295 HGT files since the other square degrees
     are 100% water. The HGT files are uncompressed and 25,934,402 bytes each.

   - I then used commands like:

gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt N58W122.hgt.tif

to convert these HGT files into TIFFs. The TIFFs vary in size, but are much
smaller.

   - I then imported the directory with the TIFFs as a store using
     ImageMosaic. This took a while, but appears to have worked, since "srtm1"
     (the name I gave it) now appears in my list of stores.

   - I then created a layer using this source, but when I use "OpenLayers"
     preview, the server hangs for a while and ultimately dies with "too many
     open files" (or similar). This seems odd, since my Linux OS had
     fs.file-max set to 13063650. I even tried doubling it to 26127300, but
     that didn't help.

At this point, I suspect I need to create an image pyramid or something so
GeoServer can handle the data efficiently, but I'm not sure exactly what to
do. I've used 'gdal_retile.py' previously to create pyramids from one large
image into tiles at various zoom levels, but I think I want to do the opposite
here (merge smaller images for lower zoom levels). The gdal_retile.py option
-pyramidsonly seems to say it does that, but I can't get it work (it just
hangs and creates no files)



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Serving SRTM1 data with GeoServer

Ian Turton
In reply to this post by Barry Carter
Have a look at https://www.ianturton.com/tutorials/bluemarble.html where I have example scripts and work flow to create a base image from a set of tiles and how to make a pyramid from it.

Ian

On Tue, 21 May 2019 at 03:16, Barry Carter <[hidden email]> wrote:
I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and data
using GeoServer, but can't figure out how to serve such a large dataset
efficiently. Here's what I've done so far:

   - Downloaded the 1 degree squared HGT files. The Earth is 360*180 = 64800
     square degrees, but only 14295 HGT files since the other square degrees
     are 100% water. The HGT files are uncompressed and 25,934,402 bytes each.

   - I then used commands like:

gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt N58W122.hgt.tif

to convert these HGT files into TIFFs. The TIFFs vary in size, but are much
smaller.

   - I then imported the directory with the TIFFs as a store using
     ImageMosaic. This took a while, but appears to have worked, since "srtm1"
     (the name I gave it) now appears in my list of stores.

   - I then created a layer using this source, but when I use "OpenLayers"
     preview, the server hangs for a while and ultimately dies with "too many
     open files" (or similar). This seems odd, since my Linux OS had
     fs.file-max set to 13063650. I even tried doubling it to 26127300, but
     that didn't help.

At this point, I suspect I need to create an image pyramid or something so
GeoServer can handle the data efficiently, but I'm not sure exactly what to
do. I've used 'gdal_retile.py' previously to create pyramids from one large
image into tiles at various zoom levels, but I think I want to do the opposite
here (merge smaller images for lower zoom levels). The gdal_retile.py option
-pyramidsonly seems to say it does that, but I can't get it work (it just
hangs and creates no files)



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--
Ian Turton


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Serving SRTM1 data with GeoServer

Barry Carter-2
Thank you, Ian (and Peter).

I had actually considered using gdal_merge.py to combine the GeoTiffs I'd
created, but thought the resulting mega-TIFF would be too large. However:

gdal_merge.py -o elevation.tif -co BIGTIFF=YES -co TILED=YES -co
  COMPRESS=DEFLATE *.hgt.tif

worked fine. GeoServer accepted this store and maps it slowly, but without
generating errors.

For reference, the resulting TIFF, elevation.tif is just over 100
gigabytes in size, and has a resolution of 1296001 x 417601, which is
slightly less than I expected because it turns out SRTM1 doesn't cover the
entire world.

On Tue, 21 May 2019, Ian Turton wrote:

> Date: Tue, 21 May 2019 08:18:23 +0100
> From: Ian Turton <[hidden email]>
> To: Barry Carter <[hidden email]>
> Cc: geoserver-users <[hidden email]>
> Subject: Re: [Geoserver-users] Serving SRTM1 data with GeoServer
>
> Have a look at https://www.ianturton.com/tutorials/bluemarble.html where I
> have example scripts and work flow to create a base image from a set of
> tiles and how to make a pyramid from it.
>
> Ian
>
> On Tue, 21 May 2019 at 03:16, Barry Carter <[hidden email]>
> wrote:
>
>> I'm trying to provide SRTM1 (1 arcsecond resolution elevation) images and
>> data
>> using GeoServer, but can't figure out how to serve such a large dataset
>> efficiently. Here's what I've done so far:
>>
>>    - Downloaded the 1 degree squared HGT files. The Earth is 360*180 =
>> 64800
>>      square degrees, but only 14295 HGT files since the other square
>> degrees
>>      are 100% water. The HGT files are uncompressed and 25,934,402 bytes
>> each.
>>
>>    - I then used commands like:
>>
>> gdal_translate -co TILED=YES -co COMPRESS=DEFLATE N58W122.hgt
>> N58W122.hgt.tif
>>
>> to convert these HGT files into TIFFs. The TIFFs vary in size, but are much
>> smaller.
>>
>>    - I then imported the directory with the TIFFs as a store using
>>      ImageMosaic. This took a while, but appears to have worked, since
>> "srtm1"
>>      (the name I gave it) now appears in my list of stores.
>>
>>    - I then created a layer using this source, but when I use "OpenLayers"
>>      preview, the server hangs for a while and ultimately dies with "too
>> many
>>      open files" (or similar). This seems odd, since my Linux OS had
>>      fs.file-max set to 13063650. I even tried doubling it to 26127300, but
>>      that didn't help.
>>
>> At this point, I suspect I need to create an image pyramid or something so
>> GeoServer can handle the data efficiently, but I'm not sure exactly what to
>> do. I've used 'gdal_retile.py' previously to create pyramids from one large
>> image into tiles at various zoom levels, but I think I want to do the
>> opposite
>> here (merge smaller images for lower zoom levels). The gdal_retile.py
>> option
>> -pyramidsonly seems to say it does that, but I can't get it work (it just
>> hangs and creates no files)
>>
>>
>>
>> _______________________________________________
>> Geoserver-users mailing list
>>
>> Please make sure you read the following two resources before posting to
>> this list:
>> - Earning your support instead of buying it, but Ian Turton:
>> http://www.ianturton.com/talks/foss4g.html#/
>> - The GeoServer user list posting guidelines:
>> http://geoserver.org/comm/userlist-guidelines.html
>>
>> If you want to request a feature or an improvement, also see this:
>> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
>>
>>
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>
>
>
> --
> Ian Turton
>



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users