Multi-channel raster reads slow

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Multi-channel raster reads slow

GeoServer - User mailing list
I have a large number of raster layers for known areas of interest with
different measurements for the same area and timestep stored in different
files.  To reduce the number of layers I've concatenated the measurements
for the same area/time into multi-channel layers with different stylesheets
to display each band.  

So let's say I used to have layers
20190114_BlueCarPark_1.tif,20190114_BlueCarPark_2.tif and so on, I now have
a single later 20190114_BlueCarPark.tif that contains each of the old layers
from Blue car Park on 20190114.  These channels are stored band sequentially
to allow channels to be read in continuous chunks.

However, I've found that this is causing WMS rendering to be quite a bit
slower, my guess is that rather than determining which pixels are required,
Geoserver is maybe reading the entire 3-D array from file and then
subsetting down to the channel specified in the stylesheet.  Is that
correct?  

I'm using the default ImageIO library rather than the extension



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Multi-channel raster reads slow

geowolf
Hi,
your assesment is correct, the band selection performed in the symbolizer is currently not pushed down into the GeoTiff reader.
It could be done, the machinery to facilitate is there, just a matter of funding to wire it up with the GeoTiff reader.

For multi-band data we normally use image mosaics with one band per file, and bringing the data back together in memory using a coverage view
(typical use case, Sentinel data, using the JP2K files as they are provided in the original packages).
For that case, band selection is effectively pushed down to the reader level, by avoiding to open the band files that are not needed. This works
at least in 2.14.x, don't remember how far back this functionality got implemented.

Cheers
Andrea

On Mon, Jan 14, 2019 at 5:36 PM ocox via Geoserver-users <[hidden email]> wrote:
I have a large number of raster layers for known areas of interest with
different measurements for the same area and timestep stored in different
files.  To reduce the number of layers I've concatenated the measurements
for the same area/time into multi-channel layers with different stylesheets
to display each band. 

So let's say I used to have layers
20190114_BlueCarPark_1.tif,20190114_BlueCarPark_2.tif and so on, I now have
a single later 20190114_BlueCarPark.tif that contains each of the old layers
from Blue car Park on 20190114.  These channels are stored band sequentially
to allow channels to be read in continuous chunks.

However, I've found that this is causing WMS rendering to be quite a bit
slower, my guess is that rather than determining which pixels are required,
Geoserver is maybe reading the entire 3-D array from file and then
subsetting down to the channel specified in the stylesheet.  Is that
correct? 

I'm using the default ImageIO library rather than the extension



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Multi-channel raster reads slow

GeoServer - User mailing list
Hey Andrea,

That's a great insight, thanks.  In the longer term I will look at
contributing to the reader, but in the short term the ImageMosaic looks a
good idea.
One final question on this, I've generated an Image Mosaic using the
individual bands, but the coverage view only recognises this as containing a
single band.  I've added a 'band' attribute into the indexer, should
something like this be used to configure the coverage view with multiple
bands?


Cheers

<http://osgeo-org.1560.x6.nabble.com/file/t384100/car_park_mosaic.png>



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Multi-channel raster reads slow

geowolf

On Wed, Jan 16, 2019 at 2:43 PM ocox via Geoserver-users <[hidden email]> wrote:
Hey Andrea,

That's a great insight, thanks.  In the longer term I will look at
contributing to the reader, but in the short term the ImageMosaic looks a
good idea.
One final question on this, I've generated an Image Mosaic using the
individual bands, but the coverage view only recognises this as containing a
single band.  I've added a 'band' attribute into the indexer, should
something like this be used to configure the coverage view with multiple
bands?


Cheers

<http://osgeo-org.1560.x6.nabble.com/file/t384100/car_park_mosaic.png>



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users