Limiting rendering for large resultsets

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Limiting rendering for large resultsets

PREVOSTO, Laurent

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Jonathan Moules-2
Hi Laurent,
One possible solution - two datasets.

- Use a urban-areas polygon to clip the data into two datasets - one with urban areas, the other rural.

- Style them both with different scale thresholds.

- Put both of them into the same layerGroup.

To your users it'll appear as a single layer while being optimised for viewing at both scales.

---
Alternately rather than splitting the dataset, add an attribute/field to the dataset that indicates if it is urban/rural. Load as a single layer with an SLD which has two rules, and use a filter to distinguish between them.

Hopefully that helps,
Jonathan




On 6 August 2014 10:39, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

geowolf
In reply to this post by PREVOSTO, Laurent
On Wed, Aug 6, 2014 at 11:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.


You can increase the max rendering time in the WMS panel
 

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.


No, there is no such functionality. It is of course possible to add it, but we'd need to make
an extension to SLD, the limit would have to be specified at the FeatureTypeStyle or Rule levels

Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

PREVOSTO, Laurent

I agree that would be the perfect solution… unfortunately it is not the way works today L

 

No dirty workaround ? damned…

 

Regards,

 

Laurent

 

De : [hidden email] [mailto:[hidden email]] De la part de Andrea Aime
Envoyé : mercredi 6 août 2014 12:24
À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

On Wed, Aug 6, 2014 at 11:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

 

You can increase the max rendering time in the WMS panel

 

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

No, there is no such functionality. It is of course possible to add it, but we'd need to make

an extension to SLD, the limit would have to be specified at the FeatureTypeStyle or Rule levels

 

Cheers

Andrea

 

--

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/NWWaa2 for more information.

==

 

Ing. Andrea Aime 

@geowolf

Technical Lead

 

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054  Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39  339 8844549

 

 

-------------------------------------------------------


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

cmaul
In reply to this post by PREVOSTO, Laurent
Laurent,

I've got the same problem with our cadastre, i.e. rural and urban properties. My way around was to filter using the 'Local Government' attribute. The result is a bit crude and the SLD is large, but it works well. If you have got a similar attribute in your dataset that would be simplest solution.

Disadvantaqe is: Users suddenly see nothing where they know that there are parcel.

Increasing the time-out is not a good idea because after 60 secs (which is the default) people are long gone and the only effect is you put a load on your DB server. Nobody is that patient anymore.

I have actually a time-out of 30 secs.

Cheers

Christian
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department Environment and Primary Industries
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002


Telephone:        +61-3-8636 2325
Telefax:              +61-3-8636 2813
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Mike Grogan
In reply to this post by PREVOSTO, Laurent
A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

For instance, 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

- Mike Grogan





On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

PREVOSTO, Laurent

Hi,

I understand your approach but i definately prefer to draw nothing or an “error” tile than just draw a random part of the network (a wrong information)

Because, users tend to believe what they see J

 

In my case, I am tempted to lower the timeout rendering of geoserver and consider that a tile that needs more than, let’s say, 10 sec to draw is not worth rendering.

But then Apache mod_jk load-balancer put that very Tomcat in FAIL state and I could not figure out a configuration that actually works.

 

Regards,

 

Laurent

 

De : Mike Grogan [mailto:[hidden email]]
Envoyé : vendredi 8 août 2014 18:04
À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

 

For instance, 

 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

 

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

 

- Mike Grogan

 

 

 

 

On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

 


------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Daniel Bevilacqua
Hi,

You can apply simplification functions from Oracle or PostGIS. These function will return a simplified objects.


On Tue, Aug 12, 2014 at 3:01 PM, PREVOSTO, Laurent <[hidden email]> wrote:

Hi,

I understand your approach but i definately prefer to draw nothing or an “error” tile than just draw a random part of the network (a wrong information)

Because, users tend to believe what they see J

 

In my case, I am tempted to lower the timeout rendering of geoserver and consider that a tile that needs more than, let’s say, 10 sec to draw is not worth rendering.

But then Apache mod_jk load-balancer put that very Tomcat in FAIL state and I could not figure out a configuration that actually works.

 

Regards,

 

Laurent

 

De : Mike Grogan [mailto:[hidden email]]
Envoyé : vendredi 8 août 2014 18:04


À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

 

For instance, 

 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

 

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

 

- Mike Grogan

 

 

 

 

On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

 


------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users




--
Daniel Bevilacqua

cel +55 19 81383157



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Chris Bennight
In reply to this post by PREVOSTO, Laurent
As another take on this same problem we've been working on a Z-order occlusion based distributed renderer.  (This is for an accumulo <-> geoserver datastore implementation).

Currently working is a point only implementation (well, it works for polygons/linestrings, but calculates occlusion for the centroid only - so only works "right" for points))

It's implemented as a WPS render transform process

Here's the query setup code

Index thinning

This implementation doesn't distribute the rendering process; rather it defines a pixel space -> index range transform function, and when a feature (point in this case) is found it's returned, and the seek function in the database skips to the next pixel

Here's a graphic example:

--

The next phase, and what I think a general solution to the problem you describe is to distribute the rendering process  (in accumulo the database "nodes" sit on top of multiple instances for easy horizontal scalability).

This portion has to take into account the SLD selected as well, as the style can have a big impact on how features are rendered.  

Basically each featurestylerule results in a distributed call for that feature type - and the tile for that rule is rendered local to the majority of the data (best effort) in the cluster.  When the tile is completely covered (or passes some heuristic, i.e. 80%, etc.)  the seek stops and the "full" tile returns.   (There is some skipping here to - when a feature is rendered an internal table of index ranges <-> pixel transforms is modified to remove the covered pixels/ranges - so the next feature always overlaps a non-colored region.)

All the tiles are now composted together in geoserver based on the ordering from the SLD - and hopefully we have now skipped reading and writing a bunch of data we don't have a "pixel budget" to actually display.

This portion is still in work - I think we might have an upcoming geoserver pull request, as we need to make a few private methods protected so we can distribute the render process across the cluster.   Our internal implementation just cuts and pastes a bunch of GPL code, and I didn't want to include that since the rest of the project is apache 2.0 and, well, I didn't want to explain the intermingled licenses.   

Our torture case right now is basically one of the openstreetmap SLD's at ~ zoom level 17 rules, rendered globally (with the full osm planet dump).   


--

That said, these methods have to be implemented at the geotools datastore level - and specifically you have to have control enough of the indexing so that you can short circuit reads.   But it's something that could conceptually (implementation would be slightly different) be to other systems as well  (certainly on postgis, since it's open source)


On Tue, Aug 12, 2014 at 2:01 PM, PREVOSTO, Laurent <[hidden email]> wrote:

Hi,

I understand your approach but i definately prefer to draw nothing or an “error” tile than just draw a random part of the network (a wrong information)

Because, users tend to believe what they see J

 

In my case, I am tempted to lower the timeout rendering of geoserver and consider that a tile that needs more than, let’s say, 10 sec to draw is not worth rendering.

But then Apache mod_jk load-balancer put that very Tomcat in FAIL state and I could not figure out a configuration that actually works.

 

Regards,

 

Laurent

 

De : Mike Grogan [mailto:[hidden email]]
Envoyé : vendredi 8 août 2014 18:04


À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

 

For instance, 

 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

 

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

 

- Mike Grogan

 

 

 

 

On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

 


------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

jody.garnett
Interesting, I am reminded of the gvSig renderer which can "draw" the shapefile index - getting off the ride when the index node gets down below the size of a pixel. Think that is one of the only optimisations they have left that we did not implement.

Note that if you are using PostGIS (or Oracle?) GeoTools will ask the database to simplify (stepping size based on the distance a pixel covers at current zoom level). This is great as it allows less data to be sent over from the database. For most other data stores it will simplify in Java before transform and rendering.

Chris it would be *great* if you could write the occasional blog for the geotools website - this is really cool stuff you are working on.

Jody Garnett


On Tue, Aug 12, 2014 at 5:36 PM, Chris Bennight <[hidden email]> wrote:
As another take on this same problem we've been working on a Z-order occlusion based distributed renderer.  (This is for an accumulo <-> geoserver datastore implementation).

Currently working is a point only implementation (well, it works for polygons/linestrings, but calculates occlusion for the centroid only - so only works "right" for points))

It's implemented as a WPS render transform process

Here's the query setup code

Index thinning

This implementation doesn't distribute the rendering process; rather it defines a pixel space -> index range transform function, and when a feature (point in this case) is found it's returned, and the seek function in the database skips to the next pixel

Here's a graphic example:

--

The next phase, and what I think a general solution to the problem you describe is to distribute the rendering process  (in accumulo the database "nodes" sit on top of multiple instances for easy horizontal scalability).

This portion has to take into account the SLD selected as well, as the style can have a big impact on how features are rendered.  

Basically each featurestylerule results in a distributed call for that feature type - and the tile for that rule is rendered local to the majority of the data (best effort) in the cluster.  When the tile is completely covered (or passes some heuristic, i.e. 80%, etc.)  the seek stops and the "full" tile returns.   (There is some skipping here to - when a feature is rendered an internal table of index ranges <-> pixel transforms is modified to remove the covered pixels/ranges - so the next feature always overlaps a non-colored region.)

All the tiles are now composted together in geoserver based on the ordering from the SLD - and hopefully we have now skipped reading and writing a bunch of data we don't have a "pixel budget" to actually display.

This portion is still in work - I think we might have an upcoming geoserver pull request, as we need to make a few private methods protected so we can distribute the render process across the cluster.   Our internal implementation just cuts and pastes a bunch of GPL code, and I didn't want to include that since the rest of the project is apache 2.0 and, well, I didn't want to explain the intermingled licenses.   

Our torture case right now is basically one of the openstreetmap SLD's at ~ zoom level 17 rules, rendered globally (with the full osm planet dump).   


--

That said, these methods have to be implemented at the geotools datastore level - and specifically you have to have control enough of the indexing so that you can short circuit reads.   But it's something that could conceptually (implementation would be slightly different) be to other systems as well  (certainly on postgis, since it's open source)


On Tue, Aug 12, 2014 at 2:01 PM, PREVOSTO, Laurent <[hidden email]> wrote:

Hi,

I understand your approach but i definately prefer to draw nothing or an “error” tile than just draw a random part of the network (a wrong information)

Because, users tend to believe what they see J

 

In my case, I am tempted to lower the timeout rendering of geoserver and consider that a tile that needs more than, let’s say, 10 sec to draw is not worth rendering.

But then Apache mod_jk load-balancer put that very Tomcat in FAIL state and I could not figure out a configuration that actually works.

 

Regards,

 

Laurent

 

De : Mike Grogan [mailto:[hidden email]]
Envoyé : vendredi 8 août 2014 18:04


À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

 

For instance, 

 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

 

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

 

- Mike Grogan

 

 

 

 

On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

 


------------------------------------------------------------------------------


_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Jonathan Moules-2
> Note that if you are using PostGIS (or Oracle?) GeoTools will ask the database to simplify
Andrea will know better (of course!), but I think last time this came up the notion was that Oracle's no-where near as good as PostGIS at this.

Cheers,
Jonathan



On 13 August 2014 07:22, Jody Garnett <[hidden email]> wrote:
Interesting, I am reminded of the gvSig renderer which can "draw" the shapefile index - getting off the ride when the index node gets down below the size of a pixel. Think that is one of the only optimisations they have left that we did not implement.

Note that if you are using PostGIS (or Oracle?) GeoTools will ask the database to simplify (stepping size based on the distance a pixel covers at current zoom level). This is great as it allows less data to be sent over from the database. For most other data stores it will simplify in Java before transform and rendering.

Chris it would be *great* if you could write the occasional blog for the geotools website - this is really cool stuff you are working on.

Jody Garnett


On Tue, Aug 12, 2014 at 5:36 PM, Chris Bennight <[hidden email]> wrote:
As another take on this same problem we've been working on a Z-order occlusion based distributed renderer.  (This is for an accumulo <-> geoserver datastore implementation).

Currently working is a point only implementation (well, it works for polygons/linestrings, but calculates occlusion for the centroid only - so only works "right" for points))

It's implemented as a WPS render transform process

Here's the query setup code

Index thinning

This implementation doesn't distribute the rendering process; rather it defines a pixel space -> index range transform function, and when a feature (point in this case) is found it's returned, and the seek function in the database skips to the next pixel

Here's a graphic example:

--

The next phase, and what I think a general solution to the problem you describe is to distribute the rendering process  (in accumulo the database "nodes" sit on top of multiple instances for easy horizontal scalability).

This portion has to take into account the SLD selected as well, as the style can have a big impact on how features are rendered.  

Basically each featurestylerule results in a distributed call for that feature type - and the tile for that rule is rendered local to the majority of the data (best effort) in the cluster.  When the tile is completely covered (or passes some heuristic, i.e. 80%, etc.)  the seek stops and the "full" tile returns.   (There is some skipping here to - when a feature is rendered an internal table of index ranges <-> pixel transforms is modified to remove the covered pixels/ranges - so the next feature always overlaps a non-colored region.)

All the tiles are now composted together in geoserver based on the ordering from the SLD - and hopefully we have now skipped reading and writing a bunch of data we don't have a "pixel budget" to actually display.

This portion is still in work - I think we might have an upcoming geoserver pull request, as we need to make a few private methods protected so we can distribute the render process across the cluster.   Our internal implementation just cuts and pastes a bunch of GPL code, and I didn't want to include that since the rest of the project is apache 2.0 and, well, I didn't want to explain the intermingled licenses.   

Our torture case right now is basically one of the openstreetmap SLD's at ~ zoom level 17 rules, rendered globally (with the full osm planet dump).   


--

That said, these methods have to be implemented at the geotools datastore level - and specifically you have to have control enough of the indexing so that you can short circuit reads.   But it's something that could conceptually (implementation would be slightly different) be to other systems as well  (certainly on postgis, since it's open source)


On Tue, Aug 12, 2014 at 2:01 PM, PREVOSTO, Laurent <[hidden email]> wrote:

Hi,

I understand your approach but i definately prefer to draw nothing or an “error” tile than just draw a random part of the network (a wrong information)

Because, users tend to believe what they see J

 

In my case, I am tempted to lower the timeout rendering of geoserver and consider that a tile that needs more than, let’s say, 10 sec to draw is not worth rendering.

But then Apache mod_jk load-balancer put that very Tomcat in FAIL state and I could not figure out a configuration that actually works.

 

Regards,

 

Laurent

 

De : Mike Grogan [mailto:[hidden email]]
Envoyé : vendredi 8 août 2014 18:04


À : PREVOSTO, Laurent
Cc : [hidden email]
Objet : Re: [Geoserver-users] Limiting rendering for large resultsets

 

A crude approach I have used in the past is to add an integer attribute to my points or lines and then to assign a random attribute value from a range that is based on the density of features in the area.  I then specify in the SLD that only features greater than some number should be displayed for particular zoom levels.

 

For instance, 

 

For your city features, you might assign a random integer attribute to each of them from a range between, say, 0 and 1000. 

 

For your countryside features, you might assign a random integer attribute that have a range between 700 and 1000. 

 

Then, say for a zoom level 5 or 6, you might specify in the SLD to only show features whose attribute > 700.  

 

Doing this, you end up keeping all of the countryside features, but randomly filtering out city features. 

 

At greater zoom levels ... just say 9, 10, or so (depending on your situation), you then show features whose attribute > 500, 400, etc., thereby allowing more city features to be shown as you zoom in. 

 

I could probably have done some statistics to tell me the exact ranges to use, but trial & error worked fine for me. 

 

- Mike Grogan

 

 

 

 

On Wed, Aug 6, 2014 at 5:39 AM, PREVOSTO, Laurent <[hidden email]> wrote:

Hello,

We have geoserver based WMS services that display a quite large network (lots of points and polylines stored in an Oracle Spatial database).

 

When in the cities, the number of items can be very large

In the countryside, of course, the density of polylines is quite low.

 

If I try to render  a whole city, since there are too many items to render, the request will timeout.

But if I set a scale condition to avoid drawing thousands of lines when in a large city, users complain that they can’t find the cables when in the countryside…

 

So my question was : is there a way to tell geoserver to render a blank image if there are more than n items in the Oracle resultset ?

That way, I could keep a large scale in my TLD without going timeout in areas where there is a high density of items.

 

Regards,

 

Laurent


------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

 


------------------------------------------------------------------------------


_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

geowolf
In reply to this post by Chris Bennight
On Wed, Aug 13, 2014 at 2:36 AM, Chris Bennight <[hidden email]> wrote:
As another take on this same problem we've been working on a Z-order occlusion based distributed renderer.  (This is for an accumulo <-> geoserver datastore implementation).

Currently working is a point only implementation (well, it works for polygons/linestrings, but calculates occlusion for the centroid only - so only works "right" for points))

It's implemented as a WPS render transform process

Here's the query setup code

Index thinning

This implementation doesn't distribute the rendering process; rather it defines a pixel space -> index range transform function, and when a feature (point in this case) is found it's returned, and the seek function in the database skips to the next pixel

Here's a graphic example:

What you are doing here in GeoTools is a rendering/store hint called ScreenMap, and it's normally applied in memory.
However, theoretically we could do the same as, say, MinVisitor, and have the store use a native and faster
implementation of it, like JDBCDataStore does (turning it into a select min).

To make it possible we'd need to have ScreenMap provide its internal information I supposed, the rendering rectangle,
and the grid to world transformation.

The advantage of doing so would be that your users would not have to add a rendering transformation in the SLD,
it would just work automatically (downsize, there would be no way to turn it off).

Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Limiting rendering for large resultsets

Chris Bennight

On Fri, Aug 15, 2014 at 10:24 AM, Andrea Aime <[hidden email]> wrote:

What you are doing here in GeoTools is a rendering/store hint called ScreenMap, and it's normally applied in memory.
However, theoretically we could do the same as, say, MinVisitor, and have the store use a native and faster
implementation of it, like JDBCDataStore does (turning it into a select min).

To make it possible we'd need to have ScreenMap provide its internal information I supposed, the rendering rectangle,
and the grid to world transformation.

The advantage of doing so would be that your users would not have to add a rendering transformation in the SLD,
it would just work automatically (downsize, there would be no way to turn it off).

Cheers
Andrea


So essentially using the ScreenMap information as a way to pass down the grid <-> world  transform function information to a place where pushdown support on a datastore can be implemented.

That definitely fits in a little more smoothly than requiring the WPS in the SLD (as you mention).  The inability to turn it off (which you also highlight) would be my concern.   To make it more "fire and forget"  I think the distributed rendering option I talked about would be required.  (To deal with symbolizers in grid space that aren't equivalent to the native geometry in that same grid space).   There's the efficiency optimization if the symbolizer is larger than the geometry - but the bigger concern is the "incorrectness" if the symbolizer is smaller than the geometry.  For points that's not really a huge issue (only case I think it would break would be with offsets). But for lines and polygons it's a definite issues.  For the distributed rendering portion this is definitely something I want to dig in to.   
I saw your latest commit, and this seems to be like a good pointer into digging into this (ScreenMap) functionality:

Thanks!  (for the idea, and for the unrelated but fortuitous fix)

------------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users