Some interesting optmizations from Jonathan's load test

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Some interesting optmizations from Jonathan's load test

geowolf
Hi,
in a separate thread Jonathan provided a load test, data and styles for a map he's
publishing.
The thread started as a comparison between PNG encoders, but, at least as far as I can
tell, there is no significant optimization that can be made at the PNG encoder level.

However, I've been toying with the data set and found some interesting results regardless
(grab some popcorn).

The maps are setup as a group of 15 different layers, with some scale dependency.
At 10 and 5 millions not all the layers are shown, but there is one thing that is evident: there
are a few layers (woodlands, lakes, and the portion of urban areas shown at this lavel) that 
contain a good number of small, but detailed polygons (in the order of 20k+)... this makes
it miss the reference "no more than 1000 elements in the map" mark by a factor of 20, which
is probably what makes it interesting :-p

We have code that simplifies them before drawing them, but they are small enough that
another optimization, geared towards drawing features smaller than a pixel, should have kicked
in, but it did not and that affected drawing times. So I've fixed it (fixes are already in, will
be part of the next stable release).
In order to get the fully benefit the styles must not use partial transparency, and I had
to fix one of the styles accordingly (believe it was lakes, not 100% sure).

To have some comparisons, I'm reporting the numbers found in the other thread without any
change. The report contains the JVM and build details, and then average response time and throughput
(for those that did not follow, this is a 15 layers group, made of shapefiles,
and the output is a 1272x1261 pixels full color png, with 10 concurrent clients hitting GeoServer, 
the benchmark is done on a core i7 820 CPU). All benchmarks are using the PNGJ encoder, 
for those not familar with benchmarking, the Oracle JDK 7 has  a renderer that does not scale up,
so it is worthy setting up a number of load balanced GeoServer instances in separate JVMs,
whilst OpenJDK has one that's quite a bit slower, but that has no scalability issues, so one
normally sets up a single instance instead.

OpenJDK 7: <not available>, 3.3r/s
JDK 7: 2867ms, 3.9r/s
JDK 7, three instances load balanced with HA proxy: 1248ms, 7.9r/s

The above shows the benchmark is heavily bottlenecked by the rendering subsystem drawing speed
(so much that the JDK 7 renderer, Ductus, can outperform OpenJDK 7 one, Pisces, even with 10
concurrent requests).

After fixing the "small polygons" optimization bug the results are:

OpenJDK 7: 2507ms, 4r/s
JDK 7: 2459ms, 4r/s
JDK 7, three instances load balanced with HA proxy: 1175ms, 8.4r/s

So, this optimization improves the rendering time of OpenJDK 7 and puts it on par with the
closed source JDK, and also improves the latter.

Then again, Jonathan has this data in Oracle... I did not want to venture there: loading
data there is a pain, the styles need fixing because of the uppercase attributes, 
and honestly, why waste an open source developer spare time on closed source databases anyways?
So I loaded all the data in PostGIS instead and run the tests again (test run with the connection pool
locked at 10 connections):

OpenJDK 7: 2630ms, 3.8r/s
JDK 7: 2422ms, 4.1r/s
JDK 7, three instances load balanced with HA proxy: 1301ms, 7.6r/s

Hum... a bit slower. And then I've remembered what Paul Ramsey used to say about Mapnik using
ST_Simplify on the geometries to get a boost (reference, http://blog.cartodb.com/post/20163722809/speeding-up-tiles-rendering). 
Which had been tried already in GeoServer without much of a benefit
in previous benchmarks (we already do the simplification on the java side), but.. we did not have a map 
with so many little polygons. So why not give it a kick? All one needs to do is to add the following in the postgis dialect:

@Override
    public void encodeGeometryColumnSimplified(GeometryDescriptor gatt, String prefix, int srid,
            StringBuffer sql, Double distance) {
        boolean geography = "geography".equals(gatt.getUserData().get(
                JDBCDataStore.JDBC_NATIVE_TYPENAME));

        if (geography) {
            sql.append("encode(ST_AsBinary(ST_Simplify(");
            encodeColumnName(prefix, gatt.getLocalName(), sql);
            sql.append(", "  + distance + ")),'base64')");
        } else {
            sql.append("encode(ST_AsBinary(ST_Simplify(ST_Force_2D(");
            encodeColumnName(prefix, gatt.getLocalName(), sql);
            sql.append("), "  + distance + ")),'base64')");
        }
    }
    
    protected void addSupportedHints(Set<Hints.Key> hints) {        
        hints.add(Hints.GEOMETRY_SIMPLIFICATION);
    }

And here are the results:

OpenJDK 7: 2014 ms, 4.9 r/s
JDK 7: 1694ms, 5.8r/s
JDK 7, three instances load balanced with HA proxy: 1046 ms, 9.4 r/s

Holy cow, on the single JVM setup that's roughly a 50% speedup (and not to throw away in the
case of 3 JVMs either)... 
Now, this is PostGIS specific (no
plain simplification in Oracle, only the topology preserving one is available, which
is supposedly more expensive than not doing it, at least, it is in PostGIS), 
and we need to verify what's the overhead when the geometry do not need simplification. 
That said, how do you see this being enabled?
* always on, that could be a good one if the overhead when simplification is not really needed
  show no regression
* have it enabled by a store parameter
* have it enabled with SLD vendor parameter (to be used only when zoomed out)

Now... another thing that I suspected made MapServer and Mapnik competitive is that
they load the data from the database in a single kick, instead of paging through them like we do.
The approach is a double edged sword:
* on the bright side, a single communication with the db, and no need for the dbms to allocate
  and manage a server side cursor
* on the dark side, no freaking way to control how much data you're loading into memory, OOM
  risk is there (I guess with a cgi/fastcgi approach you just don't care, if one instance goes boom
  it just gets replaced automatically, worst thing that can happen you get into a swap storm)

Code wise the change is small, in the PostGIS dialect we just disable the use of transactions while
reading, which breaks postgresql ability to use server side cursors, forcing it to return everything in
a single kick instead:

    @Override
    public boolean isAutoCommitQuery() {
        return true;
    }

Let's see the results:

OpenJDK 7: 1726 ms, 5.8 r/s
JDK 7: 1688ms, 5.9r/s
JDK 7, three instances load balanced with HA proxy: 998 ms, 9.9 r/s

The one that benefits the most are the ones CPU bottlenecked, which means, first OpenJDK,
and then the three instances of JDK 7, which are both using close to 100% cpu, whilst
JDK 7 stand alone instances uses like 50% cpu (the JVM wide lock in the Ductus renderer
is killing scalabiliity).
The issue is, this approach is not really usable "always on". Options I see (and suggestions
welcomed):
* test again and see if just increasing the fetch size helps (it's now set 1000 by default)
* add a hint that only the WMS renderer sets, that will enable this mode... this assumes
  the styles are always set to provide reasonable scale dependencies... and we should add
  some config to disable the usage of styles that are not associated to the layer
* other suggestions?

Now, for a final test, we know that the OpenJDK renderer is slow, but Laurent Bourges
has been providing Oracle some patches to make it faster. Which have not been accepted, so far.
However... I've made some tests and now have a jar that one can drop in a regular JDK 7
install, which just some of Laurent's improvements, and enable it by setting some JVM
parameters. How does this one fare? This are the results for a single OpenJDK 7
JVM, with also all of the other GeoServer patches mentioned above included:

OpenJDK 7 + optimized Pisces renderer: 1091 ms, 9.0 r/s

Not too bad uh? :-)
The plan is to keep on work on it a bit more before going public and releasing the jar for
everybody to play with.

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Jonathan Moules-2
Hi Andrea,
Excellent analysis and some very good prospective optimisations there.


In order to get the fully benefit the styles must not use partial transparency, and I had
to fix one of the styles accordingly (believe it was lakes, not 100% sure).
Interesting. Originally I used transparency, but for my re-made styles (which is what you're using) I got rid of that because I figured it would slow stuff down and I also realised that I could get the same colour without it. I'll definitely further-optimise my SLD's.


Then again, Jonathan has this data in Oracle... I did not want to venture there: loading
data there is a pain, the styles need fixing because of the uppercase attributes, 
and honestly, why waste an open source developer spare time on closed source databases anyways?
Completely concur. And thanks for all your work given it's your free time!
(That said, these styles are completely interchangeable between shapefile and our Oracle, with just one change - search/replace "ROAD_NO" (shapefile) to "ROAD_NUMBER" (Oracle)).

Now, this is PostGIS specific (no
plain simplification in Oracle, only the topology preserving one is available, which
is supposedly more expensive than not doing it, at least, it is in PostGIS), 

There is plain "simplify" in Oracle:
http://docs.oracle.com/cd/E11882_01/appdev.112/e11830/sdo_util.htm#BJEHCIBF - that tool is available via Oracle Locator and doesn't require Oracle Spatial. It's just a simple Douglas-Peucker implementation
"Topological characteristics of the input geometry might not be maintained after simplification."


And with MSSQL Server it seems to be "Reduce" - http://technet.microsoft.com/en-us/library/bb933814.aspx

I don't know if OGR is used as an input source in GeoServer (shapefile reading?), but it appears to have a Simplify function too (If I'm looking in the right place) - http://www.gdal.org/ogr/classOGRGeometry.html#afd3ea0ffa1e2994427032d0212206ccf

So the ST_Simplify optimisation could potentially be carried over to other datastores (although that does presume that ST_Simplify is as fast on the other implementations as it is in PostGIS).

Again, thanks for your free-time work! :-)
Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
On Tue, Dec 31, 2013 at 1:18 PM, Jonathan Moules <[hidden email]> wrote:

Then again, Jonathan has this data in Oracle... I did not want to venture there: loading
data there is a pain, the styles need fixing because of the uppercase attributes, 
and honestly, why waste an open source developer spare time on closed source databases anyways?
Completely concur. And thanks for all your work given it's your free time!
(That said, these styles are completely interchangeable between shapefile and our Oracle, with just one change - search/replace "ROAD_NO" (shapefile) to "ROAD_NUMBER" (Oracle)).

Now, this is PostGIS specific (no
plain simplification in Oracle, only the topology preserving one is available, which
is supposedly more expensive than not doing it, at least, it is in PostGIS), 

There is plain "simplify" in Oracle:
http://docs.oracle.com/cd/E11882_01/appdev.112/e11830/sdo_util.htm#BJEHCIBF - that tool is available via Oracle Locator and doesn't require Oracle Spatial. It's just a simple Douglas-Peucker implementation
"Topological characteristics of the input geometry might not be maintained after simplification."

Ah, doh, I looked at that page and for some reason I thought it was topology preserving.
Btw, how did you import your data in Oracle again? ogr2ogr?
 


And with MSSQL Server it seems to be "Reduce" - http://technet.microsoft.com/en-us/library/bb933814.aspx

Cool stuff 

I don't know if OGR is used as an input source in GeoServer (shapefile reading?), but it appears to have a Simplify function too (If I'm looking in the right place) - http://www.gdal.org/ogr/classOGRGeometry.html#afd3ea0ffa1e2994427032d0212206ccf

Shapefile reading is done via pure java code. We do have an unsupported OGR based store in GeoTools (which I've originally developed, guess what, in my spare time), but it has been difficult to make it graduate
to supported status as installations that could justify the work normally try to stay away from native packages, unless they are super-duper tested
(as they will bring down the whole JVM if they segfault). Sort of catch 22 situation...
 

So the ST_Simplify optimisation could potentially be carried over to other datastores (although that does presume that ST_Simplify is as fast on the other implementations as it is in PostGIS).

Right, that needs to be verified indeed.
 

Again, thanks for your free-time work! :-)

Always happy to spend some time looking into performance improvements :-p

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Juan Marín Otero
Andrea & Jonathan,

Thanks for this thorough benchmark, I think it's very useful. My 2c would be to introduce these options, but I'd be against simplifying geometries by default, as I know of a few use cases that require their geometries to be "exact" (or at least, exact in the sense of representing whatever they have captured as raw data). A rendering hint sounds great to me though, and I see most use cases benefiting from this. 

As for the OpenJDK optimized renderer....I'm intrigued. I'd love to see better support for it, as it makes deployment on Linux so much easier, but my impression has always been that it's a second class citizen and that we'd be losing something by not using Oracle's JDK. These optimizations certainly sound promising.




On Tue, Dec 31, 2013 at 8:50 AM, Andrea Aime <[hidden email]> wrote:
On Tue, Dec 31, 2013 at 1:18 PM, Jonathan Moules <[hidden email]> wrote:

Then again, Jonathan has this data in Oracle... I did not want to venture there: loading
data there is a pain, the styles need fixing because of the uppercase attributes, 
and honestly, why waste an open source developer spare time on closed source databases anyways?
Completely concur. And thanks for all your work given it's your free time!
(That said, these styles are completely interchangeable between shapefile and our Oracle, with just one change - search/replace "ROAD_NO" (shapefile) to "ROAD_NUMBER" (Oracle)).

Now, this is PostGIS specific (no
plain simplification in Oracle, only the topology preserving one is available, which
is supposedly more expensive than not doing it, at least, it is in PostGIS), 

There is plain "simplify" in Oracle:
http://docs.oracle.com/cd/E11882_01/appdev.112/e11830/sdo_util.htm#BJEHCIBF - that tool is available via Oracle Locator and doesn't require Oracle Spatial. It's just a simple Douglas-Peucker implementation
"Topological characteristics of the input geometry might not be maintained after simplification."

Ah, doh, I looked at that page and for some reason I thought it was topology preserving.
Btw, how did you import your data in Oracle again? ogr2ogr?
 


And with MSSQL Server it seems to be "Reduce" - http://technet.microsoft.com/en-us/library/bb933814.aspx

Cool stuff 

I don't know if OGR is used as an input source in GeoServer (shapefile reading?), but it appears to have a Simplify function too (If I'm looking in the right place) - http://www.gdal.org/ogr/classOGRGeometry.html#afd3ea0ffa1e2994427032d0212206ccf

Shapefile reading is done via pure java code. We do have an unsupported OGR based store in GeoTools (which I've originally developed, guess what, in my spare time), but it has been difficult to make it graduate
to supported status as installations that could justify the work normally try to stay away from native packages, unless they are super-duper tested
(as they will bring down the whole JVM if they segfault). Sort of catch 22 situation...
 

So the ST_Simplify optimisation could potentially be carried over to other datastores (although that does presume that ST_Simplify is as fast on the other implementations as it is in PostGIS).

Right, that needs to be verified indeed.
 

Again, thanks for your free-time work! :-)

Always happy to spend some time looking into performance improvements :-p

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272
mob: <a href="tel:%2B39%20%C2%A0339%208844549" value="+393398844549" target="_blank">+39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--
Juan Marín Otero
GIS Consultant

-------Visita mi blog en---------------------
http://guachintoneando.blogspot.com
---------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
On Tue, Dec 31, 2013 at 3:07 PM, Juan Marín Otero <[hidden email]> wrote:
Andrea & Jonathan,

Thanks for this thorough benchmark, I think it's very useful. My 2c would be to introduce these options, but I'd be against simplifying geometries by default, as I know of a few use cases that require their geometries to be "exact" (or at least, exact in the sense of representing whatever they have captured as raw data). A rendering hint sounds great to me though, and I see most use cases benefiting from this. 

Ah sorry, when I said "by default" I meant any time the already existing Hints.GEOMETRY_SIMPLIFICATION Query hint
is passed down, which happens only when rendering. 
What I'm not sure, is whether it's a good idea to do it always when rendering, or just have it be enabled in some cases.
 

As for the OpenJDK optimized renderer....I'm intrigued. I'd love to see better support for it, as it makes deployment on Linux so much easier, but my impression has always been that it's a second class citizen and that we'd be losing something by not using Oracle's JDK. These optimizations certainly sound promising.

Yes and no. The Ductus renderer is now 10 years old and afaik it has not been improved at all during these
years, because it's not something Oracle owns, they had another company build it and they are using it under
license.
For example, Oracle guys told me that for JavaFX they rewrote Pisces in C to get a 2x speedup (e.g., they did not
use Ductus). Funny enough, Laurent patch, in its full form (I've used a partial one at the moment, because the full one requires patching
the JDK), is in some rendering micro-benchmarks already 3 times faster than Pisces...

Cheers
Andrea


--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Jonathan Moules-2
In reply to this post by geowolf

Btw, how did you import your data in Oracle again? ogr2ogr?

Nothing so open source I'm afraid. We used FME for all our data loading; if you have a copy I can provide you with the loader if you're interested, otherwise I can export the Strategi tables as sql/csv (either data or structure or both).
Regards,
Jonathan
 
 


And with MSSQL Server it seems to be "Reduce" - http://technet.microsoft.com/en-us/library/bb933814.aspx

Cool stuff 

I don't know if OGR is used as an input source in GeoServer (shapefile reading?), but it appears to have a Simplify function too (If I'm looking in the right place) - http://www.gdal.org/ogr/classOGRGeometry.html#afd3ea0ffa1e2994427032d0212206ccf

Shapefile reading is done via pure java code. We do have an unsupported OGR based store in GeoTools (which I've originally developed, guess what, in my spare time), but it has been difficult to make it graduate
to supported status as installations that could justify the work normally try to stay away from native packages, unless they are super-duper tested
(as they will bring down the whole JVM if they segfault). Sort of catch 22 situation...
 

So the ST_Simplify optimisation could potentially be carried over to other datastores (although that does presume that ST_Simplify is as fast on the other implementations as it is in PostGIS).

Right, that needs to be verified indeed.
 

Again, thanks for your free-time work! :-)

Always happy to spend some time looking into performance improvements :-p

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272
mob: <a href="tel:%2B39%20%C2%A0339%208844549" value="+393398844549" target="_blank">+39  339 8844549


-------------------------------------------------------


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
In reply to this post by geowolf
On Tue, Dec 31, 2013 at 3:13 PM, Andrea Aime <[hidden email]> wrote:
On Tue, Dec 31, 2013 at 3:07 PM, Juan Marín Otero <[hidden email]> wrote:
Andrea & Jonathan,

Thanks for this thorough benchmark, I think it's very useful. My 2c would be to introduce these options, but I'd be against simplifying geometries by default, as I know of a few use cases that require their geometries to be "exact" (or at least, exact in the sense of representing whatever they have captured as raw data). A rendering hint sounds great to me though, and I see most use cases benefiting from this. 

Ah sorry, when I said "by default" I meant any time the already existing Hints.GEOMETRY_SIMPLIFICATION Query hint
is passed down, which happens only when rendering. 

And oh, we should not pass down that hint when a rendering transformation is used, because it may need valid geometries to w
work with.
Do you know of other cases?

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
In reply to this post by Jonathan Moules-2
On Tue, Dec 31, 2013 at 3:35 PM, Jonathan Moules <[hidden email]> wrote:

Btw, how did you import your data in Oracle again? ogr2ogr?

Nothing so open source I'm afraid. We used FME for all our data loading; if you have a copy I can provide you with the loader if you're interested, otherwise I can export the Strategi tables as sql/csv (either data or structure or both).

The ideal would be to have a Oracle dump... to be loaded and tested in a rainy day :-p

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
In reply to this post by geowolf
Hi all,
some follow up on this. I've run another set of benchmarks, restructuring the test so that
the maps at the various zoom levels are run in isolation, each one doing its own
benchmark: this gives us a perspective on what's slow, and what improves most
with the changes.

Also, wanted to check how the disabling of the server side cursor actually helps,
and the analysis of the results led me to add another test, one in which I don't
disable the server side cursors, but pump up the fetch size to 10k.

Here is a summary table with average response times (ms) and throughput (requests/second),
the name of the map is the scale at which the map is painted.

Comparison Unmodified master code ST_simplify ST_Simplify + no server side cursor ST_Simplify + 10000 fetch size
Avg Time Throughput Avg Time Throughput Avg Time Throughput Avg Time Throughput
10million 1325 6,93 770 10,72 747 11,15 725 11,59
5 million 1328 6,88 779 10,61 792 10,67 768 10,79
2,5 million 974 8,94 700 11,90 694 11,97 677 12,25
1250000 1210 7,43 967 8,98 1.005 8,65 982 8,85
625000 1717 5,30 1525 5,94 1.568 5,77 1.574 5,86
300000 1110 7,93 1037 8,43 1.034 8,42 1.045 8,43
150000 665 12,01 665 12,78 655 12,24 632 12,85

Observations:
* ST_simplify usage does indeed help at all scales in these benchmarks, probably because even at 1:150000 we're
  still somewhat away from the native resolution of the data (I'm setting up the foss4g 2010 data in postgis again to test that one too with st_simplify).
* Disabling the server side cursor helps most in the lower zoom levels, not much in the middle ones, and slows down things in some cases. Which led me to try out the usual setup (with server side cursors) and a higher fetch size (10000), the results from the last column seem to suggest it's not the server side cursor itself that costs, but the number of round trips between geoserver and postgis to fetch data, which is cut by a factor of 10 when switching form 1000 to 10000.
Apparently the lesson learned is that we don't need this change, but if one knows a large amount of features need to be transferred to paint a map, a large fetch size helps. It could be interesting, for the future, to make fetch size a query hint, so that we can specify it on a per layer basis. 
* The request at 1:625000 is by far the slowest one, and needs some extra investigation, as this one probably has its own bottleneck in a different place than the others.

Ah, another thing, the tests are giving quite erratic results, going up and down by 10%, I've tried to run the tests multiple times and report back the
best result. Also, these results are not comparable with the previous ones, as the way they are done it's different (before it was all zoom levels being
run all together, in this case instead each zoom level has 10 threads doing those 60 requests, then stopping, and starting with the next zoom level, and so on,
obviously resulting in lower performance due to the CPUs not being fully utilized when each run closes to an end and only a few threads are waiting for
completion).

Cheers
Andrea


--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
On Thu, Jan 2, 2014 at 10:13 AM, Andrea Aime <[hidden email]> wrote:
* The request at 1:625000 is by far the slowest one, and needs some extra investigation, as this one probably has its own bottleneck in a different place than the others.

Had a quick look , the layer seems to be slow because:
* It's painting more than others (it's richer)
* It has all the linework (roads, waterways, borders, whatever) heavily using translucency, I'm pretty sure none of these roads should really
  be translucent in a map... when do you really want to be able to see though the strokes of a road? Build the same colors without opacity instead

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Jonathan Moules-2
Hi Andrea,
  Interesting stuff.

* ST_simplify usage does indeed help at all scales in these benchmarks, probably because even at 1:150000 we're
  still somewhat away from the native resolution of the data 
OS Strategi has a desired scale range of 1:187,500 - 1:312,500, with the nominal scale being 1:250,000. Or so says their technical spec; maybe it hasn't been generalised enough.

* It has all the linework (roads, waterways, borders, whatever) heavily using translucency, I'm pretty sure none of these roads should really
  be translucent in a map
Yep. As noted in my previous email from earlier in this discussion, I was going to get rid of that, and actually did that Tuesday. My improved/optimised set of SLD's are at the below link. That was one of the first things I styled in GeoServer, so I was still learning a lot.

There's also a second set of SLD's plus the layergroup too - these are for Strategi as well but a completely different style. They should give different benchmark results (I find this neutral styling renders faster).
Both sets are in this file:

----

Also, the Oracle dump of the data is ready - http://maps.warwickshire.gov.uk/misc/os_strategi.dmp.Z

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Tom Kunicki-4
Andrea,

I did a little search on this but couldn't find anything obvious.  Has there been any work to utilize Java2D in a way that could take advantage of acceleration (OpenGL pipeline introduced in Java 5, Volatile/ManageImage) [1,2] or some 3rd party libraries [3].

I looked for constructs in GeoTools and GeoServer like:

GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsConfiguration gc = ge.getDefaultScreenDevice().getDefaultConfiguration();
BufferedImage img = gc.createCompatibleImage(width, height, Transparency.TRANSLUCENT);
img.setAccelerationPriority(1);

...but didn't see anything.  Of course there needs to be underlying hardware to support this.

I'm an OpenGL nerd by prior experience and it's a powerful tool.  Most of my work has been solving very specific rendering and analytical problems. Working in OpenGL to generate a generalized and optimized renderer is not always so easy which is why the use of a magic JDK flag or code construct would be nice.

Tom.

[1] http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-Desktop/html/java2d.html#gcgto
[2] http://docs.oracle.com/javase/7/docs/technotes/guides/2d/flags.html
[3] http://brandonborkholder.github.io/glg2d/


On Thu, Jan 2, 2014 at 5:58 AM, Jonathan Moules <[hidden email]> wrote:
Hi Andrea,
  Interesting stuff.

* ST_simplify usage does indeed help at all scales in these benchmarks, probably because even at 1:150000 we're
  still somewhat away from the native resolution of the data 
OS Strategi has a desired scale range of 1:187,500 - 1:312,500, with the nominal scale being 1:250,000. Or so says their technical spec; maybe it hasn't been generalised enough.

* It has all the linework (roads, waterways, borders, whatever) heavily using translucency, I'm pretty sure none of these roads should really
  be translucent in a map
Yep. As noted in my previous email from earlier in this discussion, I was going to get rid of that, and actually did that Tuesday. My improved/optimised set of SLD's are at the below link. That was one of the first things I styled in GeoServer, so I was still learning a lot.

There's also a second set of SLD's plus the layergroup too - these are for Strategi as well but a completely different style. They should give different benchmark results (I find this neutral styling renders faster).
Both sets are in this file:

----

Also, the Oracle dump of the data is ready - http://maps.warwickshire.gov.uk/misc/os_strategi.dmp.Z

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel




--
Tom Kunicki
Software Engineer | Boundless
[hidden email]
917-460-7212
@tomkunicki

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
On Thu, Jan 2, 2014 at 11:12 PM, Tom Kunicki <[hidden email]> wrote:
Andrea,

I did a little search on this but couldn't find anything obvious.  Has there been any work to utilize Java2D in a way that could take advantage of acceleration (OpenGL pipeline introduced in Java 5, Volatile/ManageImage) [1,2] or some 3rd party libraries [3].

I looked for constructs in GeoTools and GeoServer like:

GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsConfiguration gc = ge.getDefaultScreenDevice().getDefaultConfiguration();
BufferedImage img = gc.createCompatibleImage(width, height, Transparency.TRANSLUCENT);
img.setAccelerationPriority(1);

...but didn't see anything.  Of course there needs to be underlying hardware to support this.

Nope, we did not try much. Personally I've never had a graphics card worth using for this purpose, every time I've tried
some java + GPU combination (e.g., OpenCL, Aparapi) I've found that carefully written Java code was as fast, or faster, and more
scalable, than the GPU accelerated one... given the reports of the contrary on the net, this is probably due to the poor hardware 
at hand I guess(I've got a GeForce GTX 260 on my desktop, not sure what's on the notebook, it's one of those combined Intel + Nvidia
solutions that Linux is never too happy about).


I'm an OpenGL nerd by prior experience and it's a powerful tool.  Most of my work has been solving very specific rendering and analytical problems. Working in OpenGL to generate a generalized and optimized renderer is not always so easy which is why the use of a magic JDK flag or code construct would be nice.

Tom.

[1] http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-Desktop/html/java2d.html#gcgto
[2] http://docs.oracle.com/javase/7/docs/technotes/guides/2d/flags.html
[3] http://brandonborkholder.github.io/glg2d/

This one is interesting. Do you have time to do some testing and see if it provides any benefit?

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

geowolf
On Fri, Jan 3, 2014 at 8:25 AM, Andrea Aime <[hidden email]> wrote:

Ah, looking at the implementation status table:
the following limitations would pop up if we used it:
* no labels painted (we use drawGlyphVector, as we often need to extract the glyph vector
  first to do layouts and conflict resolution)
* no graphic fills (we use TexturePaint, but setPaint only supports Color)
* no maps integrating raster data (we use drawRendererImage)
* given the comments about BasicStroke at the very end, not sure if the implementation can support dashed lines, or not

That said, if we had a significant performance boost, it might be worth contributing to
the project and get them implemented.

Cheers
Andrea


--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549


-------------------------------------------------------

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Fwd: Some interesting optmizations from Jonathan's load test

Tom Kunicki-4

OpenGL/CL (err, the hardware behind it) can solve specific problems very well.  I was forced to try it to meet some requirements that seemed implausible, it worked and I was hooked.  The performance benefits tend to degrade quickly as attempts are made to generalize solutions (without a lot of resources).  Although, I did have some good luck with Aparapi for gridded dataset analysis.  When I used it in 2011 the generalization penalty was still pretty high (no local, only global memory) but it still offered some nice performance gains.  I've always tried to keep capable GPU(s) handy in my machines. As you said, It sounds like your GPU capabilities may have affected your results (need fast VRAM and lots of stream processors, you pay for this).  That all being said I wonder how well Sun^H^H^H Oracle was able to handle generalization in the face of optimization with Java2D.   I would love to look at hardware accelerated rendering, is the performance testing environment you've been using easy to set up?



On Fri, Jan 3, 2014 at 1:35 AM, Andrea Aime <[hidden email]> wrote:
On Fri, Jan 3, 2014 at 8:25 AM, Andrea Aime <[hidden email]> wrote:

Ah, looking at the implementation status table:
the following limitations would pop up if we used it:
* no labels painted (we use drawGlyphVector, as we often need to extract the glyph vector
  first to do layouts and conflict resolution)
* no graphic fills (we use TexturePaint, but setPaint only supports Color)
* no maps integrating raster data (we use drawRendererImage)
* given the comments about BasicStroke at the very end, not sure if the implementation can support dashed lines, or not

That said, if we had a significant performance boost, it might be worth contributing to
the project and get them implemented.

Cheers
Andrea


--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272
mob: <a href="tel:%2B39%20%C2%A0339%208844549" value="+393398844549" target="_blank">+39  339 8844549


-------------------------------------------------------



--
Tom Kunicki
Software Engineer | Boundless
[hidden email]
<a href="tel:917-460-7212" value="+19174607212" target="_blank">917-460-7212
@tomkunicki



--
Tom Kunicki
Software Engineer | Boundless
[hidden email]
917-460-7212
@tomkunicki

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Some interesting optmizations from Jonathan's load test

Tom Kunicki-4
In reply to this post by geowolf
Oops,

Didn't mean to go off list.  I'll take a peak at the rendering.  I'm a Netbeans user but w/ maven that's of no consequence...


On Fri, Jan 3, 2014 at 2:52 AM, Andrea Aime <[hidden email]> wrote:
On Fri, Jan 3, 2014 at 9:40 AM, Tom Kunicki <[hidden email]> wrote:

OpenGL/CL (err, the hardware behind it) can solve specific problems very well.  I was forced to try it to meet some requirements that seemed implausible, it worked and I was hooked.  The performance benefits tend to degrade quickly as attempts are made to generalize solutions (without a lot of resources).  Although, I did have some good luck with Aparapi for gridded dataset analysis.  When I used it in 2011 the generalization penalty was still pretty high (no local, only global memory) but it still offered some nice performance gains.  I've always tried to keep capable GPU(s) handy in my machines. As you said, It sounds like your GPU capabilities may have affected your results (need fast VRAM and lots of stream processors, you pay for this).  That all being said I wonder how well Sun^H^H^H Oracle was able to handle generalization in the face of optimization with Java2D.   I would love to look at hardware accelerated rendering, is the performance testing environment you've been using easy to set up?

Hum yes, nothing fancy. Jonathan prepared data set, relevant bits of the
data dir and a JMeter test script here:

I normally run GeoServer starting it directly from Eclipse (to make it quicker to
round trip between code changes and tests)

Btw, this ticket seems relevant to our case, we indeed need to write to a BufferedImage:

Cheers
Andrea

--
== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272
mob: <a href="tel:%2B39%20%C2%A0339%208844549" value="+393398844549" target="_blank">+39  339 8844549


-------------------------------------------------------



--
Tom Kunicki
Software Engineer | Boundless
[hidden email]
917-460-7212
@tomkunicki

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel