JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

steve.omondi
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Kind Regards, Steve Omondi GIS & DB Developer/DBA Ramani Online, Ramani Geosystems
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

geowolf
Hi Steve,
not to my knowledge, I believe JDBCConfig is a one way ticket.
That said, I guess you can build a script that pulls the XML files from the database and writes
them out to disk in the correct location.

Cheers
Andrea


On Thu, Nov 9, 2017 at 11:02 AM, Steve Omondi <[hidden email]> wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users




--

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Niels Charlier
In reply to this post by steve.omondi

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

geowolf
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users




--

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

steve.omondi
In reply to this post by Niels Charlier
Hi Niels,

I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Copied in this conversation is my colleague Nicolas, and we are very much interested in the JDBCConfig especially on the work to optimize how it handles resource caching. This would be the best for our system if we can make it perform faster.

In the mean time, however, we are more interested on how we can bring back the JDBC Catalog to XML to make our PROD Environment faster as a workaround. We'd appreciate getting the code you wrote for yourself and try to do something with it. We'll probably add a few things here and there to make it work for our scenarion and communicate that back to you. 
Are you able to share the work you've done on Exporting JDBC Catalog back to XML?

Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Tue, Nov 14, 2017 at 7:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Kind Regards, Steve Omondi GIS & DB Developer/DBA Ramani Online, Ramani Geosystems
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Niels Charlier
In reply to this post by geowolf

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

geowolf
On Wed, Nov 15, 2017 at 11:30 AM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

Nobody at the PSC meeting yesterday was dealing with a production system using it (I have helped set up
a couple in the past, making several code adjustment to give them a fighting chance of working, but have 
not used it in a long while).
 

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

Nope, you're not reading the mail correctly. The main problem is a lack of a champion backing the module, answering
mails and reviewing PRs related to it. We did not list slowness as a reason to drop it, the reason is that currently
it's more or less abandoned and we don't know what to do about mails and PRs related to it.

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Niels Charlier
On 15-11-17 11:57, Andrea Aime wrote:

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

Nope, you're not reading the mail correctly. The main problem is a lack of a champion backing the module, answering
mails and reviewing PRs related to it. We did not list slowness as a reason to drop it, the reason is that currently
it's more or less abandoned and we don't know what to do about mails and PRs related to it.

I read your mail correctly, I meant the email from the user, and I was referring to the main problem with the _use_ of the module. However that problem may indirectly explain why the module is not more successful and therefore abandoned.

Regards
Niels

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

steve.omondi
In reply to this post by Niels Charlier
Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

​Our intention is not to drop it entirely. But to run more or less ​a catalog that is based on XML for some specific workspaces that we really need to be performant urgently.


While we have found jdbcconfig to be an excellent solution for clustering, it 
also
​ ​
has made 3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1 Geoserver 2.9 all other configurations kept constant.
​ And this is an issue that we can't ignore in PROD for our case.

We intent to move a few of the workspaces and layers especially complex LayerGoups fron the JDBC catalog to achieve better performance over convenience of the DB Catalog.​

Otherwise, JDBCConfig is an excellent module in use.



Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Kind Regards, Steve Omondi GIS & DB Developer/DBA Ramani Online, Ramani Geosystems
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Niels Charlier

Steve,

Sorry for the confusion. This was not in response to you, but Andrea about dropping the module from geoserver altogether. I understand your decision because I have also found that  jdbcconfig is a problematic bottle neck (but as I mentioned, mostly because of the poor cache support and the unnecessarily repetitive queries).

In the attachment you can find the code for exporting jdbc to hard drive. The file needs to be temporarily added to the class path and scanned by the spring appcontext, then when you are logged in as an admin you can call /jdbcexport from the browser to force the caching of the catalog. I hope you can use this somehow.

Kind Regards

Niels

On 15-11-17 12:51, Steve Omondi wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

​Our intention is not to drop it entirely. But to run more or less ​a catalog that is based on XML for some specific workspaces that we really need to be performant urgently.


While we have found jdbcconfig to be an excellent solution for clustering, it 
also
​ ​
has made 3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1 Geoserver 2.9 all other configurations kept constant.
​ And this is an issue that we can't ignore in PROD for our case.

We intent to move a few of the workspaces and layers especially complex LayerGoups fron the JDBC catalog to achieve better performance over convenience of the DB Catalog.​

Otherwise, JDBCConfig is an excellent module in use.



Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users

JdbcExport.java (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

steve.omondi
Great Niels,

We're gonna get right on this and we'll get back to you on the progress and if we find something interesting.

I appreciate it.

Regards,
Steve Ochieng

On Nov 15, 2017 17:02, "Niels Charlier" <[hidden email]> wrote:

Steve,

Sorry for the confusion. This was not in response to you, but Andrea about dropping the module from geoserver altogether. I understand your decision because I have also found that  jdbcconfig is a problematic bottle neck (but as I mentioned, mostly because of the poor cache support and the unnecessarily repetitive queries).

In the attachment you can find the code for exporting jdbc to hard drive. The file needs to be temporarily added to the class path and scanned by the spring appcontext, then when you are logged in as an admin you can call /jdbcexport from the browser to force the caching of the catalog. I hope you can use this somehow.

Kind Regards

Niels

On 15-11-17 12:51, Steve Omondi wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

​Our intention is not to drop it entirely. But to run more or less ​a catalog that is based on XML for some specific workspaces that we really need to be performant urgently.


While we have found jdbcconfig to be an excellent solution for clustering, it 
also
​ ​
has made 3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1 Geoserver 2.9 all other configurations kept constant.
​ And this is an issue that we can't ignore in PROD for our case.

We intent to move a few of the workspaces and layers especially complex LayerGoups fron the JDBC catalog to achieve better performance over convenience of the DB Catalog.​

Otherwise, JDBCConfig is an excellent module in use.



Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Kind Regards, Steve Omondi GIS & DB Developer/DBA Ramani Online, Ramani Geosystems
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

steve.omondi
In reply to this post by Niels Charlier
Hi Niels,

With this code, there is the section //export jdbc config to jdbc store and  package org.geoserver.jdbcstore.rest;

Do I have to have JDBCStore enabled as well? Currently I only have JDBCConfig enaabled.

Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 5:02 PM, Niels Charlier <[hidden email]> wrote:

Steve,

Sorry for the confusion. This was not in response to you, but Andrea about dropping the module from geoserver altogether. I understand your decision because I have also found that  jdbcconfig is a problematic bottle neck (but as I mentioned, mostly because of the poor cache support and the unnecessarily repetitive queries).

In the attachment you can find the code for exporting jdbc to hard drive. The file needs to be temporarily added to the class path and scanned by the spring appcontext, then when you are logged in as an admin you can call /jdbcexport from the browser to force the caching of the catalog. I hope you can use this somehow.

Kind Regards

Niels


On 15-11-17 12:51, Steve Omondi wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

​Our intention is not to drop it entirely. But to run more or less ​a catalog that is based on XML for some specific workspaces that we really need to be performant urgently.


While we have found jdbcconfig to be an excellent solution for clustering, it 
also
​ ​
has made 3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1 Geoserver 2.9 all other configurations kept constant.
​ And this is an issue that we can't ignore in PROD for our case.

We intent to move a few of the workspaces and layers especially complex LayerGoups fron the JDBC catalog to achieve better performance over convenience of the DB Catalog.​

Otherwise, JDBCConfig is an excellent module in use.



Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Kind Regards, Steve Omondi GIS & DB Developer/DBA Ramani Online, Ramani Geosystems
Reply | Threaded
Open this post in threaded view
|

Re: JDBCConfig export Catalog to XML Catalog in GEOSERVER DAT DIR

Niels Charlier

no you can remove those parts

Regards

Niels


On 21-11-17 08:44, Steve Omondi wrote:
Hi Niels,

With this code, there is the section //export jdbc config to jdbc store and  package org.geoserver.jdbcstore.rest;

Do I have to have JDBCStore enabled as well? Currently I only have JDBCConfig enaabled.

Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 5:02 PM, Niels Charlier <[hidden email]> wrote:

Steve,

Sorry for the confusion. This was not in response to you, but Andrea about dropping the module from geoserver altogether. I understand your decision because I have also found that  jdbcconfig is a problematic bottle neck (but as I mentioned, mostly because of the poor cache support and the unnecessarily repetitive queries).

In the attachment you can find the code for exporting jdbc to hard drive. The file needs to be temporarily added to the class path and scanned by the spring appcontext, then when you are logged in as an admin you can call /jdbcexport from the browser to force the caching of the catalog. I hope you can use this somehow.

Kind Regards

Niels


On 15-11-17 12:51, Steve Omondi wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

​Our intention is not to drop it entirely. But to run more or less ​a catalog that is based on XML for some specific workspaces that we really need to be performant urgently.


While we have found jdbcconfig to be an excellent solution for clustering, it 
also
​ ​
has made 3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1 Geoserver 2.9 all other configurations kept constant.
​ And this is an issue that we can't ignore in PROD for our case.

We intent to move a few of the workspaces and layers especially complex LayerGoups fron the JDBC catalog to achieve better performance over convenience of the DB Catalog.​

Otherwise, JDBCConfig is an excellent module in use.



Virus-free. www.avast.com

Kind regards,
Steve Omondi

On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <[hidden email]> wrote:

Oh really, drop it? I thought jdbcconfig was the recommended way for clustering and is particularly useful when you are working with huge catalogs. I assumed it was actually being used in production by people.

The main problem seems to be the performance issues though, as this email also proves, as well as both of the two PR's that are open who are related to the same thing. But I believe this is rather solvable and would make a huge difference.

I can review these PRs no problem, since they are actually related to the work I am doing for this module.

Perhaps these improvements there can blow some new life into it.

Regards

Niels


On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we should just drop
JDBCConfig, since:
  • mails related to it are not really getting answered
  • pull requests related to it are sitting there not getting reviewed
We still have to see if there is some interested, but we need someone to champion the module
just enough to make it worth keeping around (I'm not talking about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.

Cheers
Andrea


On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <[hidden email]> wrote:

Hello Steve,

First I'd like to say I have been doing some work on considerable performance improvements for jdbcconfig. It  happens to be the case that jdbcconfig doesn't take good use of its cache and repeatedly sends the same queries over and over again. It looks promising but I still need to do some improvements and write some tests and I had other work coming on top of it. But this is definitely coming.

Also, I wrote some code in geoserver for myself to conveniently export the jdbc catalog back to the file system. I found a way to do this easily by firing a bunch of catalog change events to the file system catalog. However, I am not sure to make it easily available to end users at this point, I assume that just giving you the code won't be very helpful for you. Perhaps we can work something out if you are interested.

Regards

Niels



On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,

I have JDBCConfig enabled and connected to a Postgres DB. I have over 500 layers spread across different workspaces and layergroups.

After running performance tests for GetMap requests against this configuration versus XML Catalog based geoserver I really need to revert my catalog back to XML in GEOSERVER DAT DIR and probably share it using NFS.

Is their a method to export the Database Catalog (based on JDBCConfig) back to XML Catalog?

Kind regards,
Steve Omondi

Virus-free. www.avast.com


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html [hidden email] https://lists.sourceforge.net/lists/listinfo/geoserver-users
--

Regards,

Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054  Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39  339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users