geowebcache not clearing cache

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

geowebcache not clearing cache

idharper
I have inherited a system and the original people have left the company so I
need some help.

It looks like we have some layers that keep increasing in size on the disk
and for which the diskquote for size is not clearing tiles out.

They appear to be tiles which have a TIME parameter on them.

Is it possible to run a truncate for a particular layer where TIME is before
a certain date so that I can clear out a large amount of space.

Thanks




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: geowebcache not clearing cache

geowolf
Hi,
sorry to hear disk quota is not working for you, we had reports, but none that would be reproducible so far,
with small examples it normally does it job.
Another potential issue is having reported a block size that does not really match the disk one.

About dropping tiles before a certain date, I don't think there is a direct way, GWC has no understanding
of the parameters and their data type, so it cannot be instructed to drop before a certain date.
In order to do drops one has to provide the exact parameter values involved, and use an approach such as the one described here:
Another hiccup is that it's not possible to figure what what time values are in the cache, because the set of parameters gets
hashed using sha-1, which is a one way hashing algorithm, before hitting the disk (you might have seen the hashes in the folder names).

A possible approach is to clean up the quota store (the database contents, making a backup of it just in case)
and restart GWC, it should notice the quota is empty and go an collect statistics again. Of course, that might take quite some time,
as it has to go and scan all the available tiles, so maybe do the cleanup (by file/directory date?) manually before restarting the system.

Hope this helps

Cheers
Andrea


On Mon, Jan 28, 2019 at 3:17 PM idharper <[hidden email]> wrote:
I have inherited a system and the original people have left the company so I
need some help.

It looks like we have some layers that keep increasing in size on the disk
and for which the diskquote for size is not clearing tiles out.

They appear to be tiles which have a TIME parameter on them.

Is it possible to run a truncate for a particular layer where TIME is before
a certain date so that I can clear out a large amount of space.

Thanks




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: geowebcache not clearing cache

geowolf
On Mon, Jan 28, 2019 at 4:46 PM Andrea Aime <[hidden email]> wrote:
Another hiccup is that it's not possible to figure what what time values are in the cache, because the set of parameters gets
hashed using sha-1, which is a one way hashing algorithm, before hitting the disk (you might have seen the hashes in the folder names).

I actually did not remember one important detail about this, GWC keeps some breadcrumbs around, if you go in the GEOSERVER_DATA_DIR/gwc/<layername>
directory you should fine a metadata.properties that allows tracking back from the sha to the params inside of it, I tried
with a topp:states with 3 different styles, and the contents ended up being:

parameters.f0023dc7bc347fee7a3a04dc797f2223f74e3448=STYLES%3Dpolygon
parameters.7510004a12f49fdd49a2ba366e9c4594be7e4358=STYLES%3Dpophatch

If you cached on time, I'd expect to find the sha-1 alongside the time value on the right end

Hope this helps

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: geowebcache not clearing cache

idharper
Thanks i will look into these, the layers in question are satellite and radar
images so the worst case we can truncate everything and let it repopulate
periodically, but it would be nice if the diskquota system could handle
these tiles and cull one older than say 30 days.

Ian



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: geowebcache not clearing cache

geowolf
On Tue, Jan 29, 2019 at 9:56 AM idharper <[hidden email]> wrote:
Thanks i will look into these, the layers in question are satellite and radar
images so the worst case we can truncate everything and let it repopulate
periodically, but it would be nice if the diskquota system could handle
these tiles and cull one older than say 30 days.

It should and normally does, as said, often the reports of misbehavior are due
to hand manipulation of the cache, mis-configuration, or improper setup (e.g., in clusters).
It's hard to tell because normally we get called after the fact and it's not possible to reconstruct
what actually happened.

I have the feeling there might be a bug in there somewhere too, but if it's there, we haven't found
any way to reproduce it so far (attempts to cause it on purpose failed to deliver results, with
the quota working as expected).

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users