[MapProxy] Mapproxy Cleanups Speed

Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[MapProxy] Mapproxy Cleanups Speed

nsilva

Hi everybody.

I'm using Mapproxy to create a cache of a WMS layer on a production server to prevent direct access from the users to this server, I used in the past MapCache, but problems related to GetFeatureInfo and GetLegendGraphic with some WMS Servers made me try this alternative.

The problem:

I have a layer that is generated every 3-4h using as source a netcdf file...the backend server has to read that netcdf file, with high cost of processing the file, with the cache between the client and the server, only the first requests would take longer, but when I'm using the mapproxy-seed utility to do a cache cleanup, assuming that the user could use all 18 levels of that layer, is taking now 4h to complete 25% cleaning of the cache....making this process take about 16h to achieve 100% (again, I have new information being created every 3-4h)

Is there a way to achieve fast cache cleaning? I'll already set  remove_all: true , but even so it takes toooooo long even without folder timestamp compare...

I'm considering making a manual removal of the folder with rm -fR /cache_folder/* and the seeding for to: 5 just to fill the initial cache with some data.

Any suggestions/alternatives would be appreciated , thanks in advance!



--
Kind regards,
nsilva

_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy
Reply | Threaded
Open this post in threaded view
|

Re: [MapProxy] Mapproxy Cleanups Speed

Johannes Weskamm

Hi,


Try saving your cache in a database -> e.g. sqlite (https://mapproxy.org/docs/1.13.0/caches.html#sqlite) instead of a file based cache. That should speed up cache cleanup.


Greetings,

Johannes



Am 31.03.21 um 15:27 schrieb Nuno Silva:

Hi everybody.

I'm using Mapproxy to create a cache of a WMS layer on a production server to prevent direct access from the users to this server, I used in the past MapCache, but problems related to GetFeatureInfo and GetLegendGraphic with some WMS Servers made me try this alternative.

The problem:

I have a layer that is generated every 3-4h using as source a netcdf file...the backend server has to read that netcdf file, with high cost of processing the file, with the cache between the client and the server, only the first requests would take longer, but when I'm using the mapproxy-seed utility to do a cache cleanup, assuming that the user could use all 18 levels of that layer, is taking now 4h to complete 25% cleaning of the cache....making this process take about 16h to achieve 100% (again, I have new information being created every 3-4h)

Is there a way to achieve fast cache cleaning? I'll already set  remove_all: true , but even so it takes toooooo long even without folder timestamp compare...

I'm considering making a manual removal of the folder with rm -fR /cache_folder/* and the seeding for to: 5 just to fill the initial cache with some data.

Any suggestions/alternatives would be appreciated , thanks in advance!



--
Kind regards,
nsilva

_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy

_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy
Reply | Threaded
Open this post in threaded view
|

Re: [MapProxy] Mapproxy Cleanups Speed

Travis Kirstine


I think your best bet is to rm the folder using an alternative process.  In my experience it has been faster to use custom scripts to perform the cleanups especially if you are just looking to remove everything from a cache, as Johannes indicated a file cache may not be the best for this type of operation as it takes time for MP to walk the directories. 

I was thinking that you could maybe utilize dimensions to generate a new cache when you update the data however it appears MP does not support that yet, this is something you could do with MapCache.


On Wed, 31 Mar 2021 at 09:34, Johannes Weskamm <[hidden email]> wrote:

Hi,


Try saving your cache in a database -> e.g. sqlite (https://mapproxy.org/docs/1.13.0/caches.html#sqlite) instead of a file based cache. That should speed up cache cleanup.


Greetings,

Johannes



Am 31.03.21 um 15:27 schrieb Nuno Silva:

Hi everybody.

I'm using Mapproxy to create a cache of a WMS layer on a production server to prevent direct access from the users to this server, I used in the past MapCache, but problems related to GetFeatureInfo and GetLegendGraphic with some WMS Servers made me try this alternative.

The problem:

I have a layer that is generated every 3-4h using as source a netcdf file...the backend server has to read that netcdf file, with high cost of processing the file, with the cache between the client and the server, only the first requests would take longer, but when I'm using the mapproxy-seed utility to do a cache cleanup, assuming that the user could use all 18 levels of that layer, is taking now 4h to complete 25% cleaning of the cache....making this process take about 16h to achieve 100% (again, I have new information being created every 3-4h)

Is there a way to achieve fast cache cleaning? I'll already set  remove_all: true , but even so it takes toooooo long even without folder timestamp compare...

I'm considering making a manual removal of the folder with rm -fR /cache_folder/* and the seeding for to: 5 just to fill the initial cache with some data.

Any suggestions/alternatives would be appreciated , thanks in advance!



--
Kind regards,
nsilva

_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy
_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy

_______________________________________________
MapProxy mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapproxy