Re: [MapProxy] Re-enabling mapnik cache per-process while seeding

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Re: [MapProxy] Re-enabling mapnik cache per-process while seeding

Arne Babenhauserheide

Sorry for the strange format of the previous email. I’m now sending it
again in plain format.


We are using highly concurrent seeding via Mapnik and a large part of
our processing time is spent re-creating the mapnik.Map(0,0) in

To reduce these costs, I switched the code to one cache per process and
re-enabled the cache. Using 64 cores we saw a speedup of roughly factor
5 to 10, and we would like to contribute these changes back.

The current implementation is a quick hack, but I received permission to
spend a few work hours to clean it up such that it can be included in
mapproxy. Therefore I would like to ask for guidance how to contribute
this change in a way which makes it easy for you to merge it.

The following is the current hack. This hack is far from perfect and has
some side-effects for on-the-fly rendering (not seeding) which I want to
clean up before I create a pull-request, but I hope you can give some
hints how best to move forward to get it ready.

diff --git a/mapproxy/source/ b/mapproxy/source/
index 8b87bfc3..69747a48 100644
--- a/mapproxy/source/
+++ b/mapproxy/source/
@@ -18,6 +18,7 @@ from __future__ import absolute_import
 import sys
 import time
 import threading
+import multiprocessing

 from mapproxy.grid import tile_grid
 from mapproxy.image import ImageSource
@@ -26,7 +27,7 @@ from mapproxy.layer import MapExtent,
DefaultMapExtent, BlankImage, MapLayer
 from mapproxy.source import  SourceError
 from mapproxy.client.log import log_request
 from import reraise_exception
-from mapproxy.util.async_ import run_non_blocking
+from mapproxy.util.async import run_non_blocking
 from mapproxy.compat import BytesIO

@@ -57,8 +58,10 @@ class MapnikSource(MapLayer):
         self.layers = set(layers) if layers else None
         self.scale_factor = scale_factor
         self.lock = lock
-        self._map_objs = {}remove
-        self._map_objs_lock = threading.Lock()
+        global _map_objs
+        _map_objs = {}
+        global _map_objs_lock
+        _map_objs_lock = threading.Lock()
         self._cache_map_obj = reuse_map_objects
         if self.coverage:
             self.extent = MapExtent(self.coverage.bbox, self.coverage.srs)
@@ -94,20 +97,22 @@ class MapnikSource(MapLayer):
             return self.render_mapfile(mapfile, query)

     def map_obj(self, mapfile):
-        if not self._cache_map_obj:
-            m = mapnik.Map(0, 0)
-            mapnik.load_map(m, str(mapfile))
-            return m
+        proc = multiprocessing.current_process()
+        process_id = proc._identity # identity is a tuple with a number
+        cachekey = (process_id, mapfile)
         # cache loaded map objects
         # only works when a single proc/thread accesses this object
         # (forking the render process doesn't work because of open database
         #  file handles that gets passed to the child)
-        if mapfile not in self._map_objs:
-            with self._map_objs_lock:
-                if mapfile not in self._map_objs:
+        if cachekey not in _map_objs:
+            with _map_objs_lock:
+                if cachekey not in _map_objs:
                     m = mapnik.Map(0, 0)
+                    _map_objs[cachekey] = m # before load, dangerous!
                     mapnik.load_map(m, str(mapfile))
+        return _map_objs[cachekey]

     def render_mapfile(self, mapfile, query):
         return run_non_blocking(self._render_mapfile, (mapfile, query))

Best wishes,
Arne Babenhauserheide

Disy Informationssysteme GmbH
Arne Babenhauserheide
+49 721 16006 443, [hidden email]
Firmensitz: Ludwig-Erhard-Allee 6, 76131 Karlsruhe
Registergericht: Amtsgericht Mannheim, HRB 107964
Geschäftsführer: Claus Hofmann
Bitte beachten Sie folgende Informationen für Kunden, Lieferanten und Bewerber
- Datenschutz:
- Informationspflichten:  

MapProxy mailing list
[hidden email]

arne_babenhauserheide.vcf (349 bytes) Download Attachment