[MapProxy] MBtiles locks & takes ages

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[MapProxy] MBtiles locks & takes ages

Tobias Wendorff
Hi there,

I was about to preseed the area of Germany in OSM tiles and store them
in sqlite databases (MBtiles). The log shows that the sqlite-DBs are
locked every second. Isn't parallel processing possible with sqlite?

And the ETA also make problems. It would take about 2 weeks to render
Germany. Even with the old python script in OSM's repository, I can
render the data in some hours (png files only, no database). It's not
a hardware problem.

Anyone with an idea, what's going wrong here?

Best regards,
Tobias

_______________________________________________
MapProxy mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapproxy
Reply | Threaded
Open this post in threaded view
|

Re: [MapProxy] MBtiles locks & takes ages

Oliver Tonnhofer-5
Hi,


> On 22.10.2015, at 18:38, Tobias Wendorff <[hidden email]> wrote:
> I was about to preseed the area of Germany in OSM tiles and store them
> in sqlite databases (MBtiles). The log shows that the sqlite-DBs are
> locked every second. Isn't parallel processing possible with sqlite?
>
> And the ETA also make problems. It would take about 2 weeks to render
> Germany. Even with the old python script in OSM's repository, I can
> render the data in some hours (png files only, no database). It's not
> a hardware problem.
>
> Anyone with an idea, what's going wrong here?


A few users reported locking issues with SQLite, but the issue seem to be related to various system details as we are using SQLite caches without any issues.

Can you post the following details:

- Linux distribution and package versions (Kernel, Python, SQLite, MapProxy)
- Storage (local/remote, filesystem)
- MapProxy configuration
- Seeding configuration and options
- Speed of the seeding process when using the file cache (i.e. how many tiles does it write per second)
- Load of the system during seeding (load, %cpu and %wa output of top)

Please search the mailing list archive and GitHub issues for ‘sqlite’ and ‘locking’ for more information.


Regards,
Oliver

--
Oliver Tonnhofer  | Omniscale GmbH & Co KG  | http://omniscale.com
OpenStreetMap WMS and tile services         | http://maps.omniscale.com





_______________________________________________
MapProxy mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapproxy
Reply | Threaded
Open this post in threaded view
|

Re: [MapProxy] MBtiles locks & takes ages

Just van den Broecke
Hi,

One of the threads is :
http://lists.osgeo.org/pipermail/mapproxy/2015-March/002051.html

In my case the underlying file system was based on Linux LVM (under
Ubuntu 14.4) which IMO caused reproducable locking problems.

Best,

Just

On 23-10-15 12:22, Oliver Tonnhofer wrote:

> Hi,
>
>
>> On 22.10.2015, at 18:38, Tobias Wendorff <[hidden email]> wrote:
>> I was about to preseed the area of Germany in OSM tiles and store them
>> in sqlite databases (MBtiles). The log shows that the sqlite-DBs are
>> locked every second. Isn't parallel processing possible with sqlite?
>>
>> And the ETA also make problems. It would take about 2 weeks to render
>> Germany. Even with the old python script in OSM's repository, I can
>> render the data in some hours (png files only, no database). It's not
>> a hardware problem.
>>
>> Anyone with an idea, what's going wrong here?
>
>
> A few users reported locking issues with SQLite, but the issue seem to be related to various system details as we are using SQLite caches without any issues.
>
> Can you post the following details:
>
> - Linux distribution and package versions (Kernel, Python, SQLite, MapProxy)
> - Storage (local/remote, filesystem)
> - MapProxy configuration
> - Seeding configuration and options
> - Speed of the seeding process when using the file cache (i.e. how many tiles does it write per second)
> - Load of the system during seeding (load, %cpu and %wa output of top)
>
> Please search the mailing list archive and GitHub issues for ‘sqlite’ and ‘locking’ for more information.
>
>
> Regards,
> Oliver
>







_______________________________________________
MapProxy mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapproxy
Reply | Threaded
Open this post in threaded view
|

Re: [MapProxy] MBtiles locks & takes ages

Tobias Wendorff
Am Mo, 26.10.2015, 08:54 schrieb Just van den Broecke:
>
> In my case the underlying file system was based on Linux LVM (under
> Ubuntu 14.4) which IMO caused reproducable locking problems.

Thanks for the link. My test-environment uses mdadm RAID-0 SSD with
ext4 (no comments on this please, it's for testing only, hehe). I'll
do some benchmarks in the next days.

_______________________________________________
MapProxy mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapproxy