Can TileCache store tiles remotely

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Can TileCache store tiles remotely

David Alda Fernandez de Lezea-3
Hi list,

We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.

Thanks in advance.



Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

David Alda Fernandez de Lezea-3
Hi Chris,

Thanks for your response.

Why is that? I mean, does tilecache lock the directory in some way?

What would be the best approach to have several tilecache instances reading/writing simultaneously?

Thanks in advance.


De: [hidden email] [mailto:[hidden email]] En nombre de Chris Krahe
Enviado el: jueves, 08 de enero de 2015 11:49
Para: David Alda Fernandez de Lezea
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
Chris
On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
Hi list,

We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.

Thanks in advance.



Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

manel.clos
Hi David,

I'd just run more instances of TileCache, on a different servers. It looks a lot simpler and you can always proxy the requests if you don't want to change urls on clients, either with TileCache or for example nginx.

You can even generate tiles beforehand and copy/move/rsync them to the remote server running TileCache. MBTiles is your friend :)


Best,
 
Manel Clos
Ajuntament de Girona
Àrea d'Hisenda i Gestió Municipal / Règim Interior
Sistemes i Tecnologies de la Informació
Tel.: 972 419 082
Tel. intern: 1382
A/e: [hidden email] / www.girona.cat
FACEBOOK  |  TWITTER  |  YOUTUBE
 
 
 
 
 
 
 
 
 
-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]] En nombre de David Alda Fernandez de Lezea
Enviado el: jueves, 08 de enero de 2015 12:09
Para: Chris Krahe
CC: [hidden email]
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Hi Chris,

Thanks for your response.

Why is that? I mean, does tilecache lock the directory in some way?

What would be the best approach to have several tilecache instances reading/writing simultaneously?

Thanks in advance.


De: [hidden email] [mailto:[hidden email]] En nombre de Chris Krahe
Enviado el: jueves, 08 de enero de 2015 11:49
Para: David Alda Fernandez de Lezea
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
Chris
On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
Hi list,

We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.

Thanks in advance.



Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

Brian Case
In reply to this post by David Alda Fernandez de Lezea-3
David,

many tilecache instances could read and write from a ahared mount at the
same time. note you may want fiberchannel for this or very fast net
(40gbit). tilecache makes a lock before it tries to create a tile
(mkdir), this will prevent other threads or instances from writing to
the same file.

https://github.com/OSGeo/tilecache/blob/1ed36b439253d4c23612824928e522f5f45fd2d1/tilecache/TileCache/Caches/Disk.py#L147

you could also set up many instances of tilecache with memcache or redis
and point them to your existing cache


if you do choose to go with many instances of tilecache you may wish to
use the postgres config, or the memcache config with postgress behind
it.

also check out the expired config option, you can cache dynamic data by
just changing a date in the config whenever the data changes , the disk
and redis caches support this.


brian


On Thu, 2015-01-08 at 11:08 +0000, David Alda Fernandez de Lezea wrote:

> Hi Chris,
>
> Thanks for your response.
>
> Why is that? I mean, does tilecache lock the directory in some way?
>
> What would be the best approach to have several tilecache instances reading/writing simultaneously?
>
> Thanks in advance.
>
>
> De: [hidden email] [mailto:[hidden email]] En nombre de Chris Krahe
> Enviado el: jueves, 08 de enero de 2015 11:49
> Para: David Alda Fernandez de Lezea
> Asunto: Re: [Tilecache] Can TileCache store tiles remotely
>
> Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
> Chris
> On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
> Hi list,
>
> We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.
>
> Thanks in advance.
>
>
>
> Agur bero bat,
>
>
> David Alda Fernández de Lezea
> Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
> [hidden email] | www.hazi.eus
> T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba
>
>
>
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/tilecache
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/tilecache


_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

David Alda Fernandez de Lezea-3
In reply to this post by manel.clos
Hi Manel,

Thanks for your response. That's was I was thinking  to do.

But I have one question... How do I proxy the requests?

Thanks again.

Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290
Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba
 
*********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.

-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]]
Enviado el: viernes, 09 de enero de 2015 15:12
Para: David Alda Fernandez de Lezea; [hidden email]
CC: [hidden email]
Asunto: RE: [Tilecache] Can TileCache store tiles remotely

Hi David,

I'd just run more instances of TileCache, on a different servers. It looks a lot simpler and you can always proxy the requests if you don't want to change urls on clients, either with TileCache or for example nginx.

You can even generate tiles beforehand and copy/move/rsync them to the remote server running TileCache. MBTiles is your friend :)


Best,
 
Manel Clos
Ajuntament de Girona
Àrea d'Hisenda i Gestió Municipal / Règim Interior Sistemes i Tecnologies de la Informació
Tel.: 972 419 082
Tel. intern: 1382
A/e: [hidden email] / www.girona.cat FACEBOOK  |  TWITTER  |  YOUTUBE
 
 
 
 
 
 
 
 
 
-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]] En nombre de David Alda Fernandez de Lezea Enviado el: jueves, 08 de enero de 2015 12:09
Para: Chris Krahe
CC: [hidden email]
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Hi Chris,

Thanks for your response.

Why is that? I mean, does tilecache lock the directory in some way?

What would be the best approach to have several tilecache instances reading/writing simultaneously?

Thanks in advance.


De: [hidden email] [mailto:[hidden email]] En nombre de Chris Krahe Enviado el: jueves, 08 de enero de 2015 11:49
Para: David Alda Fernandez de Lezea
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
Chris
On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
Hi list,

We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.

Thanks in advance.



Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

David Alda Fernandez de Lezea-3
In reply to this post by Brian Case
Hi Brian,

Thanks for your response. We're planning to set up several servers, we haven't done anything so far.  I like the postgres config approach , is there any link to the documentation or some tutorial?

What is memchache?

Thanks,

Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290
Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba
 
*********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.


-----Mensaje original-----
De: Brian Case [mailto:[hidden email]]
Enviado el: viernes, 09 de enero de 2015 21:12
Para: David Alda Fernandez de Lezea
CC: Chris Krahe; [hidden email]
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

David,

many tilecache instances could read and write from a ahared mount at the same time. note you may want fiberchannel for this or very fast net (40gbit). tilecache makes a lock before it tries to create a tile (mkdir), this will prevent other threads or instances from writing to the same file.

https://github.com/OSGeo/tilecache/blob/1ed36b439253d4c23612824928e522f5f45fd2d1/tilecache/TileCache/Caches/Disk.py#L147

you could also set up many instances of tilecache with memcache or redis and point them to your existing cache


if you do choose to go with many instances of tilecache you may wish to use the postgres config, or the memcache config with postgress behind it.

also check out the expired config option, you can cache dynamic data by just changing a date in the config whenever the data changes , the disk and redis caches support this.


brian


On Thu, 2015-01-08 at 11:08 +0000, David Alda Fernandez de Lezea wrote:

> Hi Chris,
>
> Thanks for your response.
>
> Why is that? I mean, does tilecache lock the directory in some way?
>
> What would be the best approach to have several tilecache instances reading/writing simultaneously?
>
> Thanks in advance.
>
>
> De: [hidden email] [mailto:[hidden email]] En nombre de Chris
> Krahe Enviado el: jueves, 08 de enero de 2015 11:49
> Para: David Alda Fernandez de Lezea
> Asunto: Re: [Tilecache] Can TileCache store tiles remotely
>
> Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
> Chris
> On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
> Hi list,
>
> We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.
>
> Thanks in advance.
>
>
>
> Agur bero bat,
>
>
> David Alda Fernández de Lezea
> Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
> [hidden email] | www.hazi.eus
> T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de
> Arkaute s/n | 01192 Arkaute - Araba
>
>
>
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/tilecache
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/tilecache


_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: Can TileCache store tiles remotely

manel.clos
In reply to this post by David Alda Fernandez de Lezea-3
Hi David,

I'd personally use the nginx web server to proxy requests to the remote servers:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html

Any webserver will do.

You can expand your server capacity proxing to other servers as your needs increase. The downside is that the proxy server becomes critical as it is the only path to get the tiles.

I'd group my TileCache services in logical groups, every group running on different server, and use the url in clients. Direct access is going to be faster, no need for a proxy, and no critical point. Of course it will all depend on your scenario, as you might not be able to change URLs on clients...

Best,
 
 
Manel Clos
Ajuntament de Girona
Àrea d'Hisenda i Gestió Municipal / Règim Interior
Sistemes i Tecnologies de la Informació
Tel.: 972 419 082
Tel. intern: 1382
A/e: [hidden email] / www.girona.cat
FACEBOOK  |  TWITTER  |  YOUTUBE
 
 
 
 
 
 
 
 
 
-----Mensaje original-----
De: David Alda Fernandez de Lezea [mailto:[hidden email]]
Enviado el: lunes, 12 de enero de 2015 8:49
Para: Clos Crespo, Manel; [hidden email]
CC: [hidden email]
Asunto: RE: [Tilecache] Can TileCache store tiles remotely

Hi Manel,

Thanks for your response. That's was I was thinking  to do.

But I have one question... How do I proxy the requests?

Thanks again.

Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290
Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba
 
*********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.

-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]]
Enviado el: viernes, 09 de enero de 2015 15:12
Para: David Alda Fernandez de Lezea; [hidden email]
CC: [hidden email]
Asunto: RE: [Tilecache] Can TileCache store tiles remotely

Hi David,

I'd just run more instances of TileCache, on a different servers. It looks a lot simpler and you can always proxy the requests if you don't want to change urls on clients, either with TileCache or for example nginx.

You can even generate tiles beforehand and copy/move/rsync them to the remote server running TileCache. MBTiles is your friend :)


Best,
 
Manel Clos
Ajuntament de Girona
Àrea d'Hisenda i Gestió Municipal / Règim Interior Sistemes i Tecnologies de la Informació
Tel.: 972 419 082
Tel. intern: 1382
A/e: [hidden email] / www.girona.cat FACEBOOK  |  TWITTER  |  YOUTUBE
 
 
 
 
 
 
 
 
 
-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]] En nombre de David Alda Fernandez de Lezea Enviado el: jueves, 08 de enero de 2015 12:09
Para: Chris Krahe
CC: [hidden email]
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Hi Chris,

Thanks for your response.

Why is that? I mean, does tilecache lock the directory in some way?

What would be the best approach to have several tilecache instances reading/writing simultaneously?

Thanks in advance.


De: [hidden email] [mailto:[hidden email]] En nombre de Chris Krahe Enviado el: jueves, 08 de enero de 2015 11:49
Para: David Alda Fernandez de Lezea
Asunto: Re: [Tilecache] Can TileCache store tiles remotely

Network drive, sure. But be aware that a cache can only be used by one instance of TileCache... That is, you cannot have two or more instances reading and writing to the same set of tiles.
Chris
On Jan 8, 2015 2:05 AM, "David Alda Fernandez de Lezea" <[hidden email]> wrote:
Hi list,

We're planning to change our architecture as our data is growing and I was wondering if tilecache can store the tiles in a remote server or network drive.

Thanks in advance.



Agur bero bat,


David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
[hidden email] | www.hazi.eus
T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute - Araba



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache