DiskCache through multiple disks

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

DiskCache through multiple disks

pmgmendes
Hi,

Can anyone give me a scenario where one could set up multiple disks to store the tiles in a distributed way? All the disks are in machines in the same infrastructure.

Thanks!

Mendes.
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

Jorge Arévalo
On Mon, Jan 16, 2012 at 3:12 PM, Pedro Miguel  Mendes
<[hidden email]> wrote:

> Hi,
>
> Can anyone give me a scenario where one could set up multiple disks to store
> the tiles in a distributed way? All the disks are in machines in the same
> infrastructure.
>
> Thanks!
>
> Mendes.
>
> --
> View this message in context: http://osgeo-org.1803224.n2.nabble.com/DiskCache-through-multiple-disks-tp7192591p7192591.html
> Sent from the TileCache mailing list archive at Nabble.com.
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/tilecache

What about a distributed TMS/WMTS service? Using a Cloud infrastructure.

--
Jorge Arévalo
Internet & Mobility Division, DEIMOS
[hidden email]
http://mobility.grupodeimos.com/
http://www.libregis.org
http://geohash.org/ezjqgrgzz0g
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

christopher.schmidt

On Jan 16, 2012, at 9:18 AM, ext Jorge Arévalo wrote:

> On Mon, Jan 16, 2012 at 3:12 PM, Pedro Miguel  Mendes
> <[hidden email]> wrote:
>> Hi,
>>
>> Can anyone give me a scenario where one could set up multiple disks to store
>> the tiles in a distributed way? All the disks are in machines in the same
>> infrastructure.
>>
>
> What about a distributed TMS/WMTS service? Using a Cloud infrastructure.

... What? How is that related to the question he's asking? :p

Pedro: I don't have any suggestions on storing tiles across multiple machines.
One of the things I Know people have done before is given multiple machines
different DNS names, and then used deterministic round-robin (like the tile
generation in OpenLayers) -- then, each server only gets some of the requests
(where 'some' is the number of servers you have in the round robin).

-- Chris_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

pmgmendes
Hi Chris and Jorge,

@Jorge
I'm looking for options using only our client infrastructure.

@ Chris 
That's an option, for sure. What I was looking to know is about the possibility of splitting the tiles through different disks (on different machines or not) and only having one tilecache instance. It will be achievable having several disks in the same machine each one keeping the tiles of a different zoom level?

We're evaluating options, right now.

Thanks.

On Mon, Jan 16, 2012 at 2:33 PM, <[hidden email]> wrote:

On Jan 16, 2012, at 9:18 AM, ext Jorge Arévalo wrote:

> On Mon, Jan 16, 2012 at 3:12 PM, Pedro Miguel  Mendes
> <[hidden email]> wrote:
>> Hi,
>>
>> Can anyone give me a scenario where one could set up multiple disks to store
>> the tiles in a distributed way? All the disks are in machines in the same
>> infrastructure.
>>
>
> What about a distributed TMS/WMTS service? Using a Cloud infrastructure.

... What? How is that related to the question he's asking? :p

Pedro: I don't have any suggestions on storing tiles across multiple machines.
One of the things I Know people have done before is given multiple machines
different DNS names, and then used deterministic round-robin (like the tile
generation in OpenLayers) -- then, each server only gets some of the requests
(where 'some' is the number of servers you have in the round robin).

-- Chris


_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

Jorge Arévalo
Hi,

Ok, I misunderstood. Sorry.

I actually have a slightly different situation, with different
machines, each of one storing tiles of all the desired zoom levels but
from different parts of the world. All machines together form a big
coverage. Trying MapProxy and TileCache. But I think it doesn't fit
your problem.

Sorry again.



On Mon, Jan 16, 2012 at 3:42 PM, Pedro Mendes <[hidden email]> wrote:

> Hi Chris and Jorge,
>
> @Jorge
> I'm looking for options using only our client infrastructure.
>
> @ Chris
> That's an option, for sure. What I was looking to know is about
> the possibility of splitting the tiles through different disks (on different
> machines or not) and only having one tilecache instance. It will be
> achievable having several disks in the same machine each one keeping the
> tiles of a different zoom level?
>
> We're evaluating options, right now.
>
> Thanks.
>
>
> On Mon, Jan 16, 2012 at 2:33 PM, <[hidden email]> wrote:
>>
>>
>> On Jan 16, 2012, at 9:18 AM, ext Jorge Arévalo wrote:
>>
>> > On Mon, Jan 16, 2012 at 3:12 PM, Pedro Miguel  Mendes
>> > <[hidden email]> wrote:
>> >> Hi,
>> >>
>> >> Can anyone give me a scenario where one could set up multiple disks to
>> >> store
>> >> the tiles in a distributed way? All the disks are in machines in the
>> >> same
>> >> infrastructure.
>> >>
>> >
>> > What about a distributed TMS/WMTS service? Using a Cloud infrastructure.
>>
>> ... What? How is that related to the question he's asking? :p
>>
>> Pedro: I don't have any suggestions on storing tiles across multiple
>> machines.
>> One of the things I Know people have done before is given multiple
>> machines
>> different DNS names, and then used deterministic round-robin (like the
>> tile
>> generation in OpenLayers) -- then, each server only gets some of the
>> requests
>> (where 'some' is the number of servers you have in the round robin).
>>
>> -- Chris
>
>



--
Jorge Arévalo
Internet & Mobility Division, DEIMOS
[hidden email]
http://mobility.grupodeimos.com/
http://www.libregis.org
http://geohash.org/ezjqgrgzz0g
_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

Marco Nijdam
In reply to this post by pmgmendes
As quoted from Pedro Miguel  Mendes:
> Can anyone give me a scenario where one could set up multiple disks to
> store
> the tiles in a distributed way? All the disks are in machines in the same
> infrastructure.

How about using a RAID controller with striping across multiple disks.
That could share the write among disks?

Or implement a MultipleDiskCache.py to extend or replace the existing
DiskCache in tilecache with storing the tiles om multiple disks.
E.g. with two disks, store even-numbered tiles on one, and odd-numbered ones
on another.
Adding or removing disks will then be a challenge, since all tiles will be
at the wrong disk.
To avoid that, maybe you could work with e.g. 10 virtual locations for
tiles, and assign one or more to each disk you have. When you want to add a
disk, you only have to re-assign one or more virtual locations from one disk
to the other, and move all files in that virtual location to the other
disk.
Your MultipleDiskCache.py implementation could save the files to the
virtual locations, by using (tile-id mod 10) as the index to them.

Kind regards,
--
-- Marco Nijdam
--



_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

pmgmendes
In reply to this post by Jorge Arévalo
>From my understanding to attain that set up with TileCache you must have one TileCache instance for each region you.

In my case I'm trying upgrade only the disk storage without adding and configuring more TileCache instances. Don't if that's possible, though...

Thanks for the feedback.

2012/1/16 Jorge Arévalo <[hidden email]>
Hi,

Ok, I misunderstood. Sorry.

I actually have a slightly different situation, with different
machines, each of one storing tiles of all the desired zoom levels but
from different parts of the world. All machines together form a big
coverage. Trying MapProxy and TileCache. But I think it doesn't fit
your problem.

Sorry again.



On Mon, Jan 16, 2012 at 3:42 PM, Pedro Mendes <[hidden email]> wrote:
> Hi Chris and Jorge,
>
> @Jorge
> I'm looking for options using only our client infrastructure.
>
> @ Chris
> That's an option, for sure. What I was looking to know is about
> the possibility of splitting the tiles through different disks (on different
> machines or not) and only having one tilecache instance. It will be
> achievable having several disks in the same machine each one keeping the
> tiles of a different zoom level?
>
> We're evaluating options, right now.
>
> Thanks.
>
>
> On Mon, Jan 16, 2012 at 2:33 PM, <[hidden email]> wrote:
>>
>>
>> On Jan 16, 2012, at 9:18 AM, ext Jorge Arévalo wrote:
>>
>> > On Mon, Jan 16, 2012 at 3:12 PM, Pedro Miguel  Mendes
>> > <[hidden email]> wrote:
>> >> Hi,
>> >>
>> >> Can anyone give me a scenario where one could set up multiple disks to
>> >> store
>> >> the tiles in a distributed way? All the disks are in machines in the
>> >> same
>> >> infrastructure.
>> >>
>> >
>> > What about a distributed TMS/WMTS service? Using a Cloud infrastructure.
>>
>> ... What? How is that related to the question he's asking? :p
>>
>> Pedro: I don't have any suggestions on storing tiles across multiple
>> machines.
>> One of the things I Know people have done before is given multiple
>> machines
>> different DNS names, and then used deterministic round-robin (like the
>> tile
>> generation in OpenLayers) -- then, each server only gets some of the
>> requests
>> (where 'some' is the number of servers you have in the round robin).
>>
>> -- Chris
>
>





_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache
Reply | Threaded
Open this post in threaded view
|

Re: DiskCache through multiple disks

pmgmendes
In reply to this post by Marco Nijdam
@Marco Thanks for the hints!

_______________________________________________
Tilecache mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/tilecache