shapefiles getting corrupted

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

shapefiles getting corrupted

Nathan Farrington-2
Hi,
 
I've been using GeoServer for a multiuser application. I have about 10 tables, each with about 20 columns. There are about 10 users as well. I set up two different test servers with two different GeoServer installations, one with shapefiles and one with PostgreSQL/PostGIS. They both work fine with just one user.
 
My question is how will they scale with multiple users? I wanted to deploy using the shapefile configured GeoServer, but I started noticing problems when using it with multiple clients. Sometimes a shapefile would get corrupted and contain duplicate entries. The main advantage to using this configuration is that it is easier to distribute. If I went with PostGIS, I would need to do some remote administration, which is difficult for this application.
 
Has anyone seen problems using shapefiles with multiple simultaneous users? Would a backend like PostGIS solve this problem? Is there a different backend that you would recommend?
 
Thanks,
 
Nathan
Reply | Threaded
Open this post in threaded view
|

Re: shapefiles getting corrupted

Brent Owens
If you have two different instances of Geoserver running, and they are
both targetting the same shapefile, you will get corruption. Just like
any file access system, you need a single controlling source to prevent
errors.
With a database backed data source, it controls access with transactions
but the Geoserver instances might be out of sync. I do not recommend
having multiple Geoservers hitting the same datasource. I would use just
one instance as it can handle multiple users accessing the data. You
won't have corruption that way.

Now if you are using one instance of Geoserver, and multiple users are
editing, and you are getting corruption, then that is a bug. If that is
a case, give us some info and we will add it to the bug list.

Brent Owens
TOPP



Nathan Farrington wrote:

> Hi,
>  
> I've been using GeoServer for a multiuser application. I have about 10
> tables, each with about 20 columns. There are about 10 users as well.
> I set up two different test servers with two different GeoServer
> installations, one with shapefiles and one with PostgreSQL/PostGIS.
> They both work fine with just one user.
>  
> My question is how will they scale with multiple users? I wanted to
> deploy using the shapefile configured GeoServer, but I started
> noticing problems when using it with multiple clients. Sometimes a
> shapefile would get corrupted and contain duplicate entries. The main
> advantage to using this configuration is that it is easier to
> distribute. If I went with PostGIS, I would need to do some remote
> administration, which is difficult for this application.
>  
> Has anyone seen problems using shapefiles with multiple simultaneous
> users? Would a backend like PostGIS solve this problem? Is there a
> different backend that you would recommend?
>  
> Thanks,
>  
> Nathan



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: shapefiles getting corrupted

Chris Holmes-2
Yeah, this is exactly the kinds of reasons we don't recommend doing
transactions on shapefiles.  We should actually explicitly put this
reason in the docs.  To do transactions you want a real spatial
database - databases are created explicitly to deal with these types of
issues, so that the data doesn't get corrupted when thousands of users
are making use of it.  With GeoServer we make shapefiles available for
people to 'try out', the shortest path to get something working.  But
if you are running a production environment, backending with what is
basically a tranfer file format, you will get problems.  I actually
don't feel that it's a bug that we don't handle multiple users on one
shapefile, and it doesn't surprise me one gets corruption, I don't
think we really do locking on it (could be wrong).  I think we just
need better warnings in the docs.  Or rather, explanations, since we do
advice against it currently.

Chris

Quoting Brent Owens <[hidden email]>:

> If you have two different instances of Geoserver running, and they
> are
> both targetting the same shapefile, you will get corruption. Just
> like
> any file access system, you need a single controlling source to
> prevent
> errors.
> With a database backed data source, it controls access with
> transactions
> but the Geoserver instances might be out of sync. I do not recommend
> having multiple Geoservers hitting the same datasource. I would use
> just
> one instance as it can handle multiple users accessing the data. You
> won't have corruption that way.
>
> Now if you are using one instance of Geoserver, and multiple users
> are
> editing, and you are getting corruption, then that is a bug. If that
> is
> a case, give us some info and we will add it to the bug list.
>
> Brent Owens
> TOPP
>
>
>
> Nathan Farrington wrote:
>
> > Hi,
> >
> > I've been using GeoServer for a multiuser application. I have about
> 10
> > tables, each with about 20 columns. There are about 10 users as
> well.
> > I set up two different test servers with two different GeoServer
> > installations, one with shapefiles and one with PostgreSQL/PostGIS.
> > They both work fine with just one user.
> >
> > My question is how will they scale with multiple users? I wanted to
> > deploy using the shapefile configured GeoServer, but I started
> > noticing problems when using it with multiple clients. Sometimes a
> > shapefile would get corrupted and contain duplicate entries. The
> main
> > advantage to using this configuration is that it is easier to
> > distribute. If I went with PostGIS, I would need to do some remote
> > administration, which is difficult for this application.
> >
> > Has anyone seen problems using shapefiles with multiple
> simultaneous
> > users? Would a backend like PostGIS solve this problem? Is there a
> > different backend that you would recommend?
> >
> > Thanks,
> >
> > Nathan
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads,
> discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Geoserver-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>




----------------------------------------------------------
This mail sent through IMP: https://webmail.limegroup.com/


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: shapefiles getting corrupted

Nathan Farrington-2
Thank you Chris,
 
We ran some more tests and the shapefile gets corrupted very soon after several clients start to perform transactions. This happens every time.
 
So you would recommend using PostGIS for this application, or is there a different database you recommend?
 
Thanks again,
 
Nathan

Chris Holmes <[hidden email]> wrote:
Yeah, this is exactly the kinds of reasons we don't recommend doing
transactions on shapefiles. We should actually explicitly put this
reason in the docs. To do transactions you want a real spatial
database - databases are created explicitly to deal with these types of
issues, so that the data doesn't get corrupted when thousands of users
are making use of it. With GeoServer we make shapefiles available for
people to 'try out', the shortest path to get something working. But
if you are running a production environment, backending with what is
basically a tranfer file format, you will get problems. I actually
don't feel that it's a bug that we don't handle multiple users on one
shapefile, and it doesn't surprise me one gets corruption, I don't
think we really do locking on it (could be wrong). I think we just
need better warnings in the docs. Or rather, explanations, since we do
advice against it currently.

Chris

Quoting Brent Owens :

> If you have two different instances of Geoserver running, and they
> are
> both targetting the same shapefile, you will get corruption. Just
> like
> any file access system, you need a single controlling source to
> prevent
> errors.
> With a database backed data source, it controls access with
> transactions
> but the Geoserver instances might be out of sync. I do not recommend
> having multiple Geoservers hitting the same datasource. I would use
> just
> one instance as it can handle multiple users accessing the data. You
> won't have corruption that way.
>
> Now if you are using one instance of Geoserver, and multiple users
> are
> editing, and you are getting corruption, then that is a bug. If that
> is
> a case, give us some info and we will add it to the bug list.
>
> Brent Owens
> TOPP
>
>
>
> Nathan Farrington wrote:
>
> > Hi,
> >
> > I've been using GeoServer for a multiuser application. I have about
> 10
> > tables, each with about 20 columns. There are about 10 users as
> well.
> > I set up two different test servers with two different GeoServer
> > installations, one with shapefiles and one with PostgreSQL/PostGIS.
> > They both work fine with just one user.
> >
> > My question is how will they scale with multiple users? I wanted to
> > deploy using the shapefile configured GeoServer, but I started
> > noticing problems when using it with multiple clients. Sometimes a
> > shapefile would get corrupted and contain duplicate entries. The
> main
> > advantage to using this configuration is that it is easier to
> > distribute. If I went with PostGIS, I would need to do some remote
> > administration, which is difficult for this application.
> >
> > Has anyone seen problems using shapefiles with multiple
> simultaneous
> > users? Would a backend like PostGIS solve this problem? Is there a
> > different backend that you would recommend?
> >
> > Thanks,
> >
> > Nathan
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads,
> discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> Geoserver-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>




----------------------------------------------------------
This mail sent through IMP: https://webmail.limegroup.com/
Reply | Threaded
Open this post in threaded view
|

Re: shapefiles getting corrupted

Justin Deoliveira-3
I would say postgis is definitley your best bet. Transaction suppport is
good and its straight forward to set up.

-Justin

Nathan Farrington wrote:

> Thank you Chris,
>  
> We ran some more tests and the shapefile gets corrupted very soon after
> several clients start to perform transactions. This happens every time.
>  
> So you would recommend using PostGIS for this application, or is there a
> different database you recommend?
>  
> Thanks again,
>  
> Nathan
>
> */Chris Holmes <[hidden email]>/* wrote:
>
>     Yeah, this is exactly the kinds of reasons we don't recommend doing
>     transactions on shapefiles. We should actually explicitly put this
>     reason in the docs. To do transactions you want a real spatial
>     database - databases are created explicitly to deal with these types of
>     issues, so that the data doesn't get corrupted when thousands of users
>     are making use of it. With GeoServer we make shapefiles available for
>     people to 'try out', the shortest path to get something working. But
>     if you are running a production environment, backending with what is
>     basically a tranfer file format, you will get problems. I actually
>     don't feel that it's a bug that we don't handle multiple users on one
>     shapefile, and it doesn't surprise me one gets corruption, I don't
>     think we really do locking on it (could be wrong). I think we just
>     need better warnings in the docs. Or rather, explanations, since we do
>     advice against it currently.
>
>     Chris
>
>     Quoting Brent Owens :
>
>      > If you have two different instances of Geoserver running, and they
>      > are
>      > both targetting the same shapefile, you will get corruption. Just
>      > like
>      > any file access system, you need a single controlling source to
>      > prevent
>      > errors.
>      > With a database backed data source, it controls access with
>      > transactions
>      > but the Geoserver instances might be out of sync. I do not recommend
>      > having multiple Geoservers hitting the same datasource. I would use
>      > just
>      > one instance as it can handle multiple users accessing the data. You
>      > won't have corruption that way.
>      >
>      > Now if you are using one instance of Geoserver, and multiple users
>      > are
>      > editing, and you are getting corruption, then that is a bug. If that
>      > is
>      > a case, give us some info and we will add it to the bug list.
>      >
>      > Brent Owens
>      > TOPP
>      >
>      >
>      >
>      > Nathan Farrington wrote:
>      >
>      > > Hi,
>      > >
>      > > I've been using GeoServer for a multiuser application. I have about
>      > 10
>      > > tables, each with about 20 columns. There are about 10 users as
>      > well.
>      > > I set up two different test servers with two different GeoServer
>      > > installations, one with shapefiles and one with PostgreSQL/PostGIS.
>      > > They both work fine with just one user.
>      > >
>      > > My question is how will they scale with multiple users? I wanted to
>      > > deploy using the shapefile configured GeoServer, but I started
>      > > noticing problems when using it with multiple clients. Sometimes a
>      > > shapefile would get corrupted and contain duplicate entries. The
>      > main
>      > > advantage to using this configuration is that it is easier to
>      > > distribute. If I went with PostGIS, I would need to do some remote
>      > > administration, which is difficult for this application.
>      > >
>      > > Has anyone seen problems using shapefiles with multiple
>      > simultaneous
>      > > users? Would a backend like PostGIS solve this problem? Is there a
>      > > different backend that you would recommend?
>      > >
>      > > Thanks,
>      > >
>      > > Nathan
>      >
>      >
>      >
>      > -------------------------------------------------------
>      > This SF.Net email is sponsored by:
>      > Power Architecture Resource Center: Free content, downloads,
>      > discussions,
>      > and more. http://solutions.newsforge.com/ibmarch.tmpl
>      > _______________________________________________
>      > Geoserver-users mailing list
>      > [hidden email]
>      > https://lists.sourceforge.net/lists/listinfo/geoserver-users
>      >
>
>
>
>
>     ----------------------------------------------------------
>     This mail sent through IMP: https://webmail.limegroup.com/


--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: shapefiles getting corrupted

Brent Owens
It's also quite quick and free!
Here is the download location for PostGIS:
http://postgis.refractions.net/download/
Most people on the list here and on the PostGIS list have experience
setting it up, in case you run into troubles.

Good luck,

Brent Owens
TOPP



Justin Deoliveira wrote:

> I would say postgis is definitley your best bet. Transaction suppport
> is good and its straight forward to set up.
>
> -Justin
>
> Nathan Farrington wrote:
>
>> Thank you Chris,
>>  
>> We ran some more tests and the shapefile gets corrupted very soon
>> after several clients start to perform transactions. This happens
>> every time.
>>  
>> So you would recommend using PostGIS for this application, or is
>> there a different database you recommend?
>>  
>> Thanks again,
>>  
>> Nathan
>>
>> */Chris Holmes <[hidden email]>/* wrote:
>>
>>     Yeah, this is exactly the kinds of reasons we don't recommend doing
>>     transactions on shapefiles. We should actually explicitly put this
>>     reason in the docs. To do transactions you want a real spatial
>>     database - databases are created explicitly to deal with these
>> types of
>>     issues, so that the data doesn't get corrupted when thousands of
>> users
>>     are making use of it. With GeoServer we make shapefiles available
>> for
>>     people to 'try out', the shortest path to get something working. But
>>     if you are running a production environment, backending with what is
>>     basically a tranfer file format, you will get problems. I actually
>>     don't feel that it's a bug that we don't handle multiple users on
>> one
>>     shapefile, and it doesn't surprise me one gets corruption, I don't
>>     think we really do locking on it (could be wrong). I think we just
>>     need better warnings in the docs. Or rather, explanations, since
>> we do
>>     advice against it currently.
>>
>>     Chris
>>
>>     Quoting Brent Owens :
>>
>>      > If you have two different instances of Geoserver running, and
>> they
>>      > are
>>      > both targetting the same shapefile, you will get corruption. Just
>>      > like
>>      > any file access system, you need a single controlling source to
>>      > prevent
>>      > errors.
>>      > With a database backed data source, it controls access with
>>      > transactions
>>      > but the Geoserver instances might be out of sync. I do not
>> recommend
>>      > having multiple Geoservers hitting the same datasource. I
>> would use
>>      > just
>>      > one instance as it can handle multiple users accessing the
>> data. You
>>      > won't have corruption that way.
>>      >
>>      > Now if you are using one instance of Geoserver, and multiple
>> users
>>      > are
>>      > editing, and you are getting corruption, then that is a bug.
>> If that
>>      > is
>>      > a case, give us some info and we will add it to the bug list.
>>      >
>>      > Brent Owens
>>      > TOPP
>>      >
>>      >
>>      >
>>      > Nathan Farrington wrote:
>>      >
>>      > > Hi,
>>      > >
>>      > > I've been using GeoServer for a multiuser application. I
>> have about
>>      > 10
>>      > > tables, each with about 20 columns. There are about 10 users as
>>      > well.
>>      > > I set up two different test servers with two different
>> GeoServer
>>      > > installations, one with shapefiles and one with
>> PostgreSQL/PostGIS.
>>      > > They both work fine with just one user.
>>      > >
>>      > > My question is how will they scale with multiple users? I
>> wanted to
>>      > > deploy using the shapefile configured GeoServer, but I started
>>      > > noticing problems when using it with multiple clients.
>> Sometimes a
>>      > > shapefile would get corrupted and contain duplicate entries.
>> The
>>      > main
>>      > > advantage to using this configuration is that it is easier to
>>      > > distribute. If I went with PostGIS, I would need to do some
>> remote
>>      > > administration, which is difficult for this application.
>>      > >
>>      > > Has anyone seen problems using shapefiles with multiple
>>      > simultaneous
>>      > > users? Would a backend like PostGIS solve this problem? Is
>> there a
>>      > > different backend that you would recommend?
>>      > >
>>      > > Thanks,
>>      > >
>>      > > Nathan
>>      >
>>      >
>>      >
>>      > -------------------------------------------------------
>>      > This SF.Net email is sponsored by:
>>      > Power Architecture Resource Center: Free content, downloads,
>>      > discussions,
>>      > and more. http://solutions.newsforge.com/ibmarch.tmpl
>>      > _______________________________________________
>>      > Geoserver-users mailing list
>>      > [hidden email]
>>      > https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>      >
>>
>>
>>
>>
>>     ----------------------------------------------------------
>>     This mail sent through IMP: https://webmail.limegroup.com/
>
>
>


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users