A letter to the Postgis Developers and Packagers

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

A letter to the Postgis Developers and Packagers

Andrea Peri
Hello to all Postgis Developers and Packagers,

Tuscany Region is investing in several GFLOSS systems and right
now is building the Spatialite ISO topology interface consistent
with the one already funded for PostGIS.

It then commissioned to Sandro Santilli (strk) and Sandro Furieri
(the developer of the Spatialite system) the development of a
C language API implementing the same functions of the PostGIS
Topology module. The goal of this library was to be usable with both
spatialite and postgis, thus ensuring all the necessary compatibility.

We thought it was a good idea to insert this new API in the
liblwgeom library (after having a positive opinion by Santilli about
the utility for PostGIS of this operation) so to replace the
plpgsql topology code in PostGIS (also completed with the fundings
of Tuscany Region) with C code using the new liblwgeom-topo API.

We thought that the substitution of the PostGIS Topology functions from
plpgsql in C would not cause any problems to users, offering
instead performance improvements. But during its testing in Spatialite,
we realized that the library required several further refirements in its API.

This has created a situation of embarrassment that caused the break
of the new liblwgeom API just in the same day it was released along
with the new 2.2.0 version of PostGIS.

We apologize for causing, with our interventions in liblwgeom, the
release within PostGIS of a topological library that is not yet complete
and which definitely needs several revisions and other changes to the API.

So we've tried to find a quick fix to solve the problem we caused,
trying to avoid further inconveniences in the process of maintenance and
release of PostGIS.

Beeing in need of a more rapid test and release cycle for the topology
library, and with the aim to create a library which is compatible with
both PostGIS and Spatialite, we have identified as the best solution
to carry out the development, test and release activities in a separate
project. This choice also allows us to write the library with a thread-safe
model, which is not needed by PostGIS.

As we fear the liblwgeom-topo code might be too immature and prone
to API breakage, we offer to finance the recovery in PostGIS of the
previous plpgsql implementation of the Topology module.

The new topological library of Tuscany Region will be deposited on a
distinct GIT repository, with an open license similar to that of
liblwgeom and of course we will be pleased and honored if in the future,
when it will be more robust and reliable, the Community of PostGIS
Developers and Packagers will decide to link it as a shared library for the
topological component functions.

With cordiality, Andrea Peri


--
-----------------
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-----------------
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sandro Santilli-2
Hello Andrea,

On Sun, Oct 18, 2015 at 08:52:14PM +0200, Andrea Peri wrote:

> We thought that the substitution of the PostGIS Topology functions from
> plpgsql in C would not cause any problems to users, offering
> instead performance improvements. But during its testing in Spatialite,
> we realized that the library required several further refirements in its API.
>
> This has created a situation of embarrassment that caused the break
> of the new liblwgeom API just in the same day it was released along
> with the new 2.2.0 version of PostGIS.

For the record, I think the liblwgeom API breakage was really not
a big deal, especially as what broke was the new topological API,
which is really only used by PostGIS itself (beside the ongoing
Spatialite development).

> We apologize for causing, with our interventions in liblwgeom, the
> release within PostGIS of a topological library that is not yet complete
> and which definitely needs several revisions and other changes to the API.

I don't think you can be blamed, the modifications entered the PostGIS
code base under my own responsibility so if someone feels it was an
error to introduce such change, the fault is all on myself.

Personally, I don't see any damage introduced with those changes.
Yes, there's been a performance regression, but those kind of things
tend to happen, and we've a fix for it already committed.

> Beeing in need of a more rapid test and release cycle for the topology
> library, and with the aim to create a library which is compatible with
> both PostGIS and Spatialite, we have identified as the best solution
> to carry out the development, test and release activities in a separate
> project. This choice also allows us to write the library with a thread-safe
> model, which is not needed by PostGIS.

Both adding thread-safety to the library and having a faster release
cycle are surely good reasons to have a separate package for the new
library.

I've been hoping we could directly work at making liblwgeom itself turn
into such an external library, but I understand the evaluated risk for
increased cost of such a move makes other PSC members prefer not to
do that.

Note I do not think the fundings spent on liblwgeom integration were
wasted, as that allowed us to immediately prove the appropriateness
of the library API and its correctness against the existing PostGIS
Topology testsuite, which was also improved in the past thanks to
Tuscany Region funding. And moving the code elsewhere would now mostly
be a matter of renaming the symbols.

> As we fear the liblwgeom-topo code might be too immature and prone
> to API breakage, we offer to finance the recovery in PostGIS of the
> previous plpgsql implementation of the Topology module.

I think the API breakage issue could be as well fixed by excluding
the topological support from the published liblwgeom headers.

Having topology function implemented in C opens the door to further
improvements that aren't easy with the plpgsql.

Not knowing in advance when the external library will be ready,
I'd rather keep using the internal version until an alternative
is available.

There's a fair amount of code in the C topology module that doesn't
belong to the C topology library API (implementation of the callbacks
required by the library). That code will still be useful for using
the external library, shall it have a similar interface to the current
one (as I belive is the case).

If anyone else from the PSC prefers a revert, please speak up.

> The new topological library of Tuscany Region will be deposited on a
> distinct GIT repository, with an open license similar to that of
> liblwgeom and of course we will be pleased and honored if in the future,
> when it will be more robust and reliable, the Community of PostGIS
> Developers and Packagers will decide to link it as a shared library for the
> topological component functions.

I hope that'll be soon possible.
Thanks for your continued support to free software development!

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Regina Obe
Hello Andrea and Sandro,

I'm very appreciative of Tuscany Region's contributions to PostGIS.  As a packager maintainer and PostGIS developer, the whole lilblwgeom versioning has caused me some grief.
Largely I've worked that out on my own so it's just water under the bridge and no permanent harm done.

I would like the Topology C stuff to stay rather than being reverted back to plpgsql, even if it means we have to pay Sandro to copy liblwgeom bits from some other repo into PostGIS for updates.

1) I wasn't so bothered with liblwgeom having a life of it's own, but still within the PostGIS codebase if it was just SpatiaLite using it from PostGIS.  However now that Grass, QGIS etc are using it,
I feel things are moving too fast and will jeopardize both the future of PostGIS and the use of liblwgeom by others, unless we slow the pace down a bit.

2) Now that SFCGAL is becoming more commonly deployed, a PostGIS built with liblwgeom support means, a liblwgeom that has SFCGAL and CGAL dependencies. For other projects using liblwgeom (and not caring about PostGIS)
This extra dependency would not be a welcome addition.  Hell it wasn't welcome to me when my raster2pgsql, shp2pgsql, pgsql2shp, shp2pgsql-gui suddenly had SFCGAL/CGAL dependencies when they don't currently use any of those features.
So I had to revert back to staic compiling to get rid of the dependency.

3) Personally I don't feel that liblwgeom is mature enough as a stand-alone library to be shared by anyone without explicit caution that it's volatile and others jumping on board should be good swimmers.

4) As to whether liblwgeom gets forked for SpatiaLite use and others, I'm torn.  Yes the whole threading need and that SpatiaLite needs to release more frequently than PostGIS needs one, would suggest it just get forked and we part ways.
Though with PostgreSQL 9.6 coming soon and the whole parallel query thing coming in, I'm not sure if that may change our need for threading.

5) There is a danger that liblwgeom split from PostGIS would mean they go on their separate lives never to join again.  But I look at projects like MySQL, MariaDb, Percona and they seem to do more or less fine forking each and rejoining each other all the time.

If we could have an option so that PostGIS has it's own version of liblwgeom with a versioning scheme that matches the rest of PostGIS, with an optional switch to spit out another baby with it's own versioning that others can use, then the two can coexist using the same code base
And that code base be PostGIS.

I'm not sure how easy that is to do though.

That probably doesn't help Debian folks all that much except that liblwgeom so names won't change with each new micro release of PostGIS, but they'd still have to maintain 2 for largely identical libraries.

Thanks,
Regina


-----Original Message-----
From: postgis-devel [mailto:[hidden email]] On Behalf Of Sandro Santilli
Sent: Monday, October 19, 2015 11:23 AM
To: Andrea Peri <[hidden email]>
Cc: [hidden email]
Subject: Re: [postgis-devel] A letter to the Postgis Developers and Packagers

Hello Andrea,

On Sun, Oct 18, 2015 at 08:52:14PM +0200, Andrea Peri wrote:

> We thought that the substitution of the PostGIS Topology functions
> from plpgsql in C would not cause any problems to users, offering
> instead performance improvements. But during its testing in
> Spatialite, we realized that the library required several further refirements in its API.
>
> This has created a situation of embarrassment that caused the break of
> the new liblwgeom API just in the same day it was released along with
> the new 2.2.0 version of PostGIS.

For the record, I think the liblwgeom API breakage was really not a big deal, especially as what broke was the new topological API, which is really only used by PostGIS itself (beside the ongoing Spatialite development).

> We apologize for causing, with our interventions in liblwgeom, the
> release within PostGIS of a topological library that is not yet
> complete and which definitely needs several revisions and other changes to the API.

I don't think you can be blamed, the modifications entered the PostGIS code base under my own responsibility so if someone feels it was an error to introduce such change, the fault is all on myself.

Personally, I don't see any damage introduced with those changes.
Yes, there's been a performance regression, but those kind of things tend to happen, and we've a fix for it already committed.

> Beeing in need of a more rapid test and release cycle for the topology
> library, and with the aim to create a library which is compatible with
> both PostGIS and Spatialite, we have identified as the best solution
> to carry out the development, test and release activities in a
> separate project. This choice also allows us to write the library with
> a thread-safe model, which is not needed by PostGIS.

Both adding thread-safety to the library and having a faster release cycle are surely good reasons to have a separate package for the new library.

I've been hoping we could directly work at making liblwgeom itself turn into such an external library, but I understand the evaluated risk for increased cost of such a move makes other PSC members prefer not to do that.

Note I do not think the fundings spent on liblwgeom integration were wasted, as that allowed us to immediately prove the appropriateness of the library API and its correctness against the existing PostGIS Topology testsuite, which was also improved in the past thanks to Tuscany Region funding. And moving the code elsewhere would now mostly be a matter of renaming the symbols.

> As we fear the liblwgeom-topo code might be too immature and prone to
> API breakage, we offer to finance the recovery in PostGIS of the
> previous plpgsql implementation of the Topology module.

I think the API breakage issue could be as well fixed by excluding the topological support from the published liblwgeom headers.

Having topology function implemented in C opens the door to further improvements that aren't easy with the plpgsql.

Not knowing in advance when the external library will be ready, I'd rather keep using the internal version until an alternative is available.

There's a fair amount of code in the C topology module that doesn't belong to the C topology library API (implementation of the callbacks required by the library). That code will still be useful for using the external library, shall it have a similar interface to the current one (as I belive is the case).

If anyone else from the PSC prefers a revert, please speak up.

> The new topological library of Tuscany Region will be deposited on a
> distinct GIT repository, with an open license similar to that of
> liblwgeom and of course we will be pleased and honored if in the
> future, when it will be more robust and reliable, the Community of
> PostGIS Developers and Packagers will decide to link it as a shared
> library for the topological component functions.

I hope that'll be soon possible.
Thanks for your continued support to free software development!

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sandro Santilli-2
On Thu, Oct 22, 2015 at 12:18:22PM -0400, Paragon Corporation wrote:

> If we could have an option so that PostGIS has it's own version of liblwgeom with a versioning scheme that matches the rest of PostGIS, with an optional switch to spit out another baby with it's own versioning that others can use, then the two can coexist using the same code base
> And that code base be PostGIS.

Had you not considered the option of PostGIS dynamically linking to
the new external library ? It would be like the GEOS dependency.

> That probably doesn't help Debian folks all that much except that
> liblwgeom so names won't change with each new micro release of PostGIS,
> but they'd still have to maintain 2 for largely identical libraries.

I guess the PostGIS version would not need to be an exported library
anymore, if the external one exists.

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Greg Troxel

Sandro Santilli <[hidden email]> writes:

> On Thu, Oct 22, 2015 at 12:18:22PM -0400, Paragon Corporation wrote:
>
>> If we could have an option so that PostGIS has it's own version of
>> liblwgeom with a versioning scheme that matches the rest of PostGIS,
>> with an optional switch to spit out another baby with it's own
>> versioning that others can use, then the two can coexist using the
>> same code base
>> And that code base be PostGIS.
>
> Had you not considered the option of PostGIS dynamically linking to
> the new external library ? It would be like the GEOS dependency.
>
>> That probably doesn't help Debian folks all that much except that
>> liblwgeom so names won't change with each new micro release of PostGIS,
>> but they'd still have to maintain 2 for largely identical libraries.
>
> I guess the PostGIS version would not need to be an exported library
> anymore, if the external one exists.
From the packaging viewpoint, having two copies of the same code is a
problem in terms of security patches, as well as just duplicated space
not only in the fs but at runtime.

If liblwgeom is going to be used by other packages -- and it really
seems it is -- the only reasonable thing to do in the long term is make
it a separate package.  (I realize that this may not be reasonable in
the short term.)

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel

signature.asc (186 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Paul Ramsey
I think a great deal depends on how this new library is going to be used. 
If the expectation is that it will supplant use of the postgis/liblwgeom for spatialite, qgis, etc, then we should just re-internalize ours so we’re not installing a system library anymore.
P. 

On October 23, 2015 at 6:59:30 AM, Greg Troxel ([hidden email]) wrote:

If liblwgeom is going to be used by other packages -- and it really 
seems it is -- the only reasonable thing to do in the long term is make 
it a separate package. (I realize that this may not be reasonable in 
the short term.) 


-- 
http://postgis.net
http://cleverelephant.ca


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sandro Santilli-2
On Fri, Oct 23, 2015 at 07:10:11AM -0700, Paul Ramsey wrote:
> On October 23, 2015 at 6:59:30 AM, Greg Troxel ([hidden email]) wrote:
>
> > If liblwgeom is going to be used by other packages -- and it really 
> > seems it is -- the only reasonable thing to do in the long term is make 
> > it a separate package. (I realize that this may not be reasonable in 
> > the short term.) 
>
> I think a great deal depends on how this new library is going to be used. 
> If the expectation is that it will supplant use of the postgis/liblwgeom for spatialite, qgis, etc, then we should just re-internalize ours so we’re not installing a system library anymore.

Yes, that's the idea. All current clients should be able to switch
to use the new library, even if it has a non-zero cost due to the
forced rename of all symbols.

So I envision a period in which both libraries would exist, to still
support the existing liblwgeom clients.

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Regina Obe
In reply to this post by Andrea Peri
>> If we could have an option so that PostGIS has it's own version of
liblwgeom with a versioning scheme that matches the rest of PostGIS,
with an optional switch to spit out another baby with it's own
versioning that others can use, then the two can coexist using the same
code base
>> And that code base be PostGIS.

> Had you not considered the option of PostGIS dynamically linking to
> the new external library ? It would be like the GEOS dependency.

Yes we've done that and it does not work. It creates chaos with liblwgeom
sometimes not getting installed along side PostGIS.

GEOS is a pain but a palpable pain because it was designed from the ground
up to be a shared library.  liblwgeom as  shared library is an
after-thought.  While some after-thoughts work fine, in this case I don't
think so because we are too in-grained into thinking that is part of
PostGIS (the part we can test outside of the database). That I don't
envision changing anytime soon and will slow our release cycle.


>> That probably doesn't help Debian folks all that much except that
>> liblwgeom so names won't change with each new micro release of PostGIS,
>> but they'd still have to maintain 2 for largely identical libraries.

> I guess the PostGIS version would not need to be an exported library
> anymore, if the external one exists.

> --strk;

Yes it should not be exported, it shouldn't need to be exported.


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Regina Obe
In reply to this post by Paul Ramsey

As much as I hate agreeing with Paul on this as what he has reiterated over and over sounds selfish and unfriendly,  I think that's the best solution in the long run for everybody, that other projects fork liblwgeom into a separate repo they can share and PostGIS re-internalize our version of the library so it doesn't get installed in system.

 

Not being able to rely on an exact version of liblwgeom is very troubling for PostGIS.  Unlike GEOS where we can live with some people running say 3.4 or 3.3  (when we prefer 3.5), we absolutely have to be insured that what we test in our cunits is what's being shipped with said version of PostGIS as that is 80% of what PostGIS is. Not to mention the hassle of having to contribute to two projects just to make a change to something that probably 80% of the time affects only us.

Not having that reassurance is just too much stress, I feel  on the PostGIS project. 

 

As more projects use liblwgeom this butting of heads of our needs versus theirs will become more of an issue that I fear significantly dwarfs any contributions we might get from others.  In this case we are just better off copy sharing rather than binary sharing.

 

Thanks,

Regina

 

From: postgis-devel [mailto:[hidden email]] On Behalf Of Paul Ramsey
Sent: Friday, October 23, 2015 10:10 AM
To: Sandro Santilli <[hidden email]>; PostGIS Development Discussion <[hidden email]>; Greg Troxel <[hidden email]>
Cc: PostGIS Development Discussion <[hidden email]>
Subject: Re: [postgis-devel] A letter to the Postgis Developers and Packagers

 

I think a great deal depends on how this new library is going to be used. 

If the expectation is that it will supplant use of the postgis/liblwgeom for spatialite, qgis, etc, then we should just re-internalize ours so we’re not installing a system library anymore.

P. 

On October 23, 2015 at 6:59:30 AM, Greg Troxel ([hidden email]) wrote:

If liblwgeom is going to be used by other packages -- and it really 
seems it is -- the only reasonable thing to do in the long term is make 
it a separate package. (I realize that this may not be reasonable in 
the short term.) 


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

a.furieri
Hi PostGIS developers,

just few considerations attempting to explain how the lwgeom
problem appears as seen from outside, more precisely from the
SpatiaLite's own side; I imagine that the same considerations
will probably apply for any other FLOSS/GFOSS project excluding
PostGIS itself.

Linking lwgeom as a whole never was an exciting target,
mainly for the following reasons:
1. about 90% of lwgeom simply is an unneeded duplication of
    already existing code.
    Just consider all the wrappers built on the top of both
    GEOS and PROJ.4: SpatiaLite (exactly as QGIS, GRASS, GDAL
    and alike) already has its own, they exist since a very long
    time, they are robustly tested, and they interact very
    strictly with our main core. The same obviously is for any
    other OGC-defined SQL function and for WKT/WKB/GML/KML/etc
    parsers and writers; we already have our own since long
    time and we are not really interested in the same functions
    supported by lwgeom in an slightly different and not
    compatible form.
2. the lwgeom code is not thread-safe; this is a real PITA
    because calling any single lwgeom API always implies defining
    a critical section protected by a mutex so to properly shield
    against any risk caused by unsafe parallel processing; it's a
    really unpleasant complication strongly limiting the real
    usability of lwgeom.
3. building lwgeom as a self-standing static/dynamic library
    was a cumbersome process until recent times; happily enough
    (mainly thanks to the nice efforts of strk) now it's a
    decently plain task and this surely marked a noticeable
    usability improvement.
4. lwgeom has a very poor API/ABI long tern stability: supporting
    a new lwgeom version always implies applying many relevant
    changes and patches, both in the C wrapping code and in the
    configure scripts; and being able to correctly support several
    different versions of lwgeom at the same time can quickly
    become a real nightmare.

What is strongly attracting in lwgeom for other GFOSS projects
are just a handful of highly valuable sophisticated spatial
functions that have no equivalent implementation and that
aren't so easy and simple to be independently rewritten.
Just to mention the most relevant, they are MakeValid(), Split(),
Snap(), Node(), SelfIntersection() and few others.
Please note: all these functions directly wrap some GEOS call
but all them internally perform some kind of further complex
geometry manipulation thus introducing a relevant added value.
 From an external project naive perspective they basically are
just "super muscular GEOS"; unhappily if you wish to use such
"superGEOS" functions in any other project except PostGIS you
have to necessarily face two equally unpleasant alternatives:
A) forking specific portions of the lwgeom code (IMHO an
    option to be avoided as long as possible).
B) unenthusiastically resign to link lwgeom as the lesser
    evil, then patiently suffering from the many inconvenients
    reported before.


What the new separated library should possibly be
--------------------------------------------------
a) a simple, streamlined and light-weighted library just
    depending on GEOS alone.
b) robust thread-safe implementation.
c) canonic configure scripts fully supporting versioning,
    SONAMEs and alike so to make maintainers and packagers
    activities as easy as possible.
d) long term API/ABI rock-solid stability (with possible
    exceptions only during initial alpha/beta testing steps).
e) full support for ISO Topology, and consequently directly
    supporting all the "super-GEOS" API strictly required by
    the Topology code itself, as e.g. MakeValid, Snap etc.
no less than this, no more than this.

IMHO considering this new library as an lwgeom replacement
is someway misleading; it's a different thing, it's just
a brand new topology C engine including a handful of
"superGEOS" API currently buried within the lwgeom codebase
and not really easy to be reused in a different context.
I easily guess that a new library modelled this way should
probably attract the interest of several other GFOSS projects,
and I sincerely hope that PostGIS developers too will find
useful linking the new library once it will be robustly
tested and finally stable, mainly because this will simplify
their own development and maintenance tasks.

SpatiaLite will be obviously happy to be the first "experimental"
user of this new library: and will immediately contribute to
the new library it's own C code implementing Topology-Network
(currently unsupported by PostGIS/lwgeom).

Refactoring the whole lwgeom as a fully independent library
is a completely different task, and should never be confused
with the other approach advocated by the initial post from
Tuscany (A.Peri); future evolutions of lwgeom mainly are a
PostGIS own internal affair, and have a very limited interest
(if any) for any other GFOSS project.
What really matters is a very limited and well defined
subset: "superGEOS" and Topology.

bye Sandro



_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Even Rouault-2
> Refactoring the whole lwgeom as a fully independent library
> is a completely different task, and should never be confused
> with the other approach advocated by the initial post from
> Tuscany (A.Peri); future evolutions of lwgeom mainly are a
> PostGIS own internal affair, and have a very limited interest
> (if any) for any other GFOSS project.
> What really matters is a very limited and well defined
> subset: "superGEOS" and Topology.

There's perhaps a better forum than here to ask that, but what will be the
license of this "superGEOS" ?

>
> bye Sandro
>
>
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/postgis-devel

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

a.furieri
On Sat, 24 Oct 2015 15:13:20 +0200, Even Rouault wrote:

>> Refactoring the whole lwgeom as a fully independent library
>> is a completely different task, and should never be confused
>> with the other approach advocated by the initial post from
>> Tuscany (A.Peri); future evolutions of lwgeom mainly are a
>> PostGIS own internal affair, and have a very limited interest
>> (if any) for any other GFOSS project.
>> What really matters is a very limited and well defined
>> subset: "superGEOS" and Topology.
>
> There's perhaps a better forum than here to ask that, but what will
> be the
> license of this "superGEOS" ?
>

Hi Even,

this is a very good question.
the initial post of Andrea Peri starting this thread simply stated:
"with an open license similar to that of liblwgeom"

I personally feel that adopting the LGPL should be the optimal
solution mainly because LGPL already is the license adopted by
GEOS itself and "superGEOS" would simply be a rather obvious
GEOS complement.
anyway this is just a personal preference of my own; I suppose
that we should start a most serious discussion about the most
appropriate licensing terms.

bye Sandro
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Bborie Park-2
Assuming code is coming from lwgeom, the license should probably be GPL 2.0.

-bborie

On Sat, Oct 24, 2015 at 7:06 AM, <[hidden email]> wrote:
On Sat, 24 Oct 2015 15:13:20 +0200, Even Rouault wrote:
Refactoring the whole lwgeom as a fully independent library
is a completely different task, and should never be confused
with the other approach advocated by the initial post from
Tuscany (A.Peri); future evolutions of lwgeom mainly are a
PostGIS own internal affair, and have a very limited interest
(if any) for any other GFOSS project.
What really matters is a very limited and well defined
subset: "superGEOS" and Topology.

There's perhaps a better forum than here to ask that, but what will be the
license of this "superGEOS" ?


Hi Even,

this is a very good question.
the initial post of Andrea Peri starting this thread simply stated:
"with an open license similar to that of liblwgeom"

I personally feel that adopting the LGPL should be the optimal
solution mainly because LGPL already is the license adopted by
GEOS itself and "superGEOS" would simply be a rather obvious
GEOS complement.
anyway this is just a personal preference of my own; I suppose
that we should start a most serious discussion about the most
appropriate licensing terms.


bye Sandro
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sandro Santilli-2
On Sat, Oct 24, 2015 at 09:01:38AM -0700, Bborie Park wrote:
> Assuming code is coming from lwgeom, the license should probably be GPL 2.0.

GPL 2.0 or, at your option, later version
(see https://trac.osgeo.org/postgis/ticket/2515)

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Mark Cave-Ayland
In reply to this post by a.furieri
On 24/10/15 14:04, [hidden email] wrote:

> Hi PostGIS developers,
>
> just few considerations attempting to explain how the lwgeom
> problem appears as seen from outside, more precisely from the
> SpatiaLite's own side; I imagine that the same considerations
> will probably apply for any other FLOSS/GFOSS project excluding
> PostGIS itself.
>
> Linking lwgeom as a whole never was an exciting target,
> mainly for the following reasons:
> 1. about 90% of lwgeom simply is an unneeded duplication of
>    already existing code.
>    Just consider all the wrappers built on the top of both
>    GEOS and PROJ.4: SpatiaLite (exactly as QGIS, GRASS, GDAL
>    and alike) already has its own, they exist since a very long
>    time, they are robustly tested, and they interact very
>    strictly with our main core. The same obviously is for any
>    other OGC-defined SQL function and for WKT/WKB/GML/KML/etc
>    parsers and writers; we already have our own since long
>    time and we are not really interested in the same functions
>    supported by lwgeom in an slightly different and not
>    compatible form.
> 2. the lwgeom code is not thread-safe; this is a real PITA
>    because calling any single lwgeom API always implies defining
>    a critical section protected by a mutex so to properly shield
>    against any risk caused by unsafe parallel processing; it's a
>    really unpleasant complication strongly limiting the real
>    usability of lwgeom.
> 3. building lwgeom as a self-standing static/dynamic library
>    was a cumbersome process until recent times; happily enough
>    (mainly thanks to the nice efforts of strk) now it's a
>    decently plain task and this surely marked a noticeable
>    usability improvement.
> 4. lwgeom has a very poor API/ABI long tern stability: supporting
>    a new lwgeom version always implies applying many relevant
>    changes and patches, both in the C wrapping code and in the
>    configure scripts; and being able to correctly support several
>    different versions of lwgeom at the same time can quickly
>    become a real nightmare.
>
> What is strongly attracting in lwgeom for other GFOSS projects
> are just a handful of highly valuable sophisticated spatial
> functions that have no equivalent implementation and that
> aren't so easy and simple to be independently rewritten.
> Just to mention the most relevant, they are MakeValid(), Split(),
> Snap(), Node(), SelfIntersection() and few others.
> Please note: all these functions directly wrap some GEOS call
> but all them internally perform some kind of further complex
> geometry manipulation thus introducing a relevant added value.
> From an external project naive perspective they basically are
> just "super muscular GEOS"; unhappily if you wish to use such
> "superGEOS" functions in any other project except PostGIS you
> have to necessarily face two equally unpleasant alternatives:
> A) forking specific portions of the lwgeom code (IMHO an
>    option to be avoided as long as possible).
> B) unenthusiastically resign to link lwgeom as the lesser
>    evil, then patiently suffering from the many inconvenients
>    reported before.
>
>
> What the new separated library should possibly be
> --------------------------------------------------
> a) a simple, streamlined and light-weighted library just
>    depending on GEOS alone.
> b) robust thread-safe implementation.
> c) canonic configure scripts fully supporting versioning,
>    SONAMEs and alike so to make maintainers and packagers
>    activities as easy as possible.
> d) long term API/ABI rock-solid stability (with possible
>    exceptions only during initial alpha/beta testing steps).
> e) full support for ISO Topology, and consequently directly
>    supporting all the "super-GEOS" API strictly required by
>    the Topology code itself, as e.g. MakeValid, Snap etc.
> no less than this, no more than this.
>
> IMHO considering this new library as an lwgeom replacement
> is someway misleading; it's a different thing, it's just
> a brand new topology C engine including a handful of
> "superGEOS" API currently buried within the lwgeom codebase
> and not really easy to be reused in a different context.
> I easily guess that a new library modelled this way should
> probably attract the interest of several other GFOSS projects,
> and I sincerely hope that PostGIS developers too will find
> useful linking the new library once it will be robustly
> tested and finally stable, mainly because this will simplify
> their own development and maintenance tasks.
>
> SpatiaLite will be obviously happy to be the first "experimental"
> user of this new library: and will immediately contribute to
> the new library it's own C code implementing Topology-Network
> (currently unsupported by PostGIS/lwgeom).
>
> Refactoring the whole lwgeom as a fully independent library
> is a completely different task, and should never be confused
> with the other approach advocated by the initial post from
> Tuscany (A.Peri); future evolutions of lwgeom mainly are a
> PostGIS own internal affair, and have a very limited interest
> (if any) for any other GFOSS project.
> What really matters is a very limited and well defined
> subset: "superGEOS" and Topology.
>
> bye Sandro

Well if you're thinking of starting again from the ground up, it may be
worth considering using Boost Geometry instead of GEOS for your
liblwgeom implementation. Firstly it has a more liberal license,
secondly it has the potential to support multiple precision models and
thirdly being written native for C++ (i.e. its APIs aren't based on
Java's memory model) then it's likely to be much performant than GEOS -
I seem to remember strk doing some tests with a basic Boost Geometry
wrapper that showed a 100% speedup in some cases.


ATB,

Mark.

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Regina Obe
The other benefit of building liblwgeom anew with Boost.Geometry is that MySQL 5.7 is bulding against Boost.Geometry

If the licensing was like Boost's, Oracle would be more willing to chip in on the funding of it since they can ship it with their Enterprise version of MySQL  :)

We could even link back bits of it via SFCGAL which already relies on Boost for some things :)

Thanks,
Regina



-----Original Message-----
From: postgis-devel [mailto:[hidden email]] On Behalf Of Mark Cave-Ayland
Sent: Sunday, October 25, 2015 6:55 AM
To: [hidden email]
Subject: Re: [postgis-devel] A letter to the Postgis Developers and Packagers

On 24/10/15 14:04, [hidden email] wrote:

> Hi PostGIS developers,
>
> just few considerations attempting to explain how the lwgeom problem
> appears as seen from outside, more precisely from the SpatiaLite's own
> side; I imagine that the same considerations will probably apply for
> any other FLOSS/GFOSS project excluding PostGIS itself.
>
> Linking lwgeom as a whole never was an exciting target, mainly for the
> following reasons:
> 1. about 90% of lwgeom simply is an unneeded duplication of
>    already existing code.
>    Just consider all the wrappers built on the top of both
>    GEOS and PROJ.4: SpatiaLite (exactly as QGIS, GRASS, GDAL
>    and alike) already has its own, they exist since a very long
>    time, they are robustly tested, and they interact very
>    strictly with our main core. The same obviously is for any
>    other OGC-defined SQL function and for WKT/WKB/GML/KML/etc
>    parsers and writers; we already have our own since long
>    time and we are not really interested in the same functions
>    supported by lwgeom in an slightly different and not
>    compatible form.
> 2. the lwgeom code is not thread-safe; this is a real PITA
>    because calling any single lwgeom API always implies defining
>    a critical section protected by a mutex so to properly shield
>    against any risk caused by unsafe parallel processing; it's a
>    really unpleasant complication strongly limiting the real
>    usability of lwgeom.
> 3. building lwgeom as a self-standing static/dynamic library
>    was a cumbersome process until recent times; happily enough
>    (mainly thanks to the nice efforts of strk) now it's a
>    decently plain task and this surely marked a noticeable
>    usability improvement.
> 4. lwgeom has a very poor API/ABI long tern stability: supporting
>    a new lwgeom version always implies applying many relevant
>    changes and patches, both in the C wrapping code and in the
>    configure scripts; and being able to correctly support several
>    different versions of lwgeom at the same time can quickly
>    become a real nightmare.
>
> What is strongly attracting in lwgeom for other GFOSS projects are
> just a handful of highly valuable sophisticated spatial functions that
> have no equivalent implementation and that aren't so easy and simple
> to be independently rewritten.
> Just to mention the most relevant, they are MakeValid(), Split(),
> Snap(), Node(), SelfIntersection() and few others.
> Please note: all these functions directly wrap some GEOS call but all
> them internally perform some kind of further complex geometry
> manipulation thus introducing a relevant added value.
> From an external project naive perspective they basically are just
> "super muscular GEOS"; unhappily if you wish to use such "superGEOS"
> functions in any other project except PostGIS you have to necessarily
> face two equally unpleasant alternatives:
> A) forking specific portions of the lwgeom code (IMHO an
>    option to be avoided as long as possible).
> B) unenthusiastically resign to link lwgeom as the lesser
>    evil, then patiently suffering from the many inconvenients
>    reported before.
>
>
> What the new separated library should possibly be
> --------------------------------------------------
> a) a simple, streamlined and light-weighted library just
>    depending on GEOS alone.
> b) robust thread-safe implementation.
> c) canonic configure scripts fully supporting versioning,
>    SONAMEs and alike so to make maintainers and packagers
>    activities as easy as possible.
> d) long term API/ABI rock-solid stability (with possible
>    exceptions only during initial alpha/beta testing steps).
> e) full support for ISO Topology, and consequently directly
>    supporting all the "super-GEOS" API strictly required by
>    the Topology code itself, as e.g. MakeValid, Snap etc.
> no less than this, no more than this.
>
> IMHO considering this new library as an lwgeom replacement is someway
> misleading; it's a different thing, it's just a brand new topology C
> engine including a handful of "superGEOS" API currently buried within
> the lwgeom codebase and not really easy to be reused in a different
> context.
> I easily guess that a new library modelled this way should probably
> attract the interest of several other GFOSS projects, and I sincerely
> hope that PostGIS developers too will find useful linking the new
> library once it will be robustly tested and finally stable, mainly
> because this will simplify their own development and maintenance
> tasks.
>
> SpatiaLite will be obviously happy to be the first "experimental"
> user of this new library: and will immediately contribute to the new
> library it's own C code implementing Topology-Network (currently
> unsupported by PostGIS/lwgeom).
>
> Refactoring the whole lwgeom as a fully independent library is a
> completely different task, and should never be confused with the other
> approach advocated by the initial post from Tuscany (A.Peri); future
> evolutions of lwgeom mainly are a PostGIS own internal affair, and
> have a very limited interest (if any) for any other GFOSS project.
> What really matters is a very limited and well defined
> subset: "superGEOS" and Topology.
>
> bye Sandro

Well if you're thinking of starting again from the ground up, it may be worth considering using Boost Geometry instead of GEOS for your liblwgeom implementation. Firstly it has a more liberal license, secondly it has the potential to support multiple precision models and thirdly being written native for C++ (i.e. its APIs aren't based on Java's memory model) then it's likely to be much performant than GEOS - I seem to remember strk doing some tests with a basic Boost Geometry wrapper that showed a 100% speedup in some cases.


ATB,

Mark.

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Regina Obe
In reply to this post by Even Rouault-2
I wonder if maybe GDAL is just bundled with another package that happens to have QT.  Don't know

Here is the full IRC thread for those interested:

http://irclogs.geoapt.com/postgis/%23postgis.2015-10-18.log

starting at: 03:29:47

Too bad astrodog is not on this list to explain more.

Anyway I think the issue is one that other packagers have (so just think of QT as just one possible thing).  I've heard Devrim express similar sentiment  and his was even more worrisome with being able to offer it for newer version of CentOS but not older because the older has too  old of a GDAL or no GDAL at all upstream.

(Devrim -- I know you are here so please say something or say I heard you wrong).

Thanks,
Regina


-----Original Message-----
From: postgis-devel [mailto:[hidden email]] On Behalf Of Even Rouault
Sent: Saturday, October 24, 2015 9:13 AM
To: [hidden email]
Subject: Re: [postgis-devel] A letter to the Postgis Developers and Packagers

> Refactoring the whole lwgeom as a fully independent library is a
> completely different task, and should never be confused with the other
> approach advocated by the initial post from Tuscany (A.Peri); future
> evolutions of lwgeom mainly are a PostGIS own internal affair, and
> have a very limited interest (if any) for any other GFOSS project.
> What really matters is a very limited and well defined
> subset: "superGEOS" and Topology.

There's perhaps a better forum than here to ask that, but what will be the license of this "superGEOS" ?

>
> bye Sandro
>
>
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/postgis-devel

--
Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sebastiaan Couwenberg
In reply to this post by Andrea Peri
On 18-10-15 20:52, Andrea Peri wrote:
> The new topological library of Tuscany Region will be deposited on a
> distinct GIT repository, with an open license similar to that of
> liblwgeom and of course we will be pleased and honored if in the future,
> when it will be more robust and reliable, the Community of PostGIS
> Developers and Packagers will decide to link it as a shared library for the
> topological component functions.

Is there any news on the new topological library?

I'd like to drop the liblwgeom dependency from the spatialite Debian
package, because it introduces a circular dependency that is
complicating updates of GEOS, GDAL and related packages [0].

The circular dependency is currently preventing rebuilds with GDAL
1.11.4, making us unable to prepare the upgrade from 1.11.3.

I'm seriously considering dropping the liblwgeom dependency from
spatialite already to get rid of the circular dependency, but this will
introduce an ABI break which I'd like to avoid. Resolving this situation
will be painful either way, so we might as well get it over with now.

If the new topological library will be available soon, and spatialite is
updated to use it, the reduced functionality in spatialite will be a
short lived issue. If not, we'll just have to live with the reduced
functionality.

[0] https://bugs.debian.org/808606

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sandro Santilli-2
On Fri, Feb 05, 2016 at 02:48:17PM +0100, Sebastiaan Couwenberg wrote:
> On 18-10-15 20:52, Andrea Peri wrote:
> > The new topological library of Tuscany Region will be deposited on a
> > distinct GIT repository, with an open license similar to that of
> > liblwgeom and of course we will be pleased and honored if in the future,
> > when it will be more robust and reliable, the Community of PostGIS
> > Developers and Packagers will decide to link it as a shared library for the
> > topological component functions.
>
> Is there any news on the new topological library?

Current state is here:
https://gitlab.com/rttopo/rttopo

It is basically liblwgeom with symbols renamed to have an RT
prefix, the API reworked to be fully re-entrant and the SFCGAL
dependency removed.

> I'd like to drop the liblwgeom dependency from the spatialite Debian
> package, because it introduces a circular dependency that is
> complicating updates of GEOS, GDAL and related packages [0].

Wow, my head spins, the circularity is:

 liblwgeom -> sfcgal -> openscenegraph -> gdal -> spatialite ->
 liblwgeom

So rttopo should fix that, by cutting the "sfcgal" out.

> I'm seriously considering dropping the liblwgeom dependency from
> spatialite already to get rid of the circular dependency, but this will
> introduce an ABI break which I'd like to avoid. Resolving this situation
> will be painful either way, so we might as well get it over with now.

I feel your pain. Wouldn't it be wonderful if GDAL could use
runtime plugins for drivers ?

> If the new topological library will be available soon, and spatialite is
> updated to use it, the reduced functionality in spatialite will be a
> short lived issue. If not, we'll just have to live with the reduced
> functionality.

The first release should be available before the end of 2016, but
I dont' really have a date. It's mostly up to Alessandro Furieri
to do the spatialite side of it.

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: A letter to the Postgis Developers and Packagers

Sebastiaan Couwenberg
On 05-02-16 18:54, Sandro Santilli wrote:

> On Fri, Feb 05, 2016 at 02:48:17PM +0100, Sebastiaan Couwenberg wrote:
>> On 18-10-15 20:52, Andrea Peri wrote:
>>> The new topological library of Tuscany Region will be deposited on a
>>> distinct GIT repository, with an open license similar to that of
>>> liblwgeom and of course we will be pleased and honored if in the future,
>>> when it will be more robust and reliable, the Community of PostGIS
>>> Developers and Packagers will decide to link it as a shared library for the
>>> topological component functions.
>>
>> Is there any news on the new topological library?
>
> Current state is here:
> https://gitlab.com/rttopo/rttopo

Thanks for the link. I'll start packaging a git snapshot shortly.

> It is basically liblwgeom with symbols renamed to have an RT
> prefix, the API reworked to be fully re-entrant and the SFCGAL
> dependency removed.
>
>> I'd like to drop the liblwgeom dependency from the spatialite Debian
>> package, because it introduces a circular dependency that is
>> complicating updates of GEOS, GDAL and related packages [0].
>
> Wow, my head spins, the circularity is:
>
>  liblwgeom -> sfcgal -> openscenegraph -> gdal -> spatialite ->
>  liblwgeom
>
> So rttopo should fix that, by cutting the "sfcgal" out.

This is a recent additional circular dependency caused by the
introduction of sfcgal. The initial circular dependency is
spatialite->postgis/liblwgeom->gdal->spatialite. It was worked around
for the transition to GEOS 3.5.0, but with sfcgal the situation got worse.

>> I'm seriously considering dropping the liblwgeom dependency from
>> spatialite already to get rid of the circular dependency, but this will
>> introduce an ABI break which I'd like to avoid. Resolving this situation
>> will be painful either way, so we might as well get it over with now.
>
> I feel your pain. Wouldn't it be wonderful if GDAL could use
> runtime plugins for drivers ?

That sounds good, but GDAL drivers are not very problematic so far. The
GDAL <-> GRASS dependency is resolved by building the grass plugin
separately.

GDAL not having a direct dependency on SpatiaLite will help with this
specific issue. As will replacing liblwgeom with librttopo in spatialite.

>> If the new topological library will be available soon, and spatialite is
>> updated to use it, the reduced functionality in spatialite will be a
>> short lived issue. If not, we'll just have to live with the reduced
>> functionality.
>
> The first release should be available before the end of 2016, but
> I dont' really have a date. It's mostly up to Alessandro Furieri
> to do the spatialite side of it.

That will be too late for upcoming Ubuntu xenial (import freeze on Feb
18) and Debian stretch releases (expected freeze on Nov 5), but we'll
have plenty of time to get it into the next releases.

For now I've dealt with the circular dependency by dropping the
liblwgeom dependency from spatialite.

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel