Coming releases of PROJ.4

Next Topic
 
classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Coming releases of PROJ.4

Kristian Evers-2

All,

 

Lately I have been thinking about the future of PROJ.4. FOSS4G is coming up soon, 

and for the last couple of years a new version of PROJ.4 has been released at the same time. I

think we should follow that tradition this year, but also mix it up a bit after that. Below is a rough

plan of how I would like to see PROJ.4 evolve over the next 2-3 years.

 

The executive summary is:

 

  1. Release three times a year

  2. Introducing major breaking changes after the next release that

      a. Only has one public API

      b. Only has one build system

  3. … while keeping a maintenance version alive for two years

  4. Restructure version numbering (4.9.x -> 10.0.0)

 

And now the gory details.

 

At the moment PROJ.4 has three different public API's: projects.h, proj_api.h and proj.h. The

main driver for adding new API's has been to add a new dimension. With proj_api.h came the

vertical component and with proj.h time has been introduced to PROJ.4 as well. So in a sense

they mark the changing of eras of the needs of the geospatial community with regards to

coordinate transformation. Albeit most have yet to realize the importance of the time component, 

it is clear that it will play an important role in the not so far future.

 

I propose that we narrow that down to just one API and one build system over the coming

years. The projects.h interface has a long history and exposes most of the inner workings

of PROJ.4. Originally it was simple but during the years it has become quite complicated. I don't

think it is particularly suitable as the public interface for this library. That is also why the

proj_api.h interface exists. It was created in an effort of simplifying things and making a

more modern API. That was now roughly 15 years ago, and it is perhaps not so modern

anymore. On top of that it has some unfortunate design choices, for instance the

context-system which is rather hard to work with and comprehend. Unfortunately it is also

completely undocumented and the original ideas behind it seem to be lost.

 

PROJ.4 also has three different build-systems: Autotools, nmake and CMake. Autotools is the

authoritative build-system which the other two try to mimic as best as possible. Nmake was

introduced in order to build PROJ.4 on Windows and lately CMake has also been introduced

as there was a demand for it.

 

Due to the above I suggest that we focus our efforts entirely on the proj.h API and CMake

from the next release and onwards. For that to be a proper success we will have to abandon

the older API's and build systems. Of course that is not something that can be done from one

day to the next. Hence I think we should  phase them out over a long period of time.

 

My idea is to start working on simplifying PROJ.4 immediately after the next release, with the

aim of being able to release a fully working and simplified library in august 2018. In parallel

we should maintain a backwards compatible version for two years, to give developers amble

time to adjust to the many changes that will be introduced. I envision that new projections

and possibly other changes are applied to the maintenance version for the first years, and

after that only critical bug fixes will be applied. This should ensure that people can use the

library without problems until they make the transition to the new simplified version.

 

This is also a good time to restructure the version numbers. Today we have a system where

The major release number is fixed (or at least would be quite controversial to change), leaving

just to numbers to change at release. Ideally we should follow the structure of Semantic

Versioning [0]. I propose that the next release that breaks the API is called “PROJ.4 10.0.0”.

This way we get around the awkward situation we have today, signal to the world that

something substantial has happened and we have a better version numbering system going

forward.

 

Below is a detailed plan of how my proposed release schedule for the next three years:

 

 

Version      Time of release        Action

-------      ---------------        ----------------------------------------------------------

4.9.4        August 2017            - "Standard" release of current master branch.

                                    - Announce that this will be the final version where

                                      projects.h is publicly accessible.

                                    - Introduce proj.h as the “API of the future”.

                                      tag as experimental.

                                    - Create branch 4.9-Maintenance.

 

4.9.5        December 2017          - Backport new projections.

                                    - Bug Fixes.

 

4.9.6        April 2018             - Backport new projections.

                                    - Bug Fixes.

 

4.9.8        August 2018            - Backport new projections.

                                    - Bug Fixes.

                                    - Last version of 4.9.x that introduces new features.

                                      In maintenance mode from now on.

10.0.0       August 2018            - First version without projects.h and proj_api.h

                                    - Only use CMake as build-system.

                                    - proj.h no longer experimental.

                                    - Create branch Release-10.0

 

4.9.10       December 2018          - Maintenance release. Only bug fixes.

10.x.y       December 2018          - Regular release. New features. Bug fixes.

 

4.9.11       April 2019             - Maintenance release. Only bug fixes.

10.x.y       April 2019             - Regular release. New features. Bug fixes.

 

4.9.12       August 2019            - Final release. Only bug fixes.

                                    - The 4.9-branch will be discontinued.

10.x.y       August 2019            - Regular release. New features. Bug fixes.

 

10.x.y       December 2019          - Regular release. New features. Bug fixes.

 

10.x.y       April 2020             - Regular release. New features. Bug fixes.

 

10.x.y       August 2020            - Regular release. New features. Bug fixes.

 

 

 

This is a lot at once, but I think it is necessary to make the library easier to maintain in

the future. Please let me know what you think about this. If the description isn’t detailed

enough I am happy to elaborate my thoughts.

 

/Kristian

 

 

[0] http://semver.org/


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2

Kristian,

 

> The projects.h interface has a long history and exposes

> most of the inner workings of PROJ.4.

 

As far as I know in the FOSS ecosystem, projects.h use is very marginal, or accidental. The only occurence I've in mind is in the OGDI project that uses nad_init() in one place. But I wouldn't mind too much about OGDI...

 

> That is also why the

> proj_api.h interface exists. It was created in an effort of simplifying

> things and making a more modern API. That was now roughly 15 years ago, and

> On top of that it has some unfortunate

> design choices, for instance the context-system which is rather hard to

> work with and comprehend. Unfortunately it is also completely undocumented

> and the original ideas behind it seem to be lost.

 

I'm not completely sure but I think the main driver for the context system was to remove / deprecate use of pj_errno as a global error variable. errno is now attached to the context.

 

While just reviewing proj.h, I had a few questions/observations :

https://github.com/OSGeo/proj.4/issues/529 (I see you've began to answer to some questions while I was still editing it ;-))

We should probably make sure it is in a clean enough way before releasing it to the world

 

>

> PROJ.4 also has three different build-systems: Autotools, nmake and CMake.

> Autotools is the authoritative build-system which the other two try to

> mimic as best as possible. Nmake was introduced in order to build PROJ.4 on

> Windows and lately CMake has also been introduced as there was a demand for

> it.

>

> Due to the above I suggest that we focus our efforts entirely on the proj.h

> API and CMake from the next release and onwards.

 

Adoption of the new API and deprecation of proj_api.h will show greater resistance than dropping autoconf/nmake.

 

I'm wondering: is there a reason for having a new proj.h ? (we perhaps discussed that, but I already forgot) Imagine that I want to write code that is compatible of old and new proj versions (I'm pretty sure that a number of proj users will have to do that at some point). It could be convenient to still include proj_api.h and depending on the value of PJ_VERSION decide which API are available. Whereas if you have a new header, that make you also change your autoconf/cmake logic detection of proj4.

 

In the transition period, proj_api.h could have two sections : a section with the new API and with the ancient API clearly separated. At some point ancient API would be removed.

 

On the other hand, having the new header and removing the old one will be an obvious way of signaling "eh, breaking changes have occured !" :-)

 

Actually one of the point of having a dedicatd proj.h would be to make sure that people don't accidentally use old API that will be removed. But I see that proj.h includes proj_api.h, so this risk still exists. Why proj.h couldn't avoid the proj_api.h depedency ?

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Kristian Evers-2

Even,

 

> As far as I know in the FOSS ecosystem, projects.h use is very marginal, or accidental. The only occurence I've in mind is in the OGDI project that uses nad_init() in one place. But I wouldn't mind too much about OGDI...

 

Good, that makes it a bit easier. There might be users outside of FOSS and I think it is fair to give them some time to bring their things in order if/when we break stuff.

 

> I'm not completely sure but I think the main driver for the context system was to remove / deprecate use of pj_errno as a global error variable. errno is now attached to the context.

 

Yeah, I believe the idea was to make it thread-safe. Why regular errno is not used I can understand though.

 

> We should probably make sure it is in a clean enough way before releasing it to the world

 

I agree. I think we have time to get it in a reasonable state before august.

 

> I'm wondering: is there a reason for having a new proj.h ?

 

The idea was to produce a new clean API with more sensible types, better namespace, etc. It has been discussed earlier both here and on GitHub (for instance here https://github.com/OSGeo/proj.4/pull/388)

 

 > On the other hand, having the new header and removing the old one will be an obvious way of signaling "eh, breaking changes have occured !" :-)

 

That was exactly what I had in mind! Along with a giant leap forward in version number it should raise all the flags of people using PROJ.4.

It was also why I suggested to keep a maintenance version based on 4.9.4 for a few years. So developers have time to absorb the changes in their own time.

 

> Actually one of the point of having a dedicatd proj.h would be to make sure that people don't accidentally use old API that will be removed. But I see that proj.h includes proj_api.h, so this risk still exists. Why proj.h couldn't avoid the proj_api.h depedency ?

 

I think it should be possible to remove the dependency of proj_api.h in proj.h. Perhaps by putting the parts that are dependant on proj_api.h in a proj_internal.h as suggested in your GitHub issue?

 

Kristian

 

Fra: Even Rouault [mailto:[hidden email]]
Sendt: 22. juni 2017 18:48
Til: [hidden email]
Cc: Kristian Evers
Emne: Re: [Proj] Coming releases of PROJ.4

 

Kristian,

 

> The projects.h interface has a long history and exposes

> most of the inner workings of PROJ.4.

 

As far as I know in the FOSS ecosystem, projects.h use is very marginal, or accidental. The only occurence I've in mind is in the OGDI project that uses nad_init() in one place. But I wouldn't mind too much about OGDI...

 

> That is also why the

> proj_api.h interface exists. It was created in an effort of simplifying

> things and making a more modern API. That was now roughly 15 years ago, and

> On top of that it has some unfortunate

> design choices, for instance the context-system which is rather hard to

> work with and comprehend. Unfortunately it is also completely undocumented

> and the original ideas behind it seem to be lost.

 

I'm not completely sure but I think the main driver for the context system was to remove / deprecate use of pj_errno as a global error variable. errno is now attached to the context.

 

While just reviewing proj.h, I had a few questions/observations :

https://github.com/OSGeo/proj.4/issues/529 (I see you've began to answer to some questions while I was still editing it ;-))

We should probably make sure it is in a clean enough way before releasing it to the world

 

>

> PROJ.4 also has three different build-systems: Autotools, nmake and CMake.

> Autotools is the authoritative build-system which the other two try to

> mimic as best as possible. Nmake was introduced in order to build PROJ.4 on

> Windows and lately CMake has also been introduced as there was a demand for

> it.

>

> Due to the above I suggest that we focus our efforts entirely on the proj.h

> API and CMake from the next release and onwards.

 

Adoption of the new API and deprecation of proj_api.h will show greater resistance than dropping autoconf/nmake.

 

I'm wondering: is there a reason for having a new proj.h ? (we perhaps discussed that, but I already forgot) Imagine that I want to write code that is compatible of old and new proj versions (I'm pretty sure that a number of proj users will have to do that at some point). It could be convenient to still include proj_api.h and depending on the value of PJ_VERSION decide which API are available. Whereas if you have a new header, that make you also change your autoconf/cmake logic detection of proj4.

 

In the transition period, proj_api.h could have two sections : a section with the new API and with the ancient API clearly separated. At some point ancient API would be removed.

 

On the other hand, having the new header and removing the old one will be an obvious way of signaling "eh, breaking changes have occured !" :-)

 

Actually one of the point of having a dedicatd proj.h would be to make sure that people don't accidentally use old API that will be removed. But I see that proj.h includes proj_api.h, so this risk still exists. Why proj.h couldn't avoid the proj_api.h depedency ?

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2

 

> It was also why I suggested to keep a maintenance version based on 4.9.4 for

> a few years. So developers have time to absorb the changes in their own

> time.

 

I think that it is important to have at some point a release that has both functional proj_api.h and proj.h so that it can serve as a migration basis. If a project doesn't want to be compatible of both API and migrate directly to the new API, it must be able to rely on the new API in a release. So maintainance versions of 4.9 will have to fix potential issues in proj.h and related implementation. I'm also thinking to Linux distributions that will be able to ship only a single proj version and will perhaps have packages that still use the old API while others have already migrated.

 

I'm thinking as a potential test for the new API to use the code sprint at FOSS4G-Europe to try to make GDAL use the new API.

 

> I think it should be possible to remove the dependency of proj_api.h in

> proj.h. Perhaps by putting the parts that are dependant on proj_api.h in a

> proj_internal.h as suggested in your GitHub issue?

 

Yes, proj.h should be as clean as possible. And I wouldn't mind if there are duplicated declarations in proj.h and proj_api.h. Actually I think both should be mutually exclusive and self-contained. It makes little sense to include both in the same compilation unit. Of course that will require some care of not putting conflicting declarations for the same function...

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Kristian Evers-2

> I think that it is important to have at some point a release that has both functional proj_api.h and proj.h so that it can serve as a migration basis

 

I might not have expressed myself clear enough in the initial write-up. The next release, 4.9.4, will include both proj_api.h and proj.h. We should put in the effort to bring proj.h in a state where it can replace proj_api.h. I am able to spend a fair amount of time on this during the next month or so.

 

> I'm also thinking to Linux distributions that will be able to ship only a single proj version and will perhaps have packages that still use the old API while others have already migrated.

 

Good point.

 

> I'm thinking as a potential test for the new API to use the code sprint at FOSS4G-Europe to try to make GDAL use the new API.

 

That would be awesome!

 

>  Yes, proj.h should be as clean as possible. And I wouldn't mind if there are duplicated declarations in proj.h and proj_api.h. Actually I think both should be mutually exclusive and self-contained. It makes little sense to include both in the same compilation unit. Of course that will require some care of not putting conflicting declarations for the same function...

 

 

I agree. I think today, from the users perspective, proj.h and proj_api.h are mutually exclusive already. But yeah, the cut should be cleaner.

 

So to sum up, for the next release we will keep projects.h, proj_api.h and proj.h as public interfaces to PROJ.4. We will make sure that proj.h is a proper alternative to proj_api.h, and keep both in a maintenance version for a longer period, e.g. two years. After the 4.9.4 release we will start working on getting rid of (publicly available) projects.h, proj_api.h, autotools and nmake in version 10.0.0 and backport bug fixes and new proj.h functionality to 4.9-maintenance. Agreed?

 

Kristian

 

Fra: Even Rouault [mailto:[hidden email]]
Sendt: 22. juni 2017 20:23
Til: Kristian Evers
Cc: proj@lis
ts.maptools.org
Emne: Re: SV: [Proj] Coming releases of PROJ.4

 

 

> It was also why I suggested to keep a maintenance version based on 4.9.4 for

> a few years. So developers have time to absorb the changes in their own

> time.

 

I think that it is important to have at some point a release that has both functional proj_api.h and proj.h so that it can serve as a migration basis. If a project doesn't want to be compatible of both API and migrate directly to the new API, it must be able to rely on the new API in a release. So maintainance versions of 4.9 will have to fix potential issues in proj.h and related implementation. I'm also thinking to Linux distributions that will be able to ship only a single proj version and will perhaps have packages that still use the old API while others have already migrated.

 

I'm thinking as a potential test for the new API to use the code sprint at FOSS4G-Europe to try to make GDAL use the new API.

 

> I think it should be possible to remove the dependency of proj_api.h in

> proj.h. Perhaps by putting the parts that are dependant on proj_api.h in a

> proj_internal.h as suggested in your GitHub issue?

 

Yes, proj.h should be as clean as possible. And I wouldn't mind if there are duplicated declarations in proj.h and proj_api.h. Actually I think both should be mutually exclusive and self-contained. It makes little sense to include both in the same compilation unit. Of course that will require some care of not putting conflicting declarations for the same function...

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Sebastiaan Couwenberg
On 06/22/2017 08:49 PM, Kristian Evers wrote:
>> I'm also thinking to Linux distributions that will be able to ship only a single proj version and will perhaps have packages that still use the old API while others have already migrated.
>
> Good point.
To be more specific, there will be projects that will never move to the
new API due to lack of development manpower, etc.

Ideally the old API remains available, and gets put into an indefinite
maintenance mode.

The alternative is removing those projects from the distributions when
the old API is no longer available (i.e. after 4.9.12).

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2

On jeudi 22 juin 2017 21:15:27 CEST Sebastiaan Couwenberg wrote:

> On 06/22/2017 08:49 PM, Kristian Evers wrote:

> >> I'm also thinking to Linux distributions that will be able to ship only a

> >> single proj version and will perhaps have packages that still use the

> >> old API while others have already migrated.>

> > Good point.

>

> To be more specific, there will be projects that will never move to the

> new API due to lack of development manpower, etc.

>

> Ideally the old API remains available, and gets put into an indefinite

> maintenance mode.

 

On my ubuntu 16.04, I see

 

$ apt-cache rdepends libproj9

libproj9

Reverse Depends:

libproj-dev

libmapserver2

zygrib

xastir

thuban

therion

survex-aven

survex

sumo

spatialite-gui

sosi2osm

shapelib

saga

qmapshack

qlandkartegt

qgis

python3-pyproj

python-pyproj

proj-bin

postgresql-9.5-postgis-2.2

pdl

osm2pgsql

ogdi-bin

ncl-ncarg

merkaartor

libvtk6.2

libsqlite3-mod-spatialite

libspatialite7

libqgis-core2.8.6

libproj-java

liblwgeom-2.2-5

libogdi3.2

libmapserver2

libmapnik3.0

libmagplus3v5

cdo

libgeotiff2

libgeo-proj4-perl

libgdal1i

grass-core

gpx2shp

 

I didn't think the list was so long. Indeed the cumulative amount of effort for migrating all those projects is very likely much higher than supporting the old API.

 

As far as I'm concerned, I'm afraid I'm somehow the fallback maintainer of libgeotiff and shapelib, but I've never touched at their proj.4 dependant part (I'm in fact almost discovering that they depend on proj.4 !) I wouldn't be so enthousiastic in spending time touching that part.

 

Kristian, do you think that would be feasible to keep the proj_api API, while possibly making it internally use the new API if that simplifies design / maintainance ? A kind of compatibility layer.

 

I could potentially collect the API calls of projects I'm familiar with (GDAL, shapelib, libgeotiff, mapserver and QGIS), so we have a better picture of what is actually used in the wild. At the very least, it will give us an idea of what the capabilities the new API should cover...

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Howard Butler-3
In reply to this post by Kristian Evers-2

On Jun 22, 2017, at 1:49 PM, Kristian Evers <[hidden email]> wrote:
 
So to sum up, for the next release we will keep projects.h, proj_api.h and proj.h as public interfaces to PROJ.4. We will make sure that proj.h is a proper alternative to proj_api.h, and keep both in a maintenance version for a longer period, e.g. two years. After the 4.9.4 release we will start working on getting rid of (publicly available) projects.h, proj_api.h, autotools and nmake in version 10.0.0 and backport bug fixes and new proj.h functionality to 4.9-maintenance. Agreed? 

Ditching manually-constructed and out-of-date nmake for cmake seems fine to me (ooh, CMake gives me project files now!), but ditching autotools that has worked for people for decades does not. That change would ripple into packagers and the distros for little or no appreciable benefit. 

I'm generally enthusiastic in spirit about the rest of your proposal, but I must say I'm very skeptical that we can actually *remove* anything -- especially headers with ambiguous public vs. private behavior that people now depend upon. We can add a new one, and we can make make a much cleaner and less ambiguous API, but I don't think we can take the old ones back. There's just too much momentum and calcification that is going to make it near impossible to go back and update external code to work with a new arrangement of such a foundational library such as Proj.4. I think we are stuck with the (API) sins of our forefathers, but I'd like to hear from more people who think we are not. 

We need to be delicate about changing things for a library such as Proj.4 only for the sake of being "better". This is an unfortunate consequence of maintaining a library that everything ultimately uses (and started using 20+ years ago). 

_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Kristian Evers-2
In reply to this post by Even Rouault-2

Even,

 

> Kristian, do you think that would be feasible to keep the proj_api API, while possibly making it internally use the new API if that simplifies design / maintainance ? A kind of compatibility layer.

 

On top of my head, yes. It might be a challenge to sort out contexts and the ProjFileAPI, but I think it is doable. I’ll have to have a closer look though.

 

> I could potentially collect the API calls of projects I'm familiar with (GDAL, shapelib, libgeotiff, mapserver and QGIS), so we have a better picture of what is actually used in the wild. At the very least, it will give us an idea of what the capabilities the new API should cover...

 

That would be very helpful. I am especially interested to see if anyone has ever used the ProjFileAPI functions…

 

 

 

Howard,

 

> Ditching manually-constructed and out-of-date nmake for cmake seems fine to me (ooh, CMake gives me project files now!), but ditching autotools that has worked for people for decades does not. That change would ripple into packagers and the distros for little or no appreciable benefit. 

 

I won’t claim to be an expert  in CMake, quite the opposite really, but I was under the impression that it  would be possible to set up CMake in a way so that it mimics the autotools behavior exactly when UNIX makefiles are generated. I am completely wrong here?

The benefit would be to limit the amount of code we have to maintain. It would only benefit us, true, but it would also free time that can be spend improving the overall project instead of maintaining status quo. I don’t feel too strongly about this, I just wanted to throw it out there while I was at it.

 

> but I must say I'm very skeptical that we can actually *remove* anything -- especially headers with ambiguous public vs. private behavior that people now depend upon

 

I get that. On the other hand I fear that we will be locked in place forever if we don’t allow breaking stuff once in a while. If at least we can get away with not exposing projects.h I think there will be much more room to play in the future. I hope that Even’s assessment about it not being used much is true.

 

> We need to be delicate about changing things for a library such as Proj.4 only for the sake of being "better". This is an unfortunate consequence of maintaining a library that everything ultimately uses (and started using 20+ years ago). 

 

Perhaps the ambition should be to BE MUCH BETTER than previous versions on PROJ.4. So much better that other projects will benefit hugely from using the new API and update their codebase voluntarily. Eventually the problem will solve itself. I already think we are on the right track.

 

/Kristian

 

Fra: [hidden email] [mailto:[hidden email]] På vegne af Even Rouault
Sendt: 22. juni 2017 21:49
Til: [hidden email]
Cc: Sebastiaan Couwenberg
Emne: Re: [Proj] Coming releases of PROJ.4

 

On jeudi 22 juin 2017 21:15:27 CEST Sebastiaan Couwenberg wrote:

> On 06/22/2017 08:49 PM, Kristian Evers wrote:

> >> I'm also thinking to Linux distributions that will be able to ship only a

> >> single proj version and will perhaps have packages that still use the

> >> old API while others have already migrated.>

> > Good point.

>

> To be more specific, there will be projects that will never move to the

> new API due to lack of development manpower, etc.

>

> Ideally the old API remains available, and gets put into an indefinite

> maintenance mode.

 

On my ubuntu 16.04, I see

 

$ apt-cache rdepends libproj9

libproj9

Reverse Depends:

libproj-dev

libmapserver2

zygrib

xastir

thuban

therion

survex-aven

survex

sumo

spatialite-gui

sosi2osm

shapelib

saga

qmapshack

qlandkartegt

qgis

python3-pyproj

python-pyproj

proj-bin

postgresql-9.5-postgis-2.2

pdl

osm2pgsql

ogdi-bin

ncl-ncarg

merkaartor

libvtk6.2

libsqlite3-mod-spatialite

libspatialite7

libqgis-core2.8.6

libproj-java

liblwgeom-2.2-5

libogdi3.2

libmapserver2

libmapnik3.0

libmagplus3v5

cdo

libgeotiff2

libgeo-proj4-perl

libgdal1i

grass-core

gpx2shp

 

I didn't think the list was so long. Indeed the cumulative amount of effort for migrating all those projects is very likely much higher than supporting the old API.

 

As far as I'm concerned, I'm afraid I'm somehow the fallback maintainer of libgeotiff and shapelib, but I've never touched at their proj.4 dependant part (I'm in fact almost discovering that they depend on proj.4 !) I wouldn't be so enthousiastic in spending time touching that part.

 

Kristian, do you think that would be feasible to keep the proj_api API, while possibly making it internally use the new API if that simplifies design / maintainance ? A kind of compatibility layer.

 

I could potentially collect the API calls of projects I'm familiar with (GDAL, shapelib, libgeotiff, mapserver and QGIS), so we have a better picture of what is actually used in the wild. At the very least, it will give us an idea of what the capabilities the new API should cover...

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2
In reply to this post by Even Rouault-2

> I could potentially collect the API calls of projects I'm familiar with

> (GDAL, shapelib, libgeotiff, mapserver and QGIS), so we have a better

> picture of what is actually used in the wild. At the very least, it will

> give us an idea of what the capabilities the new API should cover...

>

 

Here's the result of my exploration of proj API use in those packages :

 

GDAL:

pj_ctx_alloc

pj_ctx_fclose (1)

pj_ctx_free

pj_ctx_get_errno

pj_ctx_set_app_data (1)

pj_ctx_set_debug (1)

pj_ctx_set_logger (1)

pj_dalloc

pj_find_file (2)

pj_free

pj_get_def

pj_get_errno_ref

pj_init --> not really used in fact

pj_init_plus

pj_init_plus_ctx

pj_open_lib (1)

pj_strerrno

pj_transform

 

Regarding (1) and (2), only apply to recent GDAL 2.2

(1) dirty use of the API available in currently released proj versions for a very particular use case of finding the full filename of a grid used by proj

(2) cleaner approch of (1) with the pj_open_lib() function I added recently in proj master version

 

mapserver:

pj_clear_initcache

pj_ctx_alloc

pj_ctx_free

pj_dalloc

pj_deallocate_grids

pj_free

pj_fwd

pj_get_def

pj_get_errno_ref

pj_init

pj_init_ctx

pj_inv

pj_is_latlong

pj_set_finder

pj_strerrno

pj_transform

 

libgeotiff:

pj_free

pj_fwd

pj_init

pj_inv

 

shapelib (not the lib itself, but by a few of the associated utilities):

pj_free

pj_init

pj_is_latlong

pj_transform

 

QGIS (libqgiscore. I don't think other QGIS libs depend directly on proj)

pj_ctx_alloc

pj_ctx_free

pj_dalloc

pj_free

pj_get_def

pj_init_plus_ctx

pj_is_latlong

pj_strerrno

pj_transform

 

 

All of them use only proj_api.h (quite logical since Frank Warmerdam was involved in all those projects at the time proj_api.h was introduced)

 

So the projFileAPI isn't used (except pj_open_lib / pj_ctx_close as a legacy way)

 

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Howard Butler-3
In reply to this post by Kristian Evers-2

On Jun 22, 2017, at 3:57 PM, Kristian Evers <[hidden email]> wrote:
 
> Ditching manually-constructed and out-of-date nmake for cmake seems fine to me (ooh, CMake gives me project files now!), but ditching autotools that has worked for people for decades does not. That change would ripple into packagers and the distros for little or no appreciable benefit. 
 
I won’t claim to be an expert  in CMake, quite the opposite really, but I was under the impression that it  would be possible to set up CMake in a way so that it mimics the autotools behavior exactly when UNIX makefiles are generated. I am completely wrong here?
The benefit would be to limit the amount of code we have to maintain. It would only benefit us, true, but it would also free time that can be spend improving the overall project instead of maintaining status quo. I don’t feel too strongly about this, I just wanted to throw it out there while I was at it.

CMake can produce a somewhat passable facsimile to autotools behavior, but there are many nuances and autotools features that CMake doesn't support or support in the same way. These things matter to the distribution packagers and maintainers, and I've found it's often more trouble to make CMake behave the way they need than it is to just provide an autotools layout implementation. That situation in 2017 is better than it was in 2008 when I started using CMake, but it is still a friction point.

but I must say I'm very skeptical that we can actually *remove* anything -- especially headers with ambiguous public vs. private behavior that people now depend upon
 
I get that. On the other hand I fear that we will be locked in place forever if we don’t allow breaking stuff once in a while. If at least we can get away with not exposing projects.h I think there will be much more room to play in the future. I hope that Even’s assessment about it not being used much is true.

Successfully burying projects.h would be enough to declare victory, if we can do so without upsetting too much. Even's analysis shows that might be possible. 

We need to be delicate about changing things for a library such as Proj.4 only for the sake of being "better". This is an unfortunate consequence of maintaining a library that everything ultimately uses (and started using 20+ years ago). 
 
Perhaps the ambition should be to BE MUCH BETTER than previous versions on PROJ.4. So much better that other projects will benefit hugely from using the new API and update their codebase voluntarily. Eventually the problem will solve itself. I already think we are on the right track.

#1: "It's ugly, but it works, it will stay working, and it requires no more software development"
#2: "It works more smoothly, but you have to write/update software to use it, and it does the same thing as #1"

There must be new features that incentivize the work required to do #2. For the sake of itself is not enough. Maybe access to transformation pipelines are enough for that, I don't know. It's a frustrating reality of a very old and well-established software project with lots of implementations. The success of Proj.4 has indeed locked itself in place.







_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Mike Toews-2
In reply to this post by Kristian Evers-2
On 23 June 2017 at 03:13, Kristian Evers <[hidden email]> wrote:
>   4. Restructure version numbering (4.9.x -> 10.0.0)

Silly question: will the software have ".4" baked into the name (i.e.
PROJ.4 version 10.x), or will it be incremented (i.e. PROJ.10, a
terrible idea), or dropped entirely (i.e. PROJ)? Ideally, I'd advise
future-proofing the name by simply reverting to the original name,
PROJ. Of course, renaming has lots of implications for 3rd party
software that have objects like "proj4text" or "proj4string".

To keep the name permanently as PROJ.4 for any future version, the
meaning of ".4" needs to change to something like 4-dimension
projection capabilities (x, y, z, t).

On a historical note, there never was PROJ.1 from 1980 or PROJ.2 from
1985, or even a PROJ.3 (as documented at the time) from 1990. The name
PROJ.4 first appeared for release 4 around 1994, possibly to help
distinguish from the previous releases.
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Sebastiaan Couwenberg
In reply to this post by Even Rouault-2
On 06/22/2017 11:45 PM, Even Rouault wrote:

>> I could potentially collect the API calls of projects I'm familiar with
>> (GDAL, shapelib, libgeotiff, mapserver and QGIS), so we have a better
>> picture of what is actually used in the wild. At the very least, it will
>> give us an idea of what the capabilities the new API should cover...
>>
>
> Here's the result of my exploration of proj API use in those packages :
>
> [...]
>
> All of them use only proj_api.h (quite logical since Frank Warmerdam was involved in all those
> projects at the time proj_api.h was introduced)
>
> So the projFileAPI isn't used (except pj_open_lib / pj_ctx_close as a legacy way)
The language bindings are likely to use projects.h, at least Geo::Proj4
(libgeo-proj4-perl) does. pyproj (python-pyproj) uses proj_api.h.

Using codesearch.debian.net shows that many of the PROJ.4 reverse
dependencies include projects.h, so getting rid of that requires
patching those projects.

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Roger Bivand
In reply to this post by Even Rouault-2
The listing for the R rgdal package is:

#include <projects.h> 

Was used until access to share/proj files was provided,
needed for rgdal (and sf below) to check for proj metadata files
on package install.

Is still used in the essential function by Barry Rowlingson:

#include <projects.h>

int inversetest(PJ *P){
  return (P->inv ? 1: 0);

}

test for the existence of an inverse, see also examples in:

https://r-forge.r-project.org/scm/viewvc.php/pkg/man/project.Rd?view=markup&root=rgdal

getPROJ4VersionInfo()
projs <- as.character(projInfo()$name)
res <- logical(length(projs))
names(res) <- projs
msgs <- character(length(projs))
names(msgs) <- projs
owarn <- options("warn")$warn
options(warn=2L)
for (i in seq(along=res)) {
  iprs <- paste("+proj=", projs[i], sep="")
  xy <- try(project(cbind(0, 0), iprs, legacy=TRUE), silent=TRUE)
  if (class(xy) == "try-error") {
    res[i] <- NA
    msgs[i] <- paste("fwd:", strsplit(xy, "\n")[[1]][2])
  } else {
    out <- try(project(xy, iprs, inv=TRUE, legacy=TRUE), silent=TRUE)
    if (class(out) == "try-error") {
      res[i] <- NA
      msgs[i] <- paste("inv:", strsplit(out, "\n")[[1]][2])
    } else res[i] <- isTRUE(all.equal(cbind(0,0), out))
  }
}
options(warn=owarn)

The problem is that pj_inv() hard fails when an inverse is attempted and none exists, so we need protection to avoid crashing the R session:

  if ( inversetest(pj) == 0) {
    pj_free(pj);
    error("No inverse for this projection");
  };


The effective usage is:

#include <proj_api.h>

pj_ctx_fclose
pj_ctx_fgets
pj_free
pj_fwd
pj_get_datums_ref
pj_get_def
pj_get_default_ctx
pj_get_ellps_ref
pj_get_errno_ref
pj_get_list_ref
pj_get_release
pj_get_units_ref
pj_init_plus
pj_inv
pj_is_latlong
pj_open_lib
pj_strerrno
pj_transform

for R package proj4:

#include <proj_api.h>

pj_free
pj_fwd
pj_get_errno_ref
pj_init_plus
pj_inv
pj_strerrno
pj_transform

and for R package sf:

#include <proj_api.h>

pj_ctx_fclose
pj_free
pj_get_datums_ref
pj_get_def
pj_get_default_ctx
pj_get_ellps_ref
pj_get_errno_ref
pj_get_list_ref
pj_get_units_ref
pj_init_plus
pj_init_plus
pj_open_lib
pj_open_lib
pj_strerrno

here with CPL_transform() used for projection through the GDAL -> Proj4 dependence.

Roger
Roger Bivand
NHH Norwegian School of Economics, Bergen, Norway
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Kristian Evers-2
Roger,

> Is still used in the essential function by Barry Rowlingson:
>
> #include <projects.h>
>
> int inversetest(PJ *P){
>   return (P->inv ? 1: 0);
>
> }

We could add an pj_has_inverse() function to proj_api.h. Would that work for you? Alternatively a similar check can be made with  the proj.h API, but that of course requires rewriting more code.



Sebastian,

> Using codesearch.debian.net shows that many of the PROJ.4 reverse
> dependencies include projects.h, so getting rid of that requires
> patching those projects.


I did the following search: https://codesearch.debian.net/search?q=%23include+%3Cprojects.h%3E&perpkg=1 

It returns 7 relevant projects. Basemap, Mapserver, Python-pyproj, Thuban, Therion, Paraview and SAGA.

Basemap, Therion and Paraview seem to include a complete copy of the PROJ.4 code base. Presumably they won't be bothered by a change. That leaves 4 projects, of which we have good access to most and will be able to help absorb the changes. I don't think it is completely outrageous to introduce a breaking change if it doesn't affect more projects than that. Of course it is likely that my search is not catching everything. Have I missed something?

 Mike,

> Silly question: will the software have ".4" baked into the name (i.e. PROJ.4 version 10.x),

My suggestion is to keep the projects name as PROJ.4 (all caps), and eventually change the versioning to 10.y.z,
so that you use version 10.y.z of PROJ.4. It's a weird bump in version number from 4 to 10, but hey, if Microsoft
can do that kind of thing, so can we :-) The idea behind the big jump would be to have full use of major, minor
and patch numbers, as well as signal to the world that a larger change has happened. Makes sense?



Howard,

> Successfully burying projects.h would be enough to declare victory, if we can do so without upsetting too much. Even's analysis shows that might be possible.

Agreed. I totally acknowledge that my viewpoint is too progressive, and others might lean a bit to the conservative side.
Settling on getting rid of projects.h is a good compromise I think.

> #1: "It's ugly, but it works, it will stay working, and it requires no more software development"
> #2: "It works more smoothly, but you have to write/update software to use it, and it does the same thing as #1"
>
> There must be new features that incentivize the work required to do #2. For the sake of itself is not enough. Maybe access to transformation pipelines are enough for that, I don't know. It's a  frustrating reality of a very old and well-established software project with lots of implementations. The success of Proj.4 has indeed locked itself in place.

Personally I believe transformation pipelines is enough of a game-changer but I am not sure that it is apparent to most users at the moment.
But there are plenty of room for other game-changing features as well, which eventually makes it a no brainer to use the new API.



/Kristian



_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2
In reply to this post by Roger Bivand

 

> The problem is that pj_inv() hard fails when an inverse is attempted and

> none exists, so we need protection to avoid crashing the R session:

 

For the record, this issue has been fixed in proj 4.9.3 per :

https://github.com/OSGeo/proj.4/commit/757a2c8f946faccf9d094d76cb79e6ebe0006564#diff-f0c3df1eafd91a86f6cd9cf68cc306ddR23

 

pj_inv will now return (HUGE_VAL,HUGE_VAL) if the inverse doesn't exist.

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Roger Bivand
On Fri, 23 Jun 2017, Even Rouault wrote:

>> The problem is that pj_inv() hard fails when an inverse is attempted and
>> none exists, so we need protection to avoid crashing the R session:
>
> For the record, this issue has been fixed in proj 4.9.3 per :
> https://github.com/OSGeo/proj.4/commit/757a2c8f946faccf9d094d76cb79e6ebe0006564#diff-f0c3df1eafd91a86f6cd9cf68cc306ddR23
>
> pj_inv will now return (HUGE_VAL,HUGE_VAL) if the inverse doesn't exist.
>

Thanks for the heads-up - I'll add a test conditioning on the PROJ4
version (plenty are still at 4.8 or earlier, I believe).

Roger

>

--
Roger Bivand
Department of Economics, Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; e-mail: [hidden email]
Editor-in-Chief of The R Journal, https://journal.r-project.org/index.html
http://orcid.org/0000-0003-2392-6140
https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Roger Bivand
NHH Norwegian School of Economics, Bergen, Norway
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Even Rouault-2
In reply to this post by Kristian Evers-2

> I did the following search:

> https://codesearch.debian.net/search?q=%23include+%3Cprojects.h%3E&perpkg=1

 

Looking also for #include "projects.h" brings more projects to your below list:

ogdi-dfsg, libgeo-proj4-perl, gpx2shp, pdl, python-pyproj

 

( I removed qtcreator, postbooks which have unrelated projects.h header. And gnudatalanguage for which the include is actually commented in favor of proj_api )

 

>

> It returns 7 relevant projects. Basemap, Mapserver, Python-pyproj, Thuban,

> Therion, Paraview and SAGA.

 

Consider mapserver out of this list. The only user is mapscript/php/php_proj.c and this file isn't is any CMakeList.txt. When adding it, compilation fails on many things, not related to projects.h itself

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Kristian Evers-2

So still a rather short list I would say. Short enough that it is feasible to individually reach out to the affected projects.

 

I just had a look at pyproj. It includes a copy of the PROJ.4 source code as well, so we should not consider that project either.

 

The list at the moment is:

 

- ogdi-dfsg

- libgeo-proj4-perl

- gpx2shp

- pdl

- basemap

- thuban

- therion

- paraview

- SAGA

 

/Kristian

 

Fra: Even Rouault [mailto:[hidden email]]
Sendt: 23. juni 2017 11:37
Til: [hidden email]
Cc: Kristian Evers
Emne: Re: [Proj] Coming releases of PROJ.4

 

> I did the following search:

> https://codesearch.debian.net/search?q=%23include+%3Cprojects.h%3E&perpkg=1

 

Looking also for #include "projects.h" brings more projects to your below list:

ogdi-dfsg, libgeo-proj4-perl, gpx2shp, pdl, python-pyproj

 

( I removed qtcreator, postbooks which have unrelated projects.h header. And gnudatalanguage for which the include is actually commented in favor of proj_api )

 

>

> It returns 7 relevant projects. Basemap, Mapserver, Python-pyproj, Thuban,

> Therion, Paraview and SAGA.

 

Consider mapserver out of this list. The only user is mapscript/php/php_proj.c and this file isn't is any CMakeList.txt. When adding it, compilation fails on many things, not related to projects.h itself

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Coming releases of PROJ.4

Sebastiaan Couwenberg
On 06/23/2017 11:53 AM, Kristian Evers wrote:
> So still a rather short list I would say. Short enough that it is feasible to individually reach out to the affected projects.
>
> I just had a look at pyproj. It includes a copy of the PROJ.4 source code as well, so we should not consider that project either.

In Debian embedded copy copies are not used, the packaged dependencies
are used in stead.

The complete list (source) packages in Debian unstable that (build)
depend on proj is:

 * josm (proj-data only)

 * gpx2shp
 * libgeo-proj4-perl
 * libgeotiff-dfsg
 * ogdi-dfsg
 * openorienteering-mapper
 * pdl
 * proj-rdnap (proj-bin & proj-data only)
 * python-cartopy
 * python-pyproj
 * shapelib
 * sosi2osm
 * spatialite
 * survex
 * zygrib

 * gdal
 * magics++
 * pyspatialite
 * spatialite-gui
 * spatialite-tools

 * cdo
 * dans-gdal-scripts
 * grass
 * libosmium
 * mapcache
 * mapnik
 * mapproxy
 * mapserver
 * merkaartor
 * metview
 * ncl
 * otb
 * pdal
 * postgis
 * qlandkartegt
 * qmapshack
 * saga
 * sumo
 * thuban
 * vtk6
 * xastir

 * ifrit
 * libgdal-grass
 * osm2pgsql
 * python-mapnik
 * qgis
 * therion

See: http://linuxminded.nl/tmp/pkg-grass-transitions/html/proj.html
(check the "good" checkbox)

> The list at the moment is:
>
> - ogdi-dfsg
> - libgeo-proj4-perl
> - gpx2shp
> - pdl
> - basemap
> - thuban
> - therion
> - paraview
> - SAGA

Searching for '#include [<"]projects.h[>"]' results in:

 * qtcreator
 * gnudatalanguage
 * mapserver
 * libgeo-proj4-perl
 * ogdi-dfsg
 * thuban
 * saga
 * postbooks
 * therion
 * python-pyproj
 * gpx2shp
 * paraview
 * basemap
 * proj
 * pdl

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
123
Loading...