Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3
Hello Even

Is there already a plan for the testing parts? I may be interested to
participate on the creation of a test suite. The Open Geospatial
Consortium (OGC) has a GeoAPI project which already provides some
implementation-independent tests (through a standard API) for WKT 2,
EPSG and GIGS (some examples below). Those tests are run for a few years
by Apache SIS. They are in Java, but Proj.4 is already able to execute
some of them through the JNI wrappers. In addition, Python bindings are
in progress.

Of course Proj will have its own tests in C. The GeoAPI tests would be a
complement. An advantage of GeoAPI tests would be that, by running the
same set of tests on different implementations, we increase (I think)
the confidence that the WKT 2 or EPSG codes are really interpreted in
the same way by those different implementations.

Some example of tests:

http://www.geoapi.org/geoapi-conformance/apidocs/org/opengis/test/referencing/gigs/GIGS3005.html#testUTM_zone31N()
http://www.geoapi.org/geoapi-conformance/apidocs/org/opengis/test/wkt/CRSParserTest.html#testGeographic3D()

    Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Even Rouault-2

Hi Martin,

 

Thanks for the pointers and interest. I'm certainly interested in seing solid testing and interoperability.

 

>

> Is there already a plan for the testing parts?

 

Not yet, but there will be definitely tests.

 

> I may be interested to

> participate on the creation of a test suite. The Open Geospatial

> Consortium (OGC) has a GeoAPI project which already provides some

> implementation-independent tests (through a standard API) for WKT 2,

> EPSG and GIGS (some examples below).

 

Where is GIGS described ? I couldn't find much online documentation. I do see proj has now a test/gigs directory with tests converted to the .gie format, but I couldn't find the source content

 

> Those tests are run for a few years

> by Apache SIS. They are in Java, but Proj.4 is already able to execute

> some of them through the JNI wrappers. In addition, Python bindings are

> in progress.

 

Python bindings over what ?

 

>

> Of course Proj will have its own tests in C. The GeoAPI tests would be a

> complement. An advantage of GeoAPI tests would be that, by running the

> same set of tests on different implementations, we increase (I think)

> the confidence that the WKT 2 or EPSG codes are really interpreted in

> the same way by those different implementations.

 

I guess the GeoAPI tests would be needed to be ported/adapted to whatever solution is adopted for proj testing. I don't really feel like using the proj JNI wrapper through Apache SIS would be an ideal solution : too many components aggegated, and I'm not sure if JNI wrapper will have all the new capabilities exposed.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Kristian Evers-2
Even,

You can get the GIGS documents here:


Unfortunately the test datasets comes in unstructured spreadsheets… Micah Cochran did a tremendous job of translating
that JSON for use with pyproj which I then a year or so later converted to the gie format.

There are still a number of tests that we can’t reliable pass in PROJ. I believe some, if not all, of them will be taken care
of during the work you have now committed to.

/Kristian


On 15 May 2018, at 19:42, Even Rouault <[hidden email]> wrote:

Hi Martin,

 

Thanks for the pointers and interest. I'm certainly interested in seing solid testing and interoperability.

 

>
> Is there already a plan for the testing parts?

 

Not yet, but there will be definitely tests.

 

> I may be interested to
> participate on the creation of a test suite. The Open Geospatial
> Consortium (OGC) has a GeoAPI project which already provides some
> implementation-independent tests (through a standard API) for WKT 2,
> EPSG and GIGS (some examples below).

 

Where is GIGS described ? I couldn't find much online documentation. I do see proj has now a test/gigs directory with tests converted to the .gie format, but I couldn't find the source content

 

> Those tests are run for a few years
> by Apache SIS. They are in Java, but Proj.4 is already able to execute
> some of them through the JNI wrappers. In addition, Python bindings are
> in progress.

 

Python bindings over what ?

 

>
> Of course Proj will have its own tests in C. The GeoAPI tests would be a
> complement. An advantage of GeoAPI tests would be that, by running the
> same set of tests on different implementations, we increase (I think)
> the confidence that the WKT 2 or EPSG codes are really interpreted in
> the same way by those different implementations.

 

I guess the GeoAPI tests would be needed to be ported/adapted to whatever solution is adopted for proj testing. I don't really feel like using the proj JNI wrapper through Apache SIS would be an ideal solution : too many components aggegated, and I'm not sure if JNI wrapper will have all the new capabilities exposed.

 

Even

 

--
Spatialys - Geospatial professional services
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3
In reply to this post by Even Rouault-2

Hello Even

Le 15/05/2018 à 19:42, Even Rouault a écrit :

Python bindings over what ?

Over a standard API. Not any particular implementation. To make a comparisons with databases:

  1. The C/C++ language has ODBC as a neutral API for accessing various databases. When accessing a database through ODBC, a C/C++ software does not need to care if the database is MS-Access, PostgreSQL, etc.
  2. Java has JDBC for the same purpose. When accessing a database through JDBC, a Java software does not need to care if the database is PostgreSQL, Derby, etc. JDBC is strongly inspired by ODBC, so both APIs are similar.
  3. Then there is JDBC-ODBC bridge. This bridge allows Java software to access any ODBC-compliant database as if it was a JDBC-compliant database. The JDBC-ODBC bridge was extensively used in early Java day, when few JDBC drivers were available.

Likewise, the Python binding is between Java interfaces of GeoAPI and Python abstract classes of GeoAPI. It is not between Apache SIS or GDAL or any particular implementation. The intent is to allow interoperability between any Python implementation of GeoAPI and any Java implementation of GeoAPI, in a way similar to JDBC-ODBC bridge.

But I guess this would be the topic for another mailing list.

I guess the GeoAPI tests would be needed to be ported/adapted to whatever solution is adopted for proj testing. I don't really feel like using the proj JNI wrapper through Apache SIS would be an ideal solution : too many components aggegated, and I'm not sure if JNI wrapper will have all the new capabilities exposed.

My proposal is independent of Apache SIS. GeoAPI is an OGC project, and as such is independent of any implementation - OGC is very strict about vendor neutrality. GeoAPI is an API derived from OGC/ISO standards. For example ISO 19111 defines an object named SC_GeographicCRS which contains an association to another object named CS_EllipsoidCS, which itself contains associations to axes, which have associations to units of measurements, etc. GeoAPI translates this abstract model into Java interfaces and Python abstract classes, not into implementation. In C, it could be structures with only pointer to functions but no code; providing the code would be up to Proj.

Having a common API allows to run the same tests with different implementations, in the same way that ODBC/JDBC allow to use different databases with the same code. So it would allow Proj and Apache SIS to run the same tests - it does not establish any dependency of one to the other.

This is not theoretical - GeoAPI implementation backed by Proj.4 through the JNI already exists for 5~10 years at http://www.geoapi.org/geoapi-proj4/index.html and has been used for testing Proj.4. At least one Proj bug has been identified by this test suite.

Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3
In reply to this post by Kristian Evers-2
Le 15/05/2018 à 20:02, Kristian Evers a écrit :
> Unfortunately the test datasets comes in unstructured spreadsheets…
> Micah Cochran did a tremendous job of translating
> that JSON for use with pyproj which I then a year or so later
> converted to the gie format.

For information, some of those tests were already converted in a more
structured format there:

http://www.geoapi.org/geoapi-conformance/apidocs/org/opengis/test/referencing/gigs/doc-files/

The related GeoAPI tests are basically GIGS tests converted to JUnit
statements.


> There are still a number of tests that we can’t reliable pass in PROJ.
> I believe some, if not all, of them will be taken care
> of during the work you have now committed to.

The GIGS test files have some errors, in part due to insufficient number
of digits in some values, in part because of the use of some EPSG codes
that are now deprecated. Some CRS or operations names have also changed
since GIGS has been published. Changes required for getting the test to
pass are documented here:

http://www.geoapi.org/geoapi-conformance/apidocs/org/opengis/test/referencing/gigs/doc-files/changes.html

This is unfinished work we started 6 years ago; we did not had the
resources to continue that work during the last few years.

    Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Even Rouault-2
In reply to this post by Martin Desruisseaux-3

On mardi 15 mai 2018 22:21:35 CEST Martin Desruisseaux wrote:

> Hello Even

>

> Le 15/05/2018 à 19:42, Even Rouault a écrit :

> > Python bindings over what ?

>

> Over a standard API. Not any particular implementation.

> Likewise, the Python binding is between Java interfaces of GeoAPI and

> Python abstract classes of GeoAPI.

 

OK, I think I understand better what you meant. From my C/C++ perspective, when folks talk about a Python binding, this is a Python wrapper over some C/C++ code.

I guess that here you are thinking to

https://github.com/opengeospatial/geoapi/tree/master/geoapi-java-python , which seems to be Java code calling Python classes, right ?

 

> It is not between Apache SIS or GDAL

> or any particular implementation. The intent is to allow

> interoperability between any Python implementation of GeoAPI and any

> Java implementation of GeoAPI, in a way similar to JDBC-ODBC bridge.

 

For the sake of my curiousity, are there such (public) implementations of GeoAPI in Python ?

 

> > I guess the GeoAPI tests would be needed to be ported/adapted to

> > whatever solution is adopted for proj testing. I don't really feel

> > like using the proj JNI wrapper through Apache SIS would be an ideal

> > solution : too many components aggegated, and I'm not sure if JNI

> > wrapper will have all the new capabilities exposed.

>

> My proposal is independent of Apache SIS. GeoAPI is an OGC project, and

> as such is independent of any implementation - OGC is very strict about

> vendor neutrality. GeoAPI is an API derived from OGC/ISO standards. For

> example ISO 19111 defines an object named SC_GeographicCRS which

> contains an association to another object named CS_EllipsoidCS, which

> itself contains associations to axes, which have associations to units

> of measurements, /etc./ GeoAPI translates this abstract model into Java

> interfaces and Python abstract classes, not into implementation. In C,

> it could be structures with only pointer to functions but no code;

> providing the code would be up to Proj.

>

> Having a common API allows to run the same tests with different

> implementations, in the same way that ODBC/JDBC allow to use different

> databases with the same code. So it would allow Proj and Apache SIS to

> run the same tests - it does not establish any dependency of one to the

> other.

 

For the CRS modelling, in the technical proposal of gdalbarn, I mentionned I will probably take inspiration from GeoAPI to transpose that to a C++ class hierarchy. That said, I'm not completely sure at that point to exactly follow the Java class hiearachy as it is pretty "verbose" (lots of interfaces at first sight [1]), but that needs to be studied further.

 

>

> This is not theoretical - GeoAPI implementation backed by Proj.4 through

> the JNI already exists for 5~10 years at

> http://www.geoapi.org/geoapi-proj4/index.html and has been used for

> testing Proj.4. At least one Proj bug has been identified by this test

> suite.

 

Interesting. And besides testing, do you know if that is used in production as well ?

 

Even

 

[1] Or perhaps I was confused when also looking at the Apache SIS javadoc where there's the implementation of the interfaces, and the mix of interface classes + implementation classes give the classic headaches of dealing with Java API. OK I know I'm getting controversial here ;-)

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3

Le 15/05/2018 à 23:02, Even Rouault a écrit :

OK, I think I understand better what you meant. From my C/C++ perspective, when folks talk about a Python binding, this is a Python wrapper over some C/C++ code. I guess that here you are thinking to https://github.com/opengeospatial/geoapi/tree/master/geoapi-java-python , which seems to be Java code calling Python classes, right ?

Yes, and conversely (this is bi-directional).

For the sake of my curiousity, are there such (public) implementations of GeoAPI in Python ?

We started a wrapper around GDAL at https://github.com/opengeospatial/geoapi/tree/master/geoapi-gdal using the GDAL-Python bindings (yes, this result in wrappers around wrappers). Current version handles only metadata, but raster data should come later.

The reason for this approach is that GDAL API is unique to GDAL. Applications using GDAL can not easily switch to another library when they need to. The purpose of a standard API is to reduce vendor lock-in. More specifically, our goal here is to explore the possibility to allow Open DataCube (a Python project using GDAL) to also work with Apache SIS as an alternative to GDAL. This is desirable when one library has features than the other does not have. A similar discussion in cloud environment can be seen at http://openeo.org/openeo/news/2018/03/17/poc.html

Real use case: since I have not yet been able to compile GDAL with the "--with-java=yes" option on MacOS (maybe it is an issue on my side), I already use all above-cited wrappers for my tests as below (where each → is a set of wrappers; the first → is implementation-neutral and the last → is from GDAL project):

(GeoAPI in Java) → (GeoAPI in Python) → (GDAL in Python) → (GDAL in C)

Of course so many indirections is not ideal, but it unblocks my situation for now. It also illustrates the power of standard API for using a library when no direct wrappers are available.

Note that using a standard API does not prevent the use of GDAL-specific API when interoperability is not a goal.

For the CRS modelling, in the technical proposal of gdalbarn, I mentionned I will probably take inspiration from GeoAPI to transpose that to a C++ class hierarchy. That said, I'm not completely sure at that point to exactly follow the Java class hiearachy as it is pretty "verbose" (lots of interfaces at first sight [1]), but that needs to be studied further.

GeoAPI interfaces are derived from ISO 19111, which is available from OGC there:

http://portal.opengeospatial.org/files/?artifact_id=39049

I suggest to use ISO 19111 as the primary source. GeoAPI interfaces may be used for saving time (e.g. with copy-and-paste) or for filling holes in areas not covered by ISO 19111, but in case of doubt the above OGC/ISO document should be the authoritative reference. Note however that ISO 19111 is going to change - its revision process is nearly completed. For example GeodeticDatum will be renamed GeodeticReferenceFrame. The WKT 2 format (ISO 19162) is also going to be updated accordingly. Those changes have not yet been reflected in GeoAPI.

If you consider defining a C/C++ API derived from ISO 19111, directly or indirectly (through current GeoAPI interfaces), I would be very interested in including this C/C++ API in GeoAPI if you agree, and propose them as an OGC standard. That way, there would be no need to define JNI wrappers for Proj anymore. JNI wrappers for the C/C++ API would be sufficient to handle any implementation of that API, with Proj being one of them, and the test suite mentioned in my previous email could be run on Proj with no additional effort.

> This is not theoretical - GeoAPI implementation backed by Proj.4 through

> the JNI already exists for 5~10 years at

> http://www.geoapi.org/geoapi-proj4/index.html and has been used for

> testing Proj.4. At least one Proj bug has been identified by this test

> suite.

 

Interesting. And besides testing, do you know if that is used in production as well ?

I don't know. The above-cited wrappers have never been formally released, because GeoAPI has been dormant until last year (we restarted the group at OGC). A variant of those wrappers are released in Apache SIS 0.8, which allow users to use Proj or SIS own implementation transparently.

Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Even Rouault-2

Hi Martin,

 

> Real use case: since I have not yet been able to compile GDAL with the

> "--with-java=yes" option on MacOS (maybe it is an issue on my side)

 

I don't have a Mac, so I cannot directly help. Probably some path to where the JDK is located is not appropriate set in configure.ac

 

But actually this --with-java option is not used at all to build the Java bindigs, bu to allow a GDAL driver (MDB) to be able to use a Java library under the hood...

 

For the Java bindings, cd to swig/java, editing java.opt if needed and 'make'

 

>

> GeoAPI interfaces are derived from ISO 19111, which is available from

> OGC there:

>

> http://portal.opengeospatial.org/files/?artifact_id=39049

>

> I suggest to use ISO 19111 as the primary source. GeoAPI interfaces may

> be used for saving time (e.g. with copy-and-paste) or for filling holes

> in areas not covered by ISO 19111, but in case of doubt the above

> OGC/ISO document should be the authoritative reference. Note however

> that ISO 19111 is going to change - its revision process is nearly

> completed. For example GeodeticDatum will be renamed

> GeodeticReferenceFrame. The WKT 2 format (ISO 19162) is also going to be

> updated accordingly. Those changes have not yet been reflected in GeoAPI.

>

> If you consider defining a C/C++ API derived from ISO 19111, directly or

> indirectly (through current GeoAPI interfaces), I would be very

> interested in including this C/C++ API in GeoAPI if you agree, and

> propose them as an OGC standard. That way, there would be no need to

> define JNI wrappers for Proj anymore. JNI wrappers for the C/C++ API

> would be sufficient to handle any implementation of that API, with Proj

> being one of them, and the test suite mentioned in my previous email

> could be run on Proj with no additional effort.

 

Thanks for the pointers. Some interesting reading to digest. Yes, if the API can itself rely on a standard, that could be a good thing.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3

Le 16/05/2018 à 22:20, Even Rouault a écrit :

Thanks for the pointers. Some interesting reading to digest. Yes, if the API can itself rely on a standard, that could be a good thing.

In a C/C++ world, it could be:

  • State that standard API in C is not a goal (unless someone volunteers for defining a large set of pointer to functions).
  • Define standard C++ API as a set of classes with virtual or pure virtual functions.
  • Define subclasses of above C++ classes with PROJ implementation (or delegating to PROJ methods in C).

Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Mateusz Loskot
On 18 May 2018 at 10:25, Martin Desruisseaux
<[hidden email]> wrote:

> Le 16/05/2018 à 22:20, Even Rouault a écrit :
>
> Thanks for the pointers. Some interesting reading to digest. Yes, if the API
> can itself rely on a standard, that could be a good thing.
>
> In a C/C++ world, it could be:
>
> State that standard API in C is not a goal (unless someone volunteers for
> defining a large set of pointer to functions).
> Define standard C++ API as a set of classes with virtual or pure virtual
> functions.
> Define subclasses of above C++ classes with PROJ implementation (or
> delegating to PROJ methods in C).

I'd just say, the proposed geo C++ API defined according to OOP design.
That should cover the goal(s) pretty clear.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: [gdal-dev] Coordinate system improvements - GDAL, libgeotiff, PROJ barn raising

Martin Desruisseaux-3
Le 18/05/2018 à 10:32, Mateusz Loskot a écrit :

> I'd just say, the proposed geo C++ API defined according to OOP
> design. That should cover the goal(s) pretty clear.
>
Indeed, with the nuance that in a normal OOP project there is no need to
create abstract versions of all major classes.

    Martin


_______________________________________________
Proj mailing list
[hidden email]
http://lists.maptools.org/mailman/listinfo/proj