[PROJ] Switch to proj-datumgrid-geotiff for PROJ 7 ?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[PROJ] Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
Hi,

This is one of the last major topic linked to RFC4: should we switch to
GeoTIFF grids as the prime format for the grids we deliver alongside the
software? While there is no strong technical requirement to do so, it could be
the right moment to do it, or we might prefer postponing that to PROJ 8.

Currently RFC4 implementation continues to refer to .gtx / .gsb files in the
grid_alternatives database table. When network access if allowed and needed,
the code patches the extension to access a .tif file on the CDN.

It would probably be cleaner to avoid this patching and having
grid_alternatives directly point to .tif files. This would also enable us to
put https://github.com/osgeo/proj-datumgrid in a pure maintainance state and
just make https://github.com/osgeo/proj-datumgrid-geotiff the new home for
grids (currently the later has to be resynchronized with the former)

Currently:
proj-datumgrid: total size: 703 MB as 5 .zip and 1.5 GB uncompressed
proj-datumgrid-geotiff: total size: 486 MB

One potential issue with switching to GeoTIFF files is for people having
pipeline strings using the current grid names in them. But similarly to what
is currently done for remote access, for local access, if we fail to find a
.gtx/.gsb file, we could just retry with the .tif extension.

If we switch to proj-datumgrid-geotiff, would we still split the content in
several archives (and if so, according to which logic...) ? Having a single
archive would be for sure the simplest solution. Some feedback from packagers
would be welcome.

A bit linked to the above, if we switch to proj-datumgrid-geotiff, in the
grid_alternatives table, instead of having each grid entry pointing to a
package (like "proj-datumgrid-europe", which itself points to a URL of a .zip
archive), we could just point to the exact file on the CDN rather than the
archive.

If we adopt proj-datumgrid-geotiff, I'd also willing to add a proj-datumgrid-
download utility that would download, in the user-writable directory (or
system directory), files from the CDN based on criteria such as bounding box,
producer name, country of the producer, etc...

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Greg Troxel-2
Even Rouault <[hidden email]> writes:

> This is one of the last major topic linked to RFC4: should we switch to
> GeoTIFF grids as the prime format for the grids we deliver alongside the
> software? While there is no strong technical requirement to do so, it could be
> the right moment to do it, or we might prefer postponing that to PROJ 8.
>
> Currently RFC4 implementation continues to refer to .gtx / .gsb files in the
> grid_alternatives database table. When network access if allowed and needed,
> the code patches the extension to access a .tif file on the CDN.

I think it's really important for the RFC4 implementation to end up
being an alternative mechanism to get the exact same bits that could
have been packaged.  As a test, this means that someone who has all the
grids in the entire set of released grid files, should have no negatives
and no different outcomes, compared to having used RFC4 instead.

I also think it's important that people not choosing to use RFC4 not be
second-class in their proj experience in any way.

So if the RFC4 world is tif (which is not objectionable; it seems
obvious we are talking a container format for the same bits -- but tell
me if I got that wrong), then it really seems simpler to just have that
as the format.

> It would probably be cleaner to avoid this patching and having
> grid_alternatives directly point to .tif files. This would also enable us to
> put https://github.com/osgeo/proj-datumgrid in a pure maintainance state and
> just make https://github.com/osgeo/proj-datumgrid-geotiff the new home for
> grids (currently the later has to be resynchronized with the former)

You say "maintenance", but would there be new releases of packages
derived from proj-datumgrid, for example for the benefit of proj5/6
users that have no upgraded?  Or do you mean "it will just sit there as
an archive"?

> Currently:
> proj-datumgrid: total size: 703 MB as 5 .zip and 1.5 GB uncompressed
> proj-datumgrid-geotiff: total size: 486 MB

Do you expect the sizes for the same data to be different?  It seems
obvious that every file in the old directory needs to be transformed to
tif and put in the new one -- but again I may be missing something.

> One potential issue with switching to GeoTIFF files is for people having
> pipeline strings using the current grid names in them. But similarly to what
> is currently done for remote access, for local access, if we fail to find a
> .gtx/.gsb file, we could just retry with the .tif extension.

Basically, you are proposing to consider a grid name to not have an
extension, and that seems sensible

> If we switch to proj-datumgrid-geotiff, would we still split the content in
> several archives (and if so, according to which logic...) ? Having a single
> archive would be for sure the simplest solution. Some feedback from packagers
> would be welcome.

A good question.  With the release of the grid archives associated with
6.3.0, the sizes have really grown.  I tend to think that not having
grids installed is a bug (given that proj will get different answers, as
opposed to throwing a missing grid file exception).  So, the proj6
package I am working on has all of the released grids.  But, as grids
get bigger (which seems expected over time), this becomes less
reasonable, and eventually I would think there would be separate
packages.

It would be good if all packaging systems that have multiple packges for
grids had the same split and more or less the same names.  So that means
that the proj project should define the split, as it does now.

So if anything, I would think the repo should be split up into more
archives.  The current regions seem sensible, and then there perhaps is
another axis of normal things vs. esoteric things.   Right now I can't
articulate that and I am not sure that makes sense.

So for now, I would advise not changing the archive split plan, until we
have a good basis for believing that some other plan is good.

> A bit linked to the above, if we switch to proj-datumgrid-geotiff, in the
> grid_alternatives table, instead of having each grid entry pointing to a
> package (like "proj-datumgrid-europe", which itself points to a URL of a .zip
> archive), we could just point to the exact file on the CDN rather than the
> archive.

A caution about "CDN".  There is nothing magic, just a way to make it
more efficient for people to download things.  I think we are really
just talking about a collection of files on a web server (that happens
to be a distributed system with spiffy caching).

It seems sensible for the database of what grid files are needed to have
a single namespace, and that one can get them either via download or by
unpacking them (bit for bit identical).

> If we adopt proj-datumgrid-geotiff, I'd also willing to add a proj-datumgrid-
> download utility that would download, in the user-writable directory (or
> system directory), files from the CDN based on criteria such as bounding box,
> producer name, country of the producer, etc...

As opposed to them being downloaded because someone asked for a
transform that chose them?  It seems really clear to me that these sorts
of asked-for operations are entirely necessary for the whole system to
make sense.  Surely there would be download by name.  It would be really
nice if one could ask "show me the transform pipelines that this request
would invoke" and also to get from that "this is the list of grids you
need and don't have for one or all" and then to be able to get them.

So what if someone has not enabled RFC4, and asks for a transform that
would use a grid if it were there.  Instead of downloading dynamically,
what happens?  I have always wanted that to throw an exception, unless
the user has disabled something, but I know i'm way on one side on the
"consistent outputs" side of things.




Overall, this makes me think that 7 is being rushed, or that the RFC4
stuff is being rushed into 7.  I don't really understand what's driving
a fixed release date for a major branch, and I don't understand why
having that fixed date instead of an approach of getting the RFC4 stuff
done and out for beta and we'll see how it goes.

But, there is virtually zero chance of me packaging 7.0, as opposed to
some later 7.x version.
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
> So if the RFC4 world is tif (which is not objectionable; it seems
> obvious we are talking a container format for the same bits -- but tell
> me if I got that wrong), then it really seems simpler to just have that
> as the format.

Yes the results are binary identical whether you use NTv2/GTX or equivalent
TIFF files.

> You say "maintenance", but would there be new releases of packages
> derived from proj-datumgrid, for example for the benefit of proj5/6
> users that have no upgraded?  Or do you mean "it will just sit there as
> an archive"?

That's an open question. As long as we maintain PROJ 6.x, we need at a minimum
to maintain proj-datumgrid. But the maintainance might be minimal. Just accept
fixes to currently delivered material, and new content would be for
proj-datumgrid-geotiff. And at some point, proj-datumgrid would be frozen.

> > Currently:
> > proj-datumgrid: total size: 703 MB as 5 .zip and 1.5 GB uncompressed
> > proj-datumgrid-geotiff: total size: 486 MB
>
> Do you expect the sizes for the same data to be different?  It seems
> obvious that every file in the old directory needs to be transformed to
> tif and put in the new one -- but again I may be missing something.

The content in both repositories is the same. proj-datumgrid-geotiff files are
smaller because they use the TIFF Predictor mechanism which increases the
efficiency of deflate compression over what .zip can do, hence 486 MB < 703
MB.

> So if anything, I would think the repo should be split up into more
> archives.  The current regions seem sensible, and then there perhaps is
> another axis of normal things vs. esoteric things.   Right now I can't
> articulate that and I am not sure that makes sense.
>
> So for now, I would advise not changing the archive split plan, until we
> have a good basis for believing that some other plan is good.

Actually, in the github repo, I've changed the organization to be based on the
producer, mostly to make area of responisibility clearer:
https://github.com/osgeo/proj-datumgrid-geotiff
But currently everything is bundled in a single archive.

> > If we adopt proj-datumgrid-geotiff, I'd also willing to add a
> > proj-datumgrid- download utility that would download, in the
> > user-writable directory (or system directory), files from the CDN based
> > on criteria such as bounding box, producer name, country of the producer,
> > etc...
>
> As opposed to them being downloaded because someone asked for a
> transform that chose them?

More as an alternative to someone not wanting to enable on-demand grid
download, but also not wanting to download the whole set of grids because he
knows that he'll work only in a given area.

> It seems really clear to me that these sorts
> of asked-for operations are entirely necessary for the whole system to
> make sense.  Surely there would be download by name.  It would be really
> nice if one could ask "show me the transform pipelines that this request
> would invoke" and also to get from that "this is the list of grids you
> need and don't have for one or all" and then to be able to get them.

projinfo and related API already report which grids are missing.

> So what if someone has not enabled RFC4, and asks for a transform that
> would use a grid if it were there.  Instead of downloading dynamically,
> what happens?  I have always wanted that to throw an exception, unless
> the user has disabled something, but I know i'm way on one side on the
> "consistent outputs" side of things.

RFC4 has not changed anything on that side. We cannot impose a particular grid
to be used, because sometimes the guess done by PROJ is not necessarily the
most appropriate (because of implementation limitations, or sometimes just
because it needs a human brain to decide).

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Greg Troxel-2
Even Rouault <[hidden email]> writes:

>> You say "maintenance", but would there be new releases of packages
>> derived from proj-datumgrid, for example for the benefit of proj5/6
>> users that have no upgraded?  Or do you mean "it will just sit there as
>> an archive"?
>
> That's an open question. As long as we maintain PROJ 6.x, we need at a minimum
> to maintain proj-datumgrid. But the maintainance might be minimal. Just accept
> fixes to currently delivered material, and new content would be for
> proj-datumgrid-geotiff. And at some point, proj-datumgrid would be frozen.

Sounds reasonable.

>> > Currently:
>> > proj-datumgrid: total size: 703 MB as 5 .zip and 1.5 GB uncompressed
>> > proj-datumgrid-geotiff: total size: 486 MB
>>
>> Do you expect the sizes for the same data to be different?  It seems
>> obvious that every file in the old directory needs to be transformed to
>> tif and put in the new one -- but again I may be missing something.
>
> The content in both repositories is the same. proj-datumgrid-geotiff files are
> smaller because they use the TIFF Predictor mechanism which increases the
> efficiency of deflate compression over what .zip can do, hence 486 MB < 703
> MB.

That'a a good reason to switch, especially as the TIFF compression
remains on people's disks, whereas unzipping on distribution does not.

>> So if anything, I would think the repo should be split up into more
>> archives.  The current regions seem sensible, and then there perhaps is
>> another axis of normal things vs. esoteric things.   Right now I can't
>> articulate that and I am not sure that makes sense.
>>
>> So for now, I would advise not changing the archive split plan, until we
>> have a good basis for believing that some other plan is good.
>
> Actually, in the github repo, I've changed the organization to be based on the
> producer, mostly to make area of responisibility clearer:
> https://github.com/osgeo/proj-datumgrid-geotiff
> But currently everything is bundled in a single archive.

I view single archive as a breaking change, whereas a mere change of
format not, from the packaging viewpoint.  As I said, pkgsrc is
currently just grabbing them all, but this is becoming less reasonable.
At 500 MB, though, it doesn't seem terrible.  4GB would be something
else.


>> It seems really clear to me that these sorts
>> of asked-for operations are entirely necessary for the whole system to
>> make sense.  Surely there would be download by name.  It would be really
>> nice if one could ask "show me the transform pipelines that this request
>> would invoke" and also to get from that "this is the list of grids you
>> need and don't have for one or all" and then to be able to get them.
>
> projinfo and related API already report which grids are missing.

Great, and it would be nice to have a programmatic way to fetch them
from having run projinfo.

>> So what if someone has not enabled RFC4, and asks for a transform that
>> would use a grid if it were there.  Instead of downloading dynamically,
>> what happens?  I have always wanted that to throw an exception, unless
>> the user has disabled something, but I know i'm way on one side on the
>> "consistent outputs" side of things.
>
> RFC4 has not changed anything on that side. We cannot impose a particular grid
> to be used, because sometimes the guess done by PROJ is not necessarily the
> most appropriate (because of implementation limitations, or sometimes just
> because it needs a human brain to decide).

I am not really following this.  I get it that sometimes humans need to
choose which approach is best.

What I meant is that in a world where all the grid files are on the
disk, one can say "show me all the plausible transforms, sorted by some
error metric".  If the grid file isn't present, I'd still like that
process to be able to operate the same way, but essentialy throwing an
execption if a grid file that's needed isn't there -- as opposed to
pruning pipelines that make sense but which cannot currently be done
locally.  I realize this is a discusion about the pre-RFC4 world, but to
me this is part of avoiding those not opting into RFC4 becoming
second-class.
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
> Great, and it would be nice to have a programmatic way to fetch them
> from having run projinfo.

To be pondered. In a Unix style philosophy where each utility has its purpose,
I wouldn't have imagined projinfo to proceed to download itself.

> What I meant is that in a world where all the grid files are on the
> disk, one can say "show me all the plausible transforms, sorted by some
> error metric".  If the grid file isn't present, I'd still like that
> process to be able to operate the same way, but essentialy throwing an
> execption if a grid file that's needed isn't there -- as opposed to
> pruning pipelines that make sense but which cannot currently be done
> locally.

What you describe is really how PROJ 6 works:

Extract from output of:
projinfo -s NAD27 -t NAD83 --spatial-test intersects -o PROJ

"""
[snip]
-------------------------------------
Operation No. 4:

DERIVED_FROM(EPSG):1313, NAD27 to NAD83 (4), 1.5 m, Canada - NAD27, at least
one grid missing

PROJ string:
+proj=pipeline +step +proj=axisswap +order=2,1 +step +proj=unitconvert
+xy_in=deg +xy_out=rad +step +proj=hgridshift +grids=ntv2_0.gsb +step
+proj=unitconvert +xy_in=rad +xy_out=deg +step +proj=axisswap +order=2,1

Grid ntv2_0.gsb needed but not found on the system. Can be obtained from the
proj-datumgrid-north-america package at https://download.osgeo.org/proj/proj-datumgrid-north-america-1.3.zip
"""

> I realize this is a discusion about the pre-RFC4 world, but to
> me this is part of avoiding those not opting into RFC4 becoming
> second-class.

RFC4 is not about making exiting usages second-class. It is about making new
usages easier/possible.
Requiring grids of "best" transformation to be available when using the black
box/high level API proj_create_crs_to_crs() (like in cs2cs) is really another
topic to the one I'd want to address in this thread :-) and a rather
controversial one. PROJ offers the API for making people aware of what is
missing and could be used, as used by QGIS 3.10

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Nyall Dawson
In reply to this post by Even Rouault-2
On Tue, 14 Jan 2020 at 01:40, Even Rouault <[hidden email]> wrote:

>
> Hi,
>
> This is one of the last major topic linked to RFC4: should we switch to
> GeoTIFF grids as the prime format for the grids we deliver alongside the
> software? While there is no strong technical requirement to do so, it could be
> the right moment to do it, or we might prefer postponing that to PROJ 8.
>
> Currently RFC4 implementation continues to refer to .gtx / .gsb files in the
> grid_alternatives database table. When network access if allowed and needed,
> the code patches the extension to access a .tif file on the CDN.
>
> It would probably be cleaner to avoid this patching and having
> grid_alternatives directly point to .tif files. This would also enable us to
> put https://github.com/osgeo/proj-datumgrid in a pure maintainance state and
> just make https://github.com/osgeo/proj-datumgrid-geotiff the new home for
> grids (currently the later has to be resynchronized with the former)

Possibly QGIS is a bit of an outlier in terms of Proj clients, but
this change **would** impact us. Not in a major way, but in the order
of 2-3 hours work to rectify (and even MORE proj version #ifdefs in
the code! Woohoo!). If the intention is to make this an api break
(given that it's v7), then that's fine, but if you're intending for
this change to be transparent to clients just be aware that it isn't!

> Currently:
> proj-datumgrid: total size: 703 MB as 5 .zip and 1.5 GB uncompressed
> proj-datumgrid-geotiff: total size: 486 MB
>
> One potential issue with switching to GeoTIFF files is for people having
> pipeline strings using the current grid names in them. But similarly to what
> is currently done for remote access, for local access, if we fail to find a
> .gtx/.gsb file, we could just retry with the .tif extension.

This would be a necessity for QGIS, otherwise existing projects will
break and reproject differently upon loading in a build based on the
newer PROJ version. (I suspect R would also see this as a necessity to
avoid changing end user's existing results)

Nyall
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Nyall Dawson
In reply to this post by Even Rouault-2
On Tue, 14 Jan 2020 at 03:48, Even Rouault <[hidden email]> wrote:
>
>
> RFC4 is not about making exiting usages second-class. It is about making new
> usages easier/possible.
> Requiring grids of "best" transformation to be available when using the black
> box/high level API proj_create_crs_to_crs() (like in cs2cs) is really another
> topic to the one I'd want to address in this thread :-) and a rather
> controversial one. PROJ offers the API for making people aware of what is
> missing and could be used, as used by QGIS 3.10

Yes -- we use this heavily, and I can attest that it all works VERY
nicely! See https://github.com/qgis/QGIS/pull/30559#issuecomment-509039902
for a screenshot.
https://www.icsm.gov.au/sites/default/files/North%20Road%20Handling%20GDA2020%20within%20Geospatial%20Software%20Development.pdf
goes into more depth about how to implement this choice from a client
application.

Nyall
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Kristian Evers-2
In reply to this post by Greg Troxel-2
Greg,

> Overall, this makes me think that 7 is being rushed, or that the RFC4
> stuff is being rushed into 7.

RFC4 is not being rushed. Even and Howard has been working on this
for a long time. The majority of the new code has been ready for more
than a month already. What you are seeing now is the final sprint to
get it polished for release.

> I don't really understand what's driving
> a fixed release date for a major branch, and I don't understand why
> having that fixed date instead of an approach of getting the RFC4 stuff
> done and out for beta and we'll see how it goes.

As you've noticed, PROJ 7 introduces a rather large API break. So did
PROJ 6. Having fixed release dates for this stuff is in my opinion crucial
information for everyone who depends on PROJ. We've announced this
break two years in advance and made a hell of a lot of noise about it so
that it wouldn't come as a surprise for you and everyone else. Overall
I think we have succeeded with that, although not everyone has caught
up yet awareness has been raised and most projects have either already
migrated or have plans to do so soon. I don't think we could have achieved
that with a loosely defined release schedule.

Another benefit of a fixed release schedule is that those of us who
regularly work on PROJ can plan around activities related to releases.
At least for me it is necessary to block a few days in the calendar at release
time. Otherwise the releases would not happen!

Also related to release schedules, recently we had a bit of a mismatch
regarding PROJ, GDAL and QGIS releases that caused some unfortunate
bugs in the 3.4-branch of QGIS. This could probably have been avoided
if we had coordinate releases better between the three projects. That is
something I would like to take into consideration in the future. We need
a discussion between OSGeo projects on the topic. I'll initiate that
sometime soon.

/Kristian
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Markus Neteler
On Tue, Jan 14, 2020 at 11:52 AM Kristian Evers <[hidden email]> wrote:
...
> Also related to release schedules, recently we had a bit of a mismatch
> regarding PROJ, GDAL and QGIS releases that caused some unfortunate
> bugs in the 3.4-branch of QGIS. This could probably have been avoided
> if we had coordinate releases better between the three projects. That is
> something I would like to take into consideration in the future. We need
> a discussion between OSGeo projects on the topic. I'll initiate that
> sometime soon.

Please keep in mind that more projects depend on PROJ, not only QGIS.
So yes, let's move this to OSGeo level, not only single projects.

Best,
Markus
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Kristian Evers-2
Markus,

Yes, don't worry. I only brought up GDAL and QGIS since they were
affected in this particular issue. The intention is to coordinate across
as many OSGeo projects as possible. I'll bring the topic up on the
"discuss" list. I can also ping individual project lists but I fear that will
results in too many diverging email threads.

/Kristian

-----Original Message-----
From: Markus Neteler <[hidden email]>
Sent: 14. januar 2020 11:56
To: Kristian Evers <[hidden email]>
Cc: Greg Troxel <[hidden email]>; Even Rouault <[hidden email]>; [hidden email]
Subject: Re: [PROJ] Switch to proj-datumgrid-geotiff for PROJ 7 ?

On Tue, Jan 14, 2020 at 11:52 AM Kristian Evers <[hidden email]> wrote:
...
> Also related to release schedules, recently we had a bit of a mismatch
> regarding PROJ, GDAL and QGIS releases that caused some unfortunate
> bugs in the 3.4-branch of QGIS. This could probably have been avoided
> if we had coordinate releases better between the three projects. That is
> something I would like to take into consideration in the future. We need
> a discussion between OSGeo projects on the topic. I'll initiate that
> sometime soon.

Please keep in mind that more projects depend on PROJ, not only QGIS.
So yes, let's move this to OSGeo level, not only single projects.

Best,
Markus
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Greg Troxel-2
In reply to this post by Kristian Evers-2
Kristian Evers <[hidden email]> writes:

> As you've noticed, PROJ 7 introduces a rather large API break. So did
> PROJ 6. Having fixed release dates for this stuff is in my opinion crucial
> information for everyone who depends on PROJ. We've announced this
> break two years in advance and made a hell of a lot of noise about it so
> that it wouldn't come as a surprise for you and everyone else. Overall
> I think we have succeeded with that, although not everyone has caught
> up yet awareness has been raised and most projects have either already
> migrated or have plans to do so soon. I don't think we could have achieved
> that with a loosely defined release schedule.

My comment about rush was not meant to be about the API break.  It's
about significant things intended for the release not yet being on
master -- but my notions regarding that are almost certainly an outlier
and are not particularly important, and this isn't a packaging issue.

I don't keep track of the future too much, as in packaging I'm mostly
concerned about releases that have happened, what has to be done
differently, and whether packaged versions of actual releases of things
that depend on PROJ can work with the new version.  The real issue is
that there are a lot of things that depend on PROJ and a number of those
are not really maintained.

So we can continue to track in the wiki page, and see where we are in
terms of PROJ-using projects having formal releases that work with PROJ
7.  Ideally they would all have had releases that will build with 7
before the release of PROJ 7, given the advance warnings, but
  https://github.com/OSGeo/PROJ/wiki/proj.h-adoption-status
says that only 10 projects are ok with PROJ 7.

I certainly understand that none of the PROJ team can deal with the
issue of other projects not having adequate maintenance.

_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
In reply to this post by Nyall Dawson
> > It would probably be cleaner to avoid this patching and having
> > grid_alternatives directly point to .tif files. This would also enable us
> > to put https://github.com/osgeo/proj-datumgrid in a pure maintainance
> > state and just make https://github.com/osgeo/proj-datumgrid-geotiff the
> > new home for grids (currently the later has to be resynchronized with the
> > former)
> Possibly QGIS is a bit of an outlier in terms of Proj clients, but
> this change **would** impact us. Not in a major way, but in the order
> of 2-3 hours work to rectify (and even MORE proj version #ifdefs in
> the code! Woohoo!). If the intention is to make this an api break
> (given that it's v7), then that's fine, but if you're intending for
> this change to be transparent to clients just be aware that it isn't!

Maybe I've missed something or wasn't clear, but I don't see this is as an API
break (as in C API). I can imagine some effect on test suites that would test
that the PROJ pipeline from A to B would return a foo.gsb filename, and now
that would be foo.tif, but not on the production code itself.
The impact as I see it would be more on the packaging side (grab the proj-
datumgrid-geotiff.zip )

>> One potential issue with switching to GeoTIFF files is for people having
>> pipeline strings using the current grid names in them. But similarly to
what
>> is currently done for remote access, for local access, if we fail to find a
>> .gtx/.gsb file, we could just retry with the .tif extension.

> This would be a necessity for QGIS, otherwise existing projects will
> break and reproject differently upon loading in a build based on the
> newer PROJ version. (I suspect R would also see this as a necessity to
> avoid changing end user's existing results)

Yes, that would be definitely something to do to make the transition smoother.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Nyall Dawson
On Wed, 15 Jan 2020 at 05:28, Even Rouault <[hidden email]> wrote:

>
> > > It would probably be cleaner to avoid this patching and having
> > > grid_alternatives directly point to .tif files. This would also enable us
> > > to put https://github.com/osgeo/proj-datumgrid in a pure maintainance
> > > state and just make https://github.com/osgeo/proj-datumgrid-geotiff the
> > > new home for grids (currently the later has to be resynchronized with the
> > > former)
> > Possibly QGIS is a bit of an outlier in terms of Proj clients, but
> > this change **would** impact us. Not in a major way, but in the order
> > of 2-3 hours work to rectify (and even MORE proj version #ifdefs in
> > the code! Woohoo!). If the intention is to make this an api break
> > (given that it's v7), then that's fine, but if you're intending for
> > this change to be transparent to clients just be aware that it isn't!
>
> Maybe I've missed something or wasn't clear, but I don't see this is as an API
> break (as in C API). I can imagine some effect on test suites that would test
> that the PROJ pipeline from A to B would return a foo.gsb filename, and now
> that would be foo.tif, but not on the production code itself.
> The impact as I see it would be more on the packaging side (grab the proj-
> datumgrid-geotiff.zip )

It's not a c api break - but it still does change things for clients
which were previously coded to use downloads from
https://github.com/osgeo/proj-datumgrid and which have code which
handles .gsb files manually (such as installing them after downloading
the grid referenced by proj)

Nyall


>
> >> One potential issue with switching to GeoTIFF files is for people having
> >> pipeline strings using the current grid names in them. But similarly to
> what
> >> is currently done for remote access, for local access, if we fail to find a
> >> .gtx/.gsb file, we could just retry with the .tif extension.
>
> > This would be a necessity for QGIS, otherwise existing projects will
> > break and reproject differently upon loading in a build based on the
> > newer PROJ version. (I suspect R would also see this as a necessity to
> > avoid changing end user's existing results)
>
> Yes, that would be definitely something to do to make the transition smoother.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
In reply to this post by Even Rouault-2
Hi,

I've tried to articulate my latest thoughts about how to switch to
proj-datumgrid-geotiff & organize it in
https://docs.google.com/document/d/1-QK5NkQ9ADJrF66_6VoUcQoRoYoIiFY8CRsg1Y_MMzc/edit?usp=sharing

Feel free to edit/comment directly in it.

We should try to reach to a quick conclusion if we want to have this ready for PROJ 7, as there
will be some bits of development to do. But I think it is the good time to do this.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Nyall Dawson
On Fri, 17 Jan 2020 at 06:19, Even Rouault <[hidden email]> wrote:

>
> Hi,
>
> I've tried to articulate my latest thoughts about how to switch to
> proj-datumgrid-geotiff & organize it in
> https://docs.google.com/document/d/1-QK5NkQ9ADJrF66_6VoUcQoRoYoIiFY8CRsg1Y_MMzc/edit?usp=sharing
>
> Feel free to edit/comment directly in it.
>
> We should try to reach to a quick conclusion if we want to have this ready for PROJ 7, as there
> will be some bits of development to do. But I think it is the good time to do this.

One thing that concerns me:

> due to the potential different subdivision in packages, the database (grid_alternatives table) should no longer refer to the URL of a package for the location where to find the grid, but to the grid itself, and a good candidate would be to point to the URL of the grid in the CDN

Could we possibly keep the api returning the grid urls from the
existing repos, mark that api as deprecated, and then add new api to
get the new geotiff download urls? Otherwise this is a functional
break for clients, and I'd rather it became a hard-compilation failure
then a silent change in behaviour.

Nyall




>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> PROJ mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/proj
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
Nyall,

> Could we possibly keep the api returning the grid urls from the
> existing repos, mark that api as deprecated, and then add new api to
> get the new geotiff download urls?

That would not be only at the API level. We would also need to keep for each
grid in the database a location for the old package, and for the new API. That
doesn't sound very good to me that we set in-stone past behaviour.

> Otherwise this is a functional
> break for clients, and I'd rather it became a hard-compilation failure
> then a silent change in behaviour.

Technically, this is not an API break. The PROJ 6 API already has a provision
for this case.

If you look at

int PROJ_DLL proj_coordoperation_get_grid_used(PJ_CONTEXT *ctx,
                                               const PJ *coordoperation,
                                               int index,
                                               const char **out_short_name,
                                               const char **out_full_name,
                                               const char **out_package_name,
                                               const char **out_url,
                                               int *out_direct_download,
                                               int *out_open_license,
                                               int *out_available);

then when *out_package_name might actually be an empty string, and *out_url is
then mean to be the URL of the grid to download, and not the URL of a package.

The PROJ6 database has actually already 2 entries following this for the
Netherlands / RDNAP grids, such as

INSERT INTO grid_alternatives(original_grid_name,
                              proj_grid_name,
                              proj_grid_format,
                              proj_method,
                              inverse_direction,
                              package_name,
                              url, direct_download, open_license, directory)
                      VALUES ('rdtrans2008.gsb',
                              'rdtrans2008.gsb',
                              'NTv2',
                              'hgridshift',
                              0,
                              NULL, -- package name
                              'https://salsa.debian.org/debian-gis-team/proj-rdnap/raw/upstream/2008/rdtrans2008.gsb',
                              1, -- direct download
                              0, -- non-freely licensed. See https://
salsa.debian.org/debian-gis-team/proj-rdnap/raw/master/debian/copyright
                              NULL);

The output of
"""
#include <stdio.h>

#include "proj.h"

int main(void) {

    PJ_CONTEXT *ctx = proj_context_create();
    auto op = proj_create_from_database(ctx, "EPSG", "7000",
                                            PJ_CATEGORY_COORDINATE_OPERATION,
true,
                                            nullptr);
    const char *shortName = nullptr;
    const char *fullName = nullptr;
    const char *packageName = nullptr;
    const char *url = nullptr;
    int directDownload = 0;
    int openLicense = 0;
    int available = 0;
    proj_coordoperation_get_grid_used(
                  ctx, op, 0, &shortName, &fullName, &packageName, &url,
                  &directDownload, &openLicense, &available);

    printf("packageName = '%s'\n", packageName);
    printf("url = '%s'\n", url);
    printf("directDownload = %d\n", directDownload);

    return 0;
}
"""

with PROJ 6.3 (and earlier) is

packageName = ''
url = 'https://salsa.debian.org/debian-gis-team/proj-rdnap/raw/upstream/2008/
rdtrans2008.gsb'
directDownload = 1


So, although admitedly the exception currently, the behaviour I propose in the
document and that would become the general rule, can already happen.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj
Reply | Threaded
Open this post in threaded view
|

Re: Switch to proj-datumgrid-geotiff for PROJ 7 ?

Even Rouault-2
> So, although admitedly the exception currently, the behaviour I propose in
> the document and that would become the general rule, can already happen.

Hum thinking however that with the ${agency_name}/${grid_name} structure I
proposed, then having the URL only wouldn't be enough to know where to put the
file exactly. You should also look at the shortName that might be "fr_ign/
ntf_r93.tif" to potentially create the appropriate subdirectory.

Darn. A bit more complicated than I'd like it to be. The flat organization
might be the less risky after all.

So, we could just have the ${agency_name}.txt files listing the grids related
to an agency to help packagers create smaller packagers.

Or an idea I had but didn't expose yet, prefix grid name with the agency name.

fr_ign_ntf_r93.tif
nz_linz_nzgd2kgrid0005.tif
etc...


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
PROJ mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/proj