[QGIS-Developer] Dealing with datum shift grids in QGIS

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[QGIS-Developer] Dealing with datum shift grids in QGIS

Martin Dobias
Hi all

Recently I needed transformation between points from high-accuracy GPS
to British National Grid (EPSG:27700) which involves datum shift (WGS
84 to OSGB 1936) using OSTN15 grid.

The process to do that with QGIS is currently... well... really
clumsy. One needs to:
1. manually download grids from the web and extract .gsb file to Proj
data directory
2. manually add an entry to tbl_datum_transform table in srs.db in
QGIS installation

There are two main problems with that:
- both steps need adjustment of QGIS installation - not everyone has
admin privileges
- doing manual changes to srs.db is just too much to ask from ordinary user

I understand that it is hard to ship datum shift grids with QGIS due
to various issues (their size and amount, sometimes licensing) and
that's fine. But we should think about how to make it easier to use
them. There is a QGIS plugin [1] that provides a set of algs for
Processing that call ogr2ogr with the right grid files and download
them if they are not yet available - however there is no integration
with QGIS CRS database, so any other transforms inside QGIS can't make
use of that.

Note for those not familiar with srs.db: it is a sqlite3 db living in
QGIS source code which has been generated at some point from GDAL data
files. Over time it has gathered some manual ad-hoc updates of
parameters. Then upon QGIS build/installation we run "crssync" utility
to parse current GDAL data files again and update/insert CRS and datum
shift parameters (see QgsCoordinateReferenceSystem::syncDatabase()).

My suggestions for QGIS:
1. any manual additions to srs.db should be listed as SQL commands
because it is difficult to track changes to srs.db which is a binary
blob (sqlite db). During the build of QGIS we would create final
srs.db
2. rather than manually adding entries about datum shift grids to
tbl_datum_transform table (which is generated from GDAL's
datum_shift.csv [2]) and contains parameters of 3/7 parameter
transform between datums, there should be a new table just for grid
files, ideally with links where they can be downloaded. (fun fact:
names of grid files in tbl_datum_transform table are stored in a
numeric column but because this is sqlite it just works)
3. it should be possible to add custom entries about datum shift grids
by users (just like it is possible to add custom CRS) - that is, such
table should be also in user's profile (in qgis.db)

My suggestions for Proj:
1. it should be possible to specify multiple data directories with
PROJ_LIB env variable, so that it is possible to use grids also from
user's home directory (not just the read-only system directory)
2. there could be a database similar to datum_shift.csv [2] that would
list well known datum shift grid files, so that QGIS does not need to
maintain its database and just pull that information out of Proj -
other projects could benefit from that as well.

I am aware of the fact that there is a big refactoring in Proj going
on [3] but I am not sure if any of the above suggestions are also in
the plans...

I would be happy to help implementing some of the above, but I would
like to first hear opinion of others - maybe I got something wrong and
maybe there are better ways to move forward. I hope gurus like
Kristian or Even read this and can offer some of their wisdom too :-)

Cheers
Martin

[1] https://github.com/NaturalGIS/naturalgis_ntv2_transformations
[2] https://github.com/OSGeo/gdal/blob/master/gdal/data/datum_shift.csv
[3] https://gdalbarn.com/
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Even Rouault-2
Martin,

It seems you want to have a read at my yesterday-written RFC:
https://github.com/OSGeo/proj.4/pull/1149

Particularly the part concerning the database:
http://even.rouault.free.fr/proj_cpp_api/rfc-2.html#database

and the data/sql/ files mentionne at

http://even.rouault.free.fr/proj_cpp_api/rfc-2.html#impacted-files

>
> My suggestions for Proj:
> 1. it should be possible to specify multiple data directories with
> PROJ_LIB env variable, so that it is possible to use grids also from
> user's home directory (not just the read-only system directory)

That's an interesting idea. Could you file a ticket to
https://github.com/OSGeo/proj.4/issues about that ?


> 2. there could be a database similar to datum_shift.csv [2] that would
> list well known datum shift grid files, so that QGIS does not need to
> maintain its database and just pull that information out of Proj -
> other projects could benefit from that as well.

Sounds very much like:

https://github.com/rouault/proj.4/blob/iso19111/data/sql/grid_alternatives.sql


It would be good if we could utlimately get rid of QGIS srs.db and just use
the future PROJ proj.db that could be used by the whole PROJ/GDAL/QGIS stack


Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Giovanni Manghi
Hi Martin,

> I understand that it is hard to ship datum shift grids with QGIS due
> to various issues (their size and amount, sometimes licensing) and
> that's fine. But we should think about how to make it easier to use
> them. There is a QGIS plugin [1] that provides a set of algs for
> Processing that call ogr2ogr with the right grid files and download
> them if they are not yet available - however there is no integration
> with QGIS CRS database, so any other transforms inside QGIS can't make
> use of that.

I will personally port the plugin to QGIS 3 as soon as I can and
continue to make it available (I could even make a PR and propose it
as a tool that comes out of the box with Processing), this because
anyway has proven a valuable tool for less advanced users that are in
the situation to create copies of their data with a new CRS and with a
datum transformation involved.

Other than this anything that makes easier handling datum
transformations in QGIS is welcome.

Cheers!

-- G --
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Martin Dobias
In reply to this post by Even Rouault-2
Hi Even

On Thu, Oct 11, 2018 at 5:31 PM Even Rouault <[hidden email]> wrote:

>
> It seems you want to have a read at my yesterday-written RFC:
> https://github.com/OSGeo/proj.4/pull/1149
>
> Particularly the part concerning the database:
> http://even.rouault.free.fr/proj_cpp_api/rfc-2.html#database
>
> and the data/sql/ files mentionne at
>
> http://even.rouault.free.fr/proj_cpp_api/rfc-2.html#impacted-files

Just read through all that. Amazing work.

This makes me wonder when we could start using this in QGIS - I am
afraid we will have to wait for some time until it is all out in a
stable Proj release and adopted by major distributions, which may take
quite some time... (a year?)


> > My suggestions for Proj:
> > 1. it should be possible to specify multiple data directories with
> > PROJ_LIB env variable, so that it is possible to use grids also from
> > user's home directory (not just the read-only system directory)
>
> That's an interesting idea. Could you file a ticket to
> https://github.com/OSGeo/proj.4/issues about that ?

Done: https://github.com/OSGeo/proj.4/issues/1150


> > 2. there could be a database similar to datum_shift.csv [2] that would
> > list well known datum shift grid files, so that QGIS does not need to
> > maintain its database and just pull that information out of Proj -
> > other projects could benefit from that as well.
>
> Sounds very much like:
>
> https://github.com/rouault/proj.4/blob/iso19111/data/sql/grid_alternatives.sql

Pretty much. Looking at the grid support, I see that the OSTN15
transformation is listed there in the grid_transformation table, but
we would also need an entry in grid_alternatives table for it +
include it in a future release of proj-datumgrid-europe package (that
particular grid is BSD licensed, so it should be fine).

Regarding contents of proj.db table, will it be acceptable for apps
like QGIS to browse its contents or should it be considered as Proj
library's implementation detail and any queries to it should go
through Proj API? I am thinking it would be nice to automatically
determine grid packages for missing grid files and to download them in
QGIS if the users wants. Another question is what to do with grids
with license that does not allow redistribution - should we have
entries with URL in grid_transformation table for such grid files as
well? (maybe with a flag that it is not direct link to a package, just
a web page about that grid)

One final thought - how about customization of proj.db by ordinary
users? In QGIS users are allowed to define their custom CRS based on
Proj string - is something like that going to be supported? E.g. to
have system proj.db and user proj.db with customizations?


> It would be good if we could utlimately get rid of QGIS srs.db and just use
> the future PROJ proj.db that could be used by the whole PROJ/GDAL/QGIS stack

That would be absolutely wonderful and I am looking forward to that day :-)

Cheers
Martin
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Kristian Evers-2


On 11 Oct 2018, at 18:55, Martin Dobias <[hidden email]> wrote:

Regarding contents of proj.db table, will it be acceptable for apps
like QGIS to browse its contents or should it be considered as Proj
library's implementation detail and any queries to it should go
through Proj API? I am thinking it would be nice to automatically
determine grid packages for missing grid files and to download them in
QGIS if the users wants. Another question is what to do with grids
with license that does not allow redistribution - should we have
entries with URL in grid_transformation table for such grid files as
well? (maybe with a flag that it is not direct link to a package, just
a web page about that grid)

Some time ago Even and I briefly discussed the possibility of having
PROJ offer a direct link to grids that are needed for a transformation
but not available on the system. This should let apps like QGIS deal
with grids quite smoothly I think.

This would compliment or perhaps even replace the current
proj-datumgrid packages that we maintain at the moment. It would
still be based on a curated set of grids making sure that they are
free to distribute. For the ones that are not available for redistribution,
well… I guess the makers of those grids don’t really want you to use
them in the first place :-) In all seriousness though, there’s not much
we can do. They are quite often behind a login so direct links can’t
be offered. We do keep a non-exhaustive list of non-free grids here:


If a feature like this  is something that might be useful to users of
PROJ I we should look into this more closely.

/Kristian

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Even Rouault-2
In reply to this post by Martin Dobias
> This makes me wonder when we could start using this in QGIS - I am
> afraid we will have to wait for some time until it is all out in a
> stable Proj release and adopted by major distributions, which may take
> quite some time... (a year?)

Yes, likely. PROJ 6 is scheduled for February 2019. GDAL 2.4 for May

> Pretty much. Looking at the grid support, I see that the OSTN15
> transformation is listed there in the grid_transformation table, but
> we would also need an entry in grid_alternatives table for it +
> include it in a future release of proj-datumgrid-europe package (that
> particular grid is BSD licensed, so it should be fine).

Yes, https://github.com/OSGeo/proj-datumgrid/issues/21

>
> Regarding contents of proj.db table, will it be acceptable for apps
> like QGIS to browse its contents or should it be considered as Proj
> library's implementation detail and any queries to it should go
> through Proj API?

Well QGIS could do it, but there's the risk that the DB structure might change
at some point.

> I am thinking it would be nice to automatically
> determine grid packages for missing grid files and to download them in
> QGIS if the users wants.

I've anticipated that need.

In the PROJ API, there will be a function to list all potential
transformations from CRS A to CRS B, independently of the availability of
grids

PJ_OPERATION_RESULT PROJ_DLL *proj_obj_create_operations(
                            PJ_OBJ* source_crs,
                            PJ_OBJ* target_crs,
                            PJ_OPERATION_FACTORY_CONTEXT* operationContext);

and then you can iterate over the potential operations returned, and ask for
each operation, which grids are needed, and if they are available, and if not,
where you can find them:

int PROJ_DLL proj_coordoperation_get_grid_used_count(PJ_OBJ* coordoperation);

int PROJ_DLL proj_coordoperation_get_grid_used(PJ_OBJ* coordoperation,
                                               int index,
                                               const char** pShortName,
                                               const char** pFullName,
                                               const char** pPackageName,
                                               const char** pPackageURL,
                                               int* pAvailable);


> Another question is what to do with grids
> with license that does not allow redistribution - should we have
> entries with URL in grid_transformation table for such grid files as
> well? (maybe with a flag that it is not direct link to a package, just
> a web page about that grid)

I guess grid_alternatives could receive additional columns, which would be
used if package_name is NULL, perhaps something like:

open_license BOOLEAN
download_url TEXT
direct_download BOOLEAN

Putting that in my TODO list

>
> One final thought - how about customization of proj.db by ordinary
> users? In QGIS users are allowed to define their custom CRS based on
> Proj string - is something like that going to be supported?

The issue is that a PROJ string is a rather low expressive way of defining a
CRS, but yes that something I've in a corner of my mind, to allow defining
custom CRS entries by PROJ strings or WKT strings, without going to the full
relational structure.

> E.g. to
> have system proj.db and user proj.db with customizations?

That's another issue. An easy solution would be to copy the system proj.db in
a editable location and point PROJ_LIB to it

Having both at the same time is more tricky. For example if you add custom
transformations in the user proj.db between official CRS of the system
proj.db, you'd have an issue with broken foreign keys, so the user proj.db
shouldn't enforce foreign keys.
and then PROJ could potentially create a in-memory database, that would ATTACH
DATABASE the official and custom one, and for every table, create a temporary
view that UNION ALL the tables of both dbs. Then that would be transparent for
the rest of the code that does database queries.
I might consider doing this, but this is not a top priority for now.


Thanks for your valuable feedback on this !


Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Marco Bernasocchi
In reply to this post by Martin Dobias

Hi Martin, I guess I'm misunderstanding you, but can't you just set the .gsb files in the CRS datum transformations option panel?

see: https://imgur.com/a/M5V88zr

ciao

Marco

--
Marco Bernasocchi
QGIS.org Co-chair
[hidden email]
<a href="tel:+41794672470">+41 (0)79 467 24 70

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

berndv.
In reply to this post by Kristian Evers-2


Am 11.10.18 um 20:31 schrieb Kristian Evers:


On 11 Oct 2018, at 18:55, Martin Dobias <[hidden email]> wrote:
...

This would compliment or perhaps even replace the current
proj-datumgrid packages that we maintain at the moment. It would
still be based on a curated set of grids making sure that they are
free to distribute. For the ones that are not available for redistribution,
well… I guess the makers of those grids don’t really want you to use
them in the first place :-) In all seriousness though, there’s not much
we can do. They are quite often behind a login so direct links can’t
be offered. We do keep a non-exhaustive list of non-free grids here:


I saw the German grid in the list of non-free grids. I already contacted the responsible administration in 2015 and they told me that the http://crs.bkg.bund.de/crseu/crs/descrtrans/BeTA/BETA2007.gsb is definately free to be distributed with any kind of software.



If a feature like this  is something that might be useful to users of
PROJ I we should look into this more closely.

/Kristian

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Even Rouault-2
> > https://proj4.org/resource_files.html#non-free-grids
>
> I saw the German grid in the list of non-free grids. I already contacted
> the responsible administration in 2015 and they told me that the
> http://crs.bkg.bund.de/crseu/crs/descrtrans/BeTA/BETA2007.gsb is
> definately free to be distributed with any kind of software.

Indeed, BETA2007.gsb has been incorporated into the
https://download.osgeo.org/proj/proj-datumgrid-1.8.zip package a few months
ago.

It looks like the docs need updates.Thanks for pointing that. I've filed
https://github.com/OSGeo/proj.4/issues/1159 about that.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Dealing with datum shift grids in QGIS

Even Rouault-2
In reply to this post by Even Rouault-2
Hi,

some updates:

> I guess grid_alternatives could receive additional columns, which would be
> used if package_name is NULL, perhaps something like:
>
> open_license BOOLEAN
> download_url TEXT
> direct_download BOOLEAN
>

Done

>
> > One final thought - how about customization of proj.db by ordinary
> > users? In QGIS users are allowed to define their custom CRS based on
> > Proj string - is something like that going to be supported?
>
> The issue is that a PROJ string is a rather low expressive way of defining a
> CRS, but yes that something I've in a corner of my mind, to allow defining
> custom CRS entries by PROJ strings or WKT strings, without going to the
> full relational structure.

Done.

The geodetic_crs and projected_crs tables have now a 'text_definition' where users can put PROJ strings or WKT strings

>
> > E.g. to
> > have system proj.db and user proj.db with customizations?
>

Done (for the reading part by PROJ)

There's now a

int PROJ_DLL proj_context_set_database_path(PJ_CONTEXT *ctx,
                                            const char *dbPath,
                                            const char *const *auxDbPaths)

where:
- dbPath is the path to the 'system' proj.db (or NULL to let it find it at its default location)
- auxDbPaths is a list of auxiliary databases. They can potentially
refer to objects of the main database (if you don't need standalone integrity)

Demo:

1) Create an empty DB (custom.db) with the table and view structure of the system DB (proj.db)

echo "SELECT sql || ';' FROM sqlite_master WHERE type IN ('table', 'view');" | sqlite3 data/proj.db | sqlite3 custom.db

2) Fill it with a custom CRS definition expressed as old-school PROJ.4 string

echo "INSERT INTO projected_crs (auth_name, code, name, coordinate_system_auth_name , coordinate_system_code , geodetic_crs_auth_name, geodetic_crs_code, conversion_auth_name, conversion_code, area_of_use_auth_name, area_of_use_code, text_definition , deprecated) VALUES ( 'QGIS', 'MY-CRS-CODE', 'My custom CRS', NULL, NULL, NULL, NULL, NULL, NULL,  NULL, NULL, '+proj=utm +zone=32 +ellps=GRS80 +units=us-ft +towgs84=1,2,3', 0);" | sqlite3 custom.db

3) Query the DB:

$ src/projinfo --main-db-path data/proj.db --aux-db-path custom.db QGIS:MY-CRS-CODE -o PROJ,PROJ4,WKT2_2015

PROJ string:
+proj=pipeline +step +proj=unitconvert +xy_in=deg +xy_out=rad +step +proj=utm +zone=32 +ellps=GRS80 +step +proj=unitconvert +xy_in=m +z_in=m +xy_out=us-ft +z_out=us-ft

PROJ.4 string:
+proj=utm +zone=32 +ellps=GRS80 +towgs84=1,2,3,0,0,0,0 +units=us-ft

WKT2_2015 string:
PROJCRS["My custom CRS",
    BASEGEODCRS["unknown",
        DATUM["Unknown based on GRS80 ellipsoid",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]]],
    CONVERSION["UTM zone 32N",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",9,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["US survey foot",0.304800609601219]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["US survey foot",0.304800609601219]],
    ID["QGIS","MY-CRS-CODE"]]


Pedantic note: the new PROJ string format doesn't capture the towgs84, which is a legacy way of embedding a coordinate transformation. Only the PROJ.4 output does that for the traditionnalists. However internally PROJ has built a BoundCRS object to model the towgs84 transformation that was in the text_definition, so for PROJ.4 output it can recover it, and for coordinate operations, it can use it as well. Not sure I'm very clear here :-), but in short, things should just work.

4) Test a reprojection

$ src/projinfo --main-db-path data/proj.db --aux-db-path custom.db -s QGIS:MY-CRS-CODE -t EPSG:32633 -o PROJ

PROJ string:
+proj=pipeline +step +proj=unitconvert +xy_in=us-ft +z_in=us-ft +xy_out=m +z_out=m +step +inv +proj=utm +zone=32 +ellps=GRS80 +step +proj=cart +ellps=GRS80 +step +proj=helmert +x=1 +y=2 +z=3 +step +inv +proj=cart +ellps=WGS84 +step +proj=utm +zone=33 +ellps=WGS84


Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
QGIS-Developer mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer