[gdal-dev] Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Sean Gillies-3
Hi,

I have a couple questions about discoverability of the new proj.db file and how I should prepare my Rasterio project for the changes. Background: I am publishing rasterio wheels on the Python package index that include GDAL and PROJ shared libraries and the data files so that users can `pip install rasterio` and get something with batteries included. Rasterio has an Env class that is to be used as a context manager and it ensures that PROJ_LIB (the env variable) and GDAL_DATA (the config option, not env variable) are set to the proper paths, paths within the installed wheel adjacent to rasterio's Python modules.

It seems like I'll be including proj.db in future wheels, which shouldn't be a problem.

When proj.db replaces the .csv files in GDAL and PROJ, will the GDAL_DATA configuration option become obsolete? If so, will there be different configuration option to point at the proj.db file? I'd prefer not to use an environment variable only, as far as I'm concerned a Python module should only patch the environment as a very last recourse.

I've just thought of a question about data file caching that I want to tack on. I don't think it's widely understood by users, but GDAL caches data read from the .csv files. For example, you can set GDAL_DATA to a custom location (like rasterio does), call a function that requires .csv data, then return and call that function again with GDAL_DATA unset and get the right answer. This is a good thing, because otherwise we'd be reading from disk every time. My question is whether or not this will change when we move over to the new PROJ. Will data be cached or will we need the path to proj.db to be configured all the time?

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Even Rouault-2
On mardi 8 janvier 2019 15:16:47 CET Sean Gillies wrote:

> Hi,
>
> I have a couple questions about discoverability of the new proj.db file and
> how I should prepare my Rasterio project for the changes. Background: I am
> publishing rasterio wheels on the Python package index that include GDAL
> and PROJ shared libraries and the data files so that users can `pip install
> rasterio` and get something with batteries included. Rasterio has an Env
> class that is to be used as a context manager and it ensures that PROJ_LIB
> (the env variable) and GDAL_DATA (the config option, not env variable) are
> set to the proper paths, paths within the installed wheel adjacent to
> rasterio's Python modules.
>
> It seems like I'll be including proj.db in future wheels, which shouldn't
> be a problem.

proj.db will be searched in PROJ_LIB if defined, or otherwise in the
${prefix}/share/proj directory if you ./configure --with-prefix=${prefix}
Is the issue that binaries of wheels aren't installed in fix locations ?

PROJ 6 new API has a proj_context_set_database_path() function but I've not
used/exposed through GDAL for now. It is something per-context (and thus
potentially per thread since in my new GDAL branch PROJ contexts are per-
thread variables, which are not exposed in GDAL API), and only for proj.db
itself, not other PROJ resources (such as grid files) normally accessible
through PROJ_LIB / ${prefix}/share/proj

There's also a related ticket in PROJ tracker:
https://github.com/OSGeo/proj.4/issues/1150
We'd probably need to expose pj_set_searchpath() in the public PROJ API.

>
> When proj.db replaces the .csv files in GDAL and PROJ, will the GDAL_DATA
> configuration option become obsolete?

GDAL_DATA will still be used by some drivers to retrieve resource files, but
no longer for CRS related operations.

>
> I've just thought of a question about data file caching that I want to tack
> on. I don't think it's widely understood by users, but GDAL caches data
> read from the .csv files. For example, you can set GDAL_DATA to a custom
> location (like rasterio does), call a function that requires .csv data,
> then return and call that function again with GDAL_DATA unset and get the
> right answer.

That's an implementation detail.

> This is a good thing, because otherwise we'd be reading from
> disk every time. My question is whether or not this will change when we
> move over to the new PROJ. Will data be cached or will we need the path to
> proj.db to be configured all the time?

There will be some caching, but on a least-recently-used way, not full
caching, and not on everything, so access to proj.db might occur at any time.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Even Rouault-2
> PROJ 6 new API has a proj_context_set_database_path() function but I've not
> used/exposed through GDAL for now. It is something per-context (and thus
> potentially per thread since in my new GDAL branch PROJ contexts are per-
> thread variables, which are not exposed in GDAL API), and only for proj.db
> itself, not other PROJ resources (such as grid files) normally accessible
> through PROJ_LIB / ${prefix}/share/proj
>
> There's also a related ticket in PROJ tracker:
> https://github.com/OSGeo/proj.4/issues/1150
> We'd probably need to expose pj_set_searchpath() in the public PROJ API.

I've just submitted a PR https://github.com/OSGeo/proj.4/pull/1218 to add a

proj_context_set_search_paths(
 PJ_CONTEXT *ctx, int count_paths,const char* const* paths)

If called on the default PROJ context (NULL) before GDAL creates it own
contexts, this should allow you to specify your own search path.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Sean Gillies-3

On Wed, Jan 9, 2019 at 11:12 AM Even Rouault <[hidden email]> wrote:
> PROJ 6 new API has a proj_context_set_database_path() function but I've not
> used/exposed through GDAL for now. It is something per-context (and thus
> potentially per thread since in my new GDAL branch PROJ contexts are per-
> thread variables, which are not exposed in GDAL API), and only for proj.db
> itself, not other PROJ resources (such as grid files) normally accessible
> through PROJ_LIB / ${prefix}/share/proj
>
> There's also a related ticket in PROJ tracker:
> https://github.com/OSGeo/proj.4/issues/1150
> We'd probably need to expose pj_set_searchpath() in the public PROJ API.

I've just submitted a PR https://github.com/OSGeo/proj.4/pull/1218 to add a

proj_context_set_search_paths(
 PJ_CONTEXT *ctx, int count_paths,const char* const* paths)

If called on the default PROJ context (NULL) before GDAL creates it own
contexts, this should allow you to specify your own search path.

Or I could call pj_set_searchpath?

I'm a little concerned about the need to race GDAL to the default context. I'm not sure how I'll be able to do this in practice. Even if Rasterio were to call pj_set_searchpath immediately on import, which I'd like to avoid for a number of reasons, any user who imported a different module that loads GDAL (osgeo.gdal, for example) before importing Rasterio would lose that insurance and may experience some trouble that is hard to diagnose. Could a GDAL API function that allowed modification of the search path in GDAL's own context help? 

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Even Rouault-2
> > I've just submitted a PR https://github.com/OSGeo/proj.4/pull/1218 to add
> > a
> >
> > proj_context_set_search_paths(
> >
> >  PJ_CONTEXT *ctx, int count_paths,const char* const* paths)
> >
> > If called on the default PROJ context (NULL) before GDAL creates it own
> > contexts, this should allow you to specify your own search path.
>
> Or I could call pj_set_searchpath?

It is part of the proj_api.h that will be deprecated in PROJ 6 and eventually
removed in PROJ 7. Its effect in PROJ 6 will be identifcal to using
proj_context_set_search_paths() with the default PROJ context.

>
> I'm a little concerned about the need to race GDAL to the default context.
> I'm not sure how I'll be able to do this in practice. Even if Rasterio were
> to call pj_set_searchpath immediately on import, which I'd like to avoid
> for a number of reasons, any user who imported a different module that
> loads GDAL (osgeo.gdal, for example) before importing Rasterio would lose
> that insurance and may experience some trouble that is hard to diagnose.

> Could a GDAL API function that allowed modification of the search path in
> GDAL's own context help?

ok, I'll probably add a

void CPL_DLL OSRSetPROJSearchPaths( const char* const* papszPaths );

that will internally call proj_context_set_search_paths() on PROJ context
managed by GDAL (affecting the already created TLS ones, and future ones)

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Discoverability of the new proj.db and GDAL_DATA/PROJ_LIB?

Sean Gillies-3
On Wed, Jan 9, 2019 at 12:40 PM Even Rouault <[hidden email]> wrote:

... 
> Could a GDAL API function that allowed modification of the search path in
> GDAL's own context help?

ok, I'll probably add a

void CPL_DLL OSRSetPROJSearchPaths( const char* const* papszPaths );

that will internally call proj_context_set_search_paths() on PROJ context
managed by GDAL (affecting the already created TLS ones, and future ones)


Sounds good! I just added a ticket about this: https://github.com/OSGeo/gdal/issues/1191.

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev