[gdal-dev] limiting drivers that get registered

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] limiting drivers that get registered

Ben Elliston
Is it possible to limit the number of drivers compiled into GDAL (I am
guessing not)?  Alternatively, would a patch be welcome to read a
confguration file to limit which drivers are registered at initialisation?

Cheers,
Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Even Rouault-2

Ben,

 

> Is it possible to limit the number of drivers compiled into GDAL (I am

> guessing not)?

 

You have some knobs with the various ./configure or nmake.opt switch, but yes generally drivers that don't require external dependencies will always be compiled in.

Editing manually the makefiles can help creating a smaller GDAL build but that can be a non-obvious task because of inter-driver dependencies.

 

There's also a wiki page about that

https://trac.osgeo.org/gdal/wiki/BuildingOnUnixWithMinimizedDrivers

(I can see that I created a patch for a minimal build 10 years ago. Likely doesn't apply out of the box :-) )

 

 

> Alternatively, would a patch be welcome to read a

> confguration file to limit which drivers are registered at initialisation?

 

If you use GDAL as a library from your own code (C/C++), you can also directly call the GDALRegister_XXXX() / OGRRegisterXXXX() functions you are interested in, instead of GDALAllRegister()

 

GDALOpenEx() also accepts a white-list of drivers to probe.

 

You are probably aware of the GDAL_SKIP configuration option to blacklist drivers at run-time.

 

That said perhaps a GDAL_DRIVERS configuration option to whitelist could have some use.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Tobias Wendorff
In reply to this post by Ben Elliston
I have created a config for minimal raster and vector.

I can uploaded later when I am at home.

--
Von einem iPhone gesendet und wird daher Fehler enthalten…

Am 08.03.2018 um 13:23 schrieb Ben Elliston <[hidden email]>:

> Is it possible to limit the number of drivers compiled into GDAL (I am
> guessing not)?  Alternatively, would a patch be welcome to read a
> confguration file to limit which drivers are registered at initialisation?
>
> Cheers,
> Ben
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Kurt Schwehr-2
I get a little more aggressive compared to what others suggested by commenting out register calls in gdalallregister.cpp and ogrregisterall.cpp and removing the code for the drivers.

There are gotchyas lurking, but it definitely works.

e.g.
Enable HDF5, but block BAG
Enable EHDR & ENVI, but leave out the rest of the raw drivers

On Thu, Mar 8, 2018 at 4:36 AM, Tobias Wendorff <[hidden email]> wrote:
I have created a config for minimal raster and vector.

I can uploaded later when I am at home.

--
Von einem iPhone gesendet und wird daher Fehler enthalten…

Am 08.03.2018 um 13:23 schrieb Ben Elliston <[hidden email]>:

> Is it possible to limit the number of drivers compiled into GDAL (I am
> guessing not)?  Alternatively, would a patch be welcome to read a
> confguration file to limit which drivers are registered at initialisation?
>
> Cheers,
> Ben
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Tobias Wendorff
Am Do, 8.03.2018, 17:47 schrieb Kurt Schwehr:
> I get a little more aggressive compared to what others suggested by
> commenting out register calls in gdalallregister.cpp and
> ogrregisterall.cpp
> and removing the code for the drivers.
>

Share, share, share :)

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Kurt Schwehr-2
I use bazel for building, so the build info isn't useful.  You'll have to hack the makefiles a bit.

I also tried to stop things from registering...
find . -name \*.cpp | egrep -v 'autotest|swig|apps|fuzzers|gdaldrivermanager.cpp|ogrregisterall.cpp|gdalallregister.cpp' | xargs egrep 'Register[(][)];|OGRRegisterAll|GDALRegister' | egrep -v 'void|: +[*]|:/'


Here are my two diffs:  (note that I don't define FRMT_raw in my builds)

diff -u a/gdalallregister.cpp  b/gdalallregister.cpp 
--- a/gdalallregister.cpp 2018-03-08 10:40:24.342388944 -0800
+++ b/gdalallregister.cpp 2017-12-28 10:13:13.000000000 -0800
@@ -67,7 +67,8 @@
 
 #ifdef FRMT_vrt
     GDALRegister_VRT();
-    GDALRegister_Derived();
+    // GDALRegister_Derived();
 #endif
 
 #ifdef FRMT_gtiff
@@ -220,7 +221,8 @@
 #endif
 
 #ifdef FRMT_netcdf
-    GDALRegister_GMT();
+    // GDALRegister_GMT();
     GDALRegister_netCDF();
 #endif
 
@@ -406,6 +408,11 @@
 /*      ASCII-header in the same directory.                             */
 /* -------------------------------------------------------------------- */
 
+    GDALRegister_EHdr();
+    GDALRegister_ENVI();
+
 #ifdef FRMT_raw
     GDALRegister_PNM();
     GDALRegister_DOQ1();
@@ -476,7 +483,8 @@
 #endif
 
 #ifdef FRMT_hdf5
-    GDALRegister_BAG();
+    // GDALRegister_BAG();
     GDALRegister_HDF5();
     GDALRegister_HDF5Image();
 #endif

diff -u a/ogrregisterall.cpp b/ogrregisterall.cpp 
--- a/ogrregisterall.cpp 2018-03-08 10:54:56.775492742 -0800
+++ b/ogrregisterall.cpp 2018-03-08 10:54:27.643856370 -0800
@@ -106,7 +106,7 @@
     RegisterOGRGMT();
 #endif
 #ifdef SQLITE_ENABLED
-    RegisterOGRGeoPackage();
+    // RegisterOGRGeoPackage();
     RegisterOGRSQLite();
 #endif
 #ifdef DODS_ENABLED


On Thu, Mar 8, 2018 at 9:21 AM, Tobias Wendorff <[hidden email]> wrote:
Am Do, 8.03.2018, 17:47 schrieb Kurt Schwehr:
> I get a little more aggressive compared to what others suggested by
> commenting out register calls in gdalallregister.cpp and
> ogrregisterall.cpp
> and removing the code for the drivers.
>

Share, share, share :)




--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

bradh
In reply to this post by Ben Elliston
I think you should feel free to make general improvements to the autotools setup 😊

Brad

-----Original Message-----
From: gdal-dev <[hidden email]> On Behalf Of Ben Elliston
Sent: Thursday, 8 March 2018 11:24 PM
To: [hidden email]
Subject: [gdal-dev] limiting drivers that get registered

Is it possible to limit the number of drivers compiled into GDAL (I am guessing not)?  Alternatively, would a patch be welcome to read a confguration file to limit which drivers are registered at initialisation?

Cheers,
Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Ben Elliston
On 09/03/18 10:43, [hidden email] wrote:

> I think you should feel free to make general improvements to the autotools setup 😊

Is this my past coming back to haunt me? :-)

Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Ben Elliston
In reply to this post by Even Rouault-2
Hi Even

On 08/03/18 23:34, Even Rouault wrote:

> If you use GDAL as a library from your own code (C/C++), you can also
> directly call the GDALRegister_XXXX() / OGRRegisterXXXX() functions you
> are interested in, instead of GDALAllRegister()

This is ideal, as it allows a handful of drivers to be registered first,
and then GDALAllRegister() to be called to register everything else down
the line. Unfortunately, it is impossible to know which drivers are
present until you call an undefined function in the GDAL shared library.
For example, GDALRegister_HDF4() is not defined if the HDF4 library
isn't found at configure time.

Is there any way to know which drivers are available at runtime?

Cheers,
Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Even Rouault-2
On mardi 27 mars 2018 21:09:41 CEST Ben Elliston wrote:

> Hi Even
>
> On 08/03/18 23:34, Even Rouault wrote:
> > If you use GDAL as a library from your own code (C/C++), you can also
> > directly call the GDALRegister_XXXX() / OGRRegisterXXXX() functions you
> > are interested in, instead of GDALAllRegister()
>
> This is ideal, as it allows a handful of drivers to be registered first,
> and then GDALAllRegister() to be called to register everything else down
> the line

Hum why would you need to call GDALAllRegister() if you already manually
registered some of them ?

> . Unfortunately, it is impossible to know which drivers are
> present until you call an undefined function in the GDAL shared library.
> For example, GDALRegister_HDF4() is not defined if the HDF4 library
> isn't found at configure time.
>
> Is there any way to know which drivers are available at runtime?

Well, call GDALAllRegister() and then query GDALGetDriverByName(driver_name).
You can unregister a driver with GDALDeregisterDriver()

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Ben Elliston
On 27/03/18 21:49, Even Rouault wrote:

> Hum why would you need to call GDALAllRegister() if you already manually
> registered some of them ?

Because reordering the driver list by placing commonly used drivers at
the head of the list makes a significant difference in the time taken to
open files.

> Well, call GDALAllRegister() and then query GDALGetDriverByName(driver_name).
> You can unregister a driver with GDALDeregisterDriver()

Thanks -- you've given me an idea ..

Cheers, Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Even Rouault-2
On mardi 27 mars 2018 21:51:46 CEST Ben Elliston wrote:
> On 27/03/18 21:49, Even Rouault wrote:
> > Hum why would you need to call GDALAllRegister() if you already manually
> > registered some of them ?
>
> Because reordering the driver list by placing commonly used drivers at
> the head of the list makes a significant difference in the time taken to
> open files.

Ah that was apparently your actual need rather than limiting the number of
drivers per-se. Do you have profiled where time was lost if you find opening
too slow ? The identification logic of drivers is supposed to operate fast
(extension based, or on the content of the first kilobyte of the header), but
there might be some drivers where improvements can be done.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: limiting drivers that get registered

Ben Elliston
On 27/03/18 22:00, Even Rouault wrote:

> Ah that was apparently your actual need rather than limiting the number of
> drivers per-se. Do you have profiled where time was lost if you find opening
> too slow ? The identification logic of drivers is supposed to operate fast
> (extension based, or on the content of the first kilobyte of the header), but
> there might be some drivers where improvements can be done.

I wrote a pathological benchmark that opened and closed an HDF5 file
10,000 times. Moving the HDF5 driver to the head of the driver list gave
about a 5x speedup.

Thanks for the idea earlier. I came up with an acceptable solution:
register all drivers, look to see which ones are present by using
GDALGetDriverShortName(), and then de-register all the drivers.

Cheers, Ben
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Extension based file identification in drivers

Andrew C Aitchison-2
In reply to this post by Even Rouault-2
On Tue, 27 Mar 2018, Even Rouault wrote:

>  The identification logic of drivers is supposed to operate fast
> (extension based, or on the content of the first kilobyte of the header),

If I understand correctly Frank Warmerdam did not favour extension based
file identification:
https://lists.osgeo.org/pipermail/gdal-dev/2013-February/035516.html
   GDAL is based on the idea that file formats are discovered by
   inspecting file contents rather than based on extension and I don't
   want to make it too easy for applications to ignore this principle
   and revert to extensions.

I have therefore tried to avoid using extension to identify file type.

Has thinking on this moved on in the last five years ?

--
Andrew C. Aitchison Cambridge, UK
  [hidden email]
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extension based file identification in drivers

Even Rouault-2
On mercredi 28 mars 2018 15:05:34 CEST Andrew C Aitchison wrote:

> On Tue, 27 Mar 2018, Even Rouault wrote:
> >  The identification logic of drivers is supposed to operate fast
> >
> > (extension based, or on the content of the first kilobyte of the header),
>
> If I understand correctly Frank Warmerdam did not favour extension based
> file identification:
> https://lists.osgeo.org/pipermail/gdal-dev/2013-February/035516.html
>    GDAL is based on the idea that file formats are discovered by
>    inspecting file contents rather than based on extension and I don't
>    want to make it too easy for applications to ignore this principle
>    and revert to extensions.
>
> I have therefore tried to avoid using extension to identify file type.
>
> Has thinking on this moved on in the last five years ?

No. Identification per file content is still the primary identification
method. There are a few drivers where this is not possible due to the file
format (absence of header with unambiguous signature), and thus extension is
used as a fallback.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev