[gdal-dev] Logging network requests/replies?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Logging network requests/replies?

Nyall Dawson
Hi list,

I'm guessing this is a long-shot, but is there any API within GDAL
which allows clients to be notified about outgoing network requests
created by the different GDAL drivers?

I'm looking to get some useful information regarding these for
end-user debugging purposes...

Nyall
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Sean Gillies-3
Hi Nyall,

Even will have the final word on this, but as far as I know, the information printed when you turn on the following option


is not capturable using the error handler from the GDAL API (see https://www.gdal.org/cpl__error_8h.html#a74d0e649d58180e621540bf73b58e4a2).



On Wed, Feb 6, 2019 at 7:14 PM Nyall Dawson <[hidden email]> wrote:
Hi list,

I'm guessing this is a long-shot, but is there any API within GDAL
which allows clients to be notified about outgoing network requests
created by the different GDAL drivers?

I'm looking to get some useful information regarding these for
end-user debugging purposes...

Nyall
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Even Rouault-2
On mercredi 6 février 2019 19:54:29 CET Sean Gillies wrote:
> Hi Nyall,
>
> Even will have the final word on this, but as far as I know, the
> information printed when you turn on the following option
>
>   https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE
>
> is not capturable using the error handler from the GDAL API

Yes CPL_CURL_VERBOSE=YES output goes to stderr by default, since this is curl
which handles this.
But I see in
https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
that we could possibly specify a callback instead of stderr so that it goes
through GDAL logging mechanism.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Nyall Dawson
On Thu, 7 Feb 2019 at 19:12, Even Rouault <[hidden email]> wrote:

>
> On mercredi 6 février 2019 19:54:29 CET Sean Gillies wrote:
> > Hi Nyall,
> >
> > Even will have the final word on this, but as far as I know, the
> > information printed when you turn on the following option
> >
> >   https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_CURL_VERBOSE
> >
> > is not capturable using the error handler from the GDAL API
>
> Yes CPL_CURL_VERBOSE=YES output goes to stderr by default, since this is curl
> which handles this.
> But I see in
> https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
> that we could possibly specify a callback instead of stderr so that it goes
> through GDAL logging mechanism.
>

Actually, what I had in mind (thinking out loud) would sit solely
within GDAL, e.g. a method to globally specify a callback function to
be called whenever GDAL creates a network request, e.g. within
CPLHTTPFetchEx (and similarly for replies). (With clients being
responsible for ensuring that their callback functions are
appropriately thread safe).

Does this sound desirable? Any concerns/blockers before I start hacking on it?

Nyall

> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Even Rouault-2
> Actually, what I had in mind (thinking out loud) would sit solely
> within GDAL, e.g. a method to globally specify a callback function to
> be called whenever GDAL creates a network request, e.g. within
> CPLHTTPFetchEx (and similarly for replies).

CPLHTTPFetchEx() is just one of the site that does network access.
cpl_vsil_curl.cpp, cpl_vsil_s3.cpp, cpl_vsil_az.cpp, cpl_vsil_swift.cpp,
cpl_vsil_webhdfs.cpp also do network access

But at the end of all them must go through CPLHTTPSetOptions() in cpl_http.cpp

> (With clients being
> responsible for ensuring that their callback functions are
> appropriately thread safe).
>
> Does this sound desirable? Any concerns/blockers before I start hacking on
> it?

I had several thoughts about this

1) yes a dedicated callback mechanism could work of course

2) As Sean mentionned, and provided we redirect curl logging to our own
callbacks instead of stderr, we could potentially just use the CPLDebug()
infrastructure with the the callback being installed with CPLSetErrorHandler()
+ CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so much
debugging information to be emitted by default (both because of verbosity and
performance), so this should still be conditionalized by the
CPL_CURL_VERBOSE=YES config option/env. variable

3) An alternative would be to have a dedicated generic logging API

void CPLLogEvent(const char* topic, const char* fmt, ...);

int CPLHasLoggerFor(const char* topic);

typedef void (*CPLLoggerCallback)(const char* topic,
                                                                                 const char* msg, void* user_data);

void* CPLInstallLogger(const char* topic,
                                                        CPLLoggerCallback cbk, void* user_data);
void CPLUninstallLogger(void*);

but I'm wondering if there wouldn't be too much confusion/intersection with
the CPLDebug() mechanism.


My current slight preference would go to 2) to reuse existing infrastructure.
Maybe others, including Robert CC'ed who has hacked in the error/debug area
recently, have thoughts on this.


Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Sean Gillies-3
Hi Even,

On Fri, Feb 8, 2019 at 4:59 AM Even Rouault <[hidden email]> wrote:
> Actually, what I had in mind (thinking out loud) would sit solely
> within GDAL, e.g. a method to globally specify a callback function to
> be called whenever GDAL creates a network request, e.g. within
> CPLHTTPFetchEx (and similarly for replies).

CPLHTTPFetchEx() is just one of the site that does network access.
cpl_vsil_curl.cpp, cpl_vsil_s3.cpp, cpl_vsil_az.cpp, cpl_vsil_swift.cpp,
cpl_vsil_webhdfs.cpp also do network access

But at the end of all them must go through CPLHTTPSetOptions() in cpl_http.cpp

> (With clients being
> responsible for ensuring that their callback functions are
> appropriately thread safe).
>
> Does this sound desirable? Any concerns/blockers before I start hacking on
> it?

I had several thoughts about this

1) yes a dedicated callback mechanism could work of course

2) As Sean mentionned, and provided we redirect curl logging to our own
callbacks instead of stderr, we could potentially just use the CPLDebug()
infrastructure with the the callback being installed with CPLSetErrorHandler()
+ CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so much
debugging information to be emitted by default (both because of verbosity and
performance), so this should still be conditionalized by the
CPL_CURL_VERBOSE=YES config option/env. variable

3) An alternative would be to have a dedicated generic logging API

void CPLLogEvent(const char* topic, const char* fmt, ...);

int CPLHasLoggerFor(const char* topic);

typedef void (*CPLLoggerCallback)(const char* topic,
                                                                                 const char* msg, void* user_data);

void* CPLInstallLogger(const char* topic,
                                                        CPLLoggerCallback cbk, void* user_data);
void CPLUninstallLogger(void*);

but I'm wondering if there wouldn't be too much confusion/intersection with
the CPLDebug() mechanism.


My current slight preference would go to 2) to reuse existing infrastructure.
Maybe others, including Robert CC'ed who has hacked in the error/debug area
recently, have thoughts on this.


Even

I'd prefer to see the existing infrastructure reused because then some logging of network activity would automatically appear for users of Rasterio (which already subscribes to CPLDebug messages).

I think the right amount of default logging would be to call CPLDebug for new connections and report the URL, but not necessarily call CPLDebug for every request made using that connection.

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Logging network requests/replies?

Robert Coup
In reply to this post by Even Rouault-2
Hi all,

On Sat, 9 Feb 2019 at 00:59, Even Rouault <[hidden email]> wrote:
I had several thoughts about this

1) yes a dedicated callback mechanism could work of course

2) As Sean mentionned, and provided we redirect curl logging to our own
callbacks instead of stderr, we could potentially just use the CPLDebug()
infrastructure with the the callback being installed with CPLSetErrorHandler()
+ CPLSetCurrentErrorHandlerCatchDebug(true). We probably wouldn't want so much
debugging information to be emitted by default (both because of verbosity and
performance), so this should still be conditionalized by the
CPL_CURL_VERBOSE=YES config option/env. variable

I'd suggest this is the best approach. Alternatively a TRACE loglevel below DEBUG like some of the Java logging frameworks have.
 

3) An alternative would be to have a dedicated generic logging API

Not sure this adds much over our existing logging API which already includes callbacks, and has Python integration/etc? The only thing it's really missing are logging at different levels via topic/categories, and even those are partially there atm (and you can make it work via a C++/Python logging library integration).

Rob :)

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev