[gdal-dev] /vsicurl/ and 3XX response code

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] /vsicurl/ and 3XX response code

aborruso
Hi all,
using /vsicurl/ is it possible to native intercept HTTP 3XX response code
and get the final URL of the source?

Thank you



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

Even Rouault-2
On mercredi 11 mars 2020 09:03:05 CET aborruso wrote:
> Hi all,
> using /vsicurl/ is it possible to native intercept HTTP 3XX response code
> and get the final URL of the source?

No, this is an implementation detail. You'd have to use the curl or other HTTP
API for that.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

aborruso
Hi Even ,


Even Rouault-2 wrote
> No, this is an implementation detail. You'd have to use the curl or other
> HTTP
> API for that.

today a lot of datasets, especially structured text data, are hosted in
portals that reply with "HTTP/1.1 301 Moved Permanently".
A classic example are the many and preciuous CSV and JSON files that you
have in github gist.

It would be great to have as feature the option to follow the link (as in
curl via "-L" option), both using the gdal/ogr /vsicurl/ via command line
and via the definition of a VRT – GDAL Virtual Format XML file.

Is this the right place where I can create a feature request?

As usual thank you very much




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

Even Rouault-2
On mercredi 11 mars 2020 09:53:55 CET aborruso wrote:

> Hi Even ,
>
>
> Even Rouault-2 wrote
>
> > No, this is an implementation detail. You'd have to use the curl or other
> > HTTP
> > API for that.
>
> today a lot of datasets, especially structured text data, are hosted in
> portals that reply with "HTTP/1.1 301 Moved Permanently".
> A classic example are the many and preciuous CSV and JSON files that you
> have in github gist.
>
> It would be great to have as feature the option to follow the link (as in
> curl via "-L" option), both using the gdal/ogr /vsicurl/ via command line
> and via the definition of a VRT – GDAL Virtual Format XML file.

/vsicurl/ should automatically follow links. This is even tested in
https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L230

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

aborruso
Hi Even,


Even Rouault-2 wrote
> /vsicurl/ should automatically follow links. This is even tested in
> https://github.com/OSGeo/gdal/blob/master/autotest/gcore/vsicurl.py#L230

I have this example CSV input source
https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk

It replies:

HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 0
Date: Wed, 11 Mar 2020 17:24:23 GMT
Location:
https://download.data.world/file_download/ondata/covid-19-italia-dati-dipartimento-protezione-civile/dpc-covid19-ita-regioni.csv?auth=eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJwcm9kLXVzZXItY2xpZW50OmFib3JydXNvIiwiaXNzIjoiYWdlbnQ6YWJvcnJ1c286OmRlNTc0ZTFlLTVhNzktNDhhZC1hMDk0LTdiZmI5YWExNTBiYyIsImlhdCI6MTU4MzkzNDg2Nywicm9sZSI6WyJ1c2VyIiwidXNlcl9hcGlfYWRtaW4iLCJ1c2VyX2FwaV9yZWFkIiwidXNlcl9hcGlfd3JpdGUiXSwiZ2VuZXJhbC1wdXJwb3NlIjpmYWxzZSwidXJsIjoiMjU5OTdiNjUwZGYzYjllOWM2ZTM1MWIxYTMwZDVmYWJhZjA0NjQxYiJ9.8-yXU5WpfkOcZ5VFMNVtaGwBy7ZaT4yIWIWQ8P1NIgR4o11jnwqH9DpcghdUcvvHNYQZwIDAYIRiwJe5zVLWLA
Server: nginx
Vary: Origin

If I set this VRT input file

<OGRVRTDataSource>
<OGRVRTLayer name="dpc-covid19-ita-regioni">
    <SrcDataSource
relativeToVRT="0">/vsicurl/https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk</SrcDataSource>
    <GeometryType>wkbPoint</GeometryType>
    <LayerSRS>WGS84</LayerSRS>
    <GeometryField encoding="PointFromColumns" x="long" y="lat"/>
</OGRVRTLayer>
</OGRVRTDataSource>

and run "ogrinfo input.vrt" I have

"ERROR 1: Failed to open datasource
`/vsicurl/https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk'"

If I run "curl -L http...." I have the CSV.

Probably there is some error of mine, but I don't know which it is.

Thank you





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

Even Rouault-2
There are 2 issues:
- the CSV driver only recognizes files with .csv extension, unless you prefix the filename with CSV:
- the underling resource doesn't support HTTP GET requests used by /vsicurl/. Fortunately in that situation, you can use the alternate /vsicurl_streaming/ implementation

So all in all, the following works:

ogrinfo -ro -al -q CSV:/vsicurl_streaming/https://query.data.world/s/q4h3mwm2acruvoc5snpp5ltsn5h2gk

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: /vsicurl/ and 3XX response code

aborruso
Hi Even,
as usual GDAL and you are really bombastic!!

Thank you very much



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev