[gdal-dev] GDAL vsicurl with query string

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] GDAL vsicurl with query string

JDzialo John

Hi All,

 

This is my first post so forgive my noobiness…

 

We are running a cropping tool that used to access data from a local hard drive and crop images.

 

Now we want to use s3 as our storage in aws and want to access files through http requests.  Great!  Enter vsicurl.

 

So when we open our s3 bucket to the world we can easily access our tif images from s3 with a straight url.  For example…

 

https://test-bucket.s3.amazonaws.com/test/key/value/string/object.tif

 

This works well using the gdalinfo vsicurl cmd…

 

gdalinfo /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/object.tif

 

However in s3 it is not secure to allow access to everyone on the internet and would rather use a presigned URL that uses a query string to authenticate access to the object. 

 

When sending a request with a query string vsicurl and gdal appear to breakdown and I’m not sure why. 

 

gdalinfo /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/object.tif?Signature=sdlfkj88345r9&Expires=12300000&AWSAccessKeyID=AKIALKSDJFOILIKJFVOSIDF

 

Fails saying it’s unable to open the file.

 

Is there a specific way to pass the query string to the VSIL API like when using curl with the –d switch?  Any way to run vsicurl with a query string or something I’m missing?

 

Thanks for any advice you can throw my way!

 

 

 

John Dzialo | Linux System Administrator

Direct 203.783.8163 | Main 800.352.0050

 

Environmental Data Resources, Inc.

440 Wheelers Farms Road, Milford, CT 06461

www.edrnet.com | commonground.edrnet.com

 

Description: Description: Description: Description: Description: EDR_logo4color_EDR_only_80px2

 


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

Even Rouault-2
Le samedi 23 août 2014 00:13:21, JDzialo John a écrit :

> Hi All,
>
> This is my first post so forgive my noobiness...
>
> We are running a cropping tool that used to access data from a local hard
> drive and crop images.
>
> Now we want to use s3 as our storage in aws and want to access files
> through http requests.  Great!  Enter vsicurl.
>
> So when we open our s3 bucket to the world we can easily access our tif
> images from s3 with a straight url.  For example...
>
> https://test-bucket.s3.amazonaws.com/test/key/value/string/object.tif
>
> This works well using the gdalinfo vsicurl cmd...
>
> gdalinfo
> /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/object
> .tif
>
> However in s3 it is not secure to allow access to everyone on the internet
> and would rather use a presigned URL that uses a query string to
> authenticate access to the object.
>
> When sending a request with a query string vsicurl and gdal appear to
> breakdown and I'm not sure why.
>
> gdalinfo
> /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/object
> .tif?Signature=sdlfkj88345r9&Expires=12300000&AWSAccessKeyID=AKIALKSDJFOILI
> KJFVOSIDF

I tried the URL in a browser and got a AccessDenied. Was it supposed to work
or something faked to just give a sense of the kind of URL ? A working example
would be helpful.

>
> Fails saying it's unable to open the file.
>
> Is there a specific way to pass the query string to the VSIL API like when
> using curl with the -d switch?  Any way to run vsicurl with a query string
> or something I'm missing?
>
> Thanks for any advice you can throw my way!
>
>
>
> John Dzialo | Linux System Administrator
> Direct 203.783.8163 | Main 800.352.0050
>
> Environmental Data Resources, Inc.
> 440 Wheelers Farms Road, Milford, CT 06461
> www.edrnet.com<http://www.edrnet.com/> |
> commonground.edrnet.com<http://commonground.edrnet.com/>
>
> [Description: Description: Description: Description: Description:
> EDR_logo4color_EDR_only_80px2]

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

JDzialo John
Hi Even

Yeah it was just a faked URL that was meant to show the URL format I was trying to use to vsicurl a tif file from AWS S3.

OK so these URLs are time restricted and will terminate after an hour, the maximum I ca set it at.

This is just our test bucket and test user.

If you can try this link...

https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=zyN053V02Y41VSKA6Yc1%2Bx0fz%2BU%3D&Expires=1408987568&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q

It will terminate around 1:30pm.



-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Friday, August 22, 2014 6:39 PM
To: [hidden email]
Cc: JDzialo John
Subject: Re: [gdal-dev] GDAL vsicurl with query string

Le samedi 23 août 2014 00:13:21, JDzialo John a écrit :

> Hi All,
>
> This is my first post so forgive my noobiness...
>
> We are running a cropping tool that used to access data from a local
> hard drive and crop images.
>
> Now we want to use s3 as our storage in aws and want to access files
> through http requests.  Great!  Enter vsicurl.
>
> So when we open our s3 bucket to the world we can easily access our
> tif images from s3 with a straight url.  For example...
>
> https://test-bucket.s3.amazonaws.com/test/key/value/string/object.tif
>
> This works well using the gdalinfo vsicurl cmd...
>
> gdalinfo
> /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/ob
> ject
> .tif
>
> However in s3 it is not secure to allow access to everyone on the
> internet and would rather use a presigned URL that uses a query string
> to authenticate access to the object.
>
> When sending a request with a query string vsicurl and gdal appear to
> breakdown and I'm not sure why.
>
> gdalinfo
> /vsicurl/https://test-bucket.s3.amazonaws.com/test/key/value/string/ob
> ject
> .tif?Signature=sdlfkj88345r9&Expires=12300000&AWSAccessKeyID=AKIALKSDJ
> FOILI
> KJFVOSIDF

I tried the URL in a browser and got a AccessDenied. Was it supposed to work or something faked to just give a sense of the kind of URL ? A working example would be helpful.

>
> Fails saying it's unable to open the file.
>
> Is there a specific way to pass the query string to the VSIL API like
> when using curl with the -d switch?  Any way to run vsicurl with a
> query string or something I'm missing?
>
> Thanks for any advice you can throw my way!
>
>
>
> John Dzialo | Linux System Administrator Direct 203.783.8163 | Main
> 800.352.0050
>
> Environmental Data Resources, Inc.
> 440 Wheelers Farms Road, Milford, CT 06461
> www.edrnet.com<http://www.edrnet.com/> |
> commonground.edrnet.com<http://commonground.edrnet.com/>
>
> [Description: Description: Description: Description: Description:
> EDR_logo4color_EDR_only_80px2]

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

Even Rouault-2
Le lundi 25 août 2014 18:26:47, JDzialo John a écrit :

> Hi Even
>
> Yeah it was just a faked URL that was meant to show the URL format I was
> trying to use to vsicurl a tif file from AWS S3.
>
> OK so these URLs are time restricted and will terminate after an hour, the
> maximum I ca set it at.
>
> This is just our test bucket and test user.
>
> If you can try this link...
>
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signa
> ture=zyN053V02Y41VSKA6Yc1%2Bx0fz%2BU%3D&Expires=1408987568&AWSAccessKeyId=A
> KIAJF3TTVKIEFQXGZ3Q
>
> It will terminate around 1:30pm.

Just saw your message now after timeout experitation... If you could try
setting up something with a longer timeout... Or you'll have to coordinate
closely with someone.

Well, for some offline debugging, what is the output of :
gdalinfo --debug on --config CPL_CURL_VERBOSE YES /vsicurl/https://.....

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

JDzialo John
Thanks.

I think an hour may be the limit but I set it for 10 hours so if it's allowed this link should be good until midnight tonight...

https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q

Thanks for the debug command I appreciate your help.  I tried the command and received the following error...

PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE YES "/vsicurl/https://parcel-test.s3.amazonaws.com/
test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T
TVKIEFQXGZ3Q"
VSICURL: GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string)
* timeout on name lookup is not supported
* About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
*   Trying 54.231.2.217... * connected
* Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
* libcurl is now using a weak random seed!
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
* timeout on name lookup is not supported
* About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
*   Trying 54.231.2.217... * connected
* Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
VSICURL: GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhj
vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q)=0  response_code=0
ERROR 4: `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q' does not exist in the file system,
and is not recognised as a supported dataset name.

gdalinfo failed - unable to open '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signatu
re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q'.




-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Monday, August 25, 2014 2:13 PM
To: JDzialo John
Cc: [hidden email]
Subject: Re: [gdal-dev] GDAL vsicurl with query string

Le lundi 25 août 2014 18:26:47, JDzialo John a écrit :

> Hi Even
>
> Yeah it was just a faked URL that was meant to show the URL format I
> was trying to use to vsicurl a tif file from AWS S3.
>
> OK so these URLs are time restricted and will terminate after an hour,
> the maximum I ca set it at.
>
> This is just our test bucket and test user.
>
> If you can try this link...
>
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?
> Signa
> ture=zyN053V02Y41VSKA6Yc1%2Bx0fz%2BU%3D&Expires=1408987568&AWSAccessKe
> yId=A
> KIAJF3TTVKIEFQXGZ3Q
>
> It will terminate around 1:30pm.

Just saw your message now after timeout experitation... If you could try setting up something with a longer timeout... Or you'll have to coordinate closely with someone.

Well, for some offline debugging, what is the output of :
gdalinfo --debug on --config CPL_CURL_VERBOSE YES /vsicurl/https://.....

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

Even Rouault-2
Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :

> Thanks.
>
> I think an hour may be the limit but I set it for 10 hours so if it's
> allowed this link should be good until midnight tonight...
>
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signa
> ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=A
> KIAJF3TTVKIEFQXGZ3Q
>
> Thanks for the debug command I appreciate your help.  I tried the command
> and received the following error...
>
> PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE YES
> "/vsicurl/https://parcel-test.s3.amazonaws.com/
> test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs
> %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> VSICURL:
> GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/s
> tring) * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
> * libcurl is now using a weak random seed!
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
> failed * Closing connection #0
> * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
> failed * Closing connection #0
> VSICURL:
> GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string/obj
> ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q)=0
> response_code=0 ERROR 4:
> `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/objec
> t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q' does
> not exist in the file system, and is not recognised as a supported dataset
> name.
>
> gdalinfo failed - unable to open
> '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/objec
> t.jpg?Signatu
> re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AK
> IAJF3TTVKIEFQXGZ3Q'.
>

Interesting. I don't have that error (but others ;-)). There might be a
problem with the certificate, and curl being not able to check it.

I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass certificate
checks.

Assuming this solves this issue, and you'll get the same errors as on my
machine, you'll see that the server doesn't apparently accept HEAD requests
(this seems to be a constant issue with S3 storage)

> HEAD
/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
HTTP/1.1
Host: parcel-test.s3.amazonaws.com
Accept: */*

< HTTP/1.1 403 Forbidden

You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO

And, optionnaly, to make it faster, add --config GDAL_DISABLE_READDIR_ON_OPEN
EMPTY_DIR

With the last 2 applied :

$ gdalinfo  "/vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q"
--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_VSIL_CURL_USE_HEAD
NO
Driver: JPEG/JPEG JFIF
Files: /vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
Size is 974, 647
Coordinate System is `'
Metadata:
  EXIF_ExifVersion=0220
  EXIF_PixelXDimension=974
  EXIF_PixelYDimension=647
  EXIF_Software=Google
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  647.0)
Upper Right (  974.0,    0.0)
Lower Right (  974.0,  647.0)
Center      (  487.0,  323.5)
Band 1 Block=974x1 Type=Byte, ColorInterp=Red
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=974x1 Type=Byte, ColorInterp=Green
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

JDzialo John
Weird...

I'm still having an issue with the certificate using GDAL_HTTP_UNSAFESSL YES.  It seems to try to verify the cert with this set or not.

I wonder why you are not seeing that error at all?  

Is there any other why of setting up gdalinfo to not verify the cert?  Is there a list of GDAL config options somewhere I could go through?



-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Monday, August 25, 2014 2:40 PM
To: JDzialo John
Cc: [hidden email]
Subject: Re: [gdal-dev] GDAL vsicurl with query string

Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :

> Thanks.
>
> I think an hour may be the limit but I set it for 10 hours so if it's
> allowed this link should be good until midnight tonight...
>
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?
> Signa
> ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKe
> yId=A
> KIAJF3TTVKIEFQXGZ3Q
>
> Thanks for the debug command I appreciate your help.  I tried the
> command and received the following error...
>
> PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE
> YES "/vsicurl/https://parcel-test.s3.amazonaws.com/
> test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%
> 2FOs %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> VSICURL:
> GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/val
> ue/s
> tring) * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> (#0)
> * libcurl is now using a weak random seed!
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> verify failed * Closing connection #0
> * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> (#0)
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> verify failed * Closing connection #0
> VSICURL:
> GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string
> /obj
> ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
> )=0
> response_code=0 ERROR 4:
> `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> bjec
> t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q'
> does not exist in the file system, and is not recognised as a
> supported dataset name.
>
> gdalinfo failed - unable to open
> '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> bjec
> t.jpg?Signatu
> re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyI
> d=AK
> IAJF3TTVKIEFQXGZ3Q'.
>

Interesting. I don't have that error (but others ;-)). There might be a problem with the certificate, and curl being not able to check it.

I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass certificate checks.

Assuming this solves this issue, and you'll get the same errors as on my machine, you'll see that the server doesn't apparently accept HEAD requests (this seems to be a constant issue with S3 storage)

> HEAD
/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
HTTP/1.1
Host: parcel-test.s3.amazonaws.com
Accept: */*

< HTTP/1.1 403 Forbidden

You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO

And, optionnaly, to make it faster, add --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR

With the last 2 applied :

$ gdalinfo  "/vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q"
--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_VSIL_CURL_USE_HEAD NO
Driver: JPEG/JPEG JFIF
Files: /vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
Size is 974, 647
Coordinate System is `'
Metadata:
  EXIF_ExifVersion=0220
  EXIF_PixelXDimension=974
  EXIF_PixelYDimension=647
  EXIF_Software=Google
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  647.0)
Upper Right (  974.0,    0.0)
Lower Right (  974.0,  647.0)
Center      (  487.0,  323.5)
Band 1 Block=974x1 Type=Byte, ColorInterp=Red
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=974x1 Type=Byte, ColorInterp=Green
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

Even Rouault-2
Le lundi 25 août 2014 21:40:51, JDzialo John a écrit :
> Weird...
>
> I'm still having an issue with the certificate using GDAL_HTTP_UNSAFESSL
> YES.  It seems to try to verify the cert with this set or not.
>
> I wonder why you are not seeing that error at all?

Not same curl version likely, or not compiled with same options. Mine is curl
7.19.7-1ubuntu1.7

>
> Is there any other why of setting up gdalinfo to not verify the cert?  

I don't think so right now. Does that work with the curl binary on the command
line ? If so, well, no further idea. If it does not work, then you likely have
to find the additional curl option needed.

Searching for "error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"  in your
favorite search engine might bring results perhaps.

> Is
> there a list of GDAL config options somewhere I could go through?

Most are documented in
http://www.gdal.org/cpl__http_8h.html#aee8368b7821300f4b81ef4da8a9c6a29 but I
see that UNSAFESSL was not.

>
>
>
> -----Original Message-----
> From: Even Rouault [mailto:[hidden email]]
> Sent: Monday, August 25, 2014 2:40 PM
> To: JDzialo John
> Cc: [hidden email]
> Subject: Re: [gdal-dev] GDAL vsicurl with query string
>
> Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :
> > Thanks.
> >
> > I think an hour may be the limit but I set it for 10 hours so if it's
> > allowed this link should be good until midnight tonight...
> >
> > https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?
> > Signa
> > ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKe
> > yId=A
> > KIAJF3TTVKIEFQXGZ3Q
> >
> > Thanks for the debug command I appreciate your help.  I tried the
> > command and received the following error...
> >
> > PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE
> > YES "/vsicurl/https://parcel-test.s3.amazonaws.com/
> > test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%
> > 2FOs %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> > VSICURL:
> > GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/val
> > ue/s
> > tring) * timeout on name lookup is not supported
> > * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> > *   Trying 54.231.2.217... * connected
> > * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> > (#0)
> > * libcurl is now using a weak random seed!
> > * SSL certificate problem, verify that the CA cert is OK. Details:
> > error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> > verify failed * Closing connection #0
> > * timeout on name lookup is not supported
> > * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> > *   Trying 54.231.2.217... * connected
> > * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> > (#0)
> > * SSL certificate problem, verify that the CA cert is OK. Details:
> > error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> > verify failed * Closing connection #0
> > VSICURL:
> > GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string
> > /obj
> > ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> > vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
> > )=0
> > response_code=0 ERROR 4:
> > `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> > bjec
> > t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> > v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q'
> > does not exist in the file system, and is not recognised as a
> > supported dataset name.
> >
> > gdalinfo failed - unable to open
> > '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> > bjec
> > t.jpg?Signatu
> > re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyI
> > d=AK
> > IAJF3TTVKIEFQXGZ3Q'.
>
> Interesting. I don't have that error (but others ;-)). There might be a
> problem with the certificate, and curl being not able to check it.
>
> I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass certificate
> checks.
>
> Assuming this solves this issue, and you'll get the same errors as on my
> machine, you'll see that the server doesn't apparently accept HEAD
> requests (this seems to be a constant issue with S3 storage)
>
> > HEAD
>
> /test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs
> %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q HTTP/1.1
> Host: parcel-test.s3.amazonaws.com
> Accept: */*
>
> < HTTP/1.1 403 Forbidden
>
> You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO
>
> And, optionnaly, to make it faster, add --config
> GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR
>
> With the last 2 applied :
>
> $ gdalinfo  "/vsicurl/https://parcel-
> test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N
> 2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQX
> GZ3Q" --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
> CPL_VSIL_CURL_USE_HEAD NO Driver: JPEG/JPEG JFIF
> Files: /vsicurl/https://parcel-
> test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N
> 2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQX
> GZ3Q Size is 974, 647
> Coordinate System is `'
> Metadata:
>   EXIF_ExifVersion=0220
>   EXIF_PixelXDimension=974
>   EXIF_PixelYDimension=647
>   EXIF_Software=Google
> Image Structure Metadata:
>   COMPRESSION=JPEG
>   INTERLEAVE=PIXEL
>   SOURCE_COLOR_SPACE=YCbCr
> Corner Coordinates:
> Upper Left  (    0.0,    0.0)
> Lower Left  (    0.0,  647.0)
> Upper Right (  974.0,    0.0)
> Lower Right (  974.0,  647.0)
> Center      (  487.0,  323.5)
> Band 1 Block=974x1 Type=Byte, ColorInterp=Red
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
> Band 2 Block=974x1 Type=Byte, ColorInterp=Green
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
> Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
>
> Even
>
> --
> Spatialys - Geospatial professional services http://www.spatialys.com

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL vsicurl with query string

JDzialo John
Thanks so much for your help.

Curl works fine it's only the gdalinfo \vsicurl\ that is throwing the error using 7.21.2 of curl and libcurl on a Windows Server 2008 server.

I'm looking through to see what I can find online.

-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Monday, August 25, 2014 3:48 PM
To: JDzialo John
Cc: [hidden email]
Subject: Re: [gdal-dev] GDAL vsicurl with query string

Le lundi 25 août 2014 21:40:51, JDzialo John a écrit :
> Weird...
>
> I'm still having an issue with the certificate using
> GDAL_HTTP_UNSAFESSL YES.  It seems to try to verify the cert with this set or not.
>
> I wonder why you are not seeing that error at all?

Not same curl version likely, or not compiled with same options. Mine is curl
7.19.7-1ubuntu1.7

>
> Is there any other why of setting up gdalinfo to not verify the cert?  

I don't think so right now. Does that work with the curl binary on the command line ? If so, well, no further idea. If it does not work, then you likely have to find the additional curl option needed.

Searching for "error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"  in your favorite search engine might bring results perhaps.

> Is
> there a list of GDAL config options somewhere I could go through?

Most are documented in
http://www.gdal.org/cpl__http_8h.html#aee8368b7821300f4b81ef4da8a9c6a29 but I see that UNSAFESSL was not.

>
>
>
> -----Original Message-----
> From: Even Rouault [mailto:[hidden email]]
> Sent: Monday, August 25, 2014 2:40 PM
> To: JDzialo John
> Cc: [hidden email]
> Subject: Re: [gdal-dev] GDAL vsicurl with query string
>
> Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :
> > Thanks.
> >
> > I think an hour may be the limit but I set it for 10 hours so if
> > it's allowed this link should be good until midnight tonight...
> >
> > https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?
> > Signa
> > ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccess
> > Ke
> > yId=A
> > KIAJF3TTVKIEFQXGZ3Q
> >
> > Thanks for the debug command I appreciate your help.  I tried the
> > command and received the following error...
> >
> > PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE
> > YES "/vsicurl/https://parcel-test.s3.amazonaws.com/
> > test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2F
> > U% 2FOs %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> > VSICURL:
> > GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/v
> > al
> > ue/s
> > tring) * timeout on name lookup is not supported
> > * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> > *   Trying 54.231.2.217... * connected
> > * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> > (#0)
> > * libcurl is now using a weak random seed!
> > * SSL certificate problem, verify that the CA cert is OK. Details:
> > error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> > verify failed * Closing connection #0
> > * timeout on name lookup is not supported
> > * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> > *   Trying 54.231.2.217... * connected
> > * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443
> > (#0)
> > * SSL certificate problem, verify that the CA cert is OK. Details:
> > error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate
> > verify failed * Closing connection #0
> > VSICURL:
> > GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/stri
> > ng
> > /obj
> > ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> > vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ
> > 3Q
> > )=0
> > response_code=0 ERROR 4:
> > `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string
> > /o
> > bjec
> > t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> > v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q'
> > does not exist in the file system, and is not recognised as a
> > supported dataset name.
> >
> > gdalinfo failed - unable to open
> > '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string
> > /o
> > bjec
> > t.jpg?Signatu
> > re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKe
> > yI
> > d=AK
> > IAJF3TTVKIEFQXGZ3Q'.
>
> Interesting. I don't have that error (but others ;-)). There might be
> a problem with the certificate, and curl being not able to check it.
>
> I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass
> certificate checks.
>
> Assuming this solves this issue, and you'll get the same errors as on
> my machine, you'll see that the server doesn't apparently accept HEAD
> requests (this seems to be a constant issue with S3 storage)
>
> > HEAD
>
> /test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU
> %2FOs %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
> HTTP/1.1
> Host: parcel-test.s3.amazonaws.com
> Accept: */*
>
> < HTTP/1.1 403 Forbidden
>
> You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO
>
> And, optionnaly, to make it faster, add --config
> GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR
>
> With the last 2 applied :
>
> $ gdalinfo  "/vsicurl/https://parcel-
> test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVo
> YI84N
> 2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVK
> IEFQX GZ3Q" --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
> CPL_VSIL_CURL_USE_HEAD NO Driver: JPEG/JPEG JFIF
> Files: /vsicurl/https://parcel-
> test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVo
> YI84N
> 2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVK
> IEFQX
> GZ3Q Size is 974, 647
> Coordinate System is `'
> Metadata:
>   EXIF_ExifVersion=0220
>   EXIF_PixelXDimension=974
>   EXIF_PixelYDimension=647
>   EXIF_Software=Google
> Image Structure Metadata:
>   COMPRESSION=JPEG
>   INTERLEAVE=PIXEL
>   SOURCE_COLOR_SPACE=YCbCr
> Corner Coordinates:
> Upper Left  (    0.0,    0.0)
> Lower Left  (    0.0,  647.0)
> Upper Right (  974.0,    0.0)
> Lower Right (  974.0,  647.0)
> Center      (  487.0,  323.5)
> Band 1 Block=974x1 Type=Byte, ColorInterp=Red
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
> Band 2 Block=974x1 Type=Byte, ColorInterp=Green
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
> Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
>   Overviews: 487x324, 244x162
>   Image Structure Metadata:
>     COMPRESSION=JPEG
>
> Even
>
> --
> Spatialys - Geospatial professional services http://www.spatialys.com

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev