[gdal-dev] gdal translate/warp goes16

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] gdal translate/warp goes16

Steven Beale
Hi gdalers,

I'm a little out of my field here, usually I work with weather/climate data,
but I need to warp some goes16 satellite data. (source is aws s3:
https://noaa-goes16.s3.amazonaws.com/ABI-L2-CMIPF/2018/079/12/OR_ABI-L2-CMIPF-M3C01_G16_s20180791200447_e20180791211214_c20180791211280.nc
- the link should work for a few months then it will be glaciered and
unavailable)

source data is netcdf, I'm using nco tools & gdal for the processing.

My work flow is:
1. clip the data to the region of interest (north atlantic):
ncks -d x,5424,8678 -d y,0,2170 <raw.nc> step1.nc
2. deflate & unscale
gdal_translate -ot float32 -unscale -CO COMPRESS=deflate NETCDF:step1.nc:CMI
step2.tiff
3. warp (in this case data is 1km resolution : ~0.009 degrees)
gdalwarp -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320
65  -r near step2.tiff step3.tiff
4. convert back to nc
gdal_translate -ot float32 -of NETCDF step3.tiff output.nc

The problem i have is that the output has the longitude shifted to the west
by about 10 degrees.
The issue seems related to the ncks clipping, if I exclude the x clip,
everything works out fine, but takes significantly longer.
Here's an example once it's loaded into an erddap server;
<http://osgeo-org.1560.x6.nabble.com/file/t382897/GOES16_L2CMIPF_M3C01_201803201200.png>
You can see greenland to the west of it's map underlay.

I'm using gdal v2.3.0, netcdf v4.4.1.1, nco v4.7.3 & proj4 v5.0.0





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal translate/warp goes16

Zachary Flamig
Hi Steven,

So the issue here is that gdal_translate doesn’t properly store the +sweep=x in the geotiff.

See http://geotiff.maptools.narkive.com/3CqwNeMd/geos-projection-sweep-parameter for some relevant discussion. I believe the best solution right now is to specify the s_srs in the gdalwarp command like -s_srs "+proj=geos +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x"

This is a bit of a mess, sorry about that.

Zac

On Mar 20, 2018, at 9:37 PM, Steven Beale <[hidden email]> wrote:

Hi gdalers,

I'm a little out of my field here, usually I work with weather/climate data,
but I need to warp some goes16 satellite data. (source is aws s3:
https://noaa-goes16.s3.amazonaws.com/ABI-L2-CMIPF/2018/079/12/OR_ABI-L2-CMIPF-M3C01_G16_s20180791200447_e20180791211214_c20180791211280.nc
- the link should work for a few months then it will be glaciered and
unavailable)

source data is netcdf, I'm using nco tools & gdal for the processing.

My work flow is:
1. clip the data to the region of interest (north atlantic):
ncks -d x,5424,8678 -d y,0,2170 <raw.nc> step1.nc
2. deflate & unscale
gdal_translate -ot float32 -unscale -CO COMPRESS=deflate NETCDF:step1.nc:CMI
step2.tiff
3. warp (in this case data is 1km resolution : ~0.009 degrees)
gdalwarp -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320
65  -r near step2.tiff step3.tiff
4. convert back to nc
gdal_translate -ot float32 -of NETCDF step3.tiff output.nc

The problem i have is that the output has the longitude shifted to the west
by about 10 degrees.
The issue seems related to the ncks clipping, if I exclude the x clip,
everything works out fine, but takes significantly longer.
Here's an example once it's loaded into an erddap server;
<http://osgeo-org.1560.x6.nabble.com/file/t382897/GOES16_L2CMIPF_M3C01_201803201200.png>
You can see greenland to the west of it's map underlay.

I'm using gdal v2.3.0, netcdf v4.4.1.1, nco v4.7.3 & proj4 v5.0.0





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal translate/warp goes16

Steven Beale

Hi Zac,

I tried to be as explicit as possible setting the source srs, including the semi-major/minor axes:

gdalwarp -s_srs "+proj=geos +a=6378137 +b=6356752.31414 +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x" -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320 65 -r near <in> <out>

 

But I still have the longitude shifting.

Out of curiosity I tried +sweep=y, just to see what would happen, and that didn’t do anything either… could the sweep be getting ignored entirely?

Just to verify that the srs was being interpreted at all, I tried adjusting lon_0, which did work. Not very helpful though.

Thinking there could possibly be some conflict in setting ellps=GRS80 and setting parameters by hand, I omitted that the +ellps. Same result.

Any other thoughts?

 

Thanks,

Steve

 

From: Zachary Flamig [mailto:[hidden email]]
Sent: Wednesday, March 21, 2018 4:04 AM
To: Steven Beale <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] gdal translate/warp goes16

 

Hi Steven,

 

So the issue here is that gdal_translate doesn’t properly store the +sweep=x in the geotiff.

 

See http://geotiff.maptools.narkive.com/3CqwNeMd/geos-projection-sweep-parameter for some relevant discussion. I believe the best solution right now is to specify the s_srs in the gdalwarp command like -s_srs "+proj=geos +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x"

 

This is a bit of a mess, sorry about that.

 

Zac



On Mar 20, 2018, at 9:37 PM, Steven Beale <[hidden email]> wrote:

 

Hi gdalers,

I'm a little out of my field here, usually I work with weather/climate data,
but I need to warp some goes16 satellite data. (source is aws s3:
https://noaa-goes16.s3.amazonaws.com/ABI-L2-CMIPF/2018/079/12/OR_ABI-L2-CMIPF-M3C01_G16_s20180791200447_e20180791211214_c20180791211280.nc
- the link should work for a few months then it will be glaciered and
unavailable)

source data is netcdf, I'm using nco tools & gdal for the processing.

My work flow is:
1. clip the data to the region of interest (north atlantic):
ncks -d x,5424,8678 -d y,0,2170 <raw.nc> step1.nc
2. deflate & unscale
gdal_translate -ot float32 -unscale -CO COMPRESS=deflate NETCDF:step1.nc:CMI
step2.tiff
3. warp (in this case data is 1km resolution : ~0.009 degrees)
gdalwarp -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320
65  -r near step2.tiff step3.tiff
4. convert back to nc
gdal_translate -ot float32 -of NETCDF step3.tiff output.nc

The problem i have is that the output has the longitude shifted to the west
by about 10 degrees.
The issue seems related to the ncks clipping, if I exclude the x clip,
everything works out fine, but takes significantly longer.
Here's an example once it's loaded into an erddap server;
<http://osgeo-org.1560.x6.nabble.com/file/t382897/GOES16_L2CMIPF_M3C01_201803201200.png>
You can see greenland to the west of it's map underlay.

I'm using gdal v2.3.0, netcdf v4.4.1.1, nco v4.7.3 & proj4 v5.0.0





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

 





This message is the property of John Wood Group PLC and/or its subsidiaries and/or affiliates and is intended only for the named recipient(s). Its contents (including any attachments) may be confidential, legally privileged or otherwise protected from disclosure by law. Unauthorised use, copying, distribution or disclosure of any of it may be unlawful and is strictly prohibited. We assume no responsibility to persons other than the intended named recipient(s) and do not accept liability for any errors or omissions which are a result of email transmission. If you have received this message in error, please notify us immediately by reply email to the sender and confirm that the original message and any attachments and copies have been destroyed and deleted from your system.



If you do not wish to receive future unsolicited commercial electronic messages from us, please forward this email to: [hidden email] and include “Unsubscribe” in the subject line. If applicable, you will continue to receive invoices, project communications and similar factual, non-commercial electronic communications.



Please click http://www.woodplc.com/email-disclaimer for notices and company information in relation to emails originating in the UK, Italy or France.






_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal translate/warp goes16

Steven Beale

I never did resolve this issue, but using the python libraries instead worked out fine.

-s

 

From: Beale, Steven [mailto:[hidden email]]
Sent: Wednesday, March 21, 2018 3:39 PM
To: Zachary Flamig <[hidden email]>; Steven Beale <[hidden email]>
Cc: [hidden email]
Subject: RE: [gdal-dev] gdal translate/warp goes16

 

Hi Zac,

I tried to be as explicit as possible setting the source srs, including the semi-major/minor axes:

gdalwarp -s_srs "+proj=geos +a=6378137 +b=6356752.31414 +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x" -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320 65 -r near <in> <out>

 

But I still have the longitude shifting.

Out of curiosity I tried +sweep=y, just to see what would happen, and that didn’t do anything either… could the sweep be getting ignored entirely?

Just to verify that the srs was being interpreted at all, I tried adjusting lon_0, which did work. Not very helpful though.

Thinking there could possibly be some conflict in setting ellps=GRS80 and setting parameters by hand, I omitted that the +ellps. Same result.

Any other thoughts?

 

Thanks,

Steve

 

From: Zachary Flamig [[hidden email]]
Sent: Wednesday, March 21, 2018 4:04 AM
To: Steven Beale <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] gdal translate/warp goes16

 

Hi Steven,

 

So the issue here is that gdal_translate doesn’t properly store the +sweep=x in the geotiff.

 

See http://geotiff.maptools.narkive.com/3CqwNeMd/geos-projection-sweep-parameter for some relevant discussion. I believe the best solution right now is to specify the s_srs in the gdalwarp command like -s_srs "+proj=geos +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x"

 

This is a bit of a mess, sorry about that.

 

Zac

 

On Mar 20, 2018, at 9:37 PM, Steven Beale <[hidden email]> wrote:

 

Hi gdalers,

I'm a little out of my field here, usually I work with weather/climate data,
but I need to warp some goes16 satellite data. (source is aws s3:
https://noaa-goes16.s3.amazonaws.com/ABI-L2-CMIPF/2018/079/12/OR_ABI-L2-CMIPF-M3C01_G16_s20180791200447_e20180791211214_c20180791211280.nc
- the link should work for a few months then it will be glaciered and
unavailable)

source data is netcdf, I'm using nco tools & gdal for the processing.

My work flow is:
1. clip the data to the region of interest (north atlantic):
ncks -d x,5424,8678 -d y,0,2170 <raw.nc> step1.nc
2. deflate & unscale
gdal_translate -ot float32 -unscale -CO COMPRESS=deflate NETCDF:step1.nc:CMI
step2.tiff
3. warp (in this case data is 1km resolution : ~0.009 degrees)
gdalwarp -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320
65  -r near step2.tiff step3.tiff
4. convert back to nc
gdal_translate -ot float32 -of NETCDF step3.tiff output.nc

The problem i have is that the output has the longitude shifted to the west
by about 10 degrees.
The issue seems related to the ncks clipping, if I exclude the x clip,
everything works out fine, but takes significantly longer.
Here's an example once it's loaded into an erddap server;
<http://osgeo-org.1560.x6.nabble.com/file/t382897/GOES16_L2CMIPF_M3C01_201803201200.png>
You can see greenland to the west of it's map underlay.

I'm using gdal v2.3.0, netcdf v4.4.1.1, nco v4.7.3 & proj4 v5.0.0





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

 

 




This message is the property of John Wood Group PLC and/or its subsidiaries and/or affiliates and is intended only for the named recipient(s). Its contents (including any attachments) may be confidential, legally privileged or otherwise protected from disclosure by law. Unauthorised use, copying, distribution or disclosure of any of it may be unlawful and is strictly prohibited. We assume no responsibility to persons other than the intended named recipient(s) and do not accept liability for any errors or omissions which are a result of email transmission. If you have received this message in error, please notify us immediately by reply email to the sender and confirm that the original message and any attachments and copies have been destroyed and deleted from your system.



If you do not wish to receive future unsolicited commercial electronic messages from us, please forward this email to: [hidden email] and include “Unsubscribe” in the subject line. If applicable, you will continue to receive invoices, project communications and similar factual, non-commercial electronic communications.



Please click http://www.woodplc.com/email-disclaimer for notices and company information in relation to emails originating in the UK, Italy or France.



 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal translate/warp goes16

Zachary Flamig
Interesting. Its working fine for me and some other people using the cli tools.

Zac

On Mar 27, 2018, at 9:44 AM, Beale, Steven <[hidden email]> wrote:

I never did resolve this issue, but using the python libraries instead worked out fine.
-s
 
From: Beale, Steven [[hidden email]] 
Sent: Wednesday, March 21, 2018 3:39 PM
To: Zachary Flamig <[hidden email]>; Steven Beale <[hidden email]>
Cc: [hidden email]
Subject: RE: [gdal-dev] gdal translate/warp goes16
 
Hi Zac,
I tried to be as explicit as possible setting the source srs, including the semi-major/minor axes:
gdalwarp -s_srs "+proj=geos +a=6378137 +b=6356752.31414 +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x" -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320 65 -r near <in> <out>
 
But I still have the longitude shifting.
Out of curiosity I tried +sweep=y, just to see what would happen, and that didn’t do anything either… could the sweep be getting ignored entirely?
Just to verify that the srs was being interpreted at all, I tried adjusting lon_0, which did work. Not very helpful though.
Thinking there could possibly be some conflict in setting ellps=GRS80 and setting parameters by hand, I omitted that the +ellps. Same result.
Any other thoughts?
 
Thanks,
Steve
 
From: Zachary Flamig [[hidden email]] 
Sent: Wednesday, March 21, 2018 4:04 AM
To: Steven Beale <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] gdal translate/warp goes16
 
Hi Steven,
 
So the issue here is that gdal_translate doesn’t properly store the +sweep=x in the geotiff.
 
See http://geotiff.maptools.narkive.com/3CqwNeMd/geos-projection-sweep-parameter for some relevant discussion. I believe the best solution right now is to specify the s_srs in the gdalwarp command like -s_srs "+proj=geos +lon_0=-75 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs  +sweep=x"
 
This is a bit of a mess, sorry about that.
 
Zac

 

On Mar 20, 2018, at 9:37 PM, Steven Beale <[hidden email]> wrote:
 
Hi gdalers, 

I'm a little out of my field here, usually I work with weather/climate data,
but I need to warp some goes16 satellite data. (source is aws s3:
https://noaa-goes16.s3.amazonaws.com/ABI-L2-CMIPF/2018/079/12/OR_ABI-L2-CMIPF-M3C01_G16_s20180791200447_e20180791211214_c20180791211280.nc
- the link should work for a few months then it will be glaciered and
unavailable) 

source data is netcdf, I'm using nco tools & gdal for the processing. 

My work flow is: 
1. clip the data to the region of interest (north atlantic): 
ncks -d x,5424,8678 -d y,0,2170 <raw.nc> step1.nc 
2. deflate & unscale 
gdal_translate -ot float32 -unscale -CO COMPRESS=deflate NETCDF:step1.nc:CMI
step2.tiff 
3. warp (in this case data is 1km resolution : ~0.009 degrees) 
gdalwarp -t_srs EPSG:4326 -dstnodata -999.0 -tr 0.009 0.009 -te 290 40 320
65  -r near step2.tiff step3.tiff 
4. convert back to nc 
gdal_translate -ot float32 -of NETCDF step3.tiff output.nc 

The problem i have is that the output has the longitude shifted to the west
by about 10 degrees. 
The issue seems related to the ncks clipping, if I exclude the x clip,
everything works out fine, but takes significantly longer. 
Here's an example once it's loaded into an erddap server; 
<http://osgeo-org.1560.x6.nabble.com/file/t382897/GOES16_L2CMIPF_M3C01_201803201200.png> 
You can see greenland to the west of it's map underlay. 

I'm using gdal v2.3.0, netcdf v4.4.1.1, nco v4.7.3 & proj4 v5.0.0 





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
 
 



This message is the property of John Wood Group PLC and/or its subsidiaries and/or affiliates and is intended only for the named recipient(s). Its contents (including any attachments) may be confidential, legally privileged or otherwise protected from disclosure by law. Unauthorised use, copying, distribution or disclosure of any of it may be unlawful and is strictly prohibited. We assume no responsibility to persons other than the intended named recipient(s) and do not accept liability for any errors or omissions which are a result of email transmission. If you have received this message in error, please notify us immediately by reply email to the sender and confirm that the original message and any attachments and copies have been destroyed and deleted from your system.



If you do not wish to receive future unsolicited commercial electronic messages from us, please forward this email to: [hidden email]and include “Unsubscribe” in the subject line. If applicable, you will continue to receive invoices, project communications and similar factual, non-commercial electronic communications.



Please click http://www.woodplc.com/email-disclaimer for notices and company information in relation to emails originating in the UK, Italy or France.



 
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev