[gdal-dev] Opening gridded xyz data that is out of order

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Opening gridded xyz data that is out of order

Andreas Neumann-4
Hi,

I received a DTM from a Swiss province (OpenData) which is of the
following format:

One coordinate per line, already gridded. See f.e.

2708001.00 1218001.00 1541.52
2708003.00 1218001.00 1542.35
2708005.00 1218001.00 1542.98
2708007.00 1218001.00 1543.58
2708009.00 1218001.00 1544.20
2708011.00 1218001.00 1545.13
2708013.00 1218001.00 1545.88

The issue is that further down the file there seems to be an issue of
ordering the coordinates. gdalinfo produces the following error:

ERROR 1: Ungridded dataset: At line 4125001, change of Y direction
gdalinfo failed - unable to open 'DTM_swissALTI3D_XYZ.txt'.

If GDAL could open the data - it would open as ungridded data - right?
Issue is, this data is already gridded as you can see above. I actually
want to save this is a gridded GeoTIFF file for later usage in QGIS.

Is there still a chance that GDAL can open this data set, even if
coordinates are out of order?

How about the gridded vs ungridded aspect of xyz files?

Thanks,

Andreas



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Andreas Neumann-4
Hi,

Forgot to mention the link to the file, just in case you have time
testing - see https://services.geo.zg.ch/temp/dtmav.zip (138 MB zipped
file).

Thanks,

Andreas


On 11.09.2017 15:43, Andreas Neumann wrote:

> Hi,
>
> I received a DTM from a Swiss province (OpenData) which is of the
> following format:
>
> One coordinate per line, already gridded. See f.e.
>
> 2708001.00 1218001.00 1541.52
> 2708003.00 1218001.00 1542.35
> 2708005.00 1218001.00 1542.98
> 2708007.00 1218001.00 1543.58
> 2708009.00 1218001.00 1544.20
> 2708011.00 1218001.00 1545.13
> 2708013.00 1218001.00 1545.88
>
> The issue is that further down the file there seems to be an issue of
> ordering the coordinates. gdalinfo produces the following error:
>
> ERROR 1: Ungridded dataset: At line 4125001, change of Y direction
> gdalinfo failed - unable to open 'DTM_swissALTI3D_XYZ.txt'.
>
> If GDAL could open the data - it would open as ungridded data - right?
> Issue is, this data is already gridded as you can see above. I
> actually want to save this is a gridded GeoTIFF file for later usage
> in QGIS.
>
> Is there still a chance that GDAL can open this data set, even if
> coordinates are out of order?
>
> How about the gridded vs ungridded aspect of xyz files?
>
> Thanks,
>
> Andreas
>
>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Kurt Schwehr-2
Can you transform the file and swap the y & x when they are in an alternative order with your choice of python/perl/awk?

On Mon, Sep 11, 2017 at 6:51 AM, Andreas Neumann <[hidden email]> wrote:
Hi,

Forgot to mention the link to the file, just in case you have time testing - see https://services.geo.zg.ch/temp/dtmav.zip (138 MB zipped file).

Thanks,

Andreas



On 11.09.2017 15:43, Andreas Neumann wrote:
Hi,

I received a DTM from a Swiss province (OpenData) which is of the following format:

One coordinate per line, already gridded. See f.e.

2708001.00 1218001.00 1541.52
2708003.00 1218001.00 1542.35
2708005.00 1218001.00 1542.98
2708007.00 1218001.00 1543.58
2708009.00 1218001.00 1544.20
2708011.00 1218001.00 1545.13
2708013.00 1218001.00 1545.88

The issue is that further down the file there seems to be an issue of ordering the coordinates. gdalinfo produces the following error:

ERROR 1: Ungridded dataset: At line 4125001, change of Y direction
gdalinfo failed - unable to open 'DTM_swissALTI3D_XYZ.txt'.

If GDAL could open the data - it would open as ungridded data - right? Issue is, this data is already gridded as you can see above. I actually want to save this is a gridded GeoTIFF file for later usage in QGIS.

Is there still a chance that GDAL can open this data set, even if coordinates are out of order?

How about the gridded vs ungridded aspect of xyz files?

Thanks,

Andreas



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Even Rouault-2
In reply to this post by Andreas Neumann-4

On lundi 11 septembre 2017 15:43:47 CEST Andreas Neumann wrote:

> Hi,

>

> I received a DTM from a Swiss province (OpenData) which is of the

> following format:

>

> One coordinate per line, already gridded. See f.e.

>

> 2708001.00 1218001.00 1541.52

> 2708003.00 1218001.00 1542.35

> 2708005.00 1218001.00 1542.98

> 2708007.00 1218001.00 1543.58

> 2708009.00 1218001.00 1544.20

> 2708011.00 1218001.00 1545.13

> 2708013.00 1218001.00 1545.88

>

> The issue is that further down the file there seems to be an issue of

> ordering the coordinates. gdalinfo produces the following error:

>

> ERROR 1: Ungridded dataset: At line 4125001, change of Y direction

> gdalinfo failed - unable to open 'DTM_swissALTI3D_XYZ.txt'.

 

Andreas,

 

the message is probably misleading. It means that from the XYZ point of view, it cannot handle the dataset since it doesn't match its expectations: ie successive lines with same Y and constant positive X step, and when changing lines, with a constant positive or negative Y step. The driver says probably a bit quickly that the dataset is ungridded.

 

>

> If GDAL could open the data - it would open as ungridded data - right?

> Issue is, this data is already gridded as you can see above. I actually

> want to save this is a gridded GeoTIFF file for later usage in QGIS.

>

> Is there still a chance that GDAL can open this data set, even if

> coordinates are out of order?

 

Not by the XYZ driver in its current design where it tries to not ingest the full file into memory. It does a preliminary scan to check that the file organization meets its criteria, but then reads its progressively efficiently. One could perhaps have a more relaxed mode that would require more memory and/or be less I/O efficient for datasets such as the one you tried.

 

You might try to open this file with the CSV driver and use gdal_grid (with a search radius of 2 since this apparently the horizontal resolution) to produce a GeoTIFF. You might possibly do a preliminary conversions of the CSV to shapefile with spatial index or GPKG to improve performance of gdal_grid.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

mj10777
You must sort the data beforehand
- the y position must be sorted properly (ASC or DESC)

Standard linux sort program:

sort -k2 -n -k1 392_5810.xyz -o 392_5810.sort.xyz

(2nd column (y) as numeric, then first column (x) -o = output file

As a zip file you can also do:

unzip -p 390_5820.zip | sort -k2 -n -k1 -o ../xyz/390_5820.dhhn92.txt

that would deal with problem in 1 step.

ogr needs to know the range/area to build (minx,miny,maxx, maxy), so when the y switches it assumes a new column.

Mark

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Andreas Neumann-4

Hi Mark (and others),

Your sort command worked like a charm. It was also very fast!

Now gdalinfo produces this nice information output:

Driver: XYZ/ASCII Gridded XYZ
Files: dtmav_sorted.xyz
Size is 5000, 5000
Coordinate System is `'
Origin = (2708000.000000000000000,1210000.000000000000000)
Pixel Size = (2.000000000000000,2.000000000000000)
Corner Coordinates:
Upper Left  ( 2708000.000, 1210000.000)
Lower Left  ( 2708000.000, 1220000.000)
Upper Right ( 2718000.000, 1210000.000)
Lower Right ( 2718000.000, 1220000.000)
Center      ( 2713000.000, 1215000.000)
Band 1 Block=5000x1 Type=Float32, ColorInterp=Undefined
  Min=728.210 Max=2294.290
  NoData Value=0

Now I can open this in QGIS as a normal data source and simply safe as a GeoTIFF file.

Thanks a lot to all who answered so quickly!

Glad that a simple sort command on the command line helped to solve the issue! I will contact the data provider and ask them if they could deliver this data in an already sorted form so that GDAL recognizes it as gridded data.

Andreas


On 11.09.2017 16:12, Mark Johnson wrote:
You must sort the data beforehand
- the y position must be sorted properly (ASC or DESC)

Standard linux sort program:

sort -k2 -n -k1 392_5810.xyz -o 392_5810.sort.xyz

(2nd column (y) as numeric, then first column (x) -o = output file

As a zip file you can also do:

unzip -p 390_5820.zip | sort -k2 -n -k1 -o ../xyz/390_5820.dhhn92.txt

that would deal with problem in 1 step.

ogr needs to know the range/area to build (minx,miny,maxx, maxy), so when the y switches it assumes a new column.

Mark


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

mj10777
In reply to this post by Even Rouault-2
This will do the same, just replace the EPSG code number:

gdal_translate -ot Float32 -a_srs EPSG:25833 ../xyz/390_5820.dhhn92.txt ../2007.390_5820.dhhn92.25833.tif

Mark

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Joaquim Luis
In reply to this post by Andreas Neumann-4
You could have done it with GMT as well

gmtinfo DTM_swissALTI3D_XYZ.txt -I2
-R2708001/2717999/1210001/1219999

xyz2grd -R2708001/2717999/1210001/1219999 -I2 -GDTM_swissALTI3D_XYZ.grd DTM_swissALTI3D_XYZ.txt

and you get a netCDF grid (it can do GeoTIFs too but would need to go see the docs)

The grid misses a tile of data at the lower right corner 

Joaquim

Hi Mark (and others),

Your sort command worked like a charm. It was also very fast!

Now gdalinfo produces this nice information output:

Driver: XYZ/ASCII Gridded XYZ
Files: dtmav_sorted.xyz
Size is 5000, 5000
Coordinate System is `'
Origin = (2708000.000000000000000,1210000.000000000000000)
Pixel Size = (2.000000000000000,2.000000000000000)
Corner Coordinates:
Upper Left  ( 2708000.000, 1210000.000)
Lower Left  ( 2708000.000, 1220000.000)
Upper Right ( 2718000.000, 1210000.000)
Lower Right ( 2718000.000, 1220000.000)
Center      ( 2713000.000, 1215000.000)
Band 1 Block=5000x1 Type=Float32, ColorInterp=Undefined
  Min=728.210 Max=2294.290
  NoData Value=0

Now I can open this in QGIS as a normal data source and simply safe as a GeoTIFF file.

Thanks a lot to all who answered so quickly!

Glad that a simple sort command on the command line helped to solve the issue! I will contact the data provider and ask them if they could deliver this data in an already sorted form so that GDAL recognizes it as gridded data.

Andreas


On 11.09.2017 16:12, Mark Johnson wrote:
You must sort the data beforehand
- the y position must be sorted properly (ASC or DESC)

Standard linux sort program:

sort -k2 -n -k1 392_5810.xyz -o 392_5810.sort.xyz

(2nd column (y) as numeric, then first column (x) -o = output file

As a zip file you can also do:

unzip -p 390_5820.zip | sort -k2 -n -k1 -o ../xyz/390_5820.dhhn92.txt

that would deal with problem in 1 step.

ogr needs to know the range/area to build (minx,miny,maxx, maxy), so when the y switches it assumes a new column.

Mark


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev





_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Andreas Neumann-4
In reply to this post by mj10777

Yes - I am aware of gdal_translate and gdalwarp to translate formats.

Thanks a lot. I am very happy that this was so easy to solve and now I can use this data in QGIS.

I think the data provider just concatenated chunks of existing files and did not care about the order. I will ask them to run the magic sort command you provided before delivering the data to their clients.

Thanks,
Andreas

On 11.09.2017 16:44, Mark Johnson wrote:
This will do the same, just replace the EPSG code number:

gdal_translate -ot Float32 -a_srs EPSG:25833 ../xyz/390_5820.dhhn92.txt ../2007.390_5820.dhhn92.25833.tif

Mark


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Andreas Neumann-4
In reply to this post by Joaquim Luis

Hi Joaquim,

Interesting. Thanks for sharing. As always with OpenSource there are more solutions than just one ;-)

Yes, I am aware about the missing data in the lower right corner - this is outside of the province and outside of my area of interest.

Thanks and greetings,

Andreas


On 11.09.2017 16:45, Joaquim Luis wrote:
You could have done it with GMT as well

gmtinfo DTM_swissALTI3D_XYZ.txt -I2
-R2708001/2717999/1210001/1219999

xyz2grd -R2708001/2717999/1210001/1219999 -I2 -GDTM_swissALTI3D_XYZ.grd DTM_swissALTI3D_XYZ.txt

and you get a netCDF grid (it can do GeoTIFs too but would need to go see the docs)

The grid misses a tile of data at the lower right corner 

Joaquim

Hi Mark (and others),

Your sort command worked like a charm. It was also very fast!

Now gdalinfo produces this nice information output:

Driver: XYZ/ASCII Gridded XYZ
Files: dtmav_sorted.xyz
Size is 5000, 5000
Coordinate System is `'
Origin = (2708000.000000000000000,1210000.000000000000000)
Pixel Size = (2.000000000000000,2.000000000000000)
Corner Coordinates:
Upper Left  ( 2708000.000, 1210000.000)
Lower Left  ( 2708000.000, 1220000.000)
Upper Right ( 2718000.000, 1210000.000)
Lower Right ( 2718000.000, 1220000.000)
Center      ( 2713000.000, 1215000.000)
Band 1 Block=5000x1 Type=Float32, ColorInterp=Undefined
  Min=728.210 Max=2294.290
  NoData Value=0

Now I can open this in QGIS as a normal data source and simply safe as a GeoTIFF file.

Thanks a lot to all who answered so quickly!

Glad that a simple sort command on the command line helped to solve the issue! I will contact the data provider and ask them if they could deliver this data in an already sorted form so that GDAL recognizes it as gridded data.

Andreas


On 11.09.2017 16:12, Mark Johnson wrote:
You must sort the data beforehand
- the y position must be sorted properly (ASC or DESC)

Standard linux sort program:

sort -k2 -n -k1 392_5810.xyz -o 392_5810.sort.xyz

(2nd column (y) as numeric, then first column (x) -o = output file

As a zip file you can also do:

unzip -p 390_5820.zip | sort -k2 -n -k1 -o ../xyz/390_5820.dhhn92.txt

that would deal with problem in 1 step.

ogr needs to know the range/area to build (minx,miny,maxx, maxy), so when the y switches it assumes a new column.

Mark


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev






_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Opening gridded xyz data that is out of order

Richard Barnes-2
In reply to this post by Andreas Neumann-4
I wonder if GDAL shouldn't suggest sorting as a means of fixing the issue, possibly with Mark's command? 

On Sep 11, 2017 7:32 AM, "Andreas Neumann" <[hidden email]> wrote:

Hi Mark (and others),

Your sort command worked like a charm. It was also very fast!

Now gdalinfo produces this nice information output:

Driver: XYZ/ASCII Gridded XYZ
Files: dtmav_sorted.xyz
Size is 5000, 5000
Coordinate System is `'
Origin = (2708000.000000000000000,1210000.000000000000000)
Pixel Size = (2.000000000000000,2.000000000000000)
Corner Coordinates:
Upper Left  ( 2708000.000, 1210000.000)
Lower Left  ( 2708000.000, 1220000.000)
Upper Right ( 2718000.000, 1210000.000)
Lower Right ( 2718000.000, 1220000.000)
Center      ( 2713000.000, 1215000.000)
Band 1 Block=5000x1 Type=Float32, ColorInterp=Undefined
  Min=728.210 Max=2294.290
  NoData Value=0

Now I can open this in QGIS as a normal data source and simply safe as a GeoTIFF file.

Thanks a lot to all who answered so quickly!

Glad that a simple sort command on the command line helped to solve the issue! I will contact the data provider and ask them if they could deliver this data in an already sorted form so that GDAL recognizes it as gridded data.

Andreas


On 11.09.2017 16:12, Mark Johnson wrote:
You must sort the data beforehand
- the y position must be sorted properly (ASC or DESC)

Standard linux sort program:

sort -k2 -n -k1 392_5810.xyz -o 392_5810.sort.xyz

(2nd column (y) as numeric, then first column (x) -o = output file

As a zip file you can also do:

unzip -p 390_5820.zip | sort -k2 -n -k1 -o ../xyz/390_5820.dhhn92.txt

that would deal with problem in 1 step.

ogr needs to know the range/area to build (minx,miny,maxx, maxy), so when the y switches it assumes a new column.

Mark


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev