[pdal] Debugging std::bad_alloc

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[pdal] Debugging std::bad_alloc

James Harrison
Hi all,

I'm running PDAL in Docker with a simple pipeline to write a single
dimension with the GDAL writer. The LAZ being consumed as input is
PDAL-generated 1.4 LAS, with the source file (LAS 1.2) to that process
coming from the TerraSolid/Bentley Microstation toolset.

I'm using Docker, the latest PDAL image (Docker SHA256 4d1b5..., PDAL
commit aea5bb) and have plenty of disk space free at least in the host's
domain.

All is well on the LAZ side, and ground.laz parses OK (with pdal info,
running in Docker). The LAZ in question has 46,468,399 points and a
number of dimensions. A smaller file runs through this process correctly.

My pipeline is thus:

{
  "pipeline": [
    "/output/ground.laz",
    {
      "type": "writers.gdal",
      "resolution": 0.015,
      "radius": 0.035,
      "window_size": 1,
      "filename": "/output/layer-Red.tif",
      "output_type": "idw",
      "dimension": "Red"
    }
  ]
}

If I run PDAL on this file I get an error after a little while - even
with verbosity 8, nothing is emitted:
[james@hostname laser_coloured_matched]$ docker run -it --rm -v
/tmp/lidar-tmp-1/las2tiff/210000_396000/:/output pdal/pdal:latest pdal
--developer-debug pipeline /output/layer-Red.json
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I'm at a bit of a loss as to how to go about getting more meaningful
error messages out of PDAL, hence the ML post rather than GH issue. Am I
missing a trick to produce more debug messages/output? Is this a known
problem with non-obvious causes like Docker doing something silly with
temporary storage? Or am I being an idiot? :-)
--
Cheers,
James Harrison


_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [pdal] Debugging std::bad_alloc

andrew.bell.ia@gmail.com
Bad alloc means that you're out of memory.  I'm sorry but I don't know anything about configuring swap space for docker, but in order to get things to run, that's what you need.  The raster is kept in memory and if you have a big raster you need lots of memory.

Hope that helps,

On Sat, May 13, 2017 at 12:16 PM, James Harrison <[hidden email]> wrote:
Hi all,

I'm running PDAL in Docker with a simple pipeline to write a single
dimension with the GDAL writer. The LAZ being consumed as input is
PDAL-generated 1.4 LAS, with the source file (LAS 1.2) to that process
coming from the TerraSolid/Bentley Microstation toolset.

I'm using Docker, the latest PDAL image (Docker SHA256 4d1b5..., PDAL
commit aea5bb) and have plenty of disk space free at least in the host's
domain.

All is well on the LAZ side, and ground.laz parses OK (with pdal info,
running in Docker). The LAZ in question has 46,468,399 points and a
number of dimensions. A smaller file runs through this process correctly.

My pipeline is thus:

{
  "pipeline": [
    "/output/ground.laz",
    {
      "type": "writers.gdal",
      "resolution": 0.015,
      "radius": 0.035,
      "window_size": 1,
      "filename": "/output/layer-Red.tif",
      "output_type": "idw",
      "dimension": "Red"
    }
  ]
}

If I run PDAL on this file I get an error after a little while - even
with verbosity 8, nothing is emitted:
[james@hostname laser_coloured_matched]$ docker run -it --rm -v
/tmp/lidar-tmp-1/las2tiff/210000_396000/:/output pdal/pdal:latest pdal
--developer-debug pipeline /output/layer-Red.json
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I'm at a bit of a loss as to how to go about getting more meaningful
error messages out of PDAL, hence the ML post rather than GH issue. Am I
missing a trick to produce more debug messages/output? Is this a known
problem with non-obvious causes like Docker doing something silly with
temporary storage? Or am I being an idiot? :-)
--
Cheers,
James Harrison


_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal



--
Andrew Bell
[hidden email]

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal
Reply | Threaded
Open this post in threaded view
|

Re: [pdal] [Non-DoD Source] Re: Debugging std::bad_alloc

Smith, Michael ERDC-RDE-CRREL-NH
Couldn't this be run in stream mode to reduce the memory usage?  Try adding --stream



Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers
 

From: pdal on behalf of Andrew Bell
Sent: Saturday, May 13, 2017 1:40:06 PM
To: James Harrison
Cc: [hidden email]
Subject: [Non-DoD Source] Re: [pdal] Debugging std::bad_alloc

Bad alloc means that you're out of memory.  I'm sorry but I don't know anything about configuring swap space for docker, but in order to get things to run, that's what you need.  The raster is kept in memory and if you have a big raster you need lots of memory.

Hope that helps,

On Sat, May 13, 2017 at 12:16 PM, James Harrison <[hidden email]> wrote:
Hi all,

I'm running PDAL in Docker with a simple pipeline to write a single
dimension with the GDAL writer. The LAZ being consumed as input is
PDAL-generated 1.4 LAS, with the source file (LAS 1.2) to that process
coming from the TerraSolid/Bentley Microstation toolset.

I'm using Docker, the latest PDAL image (Docker SHA256 4d1b5..., PDAL
commit aea5bb) and have plenty of disk space free at least in the host's
domain.

All is well on the LAZ side, and ground.laz parses OK (with pdal info,
running in Docker). The LAZ in question has 46,468,399 points and a
number of dimensions. A smaller file runs through this process correctly.

My pipeline is thus:

{
  "pipeline": [
    "/output/ground.laz",
    {
      "type": "writers.gdal",
      "resolution": 0.015,
      "radius": 0.035,
      "window_size": 1,
      "filename": "/output/layer-Red.tif",
      "output_type": "idw",
      "dimension": "Red"
    }
  ]
}

If I run PDAL on this file I get an error after a little while - even
with verbosity 8, nothing is emitted:
[james@hostname laser_coloured_matched]$ docker run -it --rm -v
/tmp/lidar-tmp-1/las2tiff/210000_396000/:/output pdal/pdal:latest pdal
--developer-debug pipeline /output/layer-Red.json
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I'm at a bit of a loss as to how to go about getting more meaningful
error messages out of PDAL, hence the ML post rather than GH issue. Am I
missing a trick to produce more debug messages/output? Is this a known
problem with non-obvious causes like Docker doing something silly with
temporary storage? Or am I being an idiot? :-)
--
Cheers,
James Harrison


_______________________________________________
pdal mailing list
[hidden email]
BlockedBlockedhttps://lists.osgeo.org/Blockedmailman/listinfo/pdal



--
Andrew Bell
[hidden email]

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal
Reply | Threaded
Open this post in threaded view
|

Re: [pdal] Debugging std::bad_alloc

andrew.bell.ia@gmail.com
Yes. That may help depending on the memory requirements of the points and that of the raster. You do need to specify bounds if you use writers.gdal in stream mode.

On May 13, 2017 3:31 PM, "Smith, Michael ERDC-RDE-CRREL-NH CIV" <[hidden email]> wrote:
Couldn't this be run in stream mode to reduce the memory usage?  Try adding --stream



Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers
 

From: pdal on behalf of Andrew Bell
Sent: Saturday, May 13, 2017 1:40:06 PM
To: James Harrison
Cc: [hidden email]
Subject: [Non-DoD Source] Re: [pdal] Debugging std::bad_alloc

Bad alloc means that you're out of memory.  I'm sorry but I don't know anything about configuring swap space for docker, but in order to get things to run, that's what you need.  The raster is kept in memory and if you have a big raster you need lots of memory.

Hope that helps,

On Sat, May 13, 2017 at 12:16 PM, James Harrison <[hidden email]> wrote:
Hi all,

I'm running PDAL in Docker with a simple pipeline to write a single
dimension with the GDAL writer. The LAZ being consumed as input is
PDAL-generated 1.4 LAS, with the source file (LAS 1.2) to that process
coming from the TerraSolid/Bentley Microstation toolset.

I'm using Docker, the latest PDAL image (Docker SHA256 4d1b5..., PDAL
commit aea5bb) and have plenty of disk space free at least in the host's
domain.

All is well on the LAZ side, and ground.laz parses OK (with pdal info,
running in Docker). The LAZ in question has 46,468,399 points and a
number of dimensions. A smaller file runs through this process correctly.

My pipeline is thus:

{
  "pipeline": [
    "/output/ground.laz",
    {
      "type": "writers.gdal",
      "resolution": 0.015,
      "radius": 0.035,
      "window_size": 1,
      "filename": "/output/layer-Red.tif",
      "output_type": "idw",
      "dimension": "Red"
    }
  ]
}

If I run PDAL on this file I get an error after a little while - even
with verbosity 8, nothing is emitted:
[james@hostname laser_coloured_matched]$ docker run -it --rm -v
/tmp/lidar-tmp-1/las2tiff/210000_396000/:/output pdal/pdal:latest pdal
--developer-debug pipeline /output/layer-Red.json
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I'm at a bit of a loss as to how to go about getting more meaningful
error messages out of PDAL, hence the ML post rather than GH issue. Am I
missing a trick to produce more debug messages/output? Is this a known
problem with non-obvious causes like Docker doing something silly with
temporary storage? Or am I being an idiot? :-)
--
Cheers,
James Harrison


_______________________________________________
pdal mailing list
[hidden email]
BlockedBlockedhttps://lists.osgeo.org/Blockedmailman/listinfo/pdal



--
Andrew Bell
[hidden email]

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal
Reply | Threaded
Open this post in threaded view
|

Re: [pdal] Debugging std::bad_alloc

James Harrison
On 13/05/2017 21:42, Andrew Bell wrote:

> Yes. That may help depending on the memory requirements of the points
> and that of the raster. You do need to specify bounds if you use
> writers.gdal in stream mode.
>
> On May 13, 2017 3:31 PM, "Smith, Michael ERDC-RDE-CRREL-NH CIV"
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Couldn't this be run in stream mode to reduce the memory usage?  Try
>     adding --stream
>
Didn't know about stream mode - having said that I'm seeing that in my
dataset this is actually being caused by fragmented point cloud data
containing geographically dispersed chunks of points - in other words,
my .las has two chunks of road with a big 5km gap in the middle. No
problem in the vector world but a real pain for raster generation.

I've tried using the cluster filter and the groupby filter in
combination to emit segmented LAZ files I can then iterate over, but the
groupby filter complains the ClusterId dimension is invalid. Is this the
right approach to take? I've considered using the chipper with a
reasonably high point count but I think this would create visual
discontinuities in the output.

--
Cheers,
James Harrison


_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal

signature.asc (836 bytes) Download Attachment