[pdal] Reading/Writing to a memory stream

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[pdal] Reading/Writing to a memory stream

jasonbeverage
Hi all!

Is there any way to have PDAL read or write LAZ format from/to a memory stream or would the best way be to read and write to a file and then just load the data into memory myself.

For example, say I wanted to store laz tiles in an sqlite database instead of on disk similar to how mbtiles works.

I'm just doing some experiments now and was curious if there was any built in capability for this.

Jason

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [pdal] Reading/Writing to a memory stream

Andrew Bell
Hi Jason,

It depends exactly what you want.  Our entwine software writes points in LAZ format in blocks using the compressor that's part of PDAL.  However, it's not writing straight LAZ files.  It's just writing the point data, not the header or the VLRs, etc.  Take a look at https://github.com/connormanning/entwine/blob/master/entwine/util/compression.cpp

We write points to databases natively, if that's what you want, but not LAZ tiles, as such.

If you want to write the entire LAZ file to a database, I don't know that we have anything that allows this directly.  You can certainly write a file, mmap the file and then just do a direct copy of the memory to whatever output source you'd like.  Most of the time, the OS won't even flush to physical disk unless you're short memory or force it with some flush()/sync() call, so there isn't much overhead.  Internally, the LAS/LAZ writer uses a std::ostream, which you could theoretically carve out of heap if you knew the size in advance (you don't, unless you want to be really pessimistic).  LAZ typically is written in chunks of points, so the memory can be limited, but if you want a ton of points written out without worrying about the LAZ chunking and such, you might have to allocate a bunch of memory.  Sometimes we do this on other formats because we have no choice, but that may not fit with your plan.

On Fri, Aug 11, 2017 at 11:14 AM, Jason Beverage <[hidden email]> wrote:
Hi all!

Is there any way to have PDAL read or write LAZ format from/to a memory stream or would the best way be to read and write to a file and then just load the data into memory myself.

For example, say I wanted to store laz tiles in an sqlite database instead of on disk similar to how mbtiles works.

I'm just doing some experiments now and was curious if there was any built in capability for this.

Jason

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal



--
Andrew Bell
[hidden email]

_______________________________________________
pdal mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pdal
Loading...