ST_AsGeobuf

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

ST_AsGeobuf

Björn Harrtell
Hi devs,

I've been working on a new output format function called ST_AsGeobuf these last months. Its purpose is to serialize full query output into the binary format Geobuf containing both geometry and attributes. The implementation is tracked by:


Geobuf is a binary format specified by Mapbox that can represent similar data and structure as GeoJSON but is more compact and is faster encode and decode. You can read more about Geobuf at https://github.com/mapbox/geobuf which also serves as the reference implementation to decode at a browser client side.

Geobuf is in turn based on Googles protobuf standard which I think is similar to a binary XML. My implementation uses the general purpose library protobuf-c to simplify the encoding.

I hope to be able use this work as a base to also implement support for vector tiles as specified by Mapbox which is also based on protobuf.

I think the implementation is ready for review and testing. If you have any interest in Geobuf or vector tiles output support I'd appreciate review and/or test result feedback.

/Björn

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: ST_AsGeobuf

Paul Norman
On 9/12/2016 12:59 AM, Björn Harrtell wrote:

> Hi devs,
>
> I've been working on a new output format function called ST_AsGeobuf
> these last months. Its purpose is to serialize full query output into
> the binary format Geobuf containing both geometry and attributes. The
> implementation is tracked by:
>
> https://trac.osgeo.org/postgis/ticket/3599
> https://git.osgeo.org/gogs/postgis/postgis/pulls/4 or alternatively
> https://github.com/bjornharrtell/postgis/tree/asgeobuf5
>
> Geobuf is a binary format specified by Mapbox that can represent
> similar data and structure as GeoJSON but is more compact and is
> faster encode and decode. You can read more about Geobuf at
> https://github.com/mapbox/geobuf which also serves as the reference
> implementation to decode at a browser client side.
>
> Geobuf is in turn based on Googles protobuf standard which I think is
> similar to a binary XML. My implementation uses the general purpose
> library protobuf-c to simplify the encoding.
>
> I hope to be able use this work as a base to also implement support
> for vector tiles as specified by Mapbox which is also based on protobuf.
>
> I think the implementation is ready for review and testing. If you
> have any interest in Geobuf or vector tiles output support I'd
> appreciate review and/or test result feedback.
>

The MapBox geobuf page states "Note that the encoding schema is not
stable yet". Is it premature to put this in PostGIS?
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

geobuf med mera

nicklas
In reply to this post by Björn Harrtell
Hej Björn

Så kul att du har gett dig på C-kodning på allvar :-)

Hur står det till?

Jag hörde på Bo Overgaard  på Foss4gNOR i Oslo i början av september.
Han sa att du skulle börja hos dem. Spännande! Har du redan lämnat Sweco
eller?

Har du haft nån mer kontakt med Carto-folket i Madrid? Jag blev bara
bortkopplad från Slacken deras plötsligt så jag har inte pratat med dem
på länge.

Spännande med geobuf-projektet. Jag har tänkt några gånger att jag
skulle sätta mig in i det och jämföra med twkb på geometri-delen, men
det har inte blivit av. Men utifrån https://github.com/mapbox/geobuf så
verkar det ju packa data extremt bra. Men det är väl någon liknande
teknik som twkb antar jag? Med delta-enkodning och variable integer.

Jag skall sätta mig in i det. Jag och en kompanjon har startat eget
företag och hoppas på att kunna försörja oss på utveckling av appar för
skogsbruket. Jag har skissat en hel del på plattform kring
twkb-formatet, men bör naturligtvis ta en ordentlig titt på geobuf
också. Det jag inte tycker om med format som lagrar alla attribut och
geometrier är att det ofta blir dåliga datamodeller. Det behöver
naturligtvis inte hänga ihop. Men resutlatet blir ofta att man lagrar
alla attribut tillsammans med geometrien for att det ger lätt åtkomst.
Man ignorerar då hela processen med normalisering av data-modellen och
identifikation av hur data faktiskt relaterar till varandra.

Jag har hållit på en del i det sista med en id±e om att packa twkb i
sqlite-databaser och rendera i openGL på klienten. Har bara kunnat
provköra i linux än. Må läsa upp mig på Android och jni för att få fart
på det i telefonen. Men jag har tro på det. Packar data och även
information om trianglar i polygoner så det går raskt att rendera. Men
har testat i Rapberry och det går ganska långsamt. Vet inte om det beror
på långsam läsning från minnekortet eller slö processor eller helt
enkelt att det är ARM-arkitektur. Omd et handlar om ARM, så går det helt
säkert att optimalisera en hel del.

Om du skulle få lust att testa kartklienten så har jag färdigpackad data
för hela Norge.

Min presentation av det hela på Foss4gNOR
https://vimeo.com/album/4125736/video/181133943

koden till klienten
https://github.com/nicklasaven/noTile

Färdigpackade kartor 1:50000 för norge (ca 2,4 GB)
http://twkb.jordogskog.no/maps


Skall du flytta till Danmark nu eller kommer du att pendla?

Vi hörs

Hälsningar

Nicklas Avén

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: geobuf med mera

nicklas
Sorry about the noise

Sent to wrong address

Regards

Nicklas



_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: ST_AsGeobuf

Sandro Santilli-4
In reply to this post by Björn Harrtell
On Mon, Sep 12, 2016 at 09:59:34AM +0200, Björn Harrtell wrote:

> https://trac.osgeo.org/postgis/ticket/3599
> https://git.osgeo.org/gogs/postgis/postgis/pulls/4 or alternatively

[...]

> I think the implementation is ready for review and testing. If you have any
> interest in Geobuf or vector tiles output support I'd appreciate review
> and/or test result feedback.

Would you consider extending the build-test Docker image used by
Drone/Gogs to run automated tests ? It would provide another automated
test for the implementation...

See
https://git.osgeo.org/gogs/postgis/postgis-docker
https://git.osgeo.org/gogs/postgis/postgis-docker/src/master/build-test/Dockerfile.trisquel

--strk;
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: ST_AsGeobuf

Björn Harrtell
In reply to this post by Paul Norman
2016-09-12 10:46 GMT+02:00 Paul Norman <[hidden email]>:
On 9/12/2016 12:59 AM, Björn Harrtell wrote:
Hi devs,

I've been working on a new output format function called ST_AsGeobuf these last months. Its purpose is to serialize full query output into the binary format Geobuf containing both geometry and attributes. The implementation is tracked by:

https://trac.osgeo.org/postgis/ticket/3599
https://git.osgeo.org/gogs/postgis/postgis/pulls/4 or alternatively https://github.com/bjornharrtell/postgis/tree/asgeobuf5

Geobuf is a binary format specified by Mapbox that can represent similar data and structure as GeoJSON but is more compact and is faster encode and decode. You can read more about Geobuf at https://github.com/mapbox/geobuf which also serves as the reference implementation to decode at a browser client side.

Geobuf is in turn based on Googles protobuf standard which I think is similar to a binary XML. My implementation uses the general purpose library protobuf-c to simplify the encoding.

I hope to be able use this work as a base to also implement support for vector tiles as specified by Mapbox which is also based on protobuf.

I think the implementation is ready for review and testing. If you have any interest in Geobuf or vector tiles output support I'd appreciate review and/or test result feedback.


The MapBox geobuf page states "Note that the encoding schema is not stable yet". Is it premature to put this in PostGIS?

Subjective, I think. I think Mapbox just want to disclaim backwards compatibility in the eventual next major version. Obviously I find it useful right now and do think it can be for others too. If needed, PostGIS could support both the current spec and future ones by parameterization, or drop support if deemed obsolete.

As far as I know, Mapbox want to evolve the spec so that it can support encoding/decoding as a stream, but there is no telling when/if that work will be completed.

And as a final observation, if TWKB could be accepted I think Geobuf should be. :)

/Björn

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: ST_AsGeobuf

Javier Santana-2
Hey, 

To give some context here about TWKB vs Geobuf: before actually pushing TWKB to a postgis release we spent a pretty interesting amount of time on closing the spec (specially Nicklass and Paul) and working on the implementation so no backwards compatibility changes could happen.

Geobuf is a "format" with no actual closed spec and controlled by just a single company, they will not mind about changing the spec without any prior notice (likely not going to happen tho)

Not agains this patch or anything like that, but just to clarify that there are actual differences between both formats.









On Mon, Sep 12, 2016 at 2:41 PM Björn Harrtell <[hidden email]> wrote:
2016-09-12 10:46 GMT+02:00 Paul Norman <[hidden email]>:
On 9/12/2016 12:59 AM, Björn Harrtell wrote:
Hi devs,

I've been working on a new output format function called ST_AsGeobuf these last months. Its purpose is to serialize full query output into the binary format Geobuf containing both geometry and attributes. The implementation is tracked by:

https://trac.osgeo.org/postgis/ticket/3599
https://git.osgeo.org/gogs/postgis/postgis/pulls/4 or alternatively https://github.com/bjornharrtell/postgis/tree/asgeobuf5

Geobuf is a binary format specified by Mapbox that can represent similar data and structure as GeoJSON but is more compact and is faster encode and decode. You can read more about Geobuf at https://github.com/mapbox/geobuf which also serves as the reference implementation to decode at a browser client side.

Geobuf is in turn based on Googles protobuf standard which I think is similar to a binary XML. My implementation uses the general purpose library protobuf-c to simplify the encoding.

I hope to be able use this work as a base to also implement support for vector tiles as specified by Mapbox which is also based on protobuf.

I think the implementation is ready for review and testing. If you have any interest in Geobuf or vector tiles output support I'd appreciate review and/or test result feedback.


The MapBox geobuf page states "Note that the encoding schema is not stable yet". Is it premature to put this in PostGIS?

Subjective, I think. I think Mapbox just want to disclaim backwards compatibility in the eventual next major version. Obviously I find it useful right now and do think it can be for others too. If needed, PostGIS could support both the current spec and future ones by parameterization, or drop support if deemed obsolete.

As far as I know, Mapbox want to evolve the spec so that it can support encoding/decoding as a stream, but there is no telling when/if that work will be completed.

And as a final observation, if TWKB could be accepted I think Geobuf should be. :)

/Björn

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: ST_AsGeobuf

Björn Harrtell
Agreed, I wasn't being totally serious about the comparison.

My main point is that regardless of the specification stability and intentions of Mapbox the format is useful and has been for upwards a year. For some use cases there are no good alternatives, in my opinion. Also consider that it's free under the ISC license.

In either case, I still would like review and test feedback as it would be helpful as input towards vector tile output support, being perhaps a more stable and generally useful output format.

/Björn

2016-09-12 15:20 GMT+02:00 Javier Santana <[hidden email]>:
Hey, 

To give some context here about TWKB vs Geobuf: before actually pushing TWKB to a postgis release we spent a pretty interesting amount of time on closing the spec (specially Nicklass and Paul) and working on the implementation so no backwards compatibility changes could happen.

Geobuf is a "format" with no actual closed spec and controlled by just a single company, they will not mind about changing the spec without any prior notice (likely not going to happen tho)

Not agains this patch or anything like that, but just to clarify that there are actual differences between both formats.


On Mon, Sep 12, 2016 at 2:41 PM Björn Harrtell <[hidden email]> wrote:
2016-09-12 10:46 GMT+02:00 Paul Norman <[hidden email]>:
On 9/12/2016 12:59 AM, Björn Harrtell wrote:
Hi devs,

I've been working on a new output format function called ST_AsGeobuf these last months. Its purpose is to serialize full query output into the binary format Geobuf containing both geometry and attributes. The implementation is tracked by:

https://trac.osgeo.org/postgis/ticket/3599
https://git.osgeo.org/gogs/postgis/postgis/pulls/4 or alternatively https://github.com/bjornharrtell/postgis/tree/asgeobuf5

Geobuf is a binary format specified by Mapbox that can represent similar data and structure as GeoJSON but is more compact and is faster encode and decode. You can read more about Geobuf at https://github.com/mapbox/geobuf which also serves as the reference implementation to decode at a browser client side.

Geobuf is in turn based on Googles protobuf standard which I think is similar to a binary XML. My implementation uses the general purpose library protobuf-c to simplify the encoding.

I hope to be able use this work as a base to also implement support for vector tiles as specified by Mapbox which is also based on protobuf.

I think the implementation is ready for review and testing. If you have any interest in Geobuf or vector tiles output support I'd appreciate review and/or test result feedback.


The MapBox geobuf page states "Note that the encoding schema is not stable yet". Is it premature to put this in PostGIS?

Subjective, I think. I think Mapbox just want to disclaim backwards compatibility in the eventual next major version. Obviously I find it useful right now and do think it can be for others too. If needed, PostGIS could support both the current spec and future ones by parameterization, or drop support if deemed obsolete.

As far as I know, Mapbox want to evolve the spec so that it can support encoding/decoding as a stream, but there is no telling when/if that work will be completed.

And as a final observation, if TWKB could be accepted I think Geobuf should be. :)

/Björn

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel