[gdal-dev] Gdal and Google's OSS Fuzzing project

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Gdal and Google's OSS Fuzzing project

Jesse McGraw
I think the gdal suite would be a perfect candidate for this project from google.  Is anyone interested in trying to integrate gdal into it?

https://opensource.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html?m=1
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Kurt Schwehr-2
Yup... https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046495.html

I'd be happy if anyone else wanted to take lead on it.


Since I'm internal to google, I've been running some fuzzer targets against gdal behind the scenes and used the results to fix a number of bugs.  I've added a number of fuzz targets to https://github.com/schwehr/gdal-autotest2/tree/master/cpp and modified GDAL to make fuzzing more productive... e.g. 

https://trac.osgeo.org/gdal/changeset/37592/ adds FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to a driver

I have ~50 bugs that I haven't gotten to.

On Mon, May 8, 2017 at 11:46 AM, Jesse McGraw <[hidden email]> wrote:
I think the gdal suite would be a perfect candidate for this project from google.  Is anyone interested in trying to integrate gdal into it?

https://opensource.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html?m=1
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Jesse McGraw
Nice to see that I'm only 3 weeks behind the curve!

On May 8, 2017, at 2:58 PM, Kurt Schwehr <[hidden email]> wrote:

Yup... https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046495.html

I'd be happy if anyone else wanted to take lead on it.


Since I'm internal to google, I've been running some fuzzer targets against gdal behind the scenes and used the results to fix a number of bugs.  I've added a number of fuzz targets to https://github.com/schwehr/gdal-autotest2/tree/master/cpp and modified GDAL to make fuzzing more productive... e.g. 

https://trac.osgeo.org/gdal/changeset/37592/ adds FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to a driver

I have ~50 bugs that I haven't gotten to.

On Mon, May 8, 2017 at 11:46 AM, Jesse McGraw <[hidden email]> wrote:
I think the gdal suite would be a perfect candidate for this project from google.  Is anyone interested in trying to integrate gdal into it?

https://opensource.googleblog.com/2017/05/oss-fuzz-five-months-later-and.html?m=1
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Mateusz Loskot
In reply to this post by Kurt Schwehr-2
On 8 May 2017 at 20:58, Kurt Schwehr <[hidden email]> wrote:
> Yup... https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046495.html
>
> I'd be happy if anyone else wanted to take lead on it.

I'd really like to, but due to newborn & family duties I'm not going
to promise anything.

> I've added a number of fuzz targets to
> https://github.com/schwehr/gdal-autotest2/tree/master/cpp and modified GDAL
> to make fuzzing more productive... e.g.
>
> https://trac.osgeo.org/gdal/changeset/37592/ adds
> FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to a driver
> https://trac.osgeo.org/gdal/changeset/37909 example fix

The autotest2 efforts are awesome, but huge'ish and without RFC(s)
and lots of work, they won't make it into GDAL any time soon, I suspect.

So, wonder if we could integrate with oss-fuzz at smaller scale:
- create /fuzzer direcotry (next to /gdal  and /autotest)
- port fuzz targets only from Kurt's
https://github.com/schwehr/gdal-autotest2/blob/master/cpp/
- add minimal integration with GDAL build config for Unix

and basically follow
https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Kurt Schwehr-2
My suggestion is to just cherry pick or use as examples the fuzzer targets in autotest2.  Google is okay with them being contributed the core GDAL code base under the GDAL license.  The LLVMFuzzerTestOneInput functions are super simple for drivers and calls that support vsimem.

As for autotest2...

I make assumptions in autotest2 that are just not valid (yet or anytime soon) in GDAL... especially C++11 support and a lot of Google's opensource libraries (e.g. gunit, gmock, logging, absl, bazel, etc.).  I haven't been able to see a near term path of getting autotest2 into mainline GDAL, so I've just not worried about it.  It's such a massive win for me, that it's okay for me now if it's stand alone.  People are welcome to use that code in their projects or as examples of API use as it's all Apache 2.0 licensed.  But if there is desire in the community, I'm definitely open to (and would prefer) if autotest2 was eventually a part of GDAL itself.

And if there are any parts of autotest2 that people want moved into GDAL, I'd be happy to commit them as a contribution to GDAL (it would then be under the GDAL license).  e.g. I find VsiMemTempWrapper and WithQuietHandler super handy for testing and the CHECK's could be ported to GDAL, replaced with something from GDAL, or the class could be changed to expose error reporting.

On Tue, May 9, 2017 at 1:21 PM, Mateusz Loskot <[hidden email]> wrote:
On 8 May 2017 at 20:58, Kurt Schwehr <[hidden email]> wrote:
> Yup... https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046495.html
>
> I'd be happy if anyone else wanted to take lead on it.

I'd really like to, but due to newborn & family duties I'm not going
to promise anything.

> I've added a number of fuzz targets to
> https://github.com/schwehr/gdal-autotest2/tree/master/cpp and modified GDAL
> to make fuzzing more productive... e.g.
>
> https://trac.osgeo.org/gdal/changeset/37592/ adds
> FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to a driver
> https://trac.osgeo.org/gdal/changeset/37909 example fix

The autotest2 efforts are awesome, but huge'ish and without RFC(s)
and lots of work, they won't make it into GDAL any time soon, I suspect.

So, wonder if we could integrate with oss-fuzz at smaller scale:
- create /fuzzer direcotry (next to /gdal  and /autotest)
- port fuzz targets only from Kurt's
https://github.com/schwehr/gdal-autotest2/blob/master/cpp/
- add minimal integration with GDAL build config for Unix

and basically follow
https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Even Rouault-2
In reply to this post by Jesse McGraw

On lundi 8 mai 2017 14:46:45 CEST Jesse McGraw wrote:

> I think the gdal suite would be a perfect candidate for this project from

> google. Is anyone interested in trying to integrate gdal into it?

 

First steps done per

https://trac.osgeo.org/gdal/ticket/6883

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Kurt Schwehr-2
Even and Mateusz,

OMG!  So many fixes... Awesome!  

-kurt



On Wed, May 10, 2017 at 5:55 AM, Even Rouault <[hidden email]> wrote:

On lundi 8 mai 2017 14:46:45 CEST Jesse McGraw wrote:

> I think the gdal suite would be a perfect candidate for this project from

> google. Is anyone interested in trying to integrate gdal into it?

 

First steps done per

https://trac.osgeo.org/gdal/ticket/6883

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Google's OSS Fuzzing project

Mateusz Loskot
I only fixed one so far, bug-security though. 

Even is on rage! Incredible :D

Mateusz 

On 14 May 2017 00:42, "Kurt Schwehr" <[hidden email]> wrote:
Even and Mateusz,

OMG!  So many fixes... Awesome!  

-kurt



On Wed, May 10, 2017 at 5:55 AM, Even Rouault <[hidden email]> wrote:

On lundi 8 mai 2017 14:46:45 CEST Jesse McGraw wrote:

> I think the gdal suite would be a perfect candidate for this project from

> google. Is anyone interested in trying to integrate gdal into it?

 

First steps done per

https://trac.osgeo.org/gdal/ticket/6883

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev