[gdal-dev] About CMake build again

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] About CMake build again

Dmitry Baryshnikov-2

Hi,

As Even said it make sense to move discussion from this ticket (https://trac.osgeo.org/gdal/ticket/7080) to the list.

First of all I would like to make small introduction to Borsch project. Here it is some useful links:

* Borsch repository: https://github.com/nextgis-borsch/borsch

* Presentation on FOSS4G 2016: http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf

* GDAL repository adapted for Borsch: https://github.com/nextgis-borsch/lib_gdal

Shortly speaking Borsch is 3 CMake scripts which add ability to include dependence library in one line of code.

It looks like:

find_anyproject(TIFF REQUIRED)

Certainly developer can add additional parameter to configure dependency in find_anyproject function.

Inside  find_anyproject work 2 cases:

1. First of all (by default, but can be overridden) CMake searches host system for dependency library (headers and lib files). This is usual CMake find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html) with some additional logic to try different options for hard cases (like Qt).

2. If library not found, find_anyproject can get the sources or prebuild library from github.

So the GDAL or any other library can be build the normal way, but developer have additional features to configure build all libraries with one compiler and one set of compiler/linker settings (with some limits). Such way we can have rather complicated scenarios to build GDAL and dependencies.

Here it is several examples of benefits of this approach:

1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64 architecture). I build all dependencies include GDAL statically and link in one fat library. The all code that do it: https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236

By the way the library also builds on Linux and Mac OS (Windows under development) and CMake try to use existed in host system libraries. If CMake find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... ) will be ignored as it already build with another parameters.

2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa: https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules

3. Build QGIS (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149), PostGIS (https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165), Formbuilder (https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)

This is main Borsch features.


There are some additional conventions like:

    * I modify all libraries included into Borsch repository to install on unix-like paths. For Linux this is usual, for Windows and Mac OS this let us to use Qt installer framework an install software mach similar like on Linux. This is about target "install" which is vary on different libraries (CMake has it own conventions about it). This is not mandatory for Borsch itself but useful. CMake can register installed libraries in system repository to simplify find them in find_package function.

    * CMake get library version from sources in all libraries included into Borsch (if applicable, otherwise set it in CMake script). This is necessary if exact version of library needed. This is not mandatory. One more benefit during building process we can see dependency library version in console.

    * We modify all libraries included into Borsch repository to find dependencies using find_anyproject. It is simple to use libraries from our borsch repository, but developer can fork them or use any other sources and build systems to have dependency library in it's host system.

One can see this is all very flexible.


What about GDAL.

1. After unification GDALDataset and OGRDatasource current sources tree is not fit for this new logic of GDAL classes. I rearranged sources more closer to GDAL classes and CMake needs. Main changes are moving raster and vector drivers inside drivers folder (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This simplify situation where different drivers need the same dependency library (libpg, libsqlite, etc.). Also there are several raster/vector drivers which need a separate directory but now presented in ogr or frmts directories. There are some bad decisions I made - for example I moved unit tests into separate repository - this was a mistake. We will return unit tests back to GDAL repository.

An example of cmake friendly way see https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt. The driver developer must only create new folder and put CMakeLists.txt file into it. The upper CMake script will find new driver and add it to GDAL build. In common cases no need to modify upper CMake scripts.

2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG etc). All this code are in separate repositories. I don't see the difference to get this code from git pull from main GDAL repository or from the separate repository via find_anyproject process. Current GDAL repository looks like the https://github.com/nextgis-borsch packed in one repository.


In conclusion:

1. Borsch added flexible and useful features and not remove existing approach

2. The current cmaked GDAL are in production in my company more than a year on Windows, Linix, Mac OS, iOS, Android.

3. I'm ready to discuss and improve current solution. Any help are welcome

4. I also will be happy to contribute directly or via PR into GDAL trunk instead of backporting in both directions improvements that we do in GDAL .


Finally:

Find the link to page with the CMake in GDAL discussion - https://trac.osgeo.org/gdal/wiki/CMake

-- 
Best regards,
    Dmitry

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Hiroshi Miura
Hi,

Dmitry, Thank you for starting the thread.
I'm Hiroshi, a newbie as user and dev of GDAL. I'm OSM mapper.

On 2017/10/28 06:06, Dmitry Baryshnikov wrote:

As Even said it make sense to move discussion from this ticket (https://trac.osgeo.org/gdal/ticket/7080) to the list.


The ticket (https://trac.osgeo.org/gdal/ticket/7080) is my proposal.
I put it because there are significant improvements in CMake ecosystem in recent days.

-  CMake support in Visual Studio 2017.
  (https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/)
      VS17 bundles CMake 3.9 into their tool chains.
      It helps editing code and debugging a C/C++ project with CMake.

- JetBrains CLion only supports CMake.
   CLion appeared in 2014.   Current CLion 2017.2 bundles CMake 3.8.


When I made a small contribution for GML driver recently,
I had a frustration to make changes on GDAL source tree.
If changing code and  unit test cycle become speedy, it would be better.

It is because I want to improve a development experience using C/C++ IDE.
After short research, I reached a conclusion to use CMake with GDAL source tree,
and work with CLion IDE on my Linux laptop.

Finally:

Find the link to page with the CMake in GDAL discussion - https://trac.osgeo.org/gdal/wiki/CMake


Wiki news point https://github.com/aashish24/gdal-svn/tree/cmake4gdal  as a  repository.

I tried it and catch up current GDAL  tree.
Unfortunately I didn't know Borsch project,
and a dependency issue discussed here in some years ago.

I eventually  updated cmake4gdal scripts to be able to read from CLion IDE on Linux/Windows.
https://github.com/miurahr/gdal/tree/compile_with_cmake

It can build a library with many drivers, python/perl bindings on Linux
and update vagrant script  to prepare clean compilation environment.
It also run autotest from CMake by specifying --target autotest

VS17 preview 2.0 can open this source tree as a VS native project on Windows10.
(compilation is not succeed)

Hiroshi Miura

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

John Daniel-2
In reply to this post by Dmitry Baryshnikov-2
I have my own build system for GDAL and its dependencies. It uses whatever build system each project uses. CMake is, by far, the most problematic. Luckily, I can still use the original autotools-based KML. 

If I understand correctly, CMake is going to require a complete restructuring of the project and a complete dependency on CMake for all dependencies. Can anyone explain how this would improve anything?

With the KML project and others, I have found CMake to be the most complicated and most problematic build system. When there is a problem, I'm helpless to resolve it on my own. The people who made the CMake scripts are equally helpless because they use a different platform. I build on a Mac with Xcode and target both macOS and iOS. Are the maintainers of the GDAL CMake build scripts going to have access to same beta versions of all the above that I am using? 

I'm not an active member of the mailing list. I've been on different projects the past couple of years but was looking forward to working on GDAL-based projects full time next year. CMake will shut me out of the community entirely. I simply don't have time to master CMake just to build projects that had never required a similar level of mastery of autotools. Instead, I will be forced to stay on the last working version I have. I won't be able to contribute any fixes I make or new features like my boost replacement for GEOS and Spatialite.

If some people like CMake and it works on their platforms, great. Then let people choose what build tools they want and what they can support on their own platforms. 

John Daniel
Etresoft, Inc.

Sent from my iPhone

On Oct 27, 2017, at 5:06 PM, Dmitry Baryshnikov <[hidden email]> wrote:

Hi,

As Even said it make sense to move discussion from this ticket (https://trac.osgeo.org/gdal/ticket/7080) to the list.

First of all I would like to make small introduction to Borsch project. Here it is some useful links:

* Borsch repository: https://github.com/nextgis-borsch/borsch

* Presentation on FOSS4G 2016: http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf

* GDAL repository adapted for Borsch: https://github.com/nextgis-borsch/lib_gdal

Shortly speaking Borsch is 3 CMake scripts which add ability to include dependence library in one line of code.

It looks like:

find_anyproject(TIFF REQUIRED)

Certainly developer can add additional parameter to configure dependency in find_anyproject function.

Inside  find_anyproject work 2 cases:

1. First of all (by default, but can be overridden) CMake searches host system for dependency library (headers and lib files). This is usual CMake find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html) with some additional logic to try different options for hard cases (like Qt).

2. If library not found, find_anyproject can get the sources or prebuild library from github.

So the GDAL or any other library can be build the normal way, but developer have additional features to configure build all libraries with one compiler and one set of compiler/linker settings (with some limits). Such way we can have rather complicated scenarios to build GDAL and dependencies.

Here it is several examples of benefits of this approach:

1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64 architecture). I build all dependencies include GDAL statically and link in one fat library. The all code that do it: https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236

By the way the library also builds on Linux and Mac OS (Windows under development) and CMake try to use existed in host system libraries. If CMake find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... ) will be ignored as it already build with another parameters.

2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa: https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules

3. Build QGIS (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149), PostGIS (https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165), Formbuilder (https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)

This is main Borsch features.


There are some additional conventions like:

    * I modify all libraries included into Borsch repository to install on unix-like paths. For Linux this is usual, for Windows and Mac OS this let us to use Qt installer framework an install software mach similar like on Linux. This is about target "install" which is vary on different libraries (CMake has it own conventions about it). This is not mandatory for Borsch itself but useful. CMake can register installed libraries in system repository to simplify find them in find_package function.

    * CMake get library version from sources in all libraries included into Borsch (if applicable, otherwise set it in CMake script). This is necessary if exact version of library needed. This is not mandatory. One more benefit during building process we can see dependency library version in console.

    * We modify all libraries included into Borsch repository to find dependencies using find_anyproject. It is simple to use libraries from our borsch repository, but developer can fork them or use any other sources and build systems to have dependency library in it's host system.

One can see this is all very flexible.


What about GDAL.

1. After unification GDALDataset and OGRDatasource current sources tree is not fit for this new logic of GDAL classes. I rearranged sources more closer to GDAL classes and CMake needs. Main changes are moving raster and vector drivers inside drivers folder (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This simplify situation where different drivers need the same dependency library (libpg, libsqlite, etc.). Also there are several raster/vector drivers which need a separate directory but now presented in ogr or frmts directories. There are some bad decisions I made - for example I moved unit tests into separate repository - this was a mistake. We will return unit tests back to GDAL repository.

An example of cmake friendly way see https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt. The driver developer must only create new folder and put CMakeLists.txt file into it. The upper CMake script will find new driver and add it to GDAL build. In common cases no need to modify upper CMake scripts.

2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG etc). All this code are in separate repositories. I don't see the difference to get this code from git pull from main GDAL repository or from the separate repository via find_anyproject process. Current GDAL repository looks like the https://github.com/nextgis-borsch packed in one repository.


In conclusion:

1. Borsch added flexible and useful features and not remove existing approach

2. The current cmaked GDAL are in production in my company more than a year on Windows, Linix, Mac OS, iOS, Android.

3. I'm ready to discuss and improve current solution. Any help are welcome

4. I also will be happy to contribute directly or via PR into GDAL trunk instead of backporting in both directions improvements that we do in GDAL .


Finally:

Find the link to page with the CMake in GDAL discussion - https://trac.osgeo.org/gdal/wiki/CMake

-- 
Best regards,
    Dmitry
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
In reply to this post by Hiroshi Miura

Hi Hiroshi,

I tried to test you solution:

$ git clone --depth 1 https://github.com/miurahr/gdal.git

$ cd gdal/

$ mkdir build

$ cd build

$ cmake ..

-- The C compiler identification is AppleClang 9.0.0.9000038

[ Skipped ...]

CMake Error at /Applications/CMake.app/Contents/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find QHULL (missing: QHULL_LIBRARY QHULL_INCLUDE_DIR)
Call Stack (most recent call first):
  /Applications/CMake.app/Contents/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:377 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindQHULL.cmake:41 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  gdal/alg/CMakeLists.txt:139 (find_package)


-- Configuring incomplete, errors occurred!


The QHULL is not mandatory for GDAL build and should not stop configuring at that moment.


Fast review you solution:

1 Don't see Python 2/3 support (CMake will choose default Python). How to configure to build Python 2 or Python 3 bindings? Don't see Python bindings install steps.

2. Loose some drivers (for example GPKG) as of result manually added here (https://github.com/miurahr/gdal/blob/compile_with_cmake/gdal/ogr/ogrsf_frmts/CMakeLists.txt) - in compare of my idea automatically add drivers (https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt)

3. In GML driver you add EXPAT and Xerces simultaneously - as I remember this options are mutually exclusive (may be I'm wrong here)

4. There are mandatory drivers which may be disabled in your solution (GeoJSON, ESRI Shape, Map Info ...)

5. There are  drivers which cannot be disabled in your solution but this option must be (ilwis, r, til, saga, couchdb ...)

6. Again dependency problems. For example plscene, plmosaic depends on CURL, but there is no check if CURL is found (https://github.com/miurahr/gdal/blob/compile_with_cmake/gdal/ogr/ogrsf_frmts/plscenes/CMakeLists.txt). Rasterlite ans OSM drivers depends on SQLite, ...

7. Self driver dependency - for example mbtiles depends on GPKG and also on SQLite but no checks exist (https://github.com/miurahr/gdal/blob/compile_with_cmake/gdal/frmts/mbtiles/CMakeLists.txt

8. A lot of command line utilities not build - gdamanage, gdalenhance, gnmmanage, gdalsrsinfo ...)

9. Your make system checks via configure.cmake in port directory (https://github.com/miurahr/gdal/blob/compile_with_cmake/gdal/port/CMakeLists.txt#L32) as the result you hides the variables set by CMake from other parts of build (ogr, frmts, apps, etc).

10. Hardcoded paths - for example https://github.com/miurahr/gdal/blob/compile_with_cmake/CMakeLists.txt#L38

11. No summary output after configure.

12. Install paths on Windows are strange (https://github.com/miurahr/gdal/blob/compile_with_cmake/gdal/CMakeLists.txt#L207-L224). It mast be some logic (as in my solution) or let CMake choose the right paths.


I would like to note that my solution (https://github.com/nextgis-borsch/lib_gdal) successfully builds on Windows (MSVC), Ubuntu, Mac OS, Android, iOS (in our build environment). Also for last two OS you need the special toolchains and dependency must support them too.  All check for dependencies are present in already cmaked drives, all command line utilities build, also bindings and pythons scripts installs in right places. It rather configurable via command line options or CMake GUI. 


Best regards,
    Dmitry
28.10.17 17:52, Hiroshi Miura пишет:
Hi,

Dmitry, Thank you for starting the thread.
I'm Hiroshi, a newbie as user and dev of GDAL. I'm OSM mapper.

On 2017/10/28 06:06, Dmitry Baryshnikov wrote:
As Even said it make sense to move discussion from this ticket (https://trac.osgeo.org/gdal/ticket/7080) to the list.

The ticket (https://trac.osgeo.org/gdal/ticket/7080) is my proposal.
I put it because there are significant improvements in CMake ecosystem in recent days.

-  CMake support in Visual Studio 2017.
  (https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/)
      VS17 bundles CMake 3.9 into their tool chains.
      It helps editing code and debugging a C/C++ project with CMake.

- JetBrains CLion only supports CMake.
   CLion appeared in 2014.   Current CLion 2017.2 bundles CMake 3.8.


When I made a small contribution for GML driver recently,
I had a frustration to make changes on GDAL source tree.
If changing code and  unit test cycle become speedy, it would be better.

It is because I want to improve a development experience using C/C++ IDE.
After short research, I reached a conclusion to use CMake with GDAL source tree,
and work with CLion IDE on my Linux laptop.

Finally:

Find the link to page with the CMake in GDAL discussion - https://trac.osgeo.org/gdal/wiki/CMake

Wiki news point https://github.com/aashish24/gdal-svn/tree/cmake4gdal  as a  repository.

I tried it and catch up current GDAL  tree.
Unfortunately I didn't know Borsch project,
and a dependency issue discussed here in some years ago.

I eventually  updated cmake4gdal scripts to be able to read from CLion IDE on Linux/Windows.
https://github.com/miurahr/gdal/tree/compile_with_cmake

It can build a library with many drivers, python/perl bindings on Linux
and update vagrant script  to prepare clean compilation environment.
It also run autotest from CMake by specifying --target autotest

VS17 preview 2.0 can open this source tree as a VS native project on Windows10.
(compilation is not succeed)

Hiroshi Miura



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

bradh
In reply to this post by Dmitry Baryshnikov-2

Is there a way to invoke that “do all the dependency work for me” (`find_anyproject`) from the cmake command line?

So if I clone your lib_gdal repo, it could build GDAL and any required dependencies?

Brad

 

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2

Hi Brad.

Yes you are right. Just do this steps:

git clone https://github.com/nextgis-borsch/lib_gdal.git gdal
cd gdal
mkdir build
cd build
cmake ..
cmake --build .

Some discussion was in this ticket (https://github.com/nextgis-borsch/lib_gdal/issues/13).

There are 3 scenarios supported by lib_gdal now:

1. Build gdal with all dependencies getting them from github (WITH_EXTERNAL). Preferable for Windows

2. Build gdal using the system libraries. Preferable for Linux

3. Build gdal using the dependency libraries build by user (out of source) and registered in CMake package registry. This is I use now. This script help me to clone all libraries, build them and register them in CMake package registry (https://github.com/nextgis-borsch/borsch/blob/master/opt/tools.py#L134).

Best regards,
    Dmitry
29.10.17 2:39, [hidden email] пишет:
Is there a way to invoke that “do all the dependency work for me” (`find_anyproject`) from the cmake command line?

So if I clone your lib_gdal repo, it could build GDAL and any required dependencies?

Brad

 



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
In reply to this post by John Daniel-2

Hi John,

The CMake don't required complete restructure the GDAL project. But I think this changes needed to fit new classes inheritance introduced in GDAL 2.0. And new sources structure more closer to CMake needs.

Also I would like to note that I have only positive experience with CMake, especially 3.x (where many things made simpler).

For iOS builds I wrap CMake into the XCode project and used it via Carthage (see https://github.com/nextgis/nextgis_datastore/tree/master/opt/ios). This is not just GDAL, but an example how GDAL can be wrap too.

Best regards,
    Dmitry
28.10.17 22:29, John Daniel пишет:
I have my own build system for GDAL and its dependencies. It uses whatever build system each project uses. CMake is, by far, the most problematic. Luckily, I can still use the original autotools-based KML.

If I understand correctly, CMake is going to require a complete restructuring of the project and a complete dependency on CMake for all dependencies. Can anyone explain how this would improve anything?

With the KML project and others, I have found CMake to be the most complicated and most problematic build system. When there is a problem, I'm helpless to resolve it on my own. The people who made the CMake scripts are equally helpless because they use a different platform. I build on a Mac with Xcode and target both macOS and iOS. Are the maintainers of the GDAL CMake build scripts going to have access to same beta versions of all the above that I am using?

I'm not an active member of the mailing list. I've been on different projects the past couple of years but was looking forward to working on GDAL-based projects full time next year. CMake will shut me out of the community entirely. I simply don't have time to master CMake just to build projects that had never required a similar level of mastery of autotools. Instead, I will be forced to stay on the last working version I have. I won't be able to contribute any fixes I make or new features like my boost replacement for GEOS and Spatialite.

If some people like CMake and it works on their platforms, great. Then let people choose what build tools they want and what they can support on their own platforms.

John Daniel
Etresoft, Inc.

Sent from my iPhone

On Oct 27, 2017, at 5:06 PM, Dmitry Baryshnikov <[hidden email][hidden email]> wrote:


Hi,

As Even said it make sense to move discussion from this ticket (https://trac.osgeo.org/gdal/ticket/7080) to the list.

First of all I would like to make small introduction to Borsch project. Here it is some useful links:

* Borsch repository: https://github.com/nextgis-borsch/borsch

* Presentation on FOSS4G 2016: http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf

* GDAL repository adapted for Borsch: https://github.com/nextgis-borsch/lib_gdal

Shortly speaking Borsch is 3 CMake scripts which add ability to include dependence library in one line of code.

It looks like:

find_anyproject(TIFF REQUIRED)

Certainly developer can add additional parameter to configure dependency in find_anyproject function.

Inside  find_anyproject work 2 cases:

1. First of all (by default, but can be overridden) CMake searches host system for dependency library (headers and lib files). This is usual CMake find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html) with some additional logic to try different options for hard cases (like Qt).

2. If library not found, find_anyproject can get the sources or prebuild library from github.

So the GDAL or any other library can be build the normal way, but developer have additional features to configure build all libraries with one compiler and one set of compiler/linker settings (with some limits). Such way we can have rather complicated scenarios to build GDAL and dependencies.

Here it is several examples of benefits of this approach:

1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64 architecture). I build all dependencies include GDAL statically and link in one fat library. The all code that do it: https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236

By the way the library also builds on Linux and Mac OS (Windows under development) and CMake try to use existed in host system libraries. If CMake find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... ) will be ignored as it already build with another parameters.

2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa: https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules

3. Build QGIS (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149), PostGIS (https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165), Formbuilder (https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)

This is main Borsch features.


There are some additional conventions like:

    * I modify all libraries included into Borsch repository to install on unix-like paths. For Linux this is usual, for Windows and Mac OS this let us to use Qt installer framework an install software mach similar like on Linux. This is about target "install" which is vary on different libraries (CMake has it own conventions about it). This is not mandatory for Borsch itself but useful. CMake can register installed libraries in system repository to simplify find them in find_package function.

    * CMake get library version from sources in all libraries included into Borsch (if applicable, otherwise set it in CMake script). This is necessary if exact version of library needed. This is not mandatory. One more benefit during building process we can see dependency library version in console.

    * We modify all libraries included into Borsch repository to find dependencies using find_anyproject. It is simple to use libraries from our borsch repository, but developer can fork them or use any other sources and build systems to have dependency library in it's host system.

One can see this is all very flexible.


What about GDAL.

1. After unification GDALDataset and OGRDatasource current sources tree is not fit for this new logic of GDAL classes. I rearranged sources more closer to GDAL classes and CMake needs. Main changes are moving raster and vector drivers inside drivers folder (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This simplify situation where different drivers need the same dependency library (libpg, libsqlite, etc.). Also there are several raster/vector drivers which need a separate directory but now presented in ogr or frmts directories. There are some bad decisions I made - for example I moved unit tests into separate repository - this was a mistake. We will return unit tests back to GDAL repository.

An example of cmake friendly way see https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt. The driver developer must only create new folder and put CMakeLists.txt file into it. The upper CMake script will find new driver and add it to GDAL build. In common cases no need to modify upper CMake scripts.

2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG etc). All this code are in separate repositories. I don't see the difference to get this code from git pull from main GDAL repository or from the separate repository via find_anyproject process. Current GDAL repository looks like the https://github.com/nextgis-borsch packed in one repository.


In conclusion:

1. Borsch added flexible and useful features and not remove existing approach

2. The current cmaked GDAL are in production in my company more than a year on Windows, Linix, Mac OS, iOS, Android.

3. I'm ready to discuss and improve current solution. Any help are welcome

4. I also will be happy to contribute directly or via PR into GDAL trunk instead of backporting in both directions improvements that we do in GDAL .


Finally:

Find the link to page with the CMake in GDAL discussion - https://trac.osgeo.org/gdal/wiki/CMake

--
Best regards,
    Dmitry

_______________________________________________
gdal-dev mailing list
[hidden email][hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Alexander Bruy
In reply to this post by Dmitry Baryshnikov-2
While I'm not GDAL developer or regular contributor (submitted only
few patches),
but still often build GDAL trunk on WIndows and Linux, and less often on Mac.

Personally I think that solution proposed in
https://trac.osgeo.org/gdal/ticket/7080
is more preferable. It does not require *all* dependencies to be build
with cmake,
it does not require maintaining forks with special directory structure for *all*
dependencies, it plays well with conventions/packages provided on all systems
out of the box. Maybe borsch is good for in-house use when all stack
is build from
scratch but it is not suitable for real-world use cases.

Of course, #7080 is not ideal, there are some issues, but as I understand the
work is not over and most (if not all) issues can be solved.

Just my 2c.

2017-10-28 0:06 GMT+03:00 Dmitry Baryshnikov <[hidden email]>:

> Hi,
>
> As Even said it make sense to move discussion from this ticket
> (https://trac.osgeo.org/gdal/ticket/7080) to the list.
>
> First of all I would like to make small introduction to Borsch project. Here
> it is some useful links:
>
> * Borsch repository: https://github.com/nextgis-borsch/borsch
>
> * Presentation on FOSS4G 2016:
> http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf
>
> * GDAL repository adapted for Borsch:
> https://github.com/nextgis-borsch/lib_gdal
>
> Shortly speaking Borsch is 3 CMake scripts which add ability to include
> dependence library in one line of code.
>
> It looks like:
>
> find_anyproject(TIFF REQUIRED)
>
> Certainly developer can add additional parameter to configure dependency in
> find_anyproject function.
>
> Inside  find_anyproject work 2 cases:
>
> 1. First of all (by default, but can be overridden) CMake searches host
> system for dependency library (headers and lib files). This is usual CMake
> find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html)
> with some additional logic to try different options for hard cases (like
> Qt).
>
> 2. If library not found, find_anyproject can get the sources or prebuild
> library from github.
>
> So the GDAL or any other library can be build the normal way, but developer
> have additional features to configure build all libraries with one compiler
> and one set of compiler/linker settings (with some limits). Such way we can
> have rather complicated scenarios to build GDAL and dependencies.
>
> Here it is several examples of benefits of this approach:
>
> 1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for
> iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64
> architecture). I build all dependencies include GDAL statically and link in
> one fat library. The all code that do it:
> https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236
>
> By the way the library also builds on Linux and Mac OS (Windows under
> development) and CMake try to use existed in host system libraries. If CMake
> find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... )
> will be ignored as it already build with another parameters.
>
> 2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa:
> https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules
>
> 3. Build QGIS
> (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149),
> PostGIS
> (https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165),
> Formbuilder
> (https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)
>
> This is main Borsch features.
>
>
> There are some additional conventions like:
>
>     * I modify all libraries included into Borsch repository to install on
> unix-like paths. For Linux this is usual, for Windows and Mac OS this let us
> to use Qt installer framework an install software mach similar like on
> Linux. This is about target "install" which is vary on different libraries
> (CMake has it own conventions about it). This is not mandatory for Borsch
> itself but useful. CMake can register installed libraries in system
> repository to simplify find them in find_package function.
>
>     * CMake get library version from sources in all libraries included into
> Borsch (if applicable, otherwise set it in CMake script). This is necessary
> if exact version of library needed. This is not mandatory. One more benefit
> during building process we can see dependency library version in console.
>
>     * We modify all libraries included into Borsch repository to find
> dependencies using find_anyproject. It is simple to use libraries from our
> borsch repository, but developer can fork them or use any other sources and
> build systems to have dependency library in it's host system.
>
> One can see this is all very flexible.
>
>
> What about GDAL.
>
> 1. After unification GDALDataset and OGRDatasource current sources tree is
> not fit for this new logic of GDAL classes. I rearranged sources more closer
> to GDAL classes and CMake needs. Main changes are moving raster and vector
> drivers inside drivers folder
> (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This
> simplify situation where different drivers need the same dependency library
> (libpg, libsqlite, etc.). Also there are several raster/vector drivers which
> need a separate directory but now presented in ogr or frmts directories.
> There are some bad decisions I made - for example I moved unit tests into
> separate repository - this was a mistake. We will return unit tests back to
> GDAL repository.
>
> An example of cmake friendly way see
> https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt.
> The driver developer must only create new folder and put CMakeLists.txt file
> into it. The upper CMake script will find new driver and add it to GDAL
> build. In common cases no need to modify upper CMake scripts.
>
> 2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG
> etc). All this code are in separate repositories. I don't see the difference
> to get this code from git pull from main GDAL repository or from the
> separate repository via find_anyproject process. Current GDAL repository
> looks like the https://github.com/nextgis-borsch packed in one repository.
>
>
> In conclusion:
>
> 1. Borsch added flexible and useful features and not remove existing
> approach
>
> 2. The current cmaked GDAL are in production in my company more than a year
> on Windows, Linix, Mac OS, iOS, Android.
>
> 3. I'm ready to discuss and improve current solution. Any help are welcome
>
> 4. I also will be happy to contribute directly or via PR into GDAL trunk
> instead of backporting in both directions improvements that we do in GDAL .
>
>
> Finally:
>
> Find the link to page with the CMake in GDAL discussion -
> https://trac.osgeo.org/gdal/wiki/CMake
>
> --
> Best regards,
>     Dmitry
>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



--
Alexander Bruy
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2

Hi Alexander,

Please read carefully that I wrote before. There are 3 options:

1. Build gdal with all dependencies getting them from github (WITH_EXTERNAL). Preferable for Windows

2. Build gdal using the system libraries. Preferable for Linux

3. Build gdal using the dependency libraries build by user (out of source) and registered in CMake package registry. This is I use now. This script help me to clone all libraries, build them and register them in CMake package registry (https://github.com/nextgis-borsch/borsch/blob/master/opt/tools.py#L134).


Again, this is the options which one can choose. So if building everything from scratch not suits you - just select another option. I introduce presets (now there is only one for option 1 - https://github.com/nextgis-borsch/lib_gdal/issues/14) and this can be expanded for different scenarios. If you need full control on build just provide command line options to CMake and get what you need.

My fast code review solution of Hiroshi shows that it very far from working solution and need much work that already done in lib_gdal from Borsch.

My opinion is that different options is much flexible and suits for many scenarios as very limited solution from https://trac.osgeo.org/gdal/ticket/7080

Best regards,
    Dmitry
29.10.17 9:11, Alexander Bruy пишет:
While I'm not GDAL developer or regular contributor (submitted only
few patches),
but still often build GDAL trunk on WIndows and Linux, and less often on Mac.

Personally I think that solution proposed in
https://trac.osgeo.org/gdal/ticket/7080
is more preferable. It does not require *all* dependencies to be build
with cmake,
it does not require maintaining forks with special directory structure for *all*
dependencies, it plays well with conventions/packages provided on all systems
out of the box. Maybe borsch is good for in-house use when all stack
is build from
scratch but it is not suitable for real-world use cases.

Of course, #7080 is not ideal, there are some issues, but as I understand the
work is not over and most (if not all) issues can be solved.

Just my 2c.

2017-10-28 0:06 GMT+03:00 Dmitry Baryshnikov [hidden email]:
Hi,

As Even said it make sense to move discussion from this ticket
(https://trac.osgeo.org/gdal/ticket/7080) to the list.

First of all I would like to make small introduction to Borsch project. Here
it is some useful links:

* Borsch repository: https://github.com/nextgis-borsch/borsch

* Presentation on FOSS4G 2016:
http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf

* GDAL repository adapted for Borsch:
https://github.com/nextgis-borsch/lib_gdal

Shortly speaking Borsch is 3 CMake scripts which add ability to include
dependence library in one line of code.

It looks like:

find_anyproject(TIFF REQUIRED)

Certainly developer can add additional parameter to configure dependency in
find_anyproject function.

Inside  find_anyproject work 2 cases:

1. First of all (by default, but can be overridden) CMake searches host
system for dependency library (headers and lib files). This is usual CMake
find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html)
with some additional logic to try different options for hard cases (like
Qt).

2. If library not found, find_anyproject can get the sources or prebuild
library from github.

So the GDAL or any other library can be build the normal way, but developer
have additional features to configure build all libraries with one compiler
and one set of compiler/linker settings (with some limits). Such way we can
have rather complicated scenarios to build GDAL and dependencies.

Here it is several examples of benefits of this approach:

1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for
iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64
architecture). I build all dependencies include GDAL statically and link in
one fat library. The all code that do it:
https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236

By the way the library also builds on Linux and Mac OS (Windows under
development) and CMake try to use existed in host system libraries. If CMake
find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... )
will be ignored as it already build with another parameters.

2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa:
https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules

3. Build QGIS
(https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149),
PostGIS
(https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165),
Formbuilder
(https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)

This is main Borsch features.


There are some additional conventions like:

    * I modify all libraries included into Borsch repository to install on
unix-like paths. For Linux this is usual, for Windows and Mac OS this let us
to use Qt installer framework an install software mach similar like on
Linux. This is about target "install" which is vary on different libraries
(CMake has it own conventions about it). This is not mandatory for Borsch
itself but useful. CMake can register installed libraries in system
repository to simplify find them in find_package function.

    * CMake get library version from sources in all libraries included into
Borsch (if applicable, otherwise set it in CMake script). This is necessary
if exact version of library needed. This is not mandatory. One more benefit
during building process we can see dependency library version in console.

    * We modify all libraries included into Borsch repository to find
dependencies using find_anyproject. It is simple to use libraries from our
borsch repository, but developer can fork them or use any other sources and
build systems to have dependency library in it's host system.

One can see this is all very flexible.


What about GDAL.

1. After unification GDALDataset and OGRDatasource current sources tree is
not fit for this new logic of GDAL classes. I rearranged sources more closer
to GDAL classes and CMake needs. Main changes are moving raster and vector
drivers inside drivers folder
(https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This
simplify situation where different drivers need the same dependency library
(libpg, libsqlite, etc.). Also there are several raster/vector drivers which
need a separate directory but now presented in ogr or frmts directories.
There are some bad decisions I made - for example I moved unit tests into
separate repository - this was a mistake. We will return unit tests back to
GDAL repository.

An example of cmake friendly way see
https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt.
The driver developer must only create new folder and put CMakeLists.txt file
into it. The upper CMake script will find new driver and add it to GDAL
build. In common cases no need to modify upper CMake scripts.

2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG
etc). All this code are in separate repositories. I don't see the difference
to get this code from git pull from main GDAL repository or from the
separate repository via find_anyproject process. Current GDAL repository
looks like the https://github.com/nextgis-borsch packed in one repository.


In conclusion:

1. Borsch added flexible and useful features and not remove existing
approach

2. The current cmaked GDAL are in production in my company more than a year
on Windows, Linix, Mac OS, iOS, Android.

3. I'm ready to discuss and improve current solution. Any help are welcome

4. I also will be happy to contribute directly or via PR into GDAL trunk
instead of backporting in both directions improvements that we do in GDAL .


Finally:

Find the link to page with the CMake in GDAL discussion -
https://trac.osgeo.org/gdal/wiki/CMake

--
Best regards,
    Dmitry


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev




_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Kurt Schwehr-2
In reply to this post by Alexander Bruy
While autoconf isn't pretty, it has done me the least amount of harm of any of the major build systems.  Without patches for the downstream packagers and CI (travis/appveyor), I would be -1.  e.g. debian/ubuntu, redhat/centos, macport/brew/fink, and others

With those patches, I'm -0.

BTW, It's not unreasonable to have separate build systems.  It's work, but very doable.  I maintain a bazel build environment.  I have to notice add or deletes of files in upstream, but it's been working for more than a decade.  

On Sat, Oct 28, 2017 at 11:11 PM, Alexander Bruy <[hidden email]> wrote:
While I'm not GDAL developer or regular contributor (submitted only
few patches),
but still often build GDAL trunk on WIndows and Linux, and less often on Mac.

Personally I think that solution proposed in
https://trac.osgeo.org/gdal/ticket/7080
is more preferable. It does not require *all* dependencies to be build
with cmake,
it does not require maintaining forks with special directory structure for *all*
dependencies, it plays well with conventions/packages provided on all systems
out of the box. Maybe borsch is good for in-house use when all stack
is build from
scratch but it is not suitable for real-world use cases.

Of course, #7080 is not ideal, there are some issues, but as I understand the
work is not over and most (if not all) issues can be solved.

Just my 2c.

2017-10-28 0:06 GMT+03:00 Dmitry Baryshnikov <[hidden email]>:
> Hi,
>
> As Even said it make sense to move discussion from this ticket
> (https://trac.osgeo.org/gdal/ticket/7080) to the list.
>
> First of all I would like to make small introduction to Borsch project. Here
> it is some useful links:
>
> * Borsch repository: https://github.com/nextgis-borsch/borsch
>
> * Presentation on FOSS4G 2016:
> http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-presentation.pdf
>
> * GDAL repository adapted for Borsch:
> https://github.com/nextgis-borsch/lib_gdal
>
> Shortly speaking Borsch is 3 CMake scripts which add ability to include
> dependence library in one line of code.
>
> It looks like:
>
> find_anyproject(TIFF REQUIRED)
>
> Certainly developer can add additional parameter to configure dependency in
> find_anyproject function.
>
> Inside  find_anyproject work 2 cases:
>
> 1. First of all (by default, but can be overridden) CMake searches host
> system for dependency library (headers and lib files). This is usual CMake
> find_package (https://cmake.org/cmake/help/v3.0/command/find_package.html)
> with some additional logic to try different options for hard cases (like
> Qt).
>
> 2. If library not found, find_anyproject can get the sources or prebuild
> library from github.
>
> So the GDAL or any other library can be build the normal way, but developer
> have additional features to configure build all libraries with one compiler
> and one set of compiler/linker settings (with some limits). Such way we can
> have rather complicated scenarios to build GDAL and dependencies.
>
> Here it is several examples of benefits of this approach:
>
> 1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library for
> iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64
> architecture). I build all dependencies include GDAL statically and link in
> one fat library. The all code that do it:
> https://github.com/nextgis/nextgis_datastore/blob/master/cmake/extlib.cmake#L118-L236
>
> By the way the library also builds on Linux and Mac OS (Windows under
> development) and CMake try to use existed in host system libraries. If CMake
> find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF ... )
> will be ignored as it already build with another parameters.
>
> 2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa:
> https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules
>
> 3. Build QGIS
> (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149),
> PostGIS
> (https://github.com/nextgis-borsch/postgis/blob/master/CMakeLists.txt#L165),
> Formbuilder
> (https://github.com/nextgis/formbuilder/blob/master/cmake/extlib.cmake#L53-L173)
>
> This is main Borsch features.
>
>
> There are some additional conventions like:
>
>     * I modify all libraries included into Borsch repository to install on
> unix-like paths. For Linux this is usual, for Windows and Mac OS this let us
> to use Qt installer framework an install software mach similar like on
> Linux. This is about target "install" which is vary on different libraries
> (CMake has it own conventions about it). This is not mandatory for Borsch
> itself but useful. CMake can register installed libraries in system
> repository to simplify find them in find_package function.
>
>     * CMake get library version from sources in all libraries included into
> Borsch (if applicable, otherwise set it in CMake script). This is necessary
> if exact version of library needed. This is not mandatory. One more benefit
> during building process we can see dependency library version in console.
>
>     * We modify all libraries included into Borsch repository to find
> dependencies using find_anyproject. It is simple to use libraries from our
> borsch repository, but developer can fork them or use any other sources and
> build systems to have dependency library in it's host system.
>
> One can see this is all very flexible.
>
>
> What about GDAL.
>
> 1. After unification GDALDataset and OGRDatasource current sources tree is
> not fit for this new logic of GDAL classes. I rearranged sources more closer
> to GDAL classes and CMake needs. Main changes are moving raster and vector
> drivers inside drivers folder
> (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This
> simplify situation where different drivers need the same dependency library
> (libpg, libsqlite, etc.). Also there are several raster/vector drivers which
> need a separate directory but now presented in ogr or frmts directories.
> There are some bad decisions I made - for example I moved unit tests into
> separate repository - this was a mistake. We will return unit tests back to
> GDAL repository.
>
> An example of cmake friendly way see
> https://github.com/nextgis-borsch/lib_gdal/blob/master/drivers/vector/CMakeLists.txt.
> The driver developer must only create new folder and put CMakeLists.txt file
> into it. The upper CMake script will find new driver and add it to GDAL
> build. In common cases no need to modify upper CMake scripts.
>
> 2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG, JPEG
> etc). All this code are in separate repositories. I don't see the difference
> to get this code from git pull from main GDAL repository or from the
> separate repository via find_anyproject process. Current GDAL repository
> looks like the https://github.com/nextgis-borsch packed in one repository.
>
>
> In conclusion:
>
> 1. Borsch added flexible and useful features and not remove existing
> approach
>
> 2. The current cmaked GDAL are in production in my company more than a year
> on Windows, Linix, Mac OS, iOS, Android.
>
> 3. I'm ready to discuss and improve current solution. Any help are welcome
>
> 4. I also will be happy to contribute directly or via PR into GDAL trunk
> instead of backporting in both directions improvements that we do in GDAL .
>
>
> Finally:
>
> Find the link to page with the CMake in GDAL discussion -
> https://trac.osgeo.org/gdal/wiki/CMake
>
> --
> Best regards,
>     Dmitry
>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



--
Alexander Bruy
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Hiroshi Miura
In reply to this post by Dmitry Baryshnikov-2
Hi Dmitry,

On 2017年10月29日 07:21, Dmitry Baryshnikov wrote:
>
> Hi Hiroshi,
>
> I tried to test you solution:
>

Thank you for testing and sharing your experience.
It is working in progress status. And it is based on different policy with your solution.
 
Now I don't write document about a policy and how-to.

In current script assumes 'configuration has a priority over dependency libraries'
So when user/developer ON the driver, user/developer should install libraries on their own.

I have not done every dependencies  clean yet, but I've been improved.
You can  use vagrant script that prepares  environment to pass the build.

$ vagrant up

I've tested with LXC container environment on Linux.
>
> The QHULL is not mandatory for GDAL build and should not stop configuring at that moment.
>

It is hard work for me to determine which driver is mandatory and which is optional.  Also I need to  determine which driver should be  ON in default.
It would be a simple rule that driver which does not require 3rd party library is ON in default. Otherwise optional.

Every your feedback is valuable to improve script. It would be good PoC activity to know  which approach is preferable  for GDAL dev community.
I think your solution is to jump to highest level.  My trial is to realize an intermediate step from current source tree.


Hiroshi



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
In reply to this post by Kurt Schwehr-2
Hi Kurt,

And how to deal with non Linux builds like Windows MSVC, mac OS XCode etc.?

Best regards,
     Dmitry

29.10.17 17:17, Kurt Schwehr пишет:

> While autoconf isn't pretty, it has done me the least amount of harm of any
> of the major build systems.  Without patches for the downstream packagers
> and CI (travis/appveyor), I would be -1.  e.g. debian/ubuntu,
> redhat/centos, macport/brew/fink, and others
>
> With those patches, I'm -0.
>
> BTW, It's not unreasonable to have separate build systems.  It's work, but
> very doable.  I maintain a bazel build environment.  I have to notice add
> or deletes of files in upstream, but it's been working for more than a
> decade.
>
> On Sat, Oct 28, 2017 at 11:11 PM, Alexander Bruy <[hidden email]>
> wrote:
>
>> While I'm not GDAL developer or regular contributor (submitted only
>> few patches),
>> but still often build GDAL trunk on WIndows and Linux, and less often on
>> Mac.
>>
>> Personally I think that solution proposed in
>> https://trac.osgeo.org/gdal/ticket/7080
>> is more preferable. It does not require *all* dependencies to be build
>> with cmake,
>> it does not require maintaining forks with special directory structure for
>> *all*
>> dependencies, it plays well with conventions/packages provided on all
>> systems
>> out of the box. Maybe borsch is good for in-house use when all stack
>> is build from
>> scratch but it is not suitable for real-world use cases.
>>
>> Of course, #7080 is not ideal, there are some issues, but as I understand
>> the
>> work is not over and most (if not all) issues can be solved.
>>
>> Just my 2c.
>>
>> 2017-10-28 0:06 GMT+03:00 Dmitry Baryshnikov <[hidden email]>:
>>> Hi,
>>>
>>> As Even said it make sense to move discussion from this ticket
>>> (https://trac.osgeo.org/gdal/ticket/7080) to the list.
>>>
>>> First of all I would like to make small introduction to Borsch project.
>> Here
>>> it is some useful links:
>>>
>>> * Borsch repository: https://github.com/nextgis-borsch/borsch
>>>
>>> * Presentation on FOSS4G 2016:
>>> http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-
>> presentation.pdf
>>> * GDAL repository adapted for Borsch:
>>> https://github.com/nextgis-borsch/lib_gdal
>>>
>>> Shortly speaking Borsch is 3 CMake scripts which add ability to include
>>> dependence library in one line of code.
>>>
>>> It looks like:
>>>
>>> find_anyproject(TIFF REQUIRED)
>>>
>>> Certainly developer can add additional parameter to configure dependency
>> in
>>> find_anyproject function.
>>>
>>> Inside  find_anyproject work 2 cases:
>>>
>>> 1. First of all (by default, but can be overridden) CMake searches host
>>> system for dependency library (headers and lib files). This is usual
>> CMake
>>> find_package (https://cmake.org/cmake/help/
>> v3.0/command/find_package.html)
>>> with some additional logic to try different options for hard cases (like
>>> Qt).
>>>
>>> 2. If library not found, find_anyproject can get the sources or prebuild
>>> library from github.
>>>
>>> So the GDAL or any other library can be build the normal way, but
>> developer
>>> have additional features to configure build all libraries with one
>> compiler
>>> and one set of compiler/linker settings (with some limits). Such way we
>> can
>>> have rather complicated scenarios to build GDAL and dependencies.
>>>
>>> Here it is several examples of benefits of this approach:
>>>
>>> 1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library
>> for
>>> iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64
>>> architecture). I build all dependencies include GDAL statically and link
>> in
>>> one fat library. The all code that do it:
>>> https://github.com/nextgis/nextgis_datastore/blob/master/
>> cmake/extlib.cmake#L118-L236
>>> By the way the library also builds on Linux and Mac OS (Windows under
>>> development) and CMake try to use existed in host system libraries. If
>> CMake
>>> find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF
>> ... )
>>> will be ignored as it already build with another parameters.
>>>
>>> 2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa:
>>> https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules
>>>
>>> 3. Build QGIS
>>> (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149
>> ),
>>> PostGIS
>>> (https://github.com/nextgis-borsch/postgis/blob/master/
>> CMakeLists.txt#L165),
>>> Formbuilder
>>> (https://github.com/nextgis/formbuilder/blob/master/cmake/
>> extlib.cmake#L53-L173)
>>> This is main Borsch features.
>>>
>>>
>>> There are some additional conventions like:
>>>
>>>      * I modify all libraries included into Borsch repository to install
>> on
>>> unix-like paths. For Linux this is usual, for Windows and Mac OS this
>> let us
>>> to use Qt installer framework an install software mach similar like on
>>> Linux. This is about target "install" which is vary on different
>> libraries
>>> (CMake has it own conventions about it). This is not mandatory for Borsch
>>> itself but useful. CMake can register installed libraries in system
>>> repository to simplify find them in find_package function.
>>>
>>>      * CMake get library version from sources in all libraries included
>> into
>>> Borsch (if applicable, otherwise set it in CMake script). This is
>> necessary
>>> if exact version of library needed. This is not mandatory. One more
>> benefit
>>> during building process we can see dependency library version in console.
>>>
>>>      * We modify all libraries included into Borsch repository to find
>>> dependencies using find_anyproject. It is simple to use libraries from
>> our
>>> borsch repository, but developer can fork them or use any other sources
>> and
>>> build systems to have dependency library in it's host system.
>>>
>>> One can see this is all very flexible.
>>>
>>>
>>> What about GDAL.
>>>
>>> 1. After unification GDALDataset and OGRDatasource current sources tree
>> is
>>> not fit for this new logic of GDAL classes. I rearranged sources more
>> closer
>>> to GDAL classes and CMake needs. Main changes are moving raster and
>> vector
>>> drivers inside drivers folder
>>> (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This
>>> simplify situation where different drivers need the same dependency
>> library
>>> (libpg, libsqlite, etc.). Also there are several raster/vector drivers
>> which
>>> need a separate directory but now presented in ogr or frmts directories.
>>> There are some bad decisions I made - for example I moved unit tests into
>>> separate repository - this was a mistake. We will return unit tests back
>> to
>>> GDAL repository.
>>>
>>> An example of cmake friendly way see
>>> https://github.com/nextgis-borsch/lib_gdal/blob/master/
>> drivers/vector/CMakeLists.txt.
>>> The driver developer must only create new folder and put CMakeLists.txt
>> file
>>> into it. The upper CMake script will find new driver and add it to GDAL
>>> build. In common cases no need to modify upper CMake scripts.
>>>
>>> 2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG,
>> JPEG
>>> etc). All this code are in separate repositories. I don't see the
>> difference
>>> to get this code from git pull from main GDAL repository or from the
>>> separate repository via find_anyproject process. Current GDAL repository
>>> looks like the https://github.com/nextgis-borsch packed in one
>> repository.
>>>
>>> In conclusion:
>>>
>>> 1. Borsch added flexible and useful features and not remove existing
>>> approach
>>>
>>> 2. The current cmaked GDAL are in production in my company more than a
>> year
>>> on Windows, Linix, Mac OS, iOS, Android.
>>>
>>> 3. I'm ready to discuss and improve current solution. Any help are
>> welcome
>>> 4. I also will be happy to contribute directly or via PR into GDAL trunk
>>> instead of backporting in both directions improvements that we do in
>> GDAL .
>>>
>>> Finally:
>>>
>>> Find the link to page with the CMake in GDAL discussion -
>>> https://trac.osgeo.org/gdal/wiki/CMake
>>>
>>> --
>>> Best regards,
>>>      Dmitry
>>>
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>
>> --
>> Alexander Bruy
>> _______________________________________________
>> gdal-dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
In reply to this post by Hiroshi Miura

Hi Hiroshi,

I think that anyhow the current logic of makefile mast be transfer to CMake. See the https://github.com/OSGeo/gdal/blob/trunk/gdal/configure.ac or how I did it in lib_gdal repository. This logic is rather complicated!

About vagrant:

$ vagrant up
bash: vagrant: command not found

Vagrant is not documented dependency and I don't understand how it will help me in may building environment and what additional benefits vagrant provide to me in compare with autoconf?

I'm sure all steps in any environment, as Mateusz Łoskot wrote, should be:

git clone .../gdal
mkdir build
cd build
cmake ..
apps/gdalinfo --version


Best regards,
    Dmitry
29.10.17 17:27, Hiroshi Miura пишет:
Hi Dmitry,

On 2017年10月29日 07:21, Dmitry Baryshnikov wrote:
Hi Hiroshi,

I tried to test you solution:

Thank you for testing and sharing your experience.
It is working in progress status. And it is based on different policy with your solution.
 
Now I don't write document about a policy and how-to.

In current script assumes 'configuration has a priority over dependency libraries'
So when user/developer ON the driver, user/developer should install libraries on their own.

I have not done every dependencies  clean yet, but I've been improved.
You can  use vagrant script that prepares  environment to pass the build.

$ vagrant up

I've tested with LXC container environment on Linux.
The QHULL is not mandatory for GDAL build and should not stop configuring at that moment.

It is hard work for me to determine which driver is mandatory and which is optional.  Also I need to  determine which driver should be  ON in default.
It would be a simple rule that driver which does not require 3rd party library is ON in default. Otherwise optional.

Every your feedback is valuable to improve script. It would be good PoC activity to know  which approach is preferable  for GDAL dev community.
I think your solution is to jump to highest level.  My trial is to realize an intermediate step from current source tree.


Hiroshi






_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Alexander Bruy
In reply to this post by Dmitry Baryshnikov-2
Hi Dmitry,

thanks for explanation. I see that there is an option to use system libraries.
Maybe I'm wrong, but this is also possible with current build system and
CMake scripts from #7080. There is an option to build deps by myself and
use them to build GDAL, but again this is also possible with current build
system and CMake scripts from #7080.

I agree that automatic fetching of missed dependencies is nice feature. And
can simplify live in some cases. But there are questions:

1. what if dependency hosted on BitBucket, SourceForge or any self-hosted
VCS and not in your repository?
2. what if dependency use build system different from CMake?
3. what if upstream does not want/does not see abny benefits in moving to
GitHub and/or switching to CMake?

Correct me if I'm wrong, but most of the borsh's benefits require
changes not only
in GDAL, but also in all upstream projects, as well as infrastructure
changes for
them. Or why you maintain forks for every lib needed by GDAL in your repository?
If everything is fine and flexible why not use upstream projects directly?

2017-10-29 15:39 GMT+02:00 Dmitry Baryshnikov <[hidden email]>:

> 1. Build gdal with all dependencies getting them from github
> (WITH_EXTERNAL). Preferable for Windows
>
> 2. Build gdal using the system libraries. Preferable for Linux
>
> 3. Build gdal using the dependency libraries build by user (out of source)
> and registered in CMake package registry. This is I use now. This script
> help me to clone all libraries, build them and register them in CMake
> package registry
> (https://github.com/nextgis-borsch/borsch/blob/master/opt/tools.py#L134).
>
> My opinion is that different options is much flexible and suits for many
> scenarios as very limited solution from
> https://trac.osgeo.org/gdal/ticket/7080.

--
Alexander Bruy
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
Hi Alexander,

CMake scripts from #7080 have no logic presented in current building
system (make/autoconf): there are several mandatory drivers, the
optional drivers, and drivers depends from other drivers.

Also there is one big problem for me in #7080 - this is third build
system additionally to GNUMakefile, makefile.vc.  And now CMakeLists.txt
must be supported too. Three files which must be synced each other with
and taking into consideration the upper scripts, it's awful!


That about your questions:

1. Now it is not supported. This is really limitations what should gone.
It can be easily fixed. I thinking on this direction - may be somehow
like in Carthage project was done (https://github.com/Carthage/Carthage).

2. Here we have 2 cases: 1 - cmaked dependency (we only remove current
build system with own CMake not touching the original sources), 2 - wrap
current build system by CMake (see how it is done for Qt4 -
https://github.com/nextgis-borsch/lib_qt4)

3. This should not be the problem of upstream developers. It may be
developed some script which periodically get sources and put them to
cmaked repositories, build and run tests to report if everything is OK.
Here we can achieve synced releases and developer branches with our owns
prepared repositories.

4. What about option to get all needed dependencies from github
(WITH_EXTERNAL) - yes we need own prepared repositories. Theoretical
there is the mechanism to make direct clone from any repository
(https://cmake.org/cmake/help/v3.8/module/ExternalProject.html) but
there are several problems here:

1. Original repository may not support needed platforms (usually Mac OS,
Android, iOS and Windows) or static/dynamic library builds.

2. No way to uniformly transfer building options to dependency builds.

3. Install paths of original projects may vary and this may be a problem
in install target of main project. In current solution upper project
installs all dependency dynamic libraries in it "install" target in
uniform way.

To solve this problems I choose more simple way to have own prepared
repositories (again, just like in Carthage approach).


Finally, thanks for your concrete questions!

Best regards,
     Dmitry

30.10.17 10:00, Alexander Bruy пишет:

> Hi Dmitry,
>
> thanks for explanation. I see that there is an option to use system libraries.
> Maybe I'm wrong, but this is also possible with current build system and
> CMake scripts from #7080. There is an option to build deps by myself and
> use them to build GDAL, but again this is also possible with current build
> system and CMake scripts from #7080.
>
> I agree that automatic fetching of missed dependencies is nice feature. And
> can simplify live in some cases. But there are questions:
>
> 1. what if dependency hosted on BitBucket, SourceForge or any self-hosted
> VCS and not in your repository?
> 2. what if dependency use build system different from CMake?
> 3. what if upstream does not want/does not see abny benefits in moving to
> GitHub and/or switching to CMake?
>
> Correct me if I'm wrong, but most of the borsh's benefits require
> changes not only
> in GDAL, but also in all upstream projects, as well as infrastructure
> changes for
> them. Or why you maintain forks for every lib needed by GDAL in your repository?
> If everything is fine and flexible why not use upstream projects directly?
>
> 2017-10-29 15:39 GMT+02:00 Dmitry Baryshnikov <[hidden email]>:
>> 1. Build gdal with all dependencies getting them from github
>> (WITH_EXTERNAL). Preferable for Windows
>>
>> 2. Build gdal using the system libraries. Preferable for Linux
>>
>> 3. Build gdal using the dependency libraries build by user (out of source)
>> and registered in CMake package registry. This is I use now. This script
>> help me to clone all libraries, build them and register them in CMake
>> package registry
>> (https://github.com/nextgis-borsch/borsch/blob/master/opt/tools.py#L134).
>>
>> My opinion is that different options is much flexible and suits for many
>> scenarios as very limited solution from
>> https://trac.osgeo.org/gdal/ticket/7080.

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Mateusz Loskot
On 30 October 2017 at 10:06, Dmitry Baryshnikov <[hidden email]> wrote:
>
> Also there is one big problem for me in #7080 - this is third build system
> additionally to GNUMakefile, makefile.vc.  And now CMakeLists.txt must be
> supported too. Three files which must be synced each other with and taking
> into consideration the upper scripts, it's awful!

Dmitry,

This issue is orthogonal to actual CMake challenge.
If core developers of a project X do not agree to switch to a new
build configuration Y,
then any initiative to develop setup for Y will live in a side car.

AFAIU, there is no plan or even will for such switch in GDAL.
Similar situation is with GEOS.

Finally, developing configuration for Y build system by completely
revolutionising structure of project X is a terrible thing to do in terms
of potential switch to Y. IOW, the bigger and deeper a revolution, the less
chance to get acceptance by core developers, also psychologically.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Dmitry Baryshnikov-2
Hi Mateusz,

If really "there is no plan or even will for such switch in GDAL" I
think nothing to discuss here.

The ticket #7080 should be closed and let's continue to live with
current developing approach.

Which solution, my or Hiroshi, will more popular show the time.

Best regards,
     Dmitry

30.10.17 12:24, Mateusz Loskot пишет:

> On 30 October 2017 at 10:06, Dmitry Baryshnikov <[hidden email]> wrote:
>> Also there is one big problem for me in #7080 - this is third build system
>> additionally to GNUMakefile, makefile.vc.  And now CMakeLists.txt must be
>> supported too. Three files which must be synced each other with and taking
>> into consideration the upper scripts, it's awful!
> Dmitry,
>
> This issue is orthogonal to actual CMake challenge.
> If core developers of a project X do not agree to switch to a new
> build configuration Y,
> then any initiative to develop setup for Y will live in a side car.
>
> AFAIU, there is no plan or even will for such switch in GDAL.
> Similar situation is with GEOS.
>
> Finally, developing configuration for Y build system by completely
> revolutionising structure of project X is a terrible thing to do in terms
> of potential switch to Y. IOW, the bigger and deeper a revolution, the less
> chance to get acceptance by core developers, also psychologically.
>
> Best regards,

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Mateusz Loskot
On 30 October 2017 at 10:42, Dmitry Baryshnikov <[hidden email]> wrote:
>
> If really "there is no plan or even will for such switch in GDAL" I think
> nothing to discuss here.

I indicated that it is "AFAIU".
I may be wrong.

Personally, I gave up about CMake for GDAL and I use the official build systems.
Luckily, GDAL still supports `--without-libtool` option.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

Ari Jolma-2
In reply to this post by Mateusz Loskot
Mateusz Loskot kirjoitti 30.10.2017 klo 11:24:

> On 30 October 2017 at 10:06, Dmitry Baryshnikov <[hidden email]> wrote:
>> Also there is one big problem for me in #7080 - this is third build system
>> additionally to GNUMakefile, makefile.vc.  And now CMakeLists.txt must be
>> supported too. Three files which must be synced each other with and taking
>> into consideration the upper scripts, it's awful!
> Dmitry,
>
> This issue is orthogonal to actual CMake challenge.
> If core developers of a project X do not agree to switch to a new
> build configuration Y,
> then any initiative to develop setup for Y will live in a side car.
>
> AFAIU, there is no plan or even will for such switch in GDAL.
> Similar situation is with GEOS.

https://github.com/OSGeo/gdal/graphs/contributors

It's basically Even who nowadays does developments for GDAL, and Kurt to
some degree. So to change the build system, one needs only to convince
Even :) I'm rather sure the rest of the PSC would not object.

I see the build system usually as a necessary evil. I'm yet to see a
developer friendly build system. I don't know if anybody really has any
passion to improve the current system of GDAL. I also don't understand
the antipathy against libtool. I'm mostly happy with the current system
since I'm on Linux and I usually don't need to mess with it. Building
for Windows was a pain (even when I used to use mingw).

I have very little experience with CMake, mostly from building QGIS. It
has a bit fancier UI than autotools but otherwise I don't have an opinion.

There are few things that could perhaps be better IMO

- the need to run make clean after updates (isn't it possible to have
good dependencies?)
- the possibility to make real light-weight builds with only few drivers
- I also hand-edit the -O2 away from GDALmake.opt when I need to use
gdb, maybe there is a configure option for that or something but I don't
know

Best,

Ari

>
> Finally, developing configuration for Y build system by completely
> revolutionising structure of project X is a terrible thing to do in terms
> of potential switch to Y. IOW, the bigger and deeper a revolution, the less
> chance to get acceptance by core developers, also psychologically.
>
> Best regards,

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: About CMake build again

John Daniel-2
In reply to this post by Dmitry Baryshnikov-2
Xcode builds, targeting both macOS and iOS, are my primary concern. I have to use CMake for MySQL and JPEG2000. MySQL is only for server-side work and testing because licensing issues keep it out of any distributed products. JPEG2000 works fine. I have tried and failed to get the current, CMake-based KML working. I can’t figure it out and neither can the KML developers. Instead, I am using the last autotools version. At some point I will port source code fixes from the new version back to my autotools branch.

I always assumed that CMake only works on Linux, and maybe Windows I guess. You can’t use macOS/iOS/Xcode as a rationale for using CMake. I can’t imagine the GDAL CMake maintainers having the Apple hardware, accounts, and expertise necessary to keep that working. I certainly don’t have the CMake expertise. My interest in all of this is working with GIS data and GDAL, not CMake.

The problem is that CMake is such an opaque black box. If it doesn’t work the first time, you have to dive very deep into the internals to figure it out. Autotools, for all its faults, needs only decent UNIX skills to get working. My build scripts are very complicated, but they do work. The only part I ever completely failed at was a CMake library.

If you want to sell CMake as a Windows solution, fine. I don’t do Windows and I don’t care. But CMake can’t handle Apple platforms.

John Daniel
Etresoft, Inc.

> On Oct 29, 2017, at 11:15 AM, Dmitry Baryshnikov <[hidden email]> wrote:
>
> Hi Kurt,
>
> And how to deal with non Linux builds like Windows MSVC, mac OS XCode etc.?
>
> Best regards,
>    Dmitry
>
> 29.10.17 17:17, Kurt Schwehr пишет:
>> While autoconf isn't pretty, it has done me the least amount of harm of any
>> of the major build systems.  Without patches for the downstream packagers
>> and CI (travis/appveyor), I would be -1.  e.g. debian/ubuntu,
>> redhat/centos, macport/brew/fink, and others
>>
>> With those patches, I'm -0.
>>
>> BTW, It's not unreasonable to have separate build systems.  It's work, but
>> very doable.  I maintain a bazel build environment.  I have to notice add
>> or deletes of files in upstream, but it's been working for more than a
>> decade.
>>
>> On Sat, Oct 28, 2017 at 11:11 PM, Alexander Bruy <[hidden email]>
>> wrote:
>>
>>> While I'm not GDAL developer or regular contributor (submitted only
>>> few patches),
>>> but still often build GDAL trunk on WIndows and Linux, and less often on
>>> Mac.
>>>
>>> Personally I think that solution proposed in
>>> https://trac.osgeo.org/gdal/ticket/7080
>>> is more preferable. It does not require *all* dependencies to be build
>>> with cmake,
>>> it does not require maintaining forks with special directory structure for
>>> *all*
>>> dependencies, it plays well with conventions/packages provided on all
>>> systems
>>> out of the box. Maybe borsch is good for in-house use when all stack
>>> is build from
>>> scratch but it is not suitable for real-world use cases.
>>>
>>> Of course, #7080 is not ideal, there are some issues, but as I understand
>>> the
>>> work is not over and most (if not all) issues can be solved.
>>>
>>> Just my 2c.
>>>
>>> 2017-10-28 0:06 GMT+03:00 Dmitry Baryshnikov <[hidden email]>:
>>>> Hi,
>>>>
>>>> As Even said it make sense to move discussion from this ticket
>>>> (https://trac.osgeo.org/gdal/ticket/7080) to the list.
>>>>
>>>> First of all I would like to make small introduction to Borsch project.
>>> Here
>>>> it is some useful links:
>>>>
>>>> * Borsch repository: https://github.com/nextgis-borsch/borsch
>>>>
>>>> * Presentation on FOSS4G 2016:
>>>> http://nextgis.ru/wp-content/uploads/2016/08/NextGIS-Borsh-
>>> presentation.pdf
>>>> * GDAL repository adapted for Borsch:
>>>> https://github.com/nextgis-borsch/lib_gdal
>>>>
>>>> Shortly speaking Borsch is 3 CMake scripts which add ability to include
>>>> dependence library in one line of code.
>>>>
>>>> It looks like:
>>>>
>>>> find_anyproject(TIFF REQUIRED)
>>>>
>>>> Certainly developer can add additional parameter to configure dependency
>>> in
>>>> find_anyproject function.
>>>>
>>>> Inside  find_anyproject work 2 cases:
>>>>
>>>> 1. First of all (by default, but can be overridden) CMake searches host
>>>> system for dependency library (headers and lib files). This is usual
>>> CMake
>>>> find_package (https://cmake.org/cmake/help/
>>> v3.0/command/find_package.html)
>>>> with some additional logic to try different options for hard cases (like
>>>> Qt).
>>>>
>>>> 2. If library not found, find_anyproject can get the sources or prebuild
>>>> library from github.
>>>>
>>>> So the GDAL or any other library can be build the normal way, but
>>> developer
>>>> have additional features to configure build all libraries with one
>>> compiler
>>>> and one set of compiler/linker settings (with some limits). Such way we
>>> can
>>>> have rather complicated scenarios to build GDAL and dependencies.
>>>>
>>>> Here it is several examples of benefits of this approach:
>>>>
>>>> 1. NextGIS Mobile SDK v3. SDK based on GDAL and need it in one library
>>> for
>>>> iOS as *.framework and for Android as *.so (arm7, arm64, i386, x86_64
>>>> architecture). I build all dependencies include GDAL statically and link
>>> in
>>>> one fat library. The all code that do it:
>>>> https://github.com/nextgis/nextgis_datastore/blob/master/
>>> cmake/extlib.cmake#L118-L236
>>>> By the way the library also builds on Linux and Mac OS (Windows under
>>>> development) and CMake try to use existed in host system libraries. If
>>> CMake
>>>> find GDAL in host system it will use it and all (-DENABLE_PLSCENES=OFF
>>> ... )
>>>> will be ignored as it already build with another parameters.
>>>>
>>>> 2. Build GDAL Windows standalone installer and GDAL Ubuntu ppa:
>>>> https://github.com/nextgis/ppa/blob/master/gdal/master/debian/rules
>>>>
>>>> 3. Build QGIS
>>>> (https://github.com/nextgis/nextgisqgis/blob/master/CMakeLists.txt#L149
>>> ),
>>>> PostGIS
>>>> (https://github.com/nextgis-borsch/postgis/blob/master/
>>> CMakeLists.txt#L165),
>>>> Formbuilder
>>>> (https://github.com/nextgis/formbuilder/blob/master/cmake/
>>> extlib.cmake#L53-L173)
>>>> This is main Borsch features.
>>>>
>>>>
>>>> There are some additional conventions like:
>>>>
>>>>     * I modify all libraries included into Borsch repository to install
>>> on
>>>> unix-like paths. For Linux this is usual, for Windows and Mac OS this
>>> let us
>>>> to use Qt installer framework an install software mach similar like on
>>>> Linux. This is about target "install" which is vary on different
>>> libraries
>>>> (CMake has it own conventions about it). This is not mandatory for Borsch
>>>> itself but useful. CMake can register installed libraries in system
>>>> repository to simplify find them in find_package function.
>>>>
>>>>     * CMake get library version from sources in all libraries included
>>> into
>>>> Borsch (if applicable, otherwise set it in CMake script). This is
>>> necessary
>>>> if exact version of library needed. This is not mandatory. One more
>>> benefit
>>>> during building process we can see dependency library version in console.
>>>>
>>>>     * We modify all libraries included into Borsch repository to find
>>>> dependencies using find_anyproject. It is simple to use libraries from
>>> our
>>>> borsch repository, but developer can fork them or use any other sources
>>> and
>>>> build systems to have dependency library in it's host system.
>>>>
>>>> One can see this is all very flexible.
>>>>
>>>>
>>>> What about GDAL.
>>>>
>>>> 1. After unification GDALDataset and OGRDatasource current sources tree
>>> is
>>>> not fit for this new logic of GDAL classes. I rearranged sources more
>>> closer
>>>> to GDAL classes and CMake needs. Main changes are moving raster and
>>> vector
>>>> drivers inside drivers folder
>>>> (https://github.com/nextgis-borsch/lib_gdal/tree/master/drivers).This
>>>> simplify situation where different drivers need the same dependency
>>> library
>>>> (libpg, libsqlite, etc.). Also there are several raster/vector drivers
>>> which
>>>> need a separate directory but now presented in ogr or frmts directories.
>>>> There are some bad decisions I made - for example I moved unit tests into
>>>> separate repository - this was a mistake. We will return unit tests back
>>> to
>>>> GDAL repository.
>>>>
>>>> An example of cmake friendly way see
>>>> https://github.com/nextgis-borsch/lib_gdal/blob/master/
>>> drivers/vector/CMakeLists.txt.
>>>> The driver developer must only create new folder and put CMakeLists.txt
>>> file
>>>> into it. The upper CMake script will find new driver and add it to GDAL
>>>> build. In common cases no need to modify upper CMake scripts.
>>>>
>>>> 2. I remove third-party code from drivers folders (TIFF, GeoTIF, PNG,
>>> JPEG
>>>> etc). All this code are in separate repositories. I don't see the
>>> difference
>>>> to get this code from git pull from main GDAL repository or from the
>>>> separate repository via find_anyproject process. Current GDAL repository
>>>> looks like the https://github.com/nextgis-borsch packed in one
>>> repository.
>>>>
>>>> In conclusion:
>>>>
>>>> 1. Borsch added flexible and useful features and not remove existing
>>>> approach
>>>>
>>>> 2. The current cmaked GDAL are in production in my company more than a
>>> year
>>>> on Windows, Linix, Mac OS, iOS, Android.
>>>>
>>>> 3. I'm ready to discuss and improve current solution. Any help are
>>> welcome
>>>> 4. I also will be happy to contribute directly or via PR into GDAL trunk
>>>> instead of backporting in both directions improvements that we do in
>>> GDAL .
>>>>
>>>> Finally:
>>>>
>>>> Find the link to page with the CMake in GDAL discussion -
>>>> https://trac.osgeo.org/gdal/wiki/CMake
>>>>
>>>> --
>>>> Best regards,
>>>>     Dmitry
>>>>
>>>>
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>>
>>> --
>>> Alexander Bruy
>>> _______________________________________________
>>> gdal-dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>
>>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
12