[gdal-dev] osx compile failure

Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] osx compile failure

Alan Stewart-2

MacOS 10.12, xcode 9.2

 

I doing ‘git clone’ of master from OSGeo/gdal and running the gdal/ci/travis/osx scripts. The install.sh script terminates because of:

 

plmosaicdataset.cpp:1111:30: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'const int' [-Werror,-Wshorten-64-to-32]

        const int nMosaics = json_object_array_length(poMosaics);

                  ~~~~~~~~   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

plmosaicdataset.cpp:1419:30: error: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long') [-Werror,-Wsign-compare]

            for(int i = 0; i < json_object_array_length(poItems); i++ )

                           ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

I could fix this with a simple cast, but I don’t know GDAL policy and I wonder how many more there will be if I fix these. Is my environment incorrect, should I be setting a flag to ignore this type of error? I would think that would be in the script…

 

Thanks for any advice.

 

Alan Stewart

Senior Software Engineer

TerraGo Technologies

3200 Windy Hill Road, Suite 1550W

Atlanta, GA 30339 USA

O.  +1 678.391.9615

 

<a href="applewebdata://B24C0762-C7C9-4431-8518-ACC915448B89/www.terragotech.com">www.terragotech.com

 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Even Rouault-2
Alan,

your environment is correct, it is just that there are different versions of
libjson-c where function signatures have changed. We have had similar issues
in the past.

>
> I doing 'git clone' of master from OSGeo/gdal and running the
> gdal/ci/travis/osx scripts. The install.sh script terminates because of:
>
> plmosaicdataset.cpp:1111:30: error: implicit conversion loses integer
> precision: 'size_t' (aka 'unsigned long') to 'const int'
> [-Werror,-Wshorten-64-to-32] const int nMosaics =
> json_object_array_length(poMosaics);

Could be fixed for all versions by using: const auto

>                   ~~~~~~~~   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> plmosaicdataset.cpp:1419:30: error: comparison of integers of different
> signs: 'int' and 'size_t' (aka 'unsigned long') [-Werror,-Wsign-compare]
> for(int i = 0; i < json_object_array_length(poItems); i++ ) ~ ^

I would do
  const auto nItemsLength = json_object_array_length(poItems);
    for( decltype(nItemsLength) i = 0; i < nItemsLength; i++ )

Can you submit a pull request with something like the above ?

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Alan Stewart-2
Even,

I've been working on this. There were a lot of small source changes across a number of formats, not just plscenes, plus some system-specific details in the scripts, to be resolved. I have a successful build now, but running the ci/travis/osx/script.sh results in:

ImportError while loading conftest '/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/autotest/conftest.py'.
/usr/local/lib/python2.7/site-packages/six.py:709: in exec_
    exec("""exec _code_ in _globs_, _locs_""")
conftest.py:9: in <module>
    from osgeo import gdal
../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:21: in <module>
    _gdal = swig_import_helper()
../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:17: in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
E   ImportError: dlopen(/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so, 2): Symbol not found: _CPLErrorSetState
E     Referenced from: /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so
E     Expected in: flat namespace
E    in /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so

Any thoughts on why CPLErrorSetState would be missing?

Alan Stewart
Senior Software Engineer
TerraGo Technologies
3200 Windy Hill Road, Suite 1550W
Atlanta, GA 30339 USA
O.  +1 678.391.9615
 
www.terragotech.com

-----Original Message-----
From: Even Rouault <[hidden email]>
Sent: Tuesday, December 18, 2018 4:20 PM
To: [hidden email]
Cc: Alan Stewart <[hidden email]>
Subject: Re: [gdal-dev] osx compile failure

Alan,

your environment is correct, it is just that there are different versions of libjson-c where function signatures have changed. We have had similar issues in the past.

>
> I doing 'git clone' of master from OSGeo/gdal and running the
> gdal/ci/travis/osx scripts. The install.sh script terminates because of:
>
> plmosaicdataset.cpp:1111:30: error: implicit conversion loses integer
> precision: 'size_t' (aka 'unsigned long') to 'const int'
> [-Werror,-Wshorten-64-to-32] const int nMosaics =
> json_object_array_length(poMosaics);

Could be fixed for all versions by using: const auto

>                   ~~~~~~~~   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> plmosaicdataset.cpp:1419:30: error: comparison of integers of
> different
> signs: 'int' and 'size_t' (aka 'unsigned long')
> [-Werror,-Wsign-compare] for(int i = 0; i <
> json_object_array_length(poItems); i++ ) ~ ^

I would do
  const auto nItemsLength = json_object_array_length(poItems);
    for( decltype(nItemsLength) i = 0; i < nItemsLength; i++ )

Can you submit a pull request with something like the above ?

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Even Rouault-2
On jeudi 3 janvier 2019 21:17:26 CET Alan Stewart wrote:

> Even,
>
> I've been working on this. There were a lot of small source changes across a
> number of formats, not just plscenes, plus some system-specific details in
> the scripts, to be resolved. I have a successful build now, but running the
> ci/travis/osx/script.sh results in:
>
> ImportError while loading conftest
> '/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/autotest/con
> ftest.py'. /usr/local/lib/python2.7/site-packages/six.py:709: in exec_
>     exec("""exec _code_ in _globs_, _locs_""")
> conftest.py:9: in <module>
>     from osgeo import gdal
> ../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:21:
> in <module> _gdal = swig_import_helper()
> ../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:17:
> in swig_import_helper _mod = imp.load_module('_gdal', fp, pathname,
> description)
> E   ImportError:
> dlopen(/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/s
> wig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so, 2): Symbol not
> found: _CPLErrorSetState E     Referenced from:
> /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/pyt
> hon/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so E     Expected in: flat
> namespace
> E    in
> /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/pyt
> hon/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so
>
> Any thoughts on why CPLErrorSetState would be missing?

I presume that this is because libgdal.dylib is not found.
Did you source "scripts/setdevenv.sh" that should set up the appropriate
environment variables ? In that particular case DYLD_LIBRARY_PATH to point to
libgdal.dylib

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Even Rouault-2
On vendredi 4 janvier 2019 19:52:36 CET you wrote:
> That script corrupts the PATH environment variable!

Works on Linux at lest. I *believe* it has been tested on Mac by some folks.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Alan Stewart-2
In reply to this post by Even Rouault-2
Apparently the problems I was having with setdevenv.sh where caused by something trashed in my environment? After being gone for a few das now sourcing the script reports seems to operate normally.

Yes, libgdal.dylib is in the gdal/.libs path, and this path is in DYLD_LIBRARY_PATH. I'm running gdal/ci/travis/osx/script.sh from another script that sources gdal/scripts/setdevenv.sh first. I'm still seeing the same error.

Output snippet:

Sourcing setdevenv.sh...

Setting PATH=/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/apps:/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/apps/.libs:/Users/astewart/.rvm/gems/ruby-2.5.3/bin:/Users/astewart/.rvm/gems/ruby-2.5.3@global/bin:/Users/astewart/.rvm/rubies/ruby-2.5.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Users/astewart/.rvm/bin
Setting DYLD_LIBRARY_PATH=/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal:/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/.libs:
Setting GDAL_DATA=/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/data
Setting PYTHONPATH=/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7:

Testing _gdal.so and libgdal.dylib for presence of CPLErrorSetState is successful.

Is there something besides DYLD_LIBRARY_PATH that needs to be set? Adding gdal/.libs to PATH did not good.

Alan Stewart
Senior Software Engineer
TerraGo Technologies
3200 Windy Hill Road, Suite 1550W
Atlanta, GA 30339 USA
O.  +1 678.391.9615
 
www.terragotech.com

-----Original Message-----
From: Even Rouault <[hidden email]>
Sent: Thursday, January 03, 2019 4:40 PM
To: Alan Stewart <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure

On jeudi 3 janvier 2019 21:17:26 CET Alan Stewart wrote:

> Even,
>
> I've been working on this. There were a lot of small source changes
> across a number of formats, not just plscenes, plus some
> system-specific details in the scripts, to be resolved. I have a
> successful build now, but running the ci/travis/osx/script.sh results in:
>
> ImportError while loading conftest
> '/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/autotes
> t/con ftest.py'. /usr/local/lib/python2.7/site-packages/six.py:709: in
> exec_
>     exec("""exec _code_ in _globs_, _locs_""")
> conftest.py:9: in <module>
>     from osgeo import gdal
> ../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:21:
> in <module> _gdal = swig_import_helper()
> ../gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/__init__.py:17:
> in swig_import_helper _mod = imp.load_module('_gdal', fp, pathname,
> description)
> E   ImportError:
> dlopen(/Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/g
> dal/s wig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so, 2):
> Symbol not
> found: _CPLErrorSetState E     Referenced from:
> /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swig/pyt
> hon/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so E     Expected in: flat
> namespace
> E    in
> /Users/astewart/Documents/GitHub/alanstewart-terragotech/gdal/gdal/swi
> g/pyt hon/build/lib.macosx-10.12-x86_64-2.7/osgeo/_gdal.so
>
> Any thoughts on why CPLErrorSetState would be missing?

I presume that this is because libgdal.dylib is not found.
Did you source "scripts/setdevenv.sh" that should set up the appropriate environment variables ? In that particular case DYLD_LIBRARY_PATH to point to libgdal.dylib

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Even Rouault-2
On lundi 14 janvier 2019 20:25:42 CET Alan Stewart wrote:
> DYLD_LIBRARY_PATH

Maybe an effect of the SIP thing you'd need to disable ?
https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Alan Stewart-2
Disabling SIP makes no difference.

Alan Stewart
Senior Software Engineer
TerraGo Technologies
3200 Windy Hill Road, Suite 1550W
Atlanta, GA 30339 USA
O.  +1 678.391.9615
 
www.terragotech.com

-----Original Message-----
From: Even Rouault <[hidden email]>
Sent: Monday, January 14, 2019 3:38 PM
To: Alan Stewart <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure

On lundi 14 janvier 2019 20:25:42 CET Alan Stewart wrote:
> DYLD_LIBRARY_PATH

Maybe an effect of the SIP thing you'd need to disable ?
https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Ivan Lucena-5
Alan,

Did you used --without-libtool on ./configure?

If you do that, I believe that the file ./apps/gdalinfo and all other executables will be real executables and the makefiles will call "install_name_tool" to register the libgdal.dylib on the executables. In that case DYLD_LIBRARY_PATH will be ignored.

But, if you don't use --without-libtool on ./configure, that means you *do use libtool* and a lot of other interesting things will happen. Like, for example, the files that we think that are executables are not:

$ file apps/gdalinfo
apps/gdalinfo: POSIX shell script text executable, ASCII text
$ more apps/gdalinfo
#! /bin/sh

# gdalinfo - temporary wrapper script for .libs/gdalinfo
# Generated by libtool (GNU libtool) 2.4.6 Debian-2.4.6-0.1
#
# The gdalinfo program cannot be directly executed until all the libtool
# libraries that it depends on are installed.
#
# This wrapper script should never be moved out of the build directory.
# If it is, it will not operate correctly.

# Sed substitution that helps us do robust quoting.  It backslashifies
# metacharacters that are still active within double-quoted strings.
...

This is just the initial comments (whatever they means by "libtool libraries are installed") and there is a lot of lines trying to set and reset DYLD_LIBRARY_PATH. The real executable will be generated on this other folder:

$ file apps/.libs/gdalinfo
apps/.libs/gdalinfo: Mach-O 64-bit executable x86_64

But once you run "make install" the makefiles will generated a "clean" distribution folder where the executable are real executables.

In either way, with or without libtool, the key point is that DYLD_LIBRARY_PATH on MacOS is becoming deprecated or even useless as it is preceded by whatever "install_name_tool" defines.

One tools that you can use to figure out where the executable is getting the shared libraries from is "otool", for example:

$ otool -L apps/.libs/gdalinfo

And then I believe you can use "install_name_path" to delete the rpath that you disagree with. Maybe DYLD_LIBRARY_PATH will work then. If you really need.

That is my experience with setdevenv.sh on MacOS in regard to the DYLD_LIBRARY_PATH variable. 

I am sure you can find a lot of discussions on that theme over the web, for example



Good luck,

Ivan








From: gdal-dev <[hidden email]> on behalf of Alan Stewart <[hidden email]>
Sent: Tuesday, January 15, 2019 2:00 PM
To: Even Rouault
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure
 
Disabling SIP makes no difference.

Alan Stewart
Senior Software Engineer
TerraGo Technologies
3200 Windy Hill Road, Suite 1550W
Atlanta, GA 30339 USA
O.  +1 678.391.9615
 
www.terragotech.com

-----Original Message-----
From: Even Rouault <[hidden email]>
Sent: Monday, January 14, 2019 3:38 PM
To: Alan Stewart <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure

On lundi 14 janvier 2019 20:25:42 CET Alan Stewart wrote:
> DYLD_LIBRARY_PATH

Maybe an effect of the SIP thing you'd need to disable ?
https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: osx compile failure

Alan Stewart-2

Thanks, Ivan, that’s very helpful.

 

I found that the .so  files in gdal/swig/python/build/lib.macosx-10.12-x86_64-2.7/osgeo/ are reference /usr/local/lib, while gdal/apps/.libs/gdalinfo references ~/install-gdal/lib/. This is so even after a fresh git clone.  That explains why pytest is not working. I see that gdal/ci/travix/osx/install.sh references the ~/install-gdal/lib/ path, so the question is why the swig files are being built with the /usr/local/lib/ path.

 

Alan Stewart

Senior Software Engineer

TerraGo Technologies

3200 Windy Hill Road, Suite 1550W

Atlanta, GA 30339 USA

O.  +1 678.391.9615

 

<a href="applewebdata://B24C0762-C7C9-4431-8518-ACC915448B89/www.terragotech.com">www.terragotech.com

 

From: Ivan Lucena <[hidden email]>
Sent: Wednesday, January 16, 2019 10:57 AM
To: Alan Stewart <[hidden email]>; Even Rouault <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure

 

Alan,

 

Did you used --without-libtool on ./configure?

 

If you do that, I believe that the file ./apps/gdalinfo and all other executables will be real executables and the makefiles will call "install_name_tool" to register the libgdal.dylib on the executables. In that case DYLD_LIBRARY_PATH will be ignored.

 

But, if you don't use --without-libtool on ./configure, that means you *do use libtool* and a lot of other interesting things will happen. Like, for example, the files that we think that are executables are not:

 

$ file apps/gdalinfo

apps/gdalinfo: POSIX shell script text executable, ASCII text

$ more apps/gdalinfo

#! /bin/sh

 

# gdalinfo - temporary wrapper script for .libs/gdalinfo

# Generated by libtool (GNU libtool) 2.4.6 Debian-2.4.6-0.1

#

# The gdalinfo program cannot be directly executed until all the libtool

# libraries that it depends on are installed.

#

# This wrapper script should never be moved out of the build directory.

# If it is, it will not operate correctly.

 

# Sed substitution that helps us do robust quoting.  It backslashifies

# metacharacters that are still active within double-quoted strings.

...

 

This is just the initial comments (whatever they means by "libtool libraries are installed") and there is a lot of lines trying to set and reset DYLD_LIBRARY_PATH. The real executable will be generated on this other folder:

 

$ file apps/.libs/gdalinfo

apps/.libs/gdalinfo: Mach-O 64-bit executable x86_64

 

But once you run "make install" the makefiles will generated a "clean" distribution folder where the executable are real executables.

 

In either way, with or without libtool, the key point is that DYLD_LIBRARY_PATH on MacOS is becoming deprecated or even useless as it is preceded by whatever "install_name_tool" defines.

 

One tools that you can use to figure out where the executable is getting the shared libraries from is "otool", for example:

 

$ otool -L apps/.libs/gdalinfo

 

And then I believe you can use "install_name_path" to delete the rpath that you disagree with. Maybe DYLD_LIBRARY_PATH will work then. If you really need.

 

That is my experience with setdevenv.sh on MacOS in regard to the DYLD_LIBRARY_PATH variable. 

 

I am sure you can find a lot of discussions on that theme over the web, for example

 

 

 

Good luck,

 

Ivan

 

 

 

 

 

 

 


From: gdal-dev <[hidden email]> on behalf of Alan Stewart <[hidden email]>
Sent: Tuesday, January 15, 2019 2:00 PM
To: Even Rouault
Cc:
[hidden email]
Subject: Re: [gdal-dev] osx compile failure

 

Disabling SIP makes no difference.

Alan Stewart
Senior Software Engineer
TerraGo Technologies
3200 Windy Hill Road, Suite 1550W
Atlanta, GA 30339 USA
O.  +1 678.391.9615
 
www.terragotech.com

-----Original Message-----
From: Even Rouault <[hidden email]>
Sent: Monday, January 14, 2019 3:38 PM
To: Alan Stewart <[hidden email]>
Cc: [hidden email]
Subject: Re: [gdal-dev] osx compile failure

On lundi 14 janvier 2019 20:25:42 CET Alan Stewart wrote:
> DYLD_LIBRARY_PATH

Maybe an effect of the SIP thing you'd need to disable ?
https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev