Upgrade to VC10 and GDAL 1.9

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi internals,

For the next release of MapGuide Open Source,  I would like to propose an upgrade to VC10 as the official compiler and a switch to GDAL 1.9 for the FDO GDAL Provider.

The official FDO 3.7 builds will be released using VC10 so I think we need to do the upgrade for compatibility purposes.  Do we really want more than one C/C++ allocator running amuck in our multi-threaded server app?

GDAL 1.7 is very old and GDAL 1.9 was released last month so a timely upgrade is in order.  I have already started work on the GDAL 1.9 upgrade for FDO http://trac.osgeo.org/fdo/ticket/810.  I'm putting the upgrade in an FDO sandbox until I can validate end-to-end functionality with MapGuide.

To get "full" GDAL format support, ie. ECW, MrSID, JPEG2000, etc., I am planning to use the same approach as MapGuide 2.2.  The standard install will contain basic file format support (TIFF, PNG, JPG) and post install replacement with a full GDAL binary drop will give support for the other formats.  Due to licensing restrictions, I think we have to use this approach anyway.

Eventually, I would like to include more out-of-the-box file format support for FDO GDAL.  That would be a topic for future discussion.

I should having a working MapGuide build by the end of the week.  A working MapGuide installer is further out and I would be happy to collaborate with any interested parties (Jackie, others?) on the installer work.

Please let me know if we need to write up an RFC for all this stuff, or if anyone has concerns with the proposed upgrades.

Regards,
Trevor

_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Hi Trevor,

By moving to VC10, we would need VC10 binaries of:

 - Apache
 - PHP
 - mod_jk.so

In both x86 and x64 flavours.

I noticed there's already a set of PHP windows binaries already checked into Oem\Php. Not sure if these are VC10 compiled. x64 binaries appears to be (depends show linkage to msvcr100). x86 binaries doesn't seem to be (depends show linkage to msvcr90)

I cannot find VC10 apache binaries anywhere (ApacheLounge, my usual port of call, does not offer VC10 apache builds). I have mixed success building these components from source, and this was just with the VC9 compiler.

The installer itself is not a big deal. Just a case of updating lots of file references.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi Jackie,

Maybe Mars or Arthur can provide some insight?  I assume Autodesk must be maintaining 64 bit VC10 Apache binaries in their internal tree.

http://trac.osgeo.org/mapguide/wiki/MapGuideRfc121

http://trac.osgeo.org/mapguide/ticket/1818


For the build.bat/build64.bat is msbuild.exe available under a pure VC10 environment?


Regards,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: January 24, 2012 7:42 PM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

Hi Trevor,

By moving to VC10, we would need VC10 binaries of:

 - Apache
 - PHP
 - mod_jk.so

In both x86 and x64 flavours.

I noticed there's already a set of PHP windows binaries already checked into
Oem\Php. Not sure if these are VC10 compiled. x64 binaries appears to be
(depends show linkage to msvcr100). x86 binaries doesn't seem to be (depends
show linkage to msvcr90)

I cannot find VC10 apache binaries anywhere (ApacheLounge, my usual port of
call, does not offer VC10 apache builds). I have mixed success building
these components from source, and this was just with the VC9 compiler.

The installer itself is not a big deal. Just a case of updating lots of file
references.

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4337086.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Hi Trevor,

build.bat/build64.bat uses whatever msbuild is defined by the env vars. So if you run this from a VS2010 command prompt, it should use the v4.0 msbuild.exe calling into the VC10 compiler.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Ok, I stand corrected.

build.bat/build64.bat will build using the VC10 compiler, but still runs msbuild against the VC9 solution files.

So I'll have to update these batch files

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi Jackie,

I have done these locally and haven't checked them in yet.  I'm trying to get a raster to show up in MapGuide / GDAL 1.9 before checking it in.

Out of curiosity, are raster bounds (for tiff with a world file) supposed to be working in trunk now, or do I have to patch something?


Regards,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: January 26, 2012 7:08 PM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

Ok, I stand corrected.

build.bat/build64.bat will build using the VC10 compiler, but still runs
msbuild against the VC9 solution files.

So I'll have to update these batch files

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4342091.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Hi Trevor,

If you're talking about the issue of the GDAL provider returning (+- 1000000) bounds, the fix has been in trunk for quite some time now.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
In reply to this post by Jackie Ng
A little update on my tale of getting VC10-compatible binaries.

I was able to successfully build Apache from source for both x86/x64, so far so good.

PHP is a whole different story and I am roadblocked at this point. The binaries under Oem\php does not seem to constitute a functional PHP distribution. Compare the file listings to the binary release of PHP 5.3.8 and you will see that it is missing a whole lot of dlls which I presume to be the compiled set of the prerequisite libraries needed by PHP (zlib, libiconv, libxml2, icu) as described here:

https://wiki.php.net/internals/windows/libs#third_party_libraries

So I tried in vain to build these from source:

 * icu - VS2005 sln was provided, converted to VS2010 and built without issues
 * zlib - No problems
 * libiconv - No msvc makefiles. No sln/vcproj files. Win32 build instructions only covers cygwin/mingw which is not acceptable!
 * libxml2 - Depends on libiconv, so I can't build this either.

I'm starting to think moving to VC10 is becoming a bad idea. We need VC10 web component binaries and I just cannot build all the required bits from scratch!

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi Jackie,

We have to move to VC10 for the MapGuide Server in order to use official FDO binaries.  FDO is not planning to release VC9 binaries.  We could keep the web tier on VC9.  This will make the build more complicated.  It looks as though PHP has not moved to VC10 anyway so this might be appropriate.

We may be able to integrate the msi work done Anindya by instead of building Apache and PHP ourselves http://www.anindya.com/2011/09.  Doing complete binaries for the entire Oem tree seems counter-productive (and hard).

As a fallback position, we could build a VC9 version of FDO for MapGuide and stay on VC9.  This makes it more difficult to integrate FDO service packs (ie. we have to do it, instead of clients just dropping in new FDO bins).

I am surprised that Autodesk has not raised this issue anywhere.  I figured they would have run across it as part of the RFC 121 work.
 
Regards,
Trevor

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: February 2, 2012 5:50 AM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

A little update on my tale of getting VC10-compatible binaries.

I was able to successfully build Apache from source for both x86/x64, so far
so good.

PHP is a whole different story and I am roadblocked at this point. The
binaries under Oem\php does not seem to constitute a functional PHP
distribution. Compare the file listings to the binary release of PHP 5.3.8
and you will see that it is missing a whole lot of dlls which I presume to
be the compiled set of the prerequisite libraries needed by PHP (zlib,
libiconv, libxml2, icu) as described here:

https://wiki.php.net/internals/windows/libs#third_party_libraries

So I tried in vain to build these from source:

 * icu - VS2005 sln was provided, converted to VS2010 and built without
issues
 * zlib - No problems
 * libiconv - No msvc makefiles. No sln/vcproj files. Win32 build
instructions only covers cygwin/mingw which is not acceptable!
 * libxml2 - Depends on libiconv, so I can't build this either.

I'm starting to think moving to VC10 is becoming a bad idea. We need VC10
web component binaries and I just cannot build all the required bits from
scratch!

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4358916.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
In reply to this post by Jackie Ng
Hey Jackie,

I think I hit the jackpot on Apache Lounge http://www.apachelounge.com/viewtopic.php?t=4228

Apache 2.2.21
PHP 5.3.8
Visual Studio 2010 64 bit

http://www.erazergermany.de/HOME/httPHPdSSL-Win64-2011-09-13.zip


The zip file contains php-cgi.exe and php5apache2_2.dll.  Since PHP no longer supports the ISAPI extension, it looks like we will need to move to FastCGI for IIS as per RFC 99 http://trac.osgeo.org/mapguide/wiki/MapGuideRfc99


Regards,
Trevor


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Hi Trevor,

The installer has already been updated to use and configure PHP to use FastCGI for IIS.

That httPHPdSSL distribution would've been ideal if it:

 - Also had the 32-bit build (which I couldn't find anywhere)
 - Apache and PHP weren't mixed together in the same directory (ugh!) because we want to use the same PHP for both Apache and IIS configurations.

Anyway, I'm gonna keep trucking along (gonna try with a slightly older libiconv which has working msvc makefiles)

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Ok another update,

I decided to give the binaries under Oem\php another look.

Doing a depends.exe scan of every compiled binary under Oem\php, it seems we only need to to compile the following libs for VS2010:

 - ICU (4.6 for x86, 4.8 for x64)
 - OpenSSL
 - PostgreSQL (for functional PHP extensions)
 - gettext
 - Oracle instant client (for functional PHP extensions). Probably can't bundle due to licensing.
 - Enchant (for functional PHP extension)
 - Interbase client (for functional PHP extensions). Need to check licensing if redistribution is possible

The problematic libs in question in my previous post have already been statically linked into the dlls that need it. So there is no need to compile them (yay!)

Most, if not all of the above libs I am confident of building for VS2010. These libs when combined with the existing binaries under Oem\php gives us something that resembles the official PHP windows binary distribution.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Got a question for the C++ experts (I humbly admit that I am not one :-) ).

dll compatibility problems concerning compiler version only come into play if dlls are exporting C++ classes and functions right?

Since Apache modules and PHP extensions do everything through C interfaces, the compiler used to build them shouldn't really matter. So in that case, we should just be able to use the VC9 Apache/PHP web tier components with a VC10 server right?

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi Jackie,

One of the biggest problems with mixing and matching compilers is that the C/C++ allocators are different for each compiler.  If a VC9 compiled dll mallocs a piece of memory and a VC10 compiled dll frees the same piece of memory then bad things can happen like memory leaks, crashes, etc.

Most libraries include a "Free this" API for memory/objects so this typically doesn't happen.  If the libraries and code written against them are a little naughty, then we can run into problems.  With half a million lines of FDO and MapGuide source and (millions?) of lines of third party libraries, I think we would be hard pressed to guarantee that all code and all APIs are being "good".

I have also seen memory checker tools get really confused with multiple C/C++ allocators in play.  I suspect they successfully hook one allocator and miss the other one.  This will make bug fixing even more problematic in a multi-threaded environment.

As a side note, the inclusion of (old) VC 2005 libraries through the GDAL ECW plug in was causing crashes in MapGuide 2.2.  That's why we had to go to a pure VS2008 for GDAL + extensions.

In summary, I would mixing if possible.

Trevor

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: February 3, 2012 9:34 AM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

Got a question for the C++ experts (I humbly admit that I am not one :-) ).

dll compatibility problems concerning compiler version only come into play
if dlls are exporting C++ classes and functions right?

Since Apache modules and PHP extensions do everything through C interfaces,
the compiler used to build them shouldn't really matter. So in that case, we
should just be able to use the VC9 Apache/PHP web tier components with a
VC10 server right?

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4362331.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
In reply to this post by Jackie Ng
Hi Jackie,

I wonder what would happen if we didn't release 32 bit on Windows?
[oh oh.. here comes the flame war]

For the GDAL 1.9 upgrade, I have posted a partial binary drop of MapGuide 2.4 64bit.  It was built from the same machine that produced the 2.2 builds.  The build process still needed some baby-sitting.  It's getting closer.

The zip is just a bunch of files so IIS will have to be configured manually.  The readme in the zip contains a few hints.  If the zip is extracted to C:\mg24x64, all of the directory references should work.  It's definitely an EARLY build.  It may be ok for development work.

If anyone has any concerns with the GDAL 1.9 upgrade, please try it out.  I think the FDO PSC will need a stamp of approval from the MapGuide PSC before we do the GDAL upgrade.

 
The drop contains:

64bit MapGuide Server for Windows
64bit PHP and .Net Web Extensions for IIS
64bit GDAL, WMS, OGR, SDF, SHP, ODBC, SQLite, and SQLServerSpatial FDO providers

http://download.osgeo.org/mapguide/testing/2.4.0/mg24x64test1.zip

The file size should be 223,608,232 bytes and the md5sum is b313f6280a0158d616b717a9d64abb0c.


Regards,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: February 2, 2012 11:25 PM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

Hi Trevor,

The installer has already been updated to use and configure PHP to use
FastCGI for IIS.

That httPHPdSSL distribution would've been ideal if it:

 - Also had the 32-bit build (which I couldn't find anywhere)
 - Apache and PHP weren't mixed together in the same directory (ugh!)
because we want to use the same PHP for both Apache and IIS configurations.

Anyway, I'm gonna keep trucking along (gonna try with a slightly older
libiconv which has working msvc makefiles)

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4360859.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Jackie Ng
Hi Trevor,

I gave this 64-bit build for a spin. Sheboygan loads up fine. There was some mapagent communication issues, which I'm guessing is due to an existing IIS-configured installation of MGOS 2.2 on my machine. Yeah, not an ideal test environment, but if anything this tells me what needs to be done to better support side-by-side IIS configured installations.

Is the build machine using SVN 1.6 and the latest build scripts from trunk? Because apart from fusion, I don't see any of the svn metadata being preserved in the web tier directories. The trunk build scripts preserve this metadata.

Also is the PHP in this build the same as the one you referenced from apachelounge? I've checked in this version (with the httpd binaries ripped out) for the installer.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade to VC10 and GDAL 1.9

Trevor Wekel
Hi Jackie,

You are correct, side by side is not 100% working due to the same ports being used in serverconfig.ini and webconfig.ini

I removed most of the SVN directories from the web tier directories and just kept the Fusion SVN directories.  I wasn't sure what we were keeping and what we were not.

The PHP build is the same one I referenced from Apache Lounge.

Regards,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: February 7, 2012 8:59 PM
To: [hidden email]
Subject: [mapguide-internals] Re: Upgrade to VC10 and GDAL 1.9

Hi Trevor,

I gave this 64-bit build for a spin. Sheboygan loads up fine. There was some
mapagent communication issues, which I'm guessing is due to an existing
IIS-configured installation of MGOS 2.2 on my machine. Yeah, not an ideal
test environment, but if anything this tells me what needs to be done to
better support side-by-side IIS configured installations.

Is the build machine using SVN 1.6 and the latest build scripts from trunk?
Because apart from fusion, I don't see any of the svn metadata being
preserved in the web tier directories. The trunk build scripts preserve this
metadata.

Also is the PHP in this build the same as the one you referenced from
apachelounge? I've checked in this version (with the httpd binaries ripped
out) for the installer.

- Jackie

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Upgrade-to-VC10-and-GDAL-1-9-tp4337041p4374819.html
Sent from the MapGuide Internals mailing list archive at Nabble.com.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals