Alternative osgeo4w infrastructure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Alternative osgeo4w infrastructure

Hugo Mercier
Hi,

We've been involved recently in windows packaging based on osgeo4w. Some
of our customers needed binaries for windows for some of our projects
which require a recent C++ compiler.

We would like to share and discuss our current infrastructure and see if
there are some shared interests.

We've setup a compilation / CI infrastructure based on Gitlab CI that
triggers Virtual box virtual machines for the actual compilation and
packaging.
We use Windows 10 "development" VM
(https://developer.microsoft.com/en-us/windows/downloads/virtual-machines)
with Visual Studio 2015.
It allows us to have a common clean compilation environment.
Here is an example of a build log:
https://gitlab.com/Oslandia/tempus_core/builds/14314756

For now, one of our server "supersedes" the official osgeo4w http
repository so that we have our own osgeo4w repository for our packages
and still benefit from updated packages from osgeo4w.
You can have a look at the packages we have so far here:
http://osgeo4w.oslandia.net/osgeo4w/x86_64/release/ (some packages have
src packages with them, but not all of them yet)

Among these new packages, some of them may be of interest for other
projects, mainly PostgreSQL and PostGIS (only minimal version of PostGIS
for now - no raster, no sfcgal).

The goal is then for us to contribute back to the osgeo4w project.

So far, we noticed some issues with the current osgeo4w project and
infrastructure:
- the main repository seems heavily loaded and hard to reach sometimes
- packaging fixes or addition of new packages relies on developers to
install and manage their own instance of a complete Windows development
environment.
- the build of the entire distribution is hard to reproduce
- the points listed above also result in a situation where there is a
mix of compiler versions used for packages. Which should be avoided
(http://siomsystems.com/mixing-visual-studio-versions/)
We actually hit a problem probably related to that
(https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to
recompile gdal and all its dependencies with msvc 2015.

So here are some propositions:
- find a way to add a mirroring system for the osgeo4w distribution. The
idea is to register a list of mirrors that would receive a copy of files
when they are uploaded to the main repository.
Maybe by using something like https://github.com/axkibe/lsyncd
- share access to our gitlab-ci infrastructure and vbox VMs (and allow
it to be copied somewhere else if needed). The idea would be to have a
unique repository with all package "sources" and all
the machinery needed to trigger a build of the whole distribution or of
some selected packages on a known environment.

Regarding the issue of mixing compiler versions, we apparently still
need to compile python2.7 modules with msvc 2010 (which is the version
used to compile python27 and all wheel binaries).
Apart from that (which may imply to maintain two versions of the same
libraries), do you see problems if all packages get compiled with the
same (last) version of msvc (2015) ?

Are there any interests for that work ? Raise your hand if so, and raise
both hands if you could contribute (time or funds) :) we also still need
to find a way to fund this alternative osgeo4w / CI server.

Hope to read your comments.

Hugo
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: Alternative osgeo4w infrastructure

André William
Hi Hugo!

I'm a C++ dev that just started working on the QGIS project, and I gotta say the OSGeo4W build system is something I had avoided so far because of its complexity.
I'd be happy to help with testing and coding in this project, although I might need some time (and some pointers) to get up and running.

André

2017-04-27 13:23 GMT-03:00 Hugo Mercier <[hidden email]>:
Hi,

We've been involved recently in windows packaging based on osgeo4w. Some
of our customers needed binaries for windows for some of our projects
which require a recent C++ compiler.

We would like to share and discuss our current infrastructure and see if
there are some shared interests.

We've setup a compilation / CI infrastructure based on Gitlab CI that
triggers Virtual box virtual machines for the actual compilation and
packaging.
We use Windows 10 "development" VM
(https://developer.microsoft.com/en-us/windows/downloads/virtual-machines)
with Visual Studio 2015.
It allows us to have a common clean compilation environment.
Here is an example of a build log:
https://gitlab.com/Oslandia/tempus_core/builds/14314756

For now, one of our server "supersedes" the official osgeo4w http
repository so that we have our own osgeo4w repository for our packages
and still benefit from updated packages from osgeo4w.
You can have a look at the packages we have so far here:
http://osgeo4w.oslandia.net/osgeo4w/x86_64/release/ (some packages have
src packages with them, but not all of them yet)

Among these new packages, some of them may be of interest for other
projects, mainly PostgreSQL and PostGIS (only minimal version of PostGIS
for now - no raster, no sfcgal).

The goal is then for us to contribute back to the osgeo4w project.

So far, we noticed some issues with the current osgeo4w project and
infrastructure:
- the main repository seems heavily loaded and hard to reach sometimes
- packaging fixes or addition of new packages relies on developers to
install and manage their own instance of a complete Windows development
environment.
- the build of the entire distribution is hard to reproduce
- the points listed above also result in a situation where there is a
mix of compiler versions used for packages. Which should be avoided
(http://siomsystems.com/mixing-visual-studio-versions/)
We actually hit a problem probably related to that
(https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to
recompile gdal and all its dependencies with msvc 2015.

So here are some propositions:
- find a way to add a mirroring system for the osgeo4w distribution. The
idea is to register a list of mirrors that would receive a copy of files
when they are uploaded to the main repository.
Maybe by using something like https://github.com/axkibe/lsyncd
- share access to our gitlab-ci infrastructure and vbox VMs (and allow
it to be copied somewhere else if needed). The idea would be to have a
unique repository with all package "sources" and all
the machinery needed to trigger a build of the whole distribution or of
some selected packages on a known environment.

Regarding the issue of mixing compiler versions, we apparently still
need to compile python2.7 modules with msvc 2010 (which is the version
used to compile python27 and all wheel binaries).
Apart from that (which may imply to maintain two versions of the same
libraries), do you see problems if all packages get compiled with the
same (last) version of msvc (2015) ?

Are there any interests for that work ? Raise your hand if so, and raise
both hands if you could contribute (time or funds) :) we also still need
to find a way to fund this alternative osgeo4w / CI server.

Hope to read your comments.

Hugo
_______________________________________________
osgeo4w-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev


_______________________________________________
osgeo4w-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev
Reply | Threaded
Open this post in threaded view
|

Re: Alternative osgeo4w infrastructure

Jürgen E. Fischer
In reply to this post by Hugo Mercier
Hi Hugo,

On Thu, 27. Apr 2017 at 18:23:35 +0200, Hugo Mercier wrote:
> We've setup a compilation / CI infrastructure based on Gitlab CI that
> triggers Virtual box virtual machines for the actual compilation and
> packaging.

Sounds like something on my "one-fine-day" TODO list for long.


> So far, we noticed some issues with the current osgeo4w project and
> infrastructure:

> - the main repository seems heavily loaded and hard to reach sometimes

limited disk space is also a constantly reoccuring issue (although not actually visible
from the outside)

> - the build of the entire distribution is hard to reproduce

Indeed.

> - the points listed above also result in a situation where there is a mix of
>   compiler versions used for packages. Which should be avoided
>   (http://siomsystems.com/mixing-visual-studio-versions/)

That's certainly a fragile issue.  We don't track which compilers were used to
produce the individual packages - so it's also unclear on which version of the
msvcrt they depend on (msvcrt used to have several, although it was meanwhile
split up, the dependencies were still not refined everywhere).

But I think it's unavoidable sometimes - I wouldn't like to have three GDALs
and all their dependencies for QGIS 2 (msvc2010/py2), 3 (msvc2015/py3) and
GRASS (MinGW) in OSGeo4W (not to mention the GRASS plugin where all three would
meet anyway; also newer version of GDAL will not work with msvc2010 because of
c++11).  I think it's meant to work and it does, still it should be avoided
where possible.

Others:
* not listing of build dependencies (libraries and tools)
* no build recipes for everything
* those that exist don't have a form that would allow automatic invocation in a
  uniform way
* source packages don't contain sources nor automatically usable pointers
  to download them.

So there's much to do on that front before all this can work.

Also there are some proprietary SDKs (esp. for use with GDAL: ECW, MrSID,
FileGDB etc.) that I'm not sure you are allowed to make them public in such a
setup - although you are allowed to ship the redistributable runtime parts.


> We actually hit a problem probably related to that
> (https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to recompile
> gdal and all its dependencies with msvc 2015.

I think that is just due to the fact that numpy wasn't installed, when the
package was built.

That that problem is not already fixed, is just because I'm working on
transitioning building gdal with msvc2015 (needed for 2.2).

Having all this in place would have certainly helped with this of course.

The nightly should already work again - 2.2 not yet.


> So here are some propositions:
> - find a way to add a mirroring system for the osgeo4w distribution. The idea
> is to register a list of mirrors that would receive a copy of files
> when they are uploaded to the main repository.

AFAIK there is also a OSUSL file server/proxy that could be set in front
download.osgeo.org to help with this - bandwidth should be better.  Not sure if
that extra mirror architecture would still be necessary.


> - share access to our gitlab-ci infrastructure and vbox VMs (and allow
>   it to be copied somewhere else if needed). The idea would be to have a
>   unique repository with all package "sources" and all the machinery needed
>   to trigger a build of the whole distribution or of some selected packages
>   on a known environment.

Not sure if that also would be candidates for osgeo hosting - but self hosting
is probably easier and more flexible (also a reason why I didn't push this).
Perhaps leased hosted servers are even be cheaper than hosting "own" hardware
at OSUSL.


> Regarding the issue of mixing compiler versions, we apparently still
> need to compile python2.7 modules with msvc 2010 (which is the version
> used to compile python27 and all wheel binaries).

> Apart from that (which may imply to maintain two versions of the same
> libraries), do you see problems if all packages get compiled with the
> same (last) version of msvc (2015) ?

For GRASS this will certainly not work - that requires MinGW - no version of
msvc will work for that matter.


> Are there any interests for that work? Raise your hand if so, and raise
> both hands if you could contribute (time or funds) :) we also still need
> to find a way to fund this alternative osgeo4w / CI server.

Both hands full ;) - but consider them raised.

Although I have somewhat mixed feelings about this.  Except for Martin Landa
who took over GRASS of course, I have been pretty much on my own on this for a
while...


Jürgen

--
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de

_______________________________________________
osgeo4w-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

signature.asc (844 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Alternative osgeo4w infrastructure

Hugo Mercier
Hi Jüergen,

On 27/04/2017 21:42, Jürgen E. Fischer wrote:

>
>> - the points listed above also result in a situation where there is a mix of
>>   compiler versions used for packages. Which should be avoided
>>   (http://siomsystems.com/mixing-visual-studio-versions/)
>
> That's certainly a fragile issue.  We don't track which compilers were used to
> produce the individual packages - so it's also unclear on which version of the
> msvcrt they depend on (msvcrt used to have several, although it was meanwhile
> split up, the dependencies were still not refined everywhere).
>
> But I think it's unavoidable sometimes - I wouldn't like to have three GDALs
> and all their dependencies for QGIS 2 (msvc2010/py2), 3 (msvc2015/py3) and
> GRASS (MinGW) in OSGeo4W (not to mention the GRASS plugin where all three would
> meet anyway; also newer version of GDAL will not work with msvc2010 because of
> c++11).  I think it's meant to work and it does, still it should be avoided
> where possible.
Yes indeed, it seems too complex to maintain 3 different versions of
base libraries.
From what I read in the pointed article, mixing msvcrt versions could
lead to problems that is hard to spot on first sight. Everything seems
to work well ... until it does not. But that's also right that the
osgeo4w stack is a good example of mixed libraries that just work. So
maybe it is not a big deal, but it would help if we know with which
versions of msvcrt packages have been compiled.

And ... thanks for mentioning mingw that I forgot.

>
> Others:
> * not listing of build dependencies (libraries and tools)
> * no build recipes for everything
> * those that exist don't have a form that would allow automatic invocation in a
>   uniform way
> * source packages don't contain sources nor automatically usable pointers
>   to download them.
>
> So there's much to do on that front before all this can work.
Yes. Having a central git repository to host package sources with PR and
reviews could help to maintain the quality of building scripts.

>
> Also there are some proprietary SDKs (esp. for use with GDAL: ECW, MrSID,
> FileGDB etc.) that I'm not sure you are allowed to make them public in such a
> setup - although you are allowed to ship the redistributable runtime parts.
>

Ah. You mean the sources cannot be made public outside of their official
servers ? We can only get binaries and package them ?

>
>> We actually hit a problem probably related to that
>> (https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to recompile
>> gdal and all its dependencies with msvc 2015.
>
> I think that is just due to the fact that numpy wasn't installed, when the
> package was built.
>
> That that problem is not already fixed, is just because I'm working on
> transitioning building gdal with msvc2015 (needed for 2.2).
>
> Having all this in place would have certainly helped with this of course.
>
> The nightly should already work again - 2.2 not yet.
Good to know, I'll have a look. Thanks !

>
>
>> So here are some propositions:
>> - find a way to add a mirroring system for the osgeo4w distribution. The idea
>> is to register a list of mirrors that would receive a copy of files
>> when they are uploaded to the main repository.
>
> AFAIK there is also a OSUSL file server/proxy that could be set in front
> download.osgeo.org to help with this - bandwidth should be better.  Not sure if
> that extra mirror architecture would still be necessary.
I don't know exactly how that works. How are files added to
download.osgeo.org/osgeo4w ? By ftp/scp I guess ? Could this proxy be
configured so that each file copied to osgeo.org gets copied to a list
of mirrors ? Or does the mirror have to synchronize periodically with
the main site ?

How much space would be required to host an osgeo4w repository ?


>> Are there any interests for that work? Raise your hand if so, and raise
>> both hands if you could contribute (time or funds) :) we also still need
>> to find a way to fund this alternative osgeo4w / CI server.
>
> Both hands full ;) - but consider them raised.
>
> Although I have somewhat mixed feelings about this.  Except for Martin Landa
> who took over GRASS of course, I have been pretty much on my own on this for a
> while...

Yes you already do a lot of things ! The intention behind this is also
to make it easier for good souls to help you :)



_______________________________________________
osgeo4w-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev

signature.asc (220 bytes) Download Attachment