Continuous Deployment for OSGeo4W

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Continuous Deployment for OSGeo4W

Matthias Kuhn 🌍

Dear PSC,

The largest population among our user base is using Windows, packages to Windows users are shipped through osgeo4w and a standalone installer which, if I am not wrong, is also built from the osgeo4w infrastructure. Currently the osgeo4w infrastructure is under the osgeo umbrella.

Looking at the core team page of the project [1], there are 3 people involved, 2 of them packaging and one of them taking the big share of things. Thank you Jürgen ;-)

The infrastructure in the background is to my knowledge not documented and we are happy if it just works. And if it breaks if there is someone with knowledge and access rights to fix things.

I just stumbled upon a repository from Oslandia [2] that streamlines the packaging and integrates it with continuous deployment apparently. Having such a repository under QGIS or OSGeo umbrella could help to take away the burden of maintaining the Windows distribution from very few people, having a pull request review scheme as we have for source code and distribute the responsibility.

In short, having a repository where package scripts and build scripts are publicly available, forkable, builds are reproducible and where it's possible to make pull requests.

The main question in the first line is probably, is that something which is considered a QGIS task, an OSGeo task or a task that individual companies should be taking?

Thanks a lot

Matthias

[1] https://trac.osgeo.org/osgeo4w/wiki/CoreTeam
[2] https://gitlab.com/Oslandia/osgeo4w/

--
Matthias Kuhn
[hidden email]
<a href="tel:+41764356763">+41 (0)76 435 67 63

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Jürgen E. Fischer
Hi Matthias,

On Mon, 11. Jun 2018 at 14:17:37 +0200, Matthias Kuhn wrote:
> Looking at the core team page of the project [1], there are 3 people
> involved, 2 of them packaging and one of them taking the big share of things.
> Thank you Jürgen ;-)

That page isn't well maintained either - Matt has been off the radar for long.

 
> The infrastructure in the background is to my knowledge not documented and we
> are happy if it just works. And if it breaks if there is someone with
> knowledge and access rights to fix things.

The OSGeo4W infrastructure is download.osgeo.org.  It's documented in the trac
wiki.  The only required access right is a osgeo id with shell access.

There's no building infrastructure.  So maintainers have to build their
packages on their own machine and upload them.  Some of the package (most of
mine for instance) also contain (-src) packages with the build recipes for the
package.

But there currently is no "standardized" way to invoke them, upload the package
and run the packaging scripts.  The build recipes are diverse and use cmake,
nmake, ninja, gnu make, batch files, shell scripts (cygwin/mingw) and use
several different version of compilers (multiple versions of MSVC & mingw gcc).
Some even require third party SDKs that cannot be redistributed.  There's no
base build system or way to install build dependencies.

AFAIK automated packages only exists for QGIS, GDAL and GRASS - those run on
internal infrastructure (here inhouse for QGIS/GDAL and I think at the
university Martin works for GRASS) and upload the results.


> I just stumbled upon a repository from Oslandia [2] that streamlines the
> packaging and integrates it with continuous deployment apparently.

Odd that you had to stumbled upon it - this was also presented and discussed at
the Madeira hackfest and it's one of the benifits of moving from github to
gitlab.

But it currently also depends on internal build servers (at oslandia).  It
doesn't upload to the osgeo4w repository, but integrates the repository and has
utility scripts to invoke the build, harvest the packages and add them to taht
repository.  So the current packaging scripts would have to be adapted.


> Having such a repository under QGIS or OSGeo umbrella could help to take away
> the burden of maintaining the Windows distribution from very few people,
> having a pull request review scheme as we have for source code and distribute
> the responsibility.

> In short, having a repository where package scripts and build scripts are
> publicly available, forkable, builds are reproducible and where it's possible
> to make pull requests.

> The main question in the first line is probably, is that something which is
> considered a QGIS task, an OSGeo task or a task that individual companies
> should be taking?

This might introduce a long enough pole to make it bearable for more devs to
touch windows and get more contributors for OSGeo4W. ;)

If we wouldn't need (to spend funds on) windows licenses to get this to fly, we
might already have public build machines.  And those seems to be the current
missing bits.


Jürgen


--
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc

signature.asc (844 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Nyall Dawson
In reply to this post by Matthias Kuhn 🌍
On Mon, 11 Jun 2018 at 22:17, Matthias Kuhn <[hidden email]> wrote:

> Looking at the core team page of the project [1], there are 3 people involved, 2 of them packaging and one of them taking the big share of things. Thank you Jürgen ;-)
>
> The infrastructure in the background is to my knowledge not documented and we are happy if it just works. And if it breaks if there is someone with knowledge and access rights to fix things.
>
> I just stumbled upon a repository from Oslandia [2] that streamlines the packaging and integrates it with continuous deployment apparently. Having such a repository under QGIS or OSGeo umbrella could help to take away the burden of maintaining the Windows distribution from very few people, having a pull request review scheme as we have for source code and distribute the responsibility.
>
> In short, having a repository where package scripts and build scripts are publicly available, forkable, builds are reproducible and where it's possible to make pull requests.
>
> The main question in the first line is probably, is that something which is considered a QGIS task, an OSGeo task or a task that individual companies should be taking?
>

Thanks Matthias for raising this, Jürgen for keeping things trucking
for so long, and Oslandia for getting on board with this too!

I'm wondering - in the very short term, would it be possible for
someone involved here to bump the GEOS version from 3.5.0 to 3.6.1
(not 3.6.2 -- see below!)? We have a *lot* of crash reports being
filed where the stack trace indicates that the crash is originating
within GEOS, and is hopefully fixed in a more recent GEOS build. From
my following of hub issues I suspect these GEOS related crashes are
the most common crashes experienced by our users on Windows, and we're
seeing an average of 1-2 reports a week with the same origin within
GEOS, yet very different causes within QGIS (e.g. changing symbol
style, creating layout maps, running processing algs, etc).

(I don't think we should go to GEOS 3.6.2 because this regression:
https://trac.osgeo.org/geos/ticket/867 would impact QGIS far too
heavily)

Nyall
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Jürgen E. Fischer
Hi Nyall,

On Tue, 12. Jun 2018 at 08:58:48 +1000, Nyall Dawson wrote:
> I'm wondering - in the very short term, would it be possible for
> someone involved here to bump the GEOS version from 3.5.0 to 3.6.1
> (not 3.6.2 -- see below!)?

Done.


Jürgen

--
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de
QGIS release manager (PSC)  Germany                    IRC: jef on FreeNode

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc

signature.asc (844 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Nyall Dawson
On Tue, 12 Jun 2018 at 18:11, Jürgen E. Fischer <[hidden email]> wrote:
>
> Hi Nyall,
>
> On Tue, 12. Jun 2018 at 08:58:48 +1000, Nyall Dawson wrote:
> > I'm wondering - in the very short term, would it be possible for
> > someone involved here to bump the GEOS version from 3.5.0 to 3.6.1
> > (not 3.6.2 -- see below!)?
>
> Done.

Thanks -- much appreciated!

Nyall
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Matthias Kuhn 🌍
In reply to this post by Jürgen E. Fischer

Hi Jürgen


On 06/11/2018 10:27 PM, Jürgen E. Fischer wrote:

The infrastructure in the background is to my knowledge not documented and we
are happy if it just works. And if it breaks if there is someone with
knowledge and access rights to fix things.
The OSGeo4W infrastructure is download.osgeo.org.  It's documented in the trac
wiki.  The only required access right is a osgeo id with shell access.

There's no building infrastructure.  So maintainers have to build their
packages on their own machine and upload them.  Some of the package (most of
mine for instance) also contain (-src) packages with the build recipes for the
package.
That's good to know, that will make it easier to go on if (and we don't hope that of course) someone disappears.

But there currently is no "standardized" way to invoke them, upload the package
and run the packaging scripts.  The build recipes are diverse and use cmake,
nmake, ninja, gnu make, batch files, shell scripts (cygwin/mingw) and use
several different version of compilers (multiple versions of MSVC & mingw gcc).
Some even require third party SDKs that cannot be redistributed.  There's no
base build system or way to install build dependencies.

AFAIK automated packages only exists for QGIS, GDAL and GRASS - those run on
internal infrastructure (here inhouse for QGIS/GDAL and I think at the
university Martin works for GRASS) and upload the results.


I just stumbled upon a repository from Oslandia [2] that streamlines the
packaging and integrates it with continuous deployment apparently.
Odd that you had to stumbled upon it - this was also presented and discussed at
the Madeira hackfest and it's one of the benifits of moving from github to
gitlab.

Then this must be caused by my absence in Madeira ;)
If required a gitlab mirror of the current github repo to trigger gitlab-ci builds should be the smallest part to setup within this infrastructure ;)

But it currently also depends on internal build servers (at oslandia).  It
doesn't upload to the osgeo4w repository, but integrates the repository and has
utility scripts to invoke the build, harvest the packages and add them to taht
repository.  So the current packaging scripts would have to be adapted.
Reading this and the above paragraph, I think it's a quite good approach here, proxying to the existing pre-built packages and building a scripted repo on top that eventually can obsolete packages one-by-one.



Having such a repository under QGIS or OSGeo umbrella could help to take away
the burden of maintaining the Windows distribution from very few people,
having a pull request review scheme as we have for source code and distribute
the responsibility.

      
In short, having a repository where package scripts and build scripts are
publicly available, forkable, builds are reproducible and where it's possible
to make pull requests.

      
The main question in the first line is probably, is that something which is
considered a QGIS task, an OSGeo task or a task that individual companies
should be taking?
This might introduce a long enough pole to make it bearable for more devs to
touch windows and get more contributors for OSGeo4W. ;)
That's my hope :)

If we wouldn't need (to spend funds on) windows licenses to get this to fly, we
might already have public build machines.  And those seems to be the current
missing bits.
I read between the lines that there have been previous (PSC) discussions about this before. And I assume the bottom line was that donations should not be used to feed MS. So I think it might be an opportunity of user groups or organizations to jump in and fill the gap here. If that's the case, I will see if I can do something to get that fixed.

Matthias




Jürgen




_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc

--
Matthias Kuhn
[hidden email]
<a href="tel:+41764356763">+41 (0)76 435 67 63

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Matthias Kuhn 🌍
In reply to this post by Nyall Dawson
Hi Jürgen

On 06/12/2018 11:56 AM, Nyall Dawson wrote:

> On Tue, 12 Jun 2018 at 18:11, Jürgen E. Fischer <[hidden email]> wrote:
>> Hi Nyall,
>>
>> On Tue, 12. Jun 2018 at 08:58:48 +1000, Nyall Dawson wrote:
>>> I'm wondering - in the very short term, would it be possible for
>>> someone involved here to bump the GEOS version from 3.5.0 to 3.6.1
>>> (not 3.6.2 -- see below!)?
>> Done.
> Thanks -- much appreciated!
>
> Nyall

I hope you don't mind if I jump on the train and ask what's required to
include setuptools as a dependency of networkx (in the standalone
installer).

See
http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-OSGeo4W-networkx-broken-td5347038.html

Thanks
Matthias
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Hugo Mercier
In reply to this post by Matthias Kuhn 🌍
Hi !

Thanks for your interest Matthias ! My few comments below.

On 12/06/2018 13:11, Matthias Kuhn wrote:

>>> I just stumbled upon a repository from Oslandia [2] that streamlines the
>>> packaging and integrates it with continuous deployment apparently.
>> Odd that you had to stumbled upon it - this was also presented and discussed at
>> the Madeira hackfest and it's one of the benifits of moving from github to
>> gitlab.
>
> Then this must be caused by my absence in Madeira ;)
> If required a gitlab mirror of the current github repo to trigger
> gitlab-ci builds should be the smallest part to setup within this
> infrastructure ;)
>

The discussion we had in Madeira has been recorded and is available here:
https://www.youtube.com/watch?v=s1UtwdMep3w

>> But it currently also depends on internal build servers (at oslandia).  It
>> doesn't upload to the osgeo4w repository, but integrates the repository and has
>> utility scripts to invoke the build, harvest the packages and add them to taht
>> repository.  So the current packaging scripts would have to be adapted.

Indeed.
The build server is still shared internally with some other customer
projects. However I would be happy to give access to a few people with
enough motivation to test and enhance what we already have.

We still plan to dedicate this server to OSGeo4W builds, once we are
done with the current customer project and would like to give access to
a wider audience among OSGeo4W packagers little by little (and maybe
also to PostGIS folks ?).

We tried to be "close enough" to current conventions that are used by
compilation scripts so that the effort of adaptation is not that high.
But I think adaptation of scripts to a new infrastructure is inevitable.

> Reading this and the above paragraph, I think it's a quite good approach
> here, proxying to the existing pre-built packages and building a
> scripted repo on top that eventually can obsolete packages one-by-one.
>

That was the initial plan. We currently have an "Oslandia" repository
that is still unmerged with the "historical" OSGeo4W main repository,
which is not an ideal situation. Some of our packages are probably not
stable / complete enough to be merged, and we probably just did not take
the time to ask for an upload for the rest of them ...


>> If we wouldn't need (to spend funds on) windows licenses to get this to fly, we
>> might already have public build machines.  And those seems to be the current
>> missing bits.
> I read between the lines that there have been previous (PSC) discussions
> about this before. And I assume the bottom line was that donations
> should not be used to feed MS. So I think it might be an opportunity of
> user groups or organizations to jump in and fill the gap here. If that's
> the case, I will see if I can do something to get that fixed.
>

Good point. However I think the license costs would be negligible
compared to the hosting / maintenance cost.
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Jürgen E. Fischer
In reply to this post by Matthias Kuhn 🌍
Hi Matthias,

On Tue, 12. Jun 2018 at 13:13:12 +0200, Matthias Kuhn wrote:
> I hope you don't mind if I jump on the train and ask what's required to
> include setuptools as a dependency of networkx (in the standalone
> installer).

> See http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-OSGeo4W-networkx-broken-td5347038.html

Odd - I thought this was an none issue and cleared up.  Should be fixed now (by
updating the setup.hint of python-networkx).


Jürgen

--
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de
QGIS release manager (PSC)  Germany                    IRC: jef on FreeNode

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc

signature.asc (844 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Matthias Kuhn 🌍
On 06/12/2018 08:52 PM, Jürgen E. Fischer wrote:

> Hi Matthias,
>
> On Tue, 12. Jun 2018 at 13:13:12 +0200, Matthias Kuhn wrote:
>> I hope you don't mind if I jump on the train and ask what's required to
>> include setuptools as a dependency of networkx (in the standalone
>> installer).
>> See http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-OSGeo4W-networkx-broken-td5347038.html
> Odd - I thought this was an none issue and cleared up.  Should be fixed now (by
> updating the setup.hint of python-networkx).
>
Thanks a lot Jürgen,

very glad to see this issue fixed!

Matthias
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Matthias Kuhn 🌍
In reply to this post by Hugo Mercier
Hi

Thanks for the information Hugo !

On 06/12/2018 05:17 PM, Hugo Mercier wrote:

> Hi !
>
> Thanks for your interest Matthias ! My few comments below.
>
> On 12/06/2018 13:11, Matthias Kuhn wrote:
>>>> I just stumbled upon a repository from Oslandia [2] that streamlines the
>>>> packaging and integrates it with continuous deployment apparently.
>>> Odd that you had to stumbled upon it - this was also presented and discussed at
>>> the Madeira hackfest and it's one of the benifits of moving from github to
>>> gitlab.
>>
>> Then this must be caused by my absence in Madeira ;)
>> If required a gitlab mirror of the current github repo to trigger
>> gitlab-ci builds should be the smallest part to setup within this
>> infrastructure ;)
>>
>
> The discussion we had in Madeira has been recorded and is available here:
> https://www.youtube.com/watch?v=s1UtwdMep3w

Nice, thanks to the video recording team :)


>
>>> But it currently also depends on internal build servers (at oslandia).  It
>>> doesn't upload to the osgeo4w repository, but integrates the repository and has
>>> utility scripts to invoke the build, harvest the packages and add them to taht
>>> repository.  So the current packaging scripts would have to be adapted.
>
> Indeed.
> The build server is still shared internally with some other customer
> projects. However I would be happy to give access to a few people with
> enough motivation to test and enhance what we already have.
That sounds interesting indeed. So you are the point of contact for this?

>
> We still plan to dedicate this server to OSGeo4W builds, once we are
> done with the current customer project and would like to give access to
> a wider audience among OSGeo4W packagers little by little (and maybe
> also to PostGIS folks ?).
>
> We tried to be "close enough" to current conventions that are used by
> compilation scripts so that the effort of adaptation is not that high.
> But I think adaptation of scripts to a new infrastructure is inevitable.

That sounds good, also I can't really see an issue with sharing things
with PostGIS folks. In contrary, that will avoid situations as seen in
the past on Ubuntu with ubuntugis repository where installing qgis and
postgis on the same platform was sometimes hard because they depended on
different gdal libs.
Is there any mid-term plan to integrate this with the main OSGeo4W
repository? And to move those servers under the osgeo or qgis umbrella?


>
>> Reading this and the above paragraph, I think it's a quite good approach
>> here, proxying to the existing pre-built packages and building a
>> scripted repo on top that eventually can obsolete packages one-by-one.
>>
>
> That was the initial plan. We currently have an "Oslandia" repository
> that is still unmerged with the "historical" OSGeo4W main repository,
> which is not an ideal situation. Some of our packages are probably not
> stable / complete enough to be merged, and we probably just did not take
> the time to ask for an upload for the rest of them ...

I see. That's exactly where an integrated continuous deployment
infrastructure would help :)

>
>
>>> If we wouldn't need (to spend funds on) windows licenses to get this to fly, we
>>> might already have public build machines.  And those seems to be the current
>>> missing bits.
>> I read between the lines that there have been previous (PSC) discussions
>> about this before. And I assume the bottom line was that donations
>> should not be used to feed MS. So I think it might be an opportunity of
>> user groups or organizations to jump in and fill the gap here. If that's
>> the case, I will see if I can do something to get that fixed.
>>
>
> Good point. However I think the license costs would be negligible
> compared to the hosting / maintenance cost.

That is more than likely, yes.
I'd count the hosting costs on the same budget than the license costs.
For the maintenance costs it's a question if it should be volunteer, if
one company should be contracted to do that and/or if maintenance is
also something that can be put in the community hands (e.g. travis right
now is 99% maintained through scripts in the repo).

Matthias
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Deployment for OSGeo4W

Hugo Mercier
Hi,

On 15/06/2018 11:06, Matthias Kuhn wrote:
> Hi
>
> Thanks for the information Hugo !

I realize I did not answer. I am sorry for that :-/

>> On 12/06/2018 13:11, Matthias Kuhn wrote:
>>
>> Indeed.
>> The build server is still shared internally with some other customer
>> projects. However I would be happy to give access to a few people with
>> enough motivation to test and enhance what we already have.
> That sounds interesting indeed. So you are the point of contact for this?
Yes I am :)

> That sounds good, also I can't really see an issue with sharing things
> with PostGIS folks. In contrary, that will avoid situations as seen in
> the past on Ubuntu with ubuntugis repository where installing qgis and
> postgis on the same platform was sometimes hard because they depended on
> different gdal libs.
> Is there any mid-term plan to integrate this with the main OSGeo4W
> repository? And to move those servers under the osgeo or qgis umbrella?

You're right.
There is no strict mid-term plan. We are on a best effort path here, and
PostgreSQL/PostGIS are good candidates for integration with the main
OSGeo4W repo, as soon as we get the compilation issues fixed :-)
https://gitlab.com/Oslandia/osgeo4w/issues/12
https://gitlab.com/Oslandia/osgeo4w/issues/16

>> Good point. However I think the license costs would be negligible
>> compared to the hosting / maintenance cost.
>
> That is more than likely, yes.
> I'd count the hosting costs on the same budget than the license costs.
> For the maintenance costs it's a question if it should be volunteer, if
> one company should be contracted to do that and/or if maintenance is
> also something that can be put in the community hands (e.g. travis right
> now is 99% maintained through scripts in the repo).

On our side, one reason we want to spend some effort on this approach is
that the current situation relies on too few people and who are not
directly paid for that.
My personal feelings are that it should be handled by the community in
the long-term with a sustainable way of funding the people involved in.
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc