Travis-CI & OSGeo

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Regina Obe

Jachym,

 

Yes.  I currently manage PostGIS Jenkins, though I'd like to stream line the process so I can then apply to a more global OSGeo use.

 

I sadly haven't been keeping up to date with all the new features that have come into Jenkins like use of pipelines and travis like CI yaml scripts (there is even one plugin that can consume travis scripts)  which would be something that would be needed for all projects to be able to use a centralized Jenkins ci.

 

So I envision an Osgeo.org/Jenkins would be able to take advantage of these.  Right now the setup I have is not very extendable yet.

 

There is also some discussion of reviving the old buildbot and buildbot itself has changed quite a bit.

https://buildbot.net/index.html#basics

 

Not sure people's interest in reviving that.  I'm guessing in long run it would be easier to maintain than Jenkins, but I haven't set it up to know.

I started using Jenkins because of its WYSIWYG features that I could figure out without having to read and my inability to stand reading any documentation for longer than 5 minutes.

 

Thanks,

Regina

 

 

From: Jachym Cepicky [mailto:[hidden email]]
Sent: Monday, April 30, 2018 7:14 AM
To: [hidden email]
Cc: Jeffrey Johnson <[hidden email]>; OSGeo-Board <[hidden email]>; Sandro Santilli <[hidden email]>
Subject: Re: [Board] Travis-CI & OSGeo

 

Hi,

 

sorry jumping so late into this discussion. 

For PyWPS, we are using travis-ci infrastructure.

But I can imagine, that having osgeo.org/jenkins instance at hand, we would move straight way. I had chance to use jenkins in the past and it works nice. And it can be used for automatic build of packages etc .. did you consider this? I hope, I have not missed someones mail to this topic already

 

Jachym

 

út 24. 4. 2018 v 7:09 odesílatel Regina Obe <[hidden email]> napsal:

>  In many cases the code is developed primarily on github with Travis running tests (and often other tools in the build) and then mirrored onto a private

> gitlab and Jenkins setup where further quality assurance, release and deploy steps are run.

 

That's kind of the thing I wanted to push for thus my reason for joining SAC.  We have a couple of things we have in pipe – which we will do once we get new hardware and also using infrastructure Funtoo org was offered us. I'd use PostGIS as a proof of concept though it needs some cleanup.

 

> I would be in favor of this kind of integrated setup for OSGeo with both sides of the equation represented,

 

> but haven’t seen evidence that we are capable of organizing that, setting cross project standards and coming up with a more coherent plan that represents

 

> the organization clearly and professionally.

 

True admittedly there hasn't been evidence which has concerned me too, thus I wanted to be part of the solution and make evidence.

There is a certain self-defeating helplessness in projects that bothers me.  It's always a SAC does nothing for me. OSGeo does nothing for me.

In a sense it's self-defeating because if you expect nothing you will get nothing and you will frustrate those of us trying with (well they certainly don't care for my input).

 

On PostGIS side there is a lot of sentiment like that too – but I think hey I depend on OSGeo for a place to put my downloads, for my mailing lists, and at least currently

 

Sure old ways like Travis are useful and good for the present, but I worry about too much reliance of a single company especially one that has to answer to investors who could care less about open source.  The bigness of them to me is an artificial thing, and I'd hate that to stifle smaller players with great ideas or Uni kids wanting to explore administration and testing and thinking (why bother cause GitHub/Amazon etc. has already solved the world's problems)

 

 

To Even's point:

 

 

> That's the kind of workaround/cheating which causes others to pay for resources you

> use. And thus increase their bills... and/or may decrease the quality of the free tier.

There are several schools of thought to this. Mine are the following, which is why I feel little guilt

 

a)  In my mind, my use is advertising they can add to their statistics.  Private companies that use my work should be footing the bill and we/they should do more to extract money from these private companies.

 

b)  These companies (not speaking necessarily of Travis) are venture funded, so their progression is not organic and could be stifling growth of smaller companies who can provide similar services who are growing organically.  I'd rather see many more Travis's, AppVeyors, and Github than one big one for the same reason I appreciate having thousands of cloud hosters to choose from.

 

 

> I also hear your argument that there are solutions like Drone or GitLab that use

> open source software, which is great. But at some point you need to make that

> software run on servers, that you must buy or rent, and that you must monitor, maintain, etc.

 

The value I see in using an open solution is not just that I can run it on my own servers and tweak it for my own needs, also it's that it is more likely to lead to me having more choices of providers I can rely on that are easy to move to. Granted this starts getting messy with the plethora of OSL licenses.

 

Take my favorite example of PostgreSQL – there are a ton of PostgreSQL service providers (big and small players) now and all the big ones Amazon, Microsoft Azure, Google now offer it as a service.  Most are offering PostGIS at this point too cause they can since it too is open source.  This would not be possible if PostgreSQL is closed.

 

So yah each has their own tweaks, but I know hey if Amazon pisses me off, Microsoft has a close enough offering I can move to.

 

Thanks,

Regina

 

 

Again as a project lead I’m not quite sure what it is we got for going through incubation other than more eyeballs perhaps. 

 

On Mon, Apr 23, 2018 at 14:50 Regina Obe <[hidden email]> wrote:

Tim,

 

Well put and sorry for letting my sanctimony get in the way of my helpfulness.

Overall I think it was a good discussion and brought out some useful approaches that all projects can benefit from.

It's good to see so much interest on gitlab (as aired here and also on postgis irc).

 

Thanks,

Regina

 

 

From: Tim Sutton [mailto:[hidden email]]
Sent: Monday, April 23, 2018 5:13 PM
To: Vicky Vergara <
[hidden email]>
Cc: Regina Obe <
[hidden email]>; osgeo-board List <[hidden email]>; Sandro Santilli <[hidden email]>


Subject: Re: [Board] Travis-CI & OSGeo

 

Hi Regina and friends

 

Regarding using a discrete GitHub org, I agree this is good advice for Even. GIS.ORG has its own org in GitHub and we also benefit from exclusive use of the 5 travis jobs, autonomy over how we manage our org, and other good things.. Even: I know that Matthias Kuhn also squeezed a lot of mileage out of travis for QGIS, so you might want to chat to him and ask for any tips and tricks he might have to share. 

 

QGIS, by the way, is going to move to GitLab for a complex of reasons. Such moves are highly disruptive and should not be recommended lightly to a project. Especially in Even’s case who, if you follow hist activities, has only just moved GDAL over to GitHub and who is, I am sure, 100% not interested in doing the whole thing over again before another 10 years have passed :-P

 

Reflecting on the more ‘flaming arrow' parts of the discussion, I do think  it is good to take on board the general sentiment of the thread: if people write for help we should try to focus on solving their problems, not derail the conversation by airing views on their ‘bad’ technology choices. Even for one is, I am sure, well aware of what open source is, the benefits to humanity it offers etc. …… and also the utility he can get from a well run, stable and richly functional service like GitHub that can help him share his great work to the world.

 

Regards

 

Tim

 

 

On 23 Apr 2018, at 20:52, Vicky Vergara <[hidden email]> wrote:

 

Hi all,

I agree with Regina,

In my own experience with pgRouting as Community project:

We have pgRouting as a "GitHub organization" and have.


it has 3 sub-projects repositories:

pgrouting  -- very active using travis  (notice the lowercase r, so pgRouting its not only the thing used in postgreSQL)

pgRoutinglayers -- not using travis

osm2pgrouting -- somehow active using travis

in addition to that we also have:

Website

Workshop -- somehow active using travis

And a series of forks that we use to make contirbutions to other projects for example:

osgeo

Forked from OSGeo/osgeo



Also we have stalled ideas in other repositories under pgRouting

In total we have 15 repositories

And we can manage with the 5 concurrent jobs

A testing build of pgRouting has 6 jobs that 5 take around 8 to 9 minutes and 1 takes less than 3 minutes.

We are making use of being open and using the "free" advantages of github.

And we are not consuming OSGeo valuable resources that can be used in other key things.

VIcky

 

 

 

 

On Mon, Apr 23, 2018 at 12:36 PM, Regina Obe <[hidden email]> wrote:

Apology for the flaming arrows.  Howard and I love to throw arrows at each other.

 

Oh where do I start here?

 

From an idealist standpoint.  Yes I am crazy enough to think I can make a difference and provide something complementary to travis and appveyor.

I do not think they satisfy all needs without going thru some crazy hoops.

As strk mention gitlab is an alternative , though you may not be interested in it.

 

From a pragmatic standpoint I think "why are we sanctioning spending $5000/yr in the name of OSGeo projects on something that people need to be under OSGeo github to take advantage of?".

 

First of all if this is something GDAL needs and Proj needs, then it should come out of GDAL's/ Proj's budget J

So if Even were to say hay GDAL needs $5000/year for this thing, then by all means yes we should give it to him and he should put it in his budget.

GDAL deserves that much per year.  Same with Proj.

 

But don't make it a  "OSGeo Projects" need this. 

I don't need it I don't want it, and I don't see the point of having an OSGeo GitHub org that doesn't even reflect all our projects.

 

It still stands that this would be a non-issue if you guys didn't create a skeletal OSGeo github group where GDAL and proj.4 are the projects eating up all the workers.

If you each had your own project group on github you'd each have 5 workers end of story.  You wouldn't even need to waste funding on this. J

 

You'd also have more room to breathe as you can create many github subprojects as QGIS has done and not crowd everyone else out J

 

https://github.com/QGIS

 

When people go to github to look for GDAL or proj.4 they could care less you are an OSGeo project.  They already know YOU and are looking for YOU.

 

If someone goes to https://github.com/OSGEO they think - so these are all the projects OSGeo has – NOOOO.  It's NEGATIVE advertising.

 

So both my idealistic and pragmatic sides are disappointed by this movement to grow the github OSGeo Org for no benefit.

 

 

Thanks,

Regina

 

 

 

From: Howard Butler [mailto:[hidden email]]
Sent: Monday, April 23, 2018 1:04 PM
To: Regina Obe <
[hidden email]>
Cc: Tim Sutton <
[hidden email]>; Jeffrey Johnson <[hidden email]>; osgeo-board List <[hidden email]>; Sandro Santilli <[hidden email]>
Subject: Re: [Board] Travis-CI & OSGeo

 

 

On Apr 23, 2018, at 9:50 AM, Regina Obe <[hidden email]> wrote:

 

My issue is not that Even shouldn't be given the freedom to manage his project the way he wants.  Of course he should.

 

Yes it is. You're arguing that you can spend $5000 on something more worthy than supporting GDAL with the project infrastructure it wants to maintain under OSGeo's umbrella.

 

The point is that 

 

1)      He is limited because he is under the OSGEO Project infrastructure on Github.  If he were on his own project space, like PostGIS or QGIS (or Geos used to be), he wouldn't be limited by the 5 worker limit. I fail to see what benefit this Org is doing us when several of aour key projects aren't even on it (e.g. QGIS, GRASS, PostGIS) and even if they are what is the point, people should be lured to the osgeo website, not github.

 

This is a really good point. Why should projects that are to be on their own for infrastructure and support bother with putting anything under an OSGeo umbrella at all?

 

Or in a less snarky tone, OSGeo needs to decide if supporting projects with infrastructure is part of its mission. Member projects have no budgetary power to put resources into infrastructure capabilities that work for them of their choice, and there's a SAC beast that must be fed with money and take on new projects to continue to have relevance. Many projects actively avoid SAC and OSGeo resources because it is lesser quality infrastructure. It is lesser quality infrastructure because it is really damn hard to be all things to everybody. Add the fact that SAC is almost entirely unrecognized volunteer effort, and it is very difficult to succeed long term with any kind of staying power. 

 

GitHub, Travis, and AppVeyor are products. They cost money. They are specialized tools. They work really well. They have organizations behind them. They didn't exist in 2006 when OSGeo was formed, so we started down the path of building our own. If we started in 2018, I'm unconvinced we would have built our own.

 

3)      I think with $5000, that's almost the size of the osgeo budget for hardware.  I think of all the good we could do with $5000/yr and something that could help all projects not just things hosted on GitHub.

 

Infrastructure is *so* much more than a piece of hardware in a subsidized data center that graciously hosts us.

 

Like building up our own CI infrastructure that would test more than just Ubuntu.

 

You can use Docker with Travis to test whatever flavor you want.

 

And what about AppVeryor.  How much are you going to have to pay for that?  Is it under the same core limitations or will you have to shell out an additional $5000/yr for that?

 

Yes, I would hope so. Sandro is very vocal about his disdain for Windows, so I'm sure he will complain, but you've made good business supporting windows prisoners, so maybe you won't. 

 

Is OSGeo a software foundation that supports projects that make software, or is it an advocacy organization for users looking to get leverage off of free and open source software? The balance has been wildly tipped toward the latter the past 5 years... The board needs to clearly signal OSGeo's relationship to its member projects in this regard. I hope the board can ignore the flaming arrows Regina and I are shooting at each other in a battle that will never end and make a decision about OSGeo's relationship to its member projects and their infrastructure. If the relationship is "you must use SAC stuff", then the board needs to dump significantly more resources into burnishing that infrastructure to an outcome that might still be unsatisfactory. If the relationship is "use whatever you want and pay for it yourself", projects with the wherewithal to do so are going to do it outside of an organization that frankly isn't providing them with anything. If the relationship is "incubated projects get XXX dollars of infrastructure funding to spend on tools of its choosing", maybe this acts as both an incentive to incubate and a ring to reach for.

 

What we have now, where the answer is "volunteer your volunteers' time to build infrastructure within SAC" is not the solution.

 

Howard

 

 

 

 


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board




--

Georepublic UG (haftungsbeschränkt)
Salzmannstraße 44, 
81739 München, Germany
 
Vicky Vergara
Operations Research
 
eMail: vicky@georepublic.de
Web: https://georepublic.info
 
Tel: +49 (089) 4161 7698-1
Fax: +49 (089) 4161 7698-9
 
Commercial register: Amtsgericht München, HRB 181428
CEO: Daniel Kastl
 

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board

 

 


 

 

Tim Sutton

 

Co-founder: Kartoza

Project chair: QGIS.org

 

Visit http://kartoza.com to find out about open source:

 

Desktop GIS programming services

Geospatial web development

GIS Training

Consulting Services

 

Skype: timlinux 

IRC: timlinux on #qgis at freenode.net

 

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board


 

--

Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

delawen
Hi,

I see this conversation is stalled so, let me summarize it a bit to see how to approach it:

 * There is a need for more CI (Travis) threads on the OSGeo organization of Github
 * Some of us are not keen to give money to private infrastructure if there is an open source solution (Gitlab CI, Jenkins) and prefer to support open source even outside OSGeo
 * Moving from Github to Gitlab will be a drawback to the projects (part of the community will be left behind and lost). So, not an option ¿¿for short term??.
 * We could add a Jenkins from OSGeo that atke care of what Travis is doing right now. Advantage: we will have no restrictions. Disadvantage: ¿¿Integration on github??

Did I miss something here?

So, our best options look like:
 * Having the Jenkins that replaces Travis
 * Pay Travis more

And to decide we need to know:
 * Is having a Jenkins really an option here? Or something will be missing?
 * Will it be cheaper?

Who can answer this two questions?


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Regina Obe

I see this conversation is stalled so, let me summarize it a bit to see how to approach it:

 > * There is a need for more CI (Travis) threads on the OSGeo organization of Github

I'll leave this to Even to discuss how pressing this need is.  I think it depends on how pressing it is. As  I understand it there are GDAL and Proj (and one more other), I can't remember that use  the travis-ci OSGeo org resources heavily.  All the other projects I can think of pgRouting, GRASS, QGIS, MapServer, and PostGIS all have their own org so are not suffering from not having enough workers.  GEOS is on OSGeo Org – but we'd be happy to move GEOS off of OSGeo org, cause it's just a mirror anyway. Then again GEOS is not as busy as GDAL so probably not hogging any workers.

 

 

I think Sandro and Even have already started taking steps to setup a GDAL gitlab repo so that GitLab-CI tests can be done on gitlab for GDAL.

 

Sandro/Even correct me if I misspoke.

 

 * Some of us are not keen to give money to private infrastructure if there is an open source solution (Gitlab CI, Jenkins) and prefer to support open source even outside OSGeo

 

Correct.  I'd be willing to put in a fair amount of sweat labor to make this happen, because for my needs I find travis is not enough and too limiting in what I can do with it.

 

 

 * Moving from Github to Gitlab will be a drawback to the projects (part of the community will be left behind and lost). So, not an option ¿¿for short term??.

That is not true.  Mirroring from Github/Gitlab is trivial.  PostGIS already does it and enjoys travis, gitlab ci, dronie, and Jenkins.

So the only difficult part is setting up an image and a gitlab-ci.yml file which is a little different from travis ci.

For reference you can look at the ones PostGIS has

 

https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/.gitlab-ci.yml

 

https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/.travis.yml

 

 

Jenkins is probably a little harder to get going but it too has a yml like script a Jenkins – ci can listen for and use.

 

 

 

 * We could add a Jenkins from OSGeo that can take care of what Travis is doing right now. Advantage: we will have no restrictions. Disadvantage: ¿¿Integration on github??

Jenkins can integrate with github fine as it's pretty agnostic and has a ton of plugins for github integration, such as the github issues plugin

 

https://wiki.jenkins.io/display/JENKINS/GitHub+Issues+Plugin

I haven't explored all those features yet, on the Jenkins side I just use scripts stored in the repo of PostGIS / pgRouting

https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/ci

https://github.com/pgRouting/pgrouting/tree/master/ci

which for example for windows,

does experimental builds for windows users to test and for the debian tests against the head of all the PostgreSQL versions.

And also has slaves for testing freebsd and so forth -

So much of the frustration with bots (travis included) is just dealing with them when they wet their pants  or someone spills water on them.

Like those inconvenient times when travis decides to up the OS or you or the repo you are relying on is broken.

 

Did I miss something here?

So, our best options look like:

 > * Having the Jenkins that replaces Travis

I think people will still want to have Travis around since it's free machines and the easiest to set up.  With Jenkins we'd be using our own hardware or have to have people slave their machines.

But it would be easier to configure to push things to Project websites.

Think OSGeoLive having docs built and pushed to OSGeoLive website at any change.

That's one job Jenkins does for us on PostGIS side, updating the docs at every commit and pushing them to the website.

 

>  * Pay Travis more

And to decide we need to know:
 * Is having a Jenkins really an option here? Or something will be missing?

Again I don't think it should be an either / or. Each has their strengths and weaknesses.

E.g. appveyor tests windows, Travis doesn't.  Jenkins can test any OS you add as a slave or be in standalone mode.

Only thing that is not clear to me if having one Jenkins is sufficient.  I don't think it will be. But we can at the very least have a formulaic setup

And images ready should people want to use it and configure for their own needs.

 

Travis CI and Gitlab CI are probably the most interchangeable with Gitlab you can pay or host your own, but there is a bit of work translating your travis script to gitlab.

Travis is the free-tier or pay of which both are closed source solutions.


 > * Will it be cheaper?

It will definitely not be cheaper in terms of labor , but will provide more flexibility, I would focus on where that flexibility is needed first.

I think you can coax travis to do things like pushing artifacts to websites like binary builds but not as straightforward as it is with Jenkins.  I could be mistaken since I know a bit more about Jenkins than Travis.

I'm sure Hobu will correct my ignorance and rave about all the goodness of Travis J

 

Thanks,

Regina



 


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Jachym Cepicky
Hi,

afaik, Jenkins can be configured, that it uses several "cores" - so one
Jenkins instance can take advantage of more servers, so  it scales up
pretty well.

Actually, in my previous work, we used jenkins quite extensively,
developers and even power users were able to setup their own jobs, which
were taking fresh code from custom git repo ... it was more or less about
writing the shell script, which did all the work

so, if there would be jenkins instance available in osgeo infrastructure, I
would prefer it over travis (which pywps is using now)

just my opinion

J
út 1. 5. 2018 v 9:39 odesílatel Regina Obe <[hidden email]> napsal:

> I see this conversation is stalled so, let me summarize it a bit to see
how to approach it:

>   > * There is a need for more CI (Travis) threads on the OSGeo
organization of Github

> I'll leave this to Even to discuss how pressing this need is.  I think it
depends on how pressing it is. As  I understand it there are GDAL and Proj
(and one more other), I can't remember that use  the travis-ci OSGeo org
resources heavily.  All the other projects I can think of pgRouting, GRASS,
QGIS, MapServer, and PostGIS all have their own org so are not suffering
from not having enough workers.  GEOS is on OSGeo Org – but we'd be happy
to move GEOS off of OSGeo org, cause it's just a mirror anyway. Then again
GEOS is not as busy as GDAL so probably not hogging any workers.





> I think Sandro and Even have already started taking steps to setup a GDAL
gitlab repo so that GitLab-CI tests can be done on gitlab for GDAL.



> Sandro/Even correct me if I misspoke.



>   * Some of us are not keen to give money to private infrastructure if
there is an open source solution (Gitlab CI, Jenkins) and prefer to support
open source even outside OSGeo



> Correct.  I'd be willing to put in a fair amount of sweat labor to make
this happen, because for my needs I find travis is not enough and too
limiting in what I can do with it.





>   * Moving from Github to Gitlab will be a drawback to the projects (part
of the community will be left behind and lost). So, not an option ¿¿for
short term??.

> That is not true.  Mirroring from Github/Gitlab is trivial.  PostGIS
already does it and enjoys travis, gitlab ci, dronie, and Jenkins.

> So the only difficult part is setting up an image and a gitlab-ci.yml
file which is a little different from travis ci.

> For reference you can look at the ones PostGIS has




https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/.gitlab-ci.yml




https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/.travis.yml





> Jenkins is probably a little harder to get going but it too has a yml
like script a Jenkins – ci can listen for and use.







>   * We could add a Jenkins from OSGeo that can take care of what Travis is
doing right now. Advantage: we will have no restrictions. Disadvantage:
¿¿Integration on github??

> Jenkins can integrate with github fine as it's pretty agnostic and has a
ton of plugins for github integration, such as the github issues plugin



> https://wiki.jenkins.io/display/JENKINS/GitHub+Issues+Plugin

> I haven't explored all those features yet, on the Jenkins side I just use
scripts stored in the repo of PostGIS / pgRouting

> https://git.osgeo.org/gitea/postgis/postgis/src/branch/svn-trunk/ci

> https://github.com/pgRouting/pgrouting/tree/master/ci

> which for example for windows,

> does experimental builds for windows users to test and for the debian
tests against the head of all the PostgreSQL versions.

> And also has slaves for testing freebsd and so forth -

> So much of the frustration with bots (travis included) is just dealing
with them when they wet their pants  or someone spills water on them.

> Like those inconvenient times when travis decides to up the OS or you or
the repo you are relying on is broken.



> Did I miss something here?

> So, our best options look like:

>   > * Having the Jenkins that replaces Travis

> I think people will still want to have Travis around since it's free
machines and the easiest to set up.  With Jenkins we'd be using our own
hardware or have to have people slave their machines.

> But it would be easier to configure to push things to Project websites.

> Think OSGeoLive having docs built and pushed to OSGeoLive website at any
change.

> That's one job Jenkins does for us on PostGIS side, updating the docs at
every commit and pushing them to the website.



> >  * Pay Travis more

> And to decide we need to know:
>   * Is having a Jenkins really an option here? Or something will be
missing?

> Again I don't think it should be an either / or. Each has their strengths
and weaknesses.

> E.g. appveyor tests windows, Travis doesn't.  Jenkins can test any OS you
add as a slave or be in standalone mode.

> Only thing that is not clear to me if having one Jenkins is sufficient.
I don't think it will be. But we can at the very least have a formulaic
setup

> And images ready should people want to use it and configure for their own
needs.



> Travis CI and Gitlab CI are probably the most interchangeable with Gitlab
you can pay or host your own, but there is a bit of work translating your
travis script to gitlab.

> Travis is the free-tier or pay of which both are closed source solutions.


>   > * Will it be cheaper?

> It will definitely not be cheaper in terms of labor , but will provide
more flexibility, I would focus on where that flexibility is needed first.

> I think you can coax travis to do things like pushing artifacts to
websites like binary builds but not as straightforward as it is with
Jenkins.  I could be mistaken since I know a bit more about Jenkins than
Travis.

> I'm sure Hobu will correct my ignorance and rave about all the goodness
of Travis J



> Thanks,

> Regina







--
Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Regina Obe

> Hi,

> afaik, Jenkins can be configured, that it uses several "cores" - so one Jenkins instance can take advantage of more servers, so  it scales up pretty well.

> Actually, in my previous work, we used jenkins quite extensively, developers and even power users were able to setup their own jobs,
> which were taking fresh code from custom git repo ... it was more or less about writing the shell script, which did all the work

> so, if there would be jenkins instance available in osgeo infrastructure, I would prefer it over travis (which pywps is using now)


Yes Jenkins can scale fairly well especially when we throw slaves into the mix.
I was more thinking about the logistics of delegating administrative rights and configuring slaves in a fashion that power users can configure their own and know what's available on each slave for their tests, cleanup of workspace etc.
And also the interoperability of the projects.

The main reason I like Jenkins is for PostGIS work we need to follow PostgreSQL developments which means we need to test against the head of not yet released PostgreSQL which is something not easy to do with travis.

We also need to test against the head of GEOS and GDAL so one of the Jenkins bots we have uses the build output from PostgreSQL, GEOS, GDAL Jenkins jobs and does a matrix job (testing 5 versions of PostgreSQL including dev head) , GEOS head, GDAL head.

There is a lot of potential for synergy, but also a bit of "If I need to test GDAL for PostGIS purposes and GDAL needs to test GDAL, but I don't necessarily need to run all the tests GDAL needs to run, where is the balance of power there".  So that's my reasoning for why a single Jenkins might not be sufficient.  It's doable but takes a bit of planning.

As far as authentication we can use OSGeo LDAP and designate administrators from each project.  I think having 10 administrators that have full control of the instance from various projects that we all feel comfortable about working together is a workable solution.


Thanks,
Regina





_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

delawen
Hi,

In my experience, I like Jenkins more than Travis and I think that
with an owned Jenkins we will be able to do much more than with the
Travis service, but I understand that I may be missing something
important here. So it is up to each project to comment if Jenkins
would be a nice solution or not.

At least for GeoNetwork, it's true that we have Travis linked to our
github (not inside OSGeo organization but on our own), but, at the
same time, what we really use is a Jenkins we have on our own
infrastructure. Every time I try to use Travis, it is unstable and
weird. We can't trust the testing results, for example. But that may
be me, or Java, or GeoNetwork, or who knows, that makes Travis
unfriendly with our project.

So, going back to the beginning of the thread: Even, will a Jenkins be enough?


On Tue, May 1, 2018 at 2:59 PM, Regina Obe <[hidden email]> wrote:

>
>> Hi,
>
>> afaik, Jenkins can be configured, that it uses several "cores" - so one Jenkins instance can take advantage of more servers, so  it scales up pretty well.
>
>> Actually, in my previous work, we used jenkins quite extensively, developers and even power users were able to setup their own jobs,
>> which were taking fresh code from custom git repo ... it was more or less about writing the shell script, which did all the work
>
>> so, if there would be jenkins instance available in osgeo infrastructure, I would prefer it over travis (which pywps is using now)
>
>
> Yes Jenkins can scale fairly well especially when we throw slaves into the mix.
> I was more thinking about the logistics of delegating administrative rights and configuring slaves in a fashion that power users can configure their own and know what's available on each slave for their tests, cleanup of workspace etc.
> And also the interoperability of the projects.
>
> The main reason I like Jenkins is for PostGIS work we need to follow PostgreSQL developments which means we need to test against the head of not yet released PostgreSQL which is something not easy to do with travis.
>
> We also need to test against the head of GEOS and GDAL so one of the Jenkins bots we have uses the build output from PostgreSQL, GEOS, GDAL Jenkins jobs and does a matrix job (testing 5 versions of PostgreSQL including dev head) , GEOS head, GDAL head.
>
> There is a lot of potential for synergy, but also a bit of "If I need to test GDAL for PostGIS purposes and GDAL needs to test GDAL, but I don't necessarily need to run all the tests GDAL needs to run, where is the balance of power there".  So that's my reasoning for why a single Jenkins might not be sufficient.  It's doable but takes a bit of planning.
>
> As far as authentication we can use OSGeo LDAP and designate administrators from each project.  I think having 10 administrators that have full control of the instance from various projects that we all feel comfortable about working together is a workable solution.
>
>
> Thanks,
> Regina
>
>
>
>
>
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

michael terner-2
At the risk of chiming in as a non-developer, I'd observe that I'm not sure that Maria's final question ("will Jenkins be enough") accurately summarizes what I hear as the core take home of this thread. Rather, I think there is a legitimate difference of opinion on:
  1. Should all OSGeo projects use the same platform (whether it be Jenkins, or Travis, or other tooling)?
  2. Or, should projects be free to use the tools that work best from their points-of-view, and in light of the history of those projects?
  3. And, there's a subtext of what is "more important", having the "purity" of more open solutions; or the practicality of getting work done with tools that the people involved with projects may be familiar with, or simply prefer?
And of course, the overarching question is what should OSGeo do about it? 
  • Should there be a policy about what tools projects may be required to use?
  • Should OSGeo setup an open platform, that it supervises and makes available (on a voluntary basis) to projects?
  • Should projects be allowed to spend project money on commercial tools (i.e., if they include those expenses in their budgets; and even if some of those moneys come from sources like OSGeo)?
I think these bigger issues are the most important part of this thread, and from my reading there is not yet a clear consensus. This seems like an important issue (and policy; and direction statement) for OSGeo/the Board to address.

Count me squarely in the "practicality is important" camp. Indeed, to me producing great, open source software is the end goal. And the people that manage those projects should have a large say in what tools are best for them; as well as the freedom to spend money on those tools when appropriate (and with OSGeo's support). Indeed, there is no such thing as open source purity (well, maybe Stallman approaches that?), as significant parts of the software development ecosystem ranging from hardware to internet access are paid for commercially.

MT

On Tue, May 1, 2018 at 11:05 AM, María Arias de Reyna <[hidden email]> wrote:
Hi,

In my experience, I like Jenkins more than Travis and I think that
with an owned Jenkins we will be able to do much more than with the
Travis service, but I understand that I may be missing something
important here. So it is up to each project to comment if Jenkins
would be a nice solution or not.

At least for GeoNetwork, it's true that we have Travis linked to our
github (not inside OSGeo organization but on our own), but, at the
same time, what we really use is a Jenkins we have on our own
infrastructure. Every time I try to use Travis, it is unstable and
weird. We can't trust the testing results, for example. But that may
be me, or Java, or GeoNetwork, or who knows, that makes Travis
unfriendly with our project.

So, going back to the beginning of the thread: Even, will a Jenkins be enough?


On Tue, May 1, 2018 at 2:59 PM, Regina Obe <[hidden email]> wrote:
>
>> Hi,
>
>> afaik, Jenkins can be configured, that it uses several "cores" - so one Jenkins instance can take advantage of more servers, so  it scales up pretty well.
>
>> Actually, in my previous work, we used jenkins quite extensively, developers and even power users were able to setup their own jobs,
>> which were taking fresh code from custom git repo ... it was more or less about writing the shell script, which did all the work
>
>> so, if there would be jenkins instance available in osgeo infrastructure, I would prefer it over travis (which pywps is using now)
>
>
> Yes Jenkins can scale fairly well especially when we throw slaves into the mix.
> I was more thinking about the logistics of delegating administrative rights and configuring slaves in a fashion that power users can configure their own and know what's available on each slave for their tests, cleanup of workspace etc.
> And also the interoperability of the projects.
>
> The main reason I like Jenkins is for PostGIS work we need to follow PostgreSQL developments which means we need to test against the head of not yet released PostgreSQL which is something not easy to do with travis.
>
> We also need to test against the head of GEOS and GDAL so one of the Jenkins bots we have uses the build output from PostgreSQL, GEOS, GDAL Jenkins jobs and does a matrix job (testing 5 versions of PostgreSQL including dev head) , GEOS head, GDAL head.
>
> There is a lot of potential for synergy, but also a bit of "If I need to test GDAL for PostGIS purposes and GDAL needs to test GDAL, but I don't necessarily need to run all the tests GDAL needs to run, where is the balance of power there".  So that's my reasoning for why a single Jenkins might not be sufficient.  It's doable but takes a bit of planning.
>
> As far as authentication we can use OSGeo LDAP and designate administrators from each project.  I think having 10 administrators that have full control of the instance from various projects that we all feel comfortable about working together is a workable solution.
>
>
> Thanks,
> Regina
>
>
>
>
>
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board



--
Michael Terner
(M) 978-631-6602

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Jachym Cepicky
my understanding is, that having cuatom jenkins instace, would make it possible to address some of the issues we have with free travis-ci. and having jenkins at hand, would make it possible to some of the projects (like pywps and similar small) to make few steps towards CI. combination of both is possible.

On Tue, 1 May 2018, 17:59 michael terner, <[hidden email]> wrote:
At the risk of chiming in as a non-developer, I'd observe that I'm not sure that Maria's final question ("will Jenkins be enough") accurately summarizes what I hear as the core take home of this thread. Rather, I think there is a legitimate difference of opinion on:
  1. Should all OSGeo projects use the same platform (whether it be Jenkins, or Travis, or other tooling)?
  2. Or, should projects be free to use the tools that work best from their points-of-view, and in light of the history of those projects?
  3. And, there's a subtext of what is "more important", having the "purity" of more open solutions; or the practicality of getting work done with tools that the people involved with projects may be familiar with, or simply prefer?
And of course, the overarching question is what should OSGeo do about it? 
  • Should there be a policy about what tools projects may be required to use?
  • Should OSGeo setup an open platform, that it supervises and makes available (on a voluntary basis) to projects?
  • Should projects be allowed to spend project money on commercial tools (i.e., if they include those expenses in their budgets; and even if some of those moneys come from sources like OSGeo)?
I think these bigger issues are the most important part of this thread, and from my reading there is not yet a clear consensus. This seems like an important issue (and policy; and direction statement) for OSGeo/the Board to address.

Count me squarely in the "practicality is important" camp. Indeed, to me producing great, open source software is the end goal. And the people that manage those projects should have a large say in what tools are best for them; as well as the freedom to spend money on those tools when appropriate (and with OSGeo's support). Indeed, there is no such thing as open source purity (well, maybe Stallman approaches that?), as significant parts of the software development ecosystem ranging from hardware to internet access are paid for commercially.

MT

On Tue, May 1, 2018 at 11:05 AM, María Arias de Reyna <[hidden email]> wrote:
Hi,

In my experience, I like Jenkins more than Travis and I think that
with an owned Jenkins we will be able to do much more than with the
Travis service, but I understand that I may be missing something
important here. So it is up to each project to comment if Jenkins
would be a nice solution or not.

At least for GeoNetwork, it's true that we have Travis linked to our
github (not inside OSGeo organization but on our own), but, at the
same time, what we really use is a Jenkins we have on our own
infrastructure. Every time I try to use Travis, it is unstable and
weird. We can't trust the testing results, for example. But that may
be me, or Java, or GeoNetwork, or who knows, that makes Travis
unfriendly with our project.

So, going back to the beginning of the thread: Even, will a Jenkins be enough?


On Tue, May 1, 2018 at 2:59 PM, Regina Obe <[hidden email]> wrote:
>
>> Hi,
>
>> afaik, Jenkins can be configured, that it uses several "cores" - so one Jenkins instance can take advantage of more servers, so  it scales up pretty well.
>
>> Actually, in my previous work, we used jenkins quite extensively, developers and even power users were able to setup their own jobs,
>> which were taking fresh code from custom git repo ... it was more or less about writing the shell script, which did all the work
>
>> so, if there would be jenkins instance available in osgeo infrastructure, I would prefer it over travis (which pywps is using now)
>
>
> Yes Jenkins can scale fairly well especially when we throw slaves into the mix.
> I was more thinking about the logistics of delegating administrative rights and configuring slaves in a fashion that power users can configure their own and know what's available on each slave for their tests, cleanup of workspace etc.
> And also the interoperability of the projects.
>
> The main reason I like Jenkins is for PostGIS work we need to follow PostgreSQL developments which means we need to test against the head of not yet released PostgreSQL which is something not easy to do with travis.
>
> We also need to test against the head of GEOS and GDAL so one of the Jenkins bots we have uses the build output from PostgreSQL, GEOS, GDAL Jenkins jobs and does a matrix job (testing 5 versions of PostgreSQL including dev head) , GEOS head, GDAL head.
>
> There is a lot of potential for synergy, but also a bit of "If I need to test GDAL for PostGIS purposes and GDAL needs to test GDAL, but I don't necessarily need to run all the tests GDAL needs to run, where is the balance of power there".  So that's my reasoning for why a single Jenkins might not be sufficient.  It's doable but takes a bit of planning.
>
> As far as authentication we can use OSGeo LDAP and designate administrators from each project.  I think having 10 administrators that have full control of the instance from various projects that we all feel comfortable about working together is a workable solution.
>
>
> Thanks,
> Regina
>
>
>
>
>
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board



--
Michael Terner
(M) 978-631-6602
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Howard Butler-3
In reply to this post by Regina Obe


On 5/1/18 2:39 AM, Regina Obe wrote:
> I'm sure Hobu will correct my ignorance and rave about all the goodness of Travis :)
My point(s) about Travis are not about its technical non-superiority.
Ten years ago, Mateusz Loskot and I built and maintained an OSGeo
buildbot instance farm to do the same thing as you're proposing with
Jenkins. It's not about which approach is technically better. It's the
fact that buildbot required Mateusz and I to never burn out. We did.
Because we burned out, the projects could not depend on us or the
infrastructure we provided. It rotted and fell down.

Paying for Travis vs. building our own is about sustainability, not
technical prowess. *Travis never burns out* because we would be paying
for it. If it goes out of business, we can find another (Circle,
whatever) to pay for. Should OSGeo pay you so you never burn out (you
would anyway, suffering people's build farms is a thankless task)? Or
should it pay for specialists who only run build farms for many more
organizations and users than OSGeo's?

Projects that depend upon unstable infrastructure are unstable.
Infrastructure that is built, maintained, operated, and supported by
volunteer effort is more unstable than commercially supported
infrastructure when the volunteer pool is very small and sensitive to
heroes and burnout.

Howard


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board

signature.asc (536 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Regina Obe
Howard,

Very well stated. Yes that's what is in the back of my mind too.  I'm willing to put in quite a bit of time, but I don't want it to be a heroic one off thing that is not sustainable like you said as that would quickly lead to burn-out.

So yes we do need to come up with a way to make this sustainable and not be a heroic effort.  I'm not convinced that we can't have people take part ownership of this and feel a part of it or get people contributing for funding of a maintainer.  Seems to work pretty well for the PostgreSQL group and that's pretty much all volunteers.

I would first of all not start with hey everybody lets jump in and do this.  I would start off with folks where travis is not sufficient enough and eat the apple one bite at a time slowly getting people on board who have expressed interest not only in using but contributing to the workload and understanding the tool.

Like I said to me it's not a this or that. You can have both.  Travis is great for testing pull requests, and various gc etc.  When I need to control more I look to something like Jenkins or buildbot, but I still value what travis offers and wouldn't throw it out the window.

10 years was a long time ago. History repeats itself every 10 years with a slight twist.
I was pretty impressed with what you guys had done.  I'd like to try it again, but this time avoiding the "Burn-out" cliff.

Thanks,
Regina









-----Original Message-----
From: Howard Butler [mailto:[hidden email]]
Sent: Tuesday, May 01, 2018 1:22 PM
To: Regina Obe <[hidden email]>; 'María Arias de Reyna' <[hidden email]>
Cc: 'OSGeo-Board' <[hidden email]>; 'Sandro Santilli' <[hidden email]>
Subject: Re: [Board] Travis-CI & OSGeo



On 5/1/18 2:39 AM, Regina Obe wrote:
> I'm sure Hobu will correct my ignorance and rave about all the
> goodness of Travis :)
My point(s) about Travis are not about its technical non-superiority.
Ten years ago, Mateusz Loskot and I built and maintained an OSGeo buildbot instance farm to do the same thing as you're proposing with Jenkins. It's not about which approach is technically better. It's the fact that buildbot required Mateusz and I to never burn out. We did.
Because we burned out, the projects could not depend on us or the infrastructure we provided. It rotted and fell down.

Paying for Travis vs. building our own is about sustainability, not technical prowess. *Travis never burns out* because we would be paying for it. If it goes out of business, we can find another (Circle,
whatever) to pay for. Should OSGeo pay you so you never burn out (you would anyway, suffering people's build farms is a thankless task)? Or should it pay for specialists who only run build farms for many more organizations and users than OSGeo's?

Projects that depend upon unstable infrastructure are unstable.
Infrastructure that is built, maintained, operated, and supported by volunteer effort is more unstable than commercially supported infrastructure when the volunteer pool is very small and sensitive to heroes and burnout.

Howard


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Even Rouault-2
In reply to this post by Howard Butler-3

On mardi 1 mai 2018 12:22:26 CEST Howard Butler wrote:

> On 5/1/18 2:39 AM, Regina Obe wrote:

> > I'm sure Hobu will correct my ignorance and rave about all the goodness of

> > Travis :)

> My point(s) about Travis are not about its technical non-superiority.

 

I would not be able to articulate things more clearly than Howard did and due to his experience on this very precise topic, the value of his testimony should be be considered with care.

 

Due to lack of recent experience with Jenkins, I can't really comment if it is appropriate for the task. I think it can trigger builds on pull requests, can't it ? What about configuration of builds: is there the equivalent of a config file hosted in the source code repo (ala .travis.yml) to which any contributor can contribute changes to the CI scripts: fix them, add new configurations, etc ? If it is a task that can only be done by a select administrator of each project, then we add extra burden on its shoulders.

But the above is just details compared to a more fundamental difference when comparing Jenkins to Travis-CI/CircleCI/GitLab-CI/etc... Jenkins is just a software. Which machines would run it ? Has OSGeo a stock of 20 unused cores somewhere to support 10 concurrent builds ?

I don't really believe either in solutions where people bring their machines. I think that was how buildbot worked with the build slaves, right ?

My mention of having sufficiently hardware power is not just a theoretical concern: ironically a lot of flakes I've seen with GDAL and Travis were due to the GDAL autotest suite doing the mundane task of downloading (small) files from download.osgeo.org and the server being unresponsive due to probably being overloaded by other tasks. I've finally modified those tests to pull from rawcontent.github.com and those flakes have gone away.

 

And even if you have the appropriate software + the appropriate machines, the crucial human factor as raised by Howard remains. I feel my time is better spent at taking care at the sofware I'm knowleadgable of rather than at doing system administration/maintainance work for which I've little expertise/interest, and which implies to be available at the moment when things break.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Regina Obe

Even,

 

I'm going to say this to save people some time.  I don't think Jenkins is right for GDAL project or at least not yet. It might be useful to augment GDAL testing,

but I wouldn't expect GDAL to give up using travis.  So the only question is if you are being seriously inconvenienced by the lack of workers right now or is it something you are willing to wait a month or 2 to revisit.

 

If you feel it's holding you back, then by all means yes OSGeo should pay for it , and maybe add it as a GDAL/Proj.4 budget item.

 

My concern with having brought it up is my fear that people will start thinking they should get into the github/OSGeo.org group and then the pricing becomes exorbitant and starts straining other funds.

 

I can say for me, I am not happy with having just travis, I want to use other tools gitlab, Jenkins, etc. in addition to travis.  I enjoy tinkering with configuring software just as much as I like programming. I think others feel the same way that travis is insufficient given what people have said and that has surprised me, as I thought I was alone feeling travis was inadequate.

 

I also think as Mike Ternier says whether we should be in the business of dictating what projects can and can't use.  I definitely don't want to be in that business.

Each project should have a budget they can use for what they think they need, and be allowed to pool their budgets with projects that have similar needs.

 

 

> On mardi 1 mai 2018 12:22:26 CEST Howard Butler wrote:

>> On 5/1/18 2:39 AM, Regina Obe wrote:

>> > I'm sure Hobu will correct my ignorance and rave about all the goodness of

>> > Travis :)

>> My point(s) about Travis are not about its technical non-superiority.

 

> I would not be able to articulate things more clearly than Howard did and due to his experience on this very precise topic, the value of his testimony should be considered with care.

Yes Howard's testimony was very moving and I am thinking about it earnestly. 

 

> Due to lack of recent experience with Jenkins, I can't really comment if it is appropriate for the task. I think it can trigger builds on pull requests, can't it ?

               Yes

 

> What about configuration of builds: is there the equivalent of a config file hosted in the source code repo (ala .travis.yml) to which any contributor can contribute changes to the CI scripts:

 

>  fix them, add new configurations, etc ? If it is a task that can only be done by a select administrator of each project, then we add extra burden on its shoulders.

 

Yes it can via a Jenkins file and that is what we would need to do if we start growing Jenkins yes.  I admit to not having experience using JenkinsFile pipeline approach. That's the other reason I was thinking one instance might not be sufficient as people works with Jenkins a little differently   That said the Jenkins file structure is a very different from travis and gitlab, so travis and gitlab I feel are probably more comparable.  It can also use docker images

 

https://jenkins.io/doc/book/pipeline/syntax/

 

 

> But the above is just details compared to a more fundamental difference when comparing Jenkins to Travis-CI/CircleCI/GitLab-CI/etc... Jenkins is just a software. Which machines would run it ?

 

> Has OSGeo a stock of 20 unused cores somewhere to support 10 concurrent builds ?

               I'm not absolutely sure how many cores we have to spare.  We just ordered a new server, and much of the stuff on our old servers will be moved to this which would free up those machines.  Funtoo has also provided us hardware – I forget the details of how many cores that is. 

 

I was also thinking that now companies like Amazon, Microsoft Azure, Google Cloud are making serious money from selling things like PostGIS. With the right recipe for coaxing  we can get hardware and bandwidth since our work will make their product stronger. Since PostGIS relies on GDAL, well it's in their best interest to make sure GDAL has massive amounts of testing.  We need someone with good coaxing skills.  I'm looking at you Mike Ternier J

 

I don't really believe either in solutions where people bring their machines. I think that was how buildbot worked with the build slaves, right ?

I have my reservations on that too.

 

> My mention of having sufficiently hardware power is not just a theoretical concern: ironically a lot of flakes I've seen with GDAL and Travis were due to the GDAL autotest suite doing the mundane task of downloading (small) files from download.osgeo.org

>

 and the server being unresponsive due to probably being overloaded by other tasks. I've finally modified those tests to pull from rawcontent.github.com and those flakes have gone away.

 

That's a bit of a concern there and something we should investigate.  Sounds more like a bandwidth issue than hardware.

 

 

 

> And even if you have the appropriate software + the appropriate machines, the crucial human factor as raised by Howard remains. I feel my time is better spent at taking care at the sofware I'm

 

> knowleadgable of rather than at doing system administration/maintainance work for which I've little expertise/interest, and which implies to be available at the moment when things break.

 

> Even

 

I wouldn't have it any other way – keep GDAL going and focus on that.

 

 

Thanks,

Regina


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Angelos Tzotsos
Thank you all for the exciting discussion.

In my opinion we definitely do not want to dictate / force projects on how to operate.
I agree that it would be preferable for our projects to use free and open source tools for their needs and in this case Travis is Open Source (just not that easy to deploy on our own infrastructure).

The Board discussed this issue yesterday and there is an open motion:

It would be great if our projects would make the small effort to apply for a budget at the begining of the year (as we call them to) in order to have the money to spend in such cases.

Best,
Angelos

On Wed, May 2, 2018 at 8:00 AM, Regina Obe <[hidden email]> wrote:

Even,

 

I'm going to say this to save people some time.  I don't think Jenkins is right for GDAL project or at least not yet. It might be useful to augment GDAL testing,

but I wouldn't expect GDAL to give up using travis.  So the only question is if you are being seriously inconvenienced by the lack of workers right now or is it something you are willing to wait a month or 2 to revisit.

 

If you feel it's holding you back, then by all means yes OSGeo should pay for it , and maybe add it as a GDAL/Proj.4 budget item.

 

My concern with having brought it up is my fear that people will start thinking they should get into the github/OSGeo.org group and then the pricing becomes exorbitant and starts straining other funds.

 

I can say for me, I am not happy with having just travis, I want to use other tools gitlab, Jenkins, etc. in addition to travis.  I enjoy tinkering with configuring software just as much as I like programming. I think others feel the same way that travis is insufficient given what people have said and that has surprised me, as I thought I was alone feeling travis was inadequate.

 

I also think as Mike Ternier says whether we should be in the business of dictating what projects can and can't use.  I definitely don't want to be in that business.

Each project should have a budget they can use for what they think they need, and be allowed to pool their budgets with projects that have similar needs.

 

 

> On mardi 1 mai 2018 12:22:26 CEST Howard Butler wrote:

>> On 5/1/18 2:39 AM, Regina Obe wrote:

>> > I'm sure Hobu will correct my ignorance and rave about all the goodness of

>> > Travis :)

>> My point(s) about Travis are not about its technical non-superiority.

 

> I would not be able to articulate things more clearly than Howard did and due to his experience on this very precise topic, the value of his testimony should be considered with care.

Yes Howard's testimony was very moving and I am thinking about it earnestly. 

 

> Due to lack of recent experience with Jenkins, I can't really comment if it is appropriate for the task. I think it can trigger builds on pull requests, can't it ?

               Yes

 

> What about configuration of builds: is there the equivalent of a config file hosted in the source code repo (ala .travis.yml) to which any contributor can contribute changes to the CI scripts:

 

>  fix them, add new configurations, etc ? If it is a task that can only be done by a select administrator of each project, then we add extra burden on its shoulders.

 

Yes it can via a Jenkins file and that is what we would need to do if we start growing Jenkins yes.  I admit to not having experience using JenkinsFile pipeline approach. That's the other reason I was thinking one instance might not be sufficient as people works with Jenkins a little differently   That said the Jenkins file structure is a very different from travis and gitlab, so travis and gitlab I feel are probably more comparable.  It can also use docker images

 

https://jenkins.io/doc/book/pipeline/syntax/

 

 

> But the above is just details compared to a more fundamental difference when comparing Jenkins to Travis-CI/CircleCI/GitLab-CI/etc... Jenkins is just a software. Which machines would run it ?

 

> Has OSGeo a stock of 20 unused cores somewhere to support 10 concurrent builds ?

               I'm not absolutely sure how many cores we have to spare.  We just ordered a new server, and much of the stuff on our old servers will be moved to this which would free up those machines.  Funtoo has also provided us hardware – I forget the details of how many cores that is. 

 

I was also thinking that now companies like Amazon, Microsoft Azure, Google Cloud are making serious money from selling things like PostGIS. With the right recipe for coaxing  we can get hardware and bandwidth since our work will make their product stronger. Since PostGIS relies on GDAL, well it's in their best interest to make sure GDAL has massive amounts of testing.  We need someone with good coaxing skills.  I'm looking at you Mike Ternier J

 

I don't really believe either in solutions where people bring their machines. I think that was how buildbot worked with the build slaves, right ?

I have my reservations on that too.

 

> My mention of having sufficiently hardware power is not just a theoretical concern: ironically a lot of flakes I've seen with GDAL and Travis were due to the GDAL autotest suite doing the mundane task of downloading (small) files from download.osgeo.org

>

 and the server being unresponsive due to probably being overloaded by other tasks. I've finally modified those tests to pull from rawcontent.github.com and those flakes have gone away.

 

That's a bit of a concern there and something we should investigate.  Sounds more like a bandwidth issue than hardware.

 

 

 

> And even if you have the appropriate software + the appropriate machines, the crucial human factor as raised by Howard remains. I feel my time is better spent at taking care at the sofware I'm

 

> knowleadgable of rather than at doing system administration/maintainance work for which I've little expertise/interest, and which implies to be available at the moment when things break.

 

> Even

 

I wouldn't have it any other way – keep GDAL going and focus on that.

 

 

Thanks,

Regina


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board



--
Angelos Tzotsos, PhD
OSGeo Charter Member

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Sandro Santilli-4
In reply to this post by michael terner-2
On Tue, May 01, 2018 at 11:58:50AM -0400, michael terner wrote:

> Indeed, there is no such thing as open source
> purity (well, maybe Stallman approaches that?), as significant parts of the
> software development ecosystem ranging from hardware to internet access are
> paid for commercially.

Please do not confuse freedom of the software with commercial nature
of services. There are free software based services which are sold
commercially, and I think OSGeo should have no problem at all entering
that kind of commerce.

Producing great open source software is one goal, but I believe what
makes open source software GREAT is its community, and considering
yourself as part of a bigger community (the free software community)
adds to that value. This means helping each other to horizontally
improve the availability of free software. So helping a non-free
software spread is IMHO contrary to the goal of enhancing greatness
of our open source software. Compare that to help improving Jenkins
or Drone or GitLab-CE...

--strk;
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Sandro Santilli-4
In reply to this post by Howard Butler-3
On Tue, May 01, 2018 at 12:22:26PM -0500, Howard Butler wrote:

> Should OSGeo pay you so you never burn out (you
> would anyway, suffering people's build farms is a thankless task)? Or
> should it pay for specialists who only run build farms for many more
> organizations and users than OSGeo's?

When available (in this case it is) I'd be in favour of OSGeo paying
specialists who run build farms based on free software, so that when
that service goes out of business (the corporate version of "burn
out") we don't need to change our habits but can just change provider
of the _same_ (free software based) service.

--strk;
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

delawen
In reply to this post by Sandro Santilli-4
On Wed, May 2, 2018 at 11:33 AM, Sandro Santilli <[hidden email]> wrote:

> On Tue, May 01, 2018 at 11:58:50AM -0400, michael terner wrote:
>
>> Indeed, there is no such thing as open source
>> purity (well, maybe Stallman approaches that?), as significant parts of the
>> software development ecosystem ranging from hardware to internet access are
>> paid for commercially.
>
> Please do not confuse freedom of the software with commercial nature
> of services. There are free software based services which are sold
> commercially, and I think OSGeo should have no problem at all entering
> that kind of commerce.
>
> Producing great open source software is one goal, but I believe what
> makes open source software GREAT is its community, and considering
> yourself as part of a bigger community (the free software community)
> adds to that value. This means helping each other to horizontally
> improve the availability of free software. So helping a non-free
> software spread is IMHO contrary to the goal of enhancing greatness
> of our open source software. Compare that to help improving Jenkins
> or Drone or GitLab-CE...
>
> --strk;


Sidenote: Travis is, in fact, open source. An obscure open source
difficult to use, but open source in the end:
https://github.com/travis-ci

I believed all this time it was privative. Probably because it won't
qualify for our open source definition where anyone can really
download, build and use it easily...
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

delawen
In reply to this post by Sandro Santilli-4
As Angelos noted, the board already moved in favor of buying the
subscription to Travis CI, at least until we have a better choice (if
we ever have it). Please, continue this discussion and consider:

* If a project has a need, the best moment to ask for money is at the
end of the year, when we write the budget. This can be modified later,
as we are doing now, but that makes it more difficult to be fair
distributing money.
* If it is something that is going to be shared with more projects, it
won't be a burden for the project that request the money. We just need
to know this is a shared resource and it would be good knowing who is
going to benefit from it.
* If some other projects have some alternative, please document and
share it. Like having a mirror on gitlab and use gitlab-ci.

Personally, I prefer to have an owned Jenkins. Specially if we note
that the maximum number of threads in Travis is 10 (even paying),
which is easily done with a simple Jenkins on a decent hardware, no
need for a farm of slaves. But I understand your concerns and your
will to simplify the work and just rent something which is already
working for you.


On Wed, May 2, 2018 at 11:36 AM, Sandro Santilli <[hidden email]> wrote:

> On Tue, May 01, 2018 at 12:22:26PM -0500, Howard Butler wrote:
>
>> Should OSGeo pay you so you never burn out (you
>> would anyway, suffering people's build farms is a thankless task)? Or
>> should it pay for specialists who only run build farms for many more
>> organizations and users than OSGeo's?
>
> When available (in this case it is) I'd be in favour of OSGeo paying
> specialists who run build farms based on free software, so that when
> that service goes out of business (the corporate version of "burn
> out") we don't need to change our habits but can just change provider
> of the _same_ (free software based) service.
>
> --strk;
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Sandro Santilli-4
In reply to this post by delawen
On Wed, May 02, 2018 at 11:40:52AM +0200, María Arias de Reyna wrote:
> On Wed, May 2, 2018 at 11:33 AM, Sandro Santilli <[hidden email]> wrote:

> >helping a non-free
> > software spread is IMHO contrary to the goal of enhancing greatness
> > of our open source software.
>
> Sidenote: Travis is, in fact, open source. An obscure open source
> difficult to use, but open source in the end:
> https://github.com/travis-ci

Ok, so this is one of those cases of "open-source as marketing".
That is, it is "open" by marketing but effectively "closed" community
wise. Common to many "recent" companies.

> I believed all this time it was privative. Probably because it won't
> qualify for our open source definition where anyone can really
> download, build and use it easily...

Right. "How do we evaluate open source" was an interesting thread on
the "discuss" mailing list, must be still in the archives. It's a
community we're after, not just a tool (imho).

--strk;
_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Even Rouault-2
In reply to this post by Angelos Tzotsos

Hi,

 

> The Board discussed this issue yesterday and there is an open motion:

> https://www.loomio.org/d/6xBUHvaV/approve-an-investment-of-5000-for-a-yearly

> -travis-support

>

 

Has the motion been approved ?

 

 

> It would be great if our projects would make the small effort to apply for

> a budget at the begining of the year (as we call them to) in order to have

> the money to spend in such cases.

 

Yep, that will be a good item to put for next year budget

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
Reply | Threaded
Open this post in threaded view
|

Re: Travis-CI & OSGeo

Angelos Tzotsos

On Fri, May 11, 2018 at 2:46 PM, Even Rouault <[hidden email]> wrote:

Hi,

 

> The Board discussed this issue yesterday and there is an open motion:

> https://www.loomio.org/d/6xBUHvaV/approve-an-investment-of-5000-for-a-yearly

> -travis-support

>

 

Has the motion been approved ?

 

 

> It would be great if our projects would make the small effort to apply for

> a budget at the begining of the year (as we call them to) in order to have

> the money to spend in such cases.

 

Yep, that will be a good item to put for next year budget

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




--
Angelos Tzotsos, PhD
OSGeo Charter Member

_______________________________________________
Board mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/board
123