[OSGeo] #2411: Jenkins build server to replace build.geoserver.org

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+-------------------
 Reporter:  jive           |      Owner:  sac@…
     Type:  task           |     Status:  new
 Priority:  normal         |  Milestone:
Component:  Systems Admin  |   Keywords:
---------------------------+-------------------
 Planet has informed us that they would like to retire build.geoserver.org
 (currently hosted on AWS).

 The GeoServer? PSC would like to ask explore options with SAC:

 - Is it possible to setup Jenkins on an OSGeo virtual machine(s)
 - Some care needs to be taken with both resource use and security
 - Or is it recommended to take over AWS hosting (will ask for an estimate
 of monthly costs)

 We do have the jenkins configuration and so on saved in private OSGeo
 gitlab repo presently. The server is setup to create and throwaway
 workers.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 Long term it would be good to have windows and macOS jenkins servers (to
 assist open source projects in creating signed installers). We should plan
 for this needed even if we do not implement right now.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:1>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 Discussion: It may be appropriate to use Microsoft Azure pipelines with
 offer linux, windows and macOS builds. With enough automation that could
 be used to sign an installer.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:2>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

geoserver automated builds, OSGeo SAC and critical long-term services

Brian M Hamlin
Dear OSGeo SAC, admins, volunteers and stakeholders

In email today there is a topic and SAC ticket, related to a Geoserver
build-server being discontinued  by Planet .. This is a great example of
a modern situation. Indeed, a suggestion followed to move the same
services, plus build services, plus signing services, to yet another
Giant Corp's infrastructure-as-a-service.

   SAC at OSGeo -- own your own hardware, run your own AUTH, sign your
own software

   Anyone who has been unjustly treated in business will know what this
means. "Business is never fair" heard this?

  Owning your Own Hardware is the right move over years, not hours
     why?   *  much harder to be shut down at short notice;  * much less
expensive over years (not hours);
  * platform access decisions, such as authentication (AUTH) are in the
hands of the group involved, not a for-profit, de-personal Corporate
entity that does not have your interests in their decision making;  * 
legacy software decisions are in the hands of the group involved;  "The
King Decides" .. a system of charity, leads to awkward situations, or
worse, as things change, on the other hand, owning the hardware and
internalizing important decisions, leads to autonomy;

Current news about Google Cloud platform .. running a a compute node on
their platform means, you use a Linux with Google modifications !   what
could go wrong.

   Deal with occassional breakins and re-build the server
   Build a trust network that is centered on OSGeo, not Giant Corp

   .. the right decision over years, not hours

   best regards from California  --Brian M Hamlin


_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by wildintellect):

 Can someone chime in how this might relate to our previous and ongoing
 build infrastructure? strk or robe?

 We should make a list of options to pick between. There seems to be many
 ways to go, and something on a cloud provider may scale better. I would
 like to rename the ticket to need Build, or CI service to highlight it
 need not be Jenkins (which I've had issues securing in the past).

 Jody can you provide a timeline on which this needs to happen.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:3>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by strk):

 The only CI service currently offered by OSGeo is http://dronie.osgeo.org.

 It is a "Drone CI" server and an handful of "agents" either donated
 by devs/users or hosted in OSGeo infrastructure. See
 https://wiki.osgeo.org/wiki/Dronie

 It is currently used by (to my knowledge):

   - PostGIS
   - GEOS
   - librttopo

 The server integrates with Gitea, automatically building what's
 pushed to a gitea repository (git.osgeo.org/gitea) and cannot be
 connected to other sources (each server connects to a single source).

 Windows runners are supported, although we don't currently run any:
 https://readme.drone.io/runner/docker/overview/

 Jenkins is something PostGIS/GEOS also do use, but outside of OSGeo
 infrastructure. It may be good to provide it as a shared facility for
 different users. I would note that Jenkins is also known to possibly
 integrate with Gitea.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:4>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by robe):

 I have been thinking about this a bit.  Sadly I don't think a shared
 Jenkins would work too well or at least not in my skill set.  We could
 setup a dedicate VM/Container for Jenkins.

 PostGIS does run it's main jenkins in LXD/LXC container but it doesn't use
 throw away slaves.

 jive, can we discuss on OSGeo SAC irc the needs of GeoServer.  Depending
 on your bandwidth needs Amazon or some other Cloud provider might be your
 best fit and off course the funds would be paid by OSGeo.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:5>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 GeoCat has volunteered a build server also, going to discuss in tomorrow's
 GeoServer meeting.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:6>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 A shared build server was used previously for geotools, geowebcache,
 geoserver, geoscript, geogig, etc...

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:7>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 I am ready to work on this if a Nexus instance can be made available.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:8>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by robe):

 jive are we talking about nexus here or jenkins or both?

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:9>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by robe):

 jive it just occurred to me that I can take an image of your geotools
 jenkins server like I did with the old osgeo servers when I migrated them.
 That might save some work -- if you can give me access or let me know
 someone to talk to to orchestrate this, that might be the easiest course
 of action.

 I also have osgeo3 all setup with lxd installed, so we can use that.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:10>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 The new jenkins server is now available at build.geoserver.org ...

 But we are having some kind of problem accessing repo.osgeo.org with
 consistent performance and wanted to ask if there is any network
 throttling or limiting in place on repo.osgeo.org?  A maven build and a
 denial of service attack look very similar with bursty traffic patterns.

 Our builds are failing after a sustained load on repo.osgeo.org with
 connection timeouts:
 <pre>
 [ERROR] Failed to execute goal on project geoserver:
 Could not resolve dependencies for project
 org.geoserver:geoserver:pom:2.18-SNAPSHOT:
 Could not transfer artifact org.geotools:gt-
 opengis:jar:24-20200503.120242-25 from/to osgeo-snapshots
 (https://repo.osgeo.org/repository/snapshot/):
 Connect to repo.osgeo.org:443 [repo.osgeo.org/140.211.15.6] failed:
 Connection timed out (Connection timed out) -> [Help 1]
 </pre>

 We have:

 - Observed this from build.geoserver.org
 - Set up a second build on a different cloud provider to verify the
 problem
 - however building as individuals around the world does not show this
 problem
 - builds of other downstream projects (geomea) is are not having a problem

 Next steps:

 - we are going to test a build that does not use repo.osgeo.org in order
 to double check this is an osgeo networking issue

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:11>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 I have also looked at the stats from monitor.osgeo.org and from
 repo.osgeo.org and cannot see any problems (but I do not know what I am
 looking for).

 Chat for this activity is on https://gitter.im/geotools/geotools

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:12>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 Okay so we ended up where:

 a) build job was stuck
 b) nexus was not doing anything

 And then it got unstuck!

 Here’s the chunk of log around where it was downloading the metadata:

 {{{
 [INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ geotools ---
 [INFO] Downloading from nexus: https://repo.osgeo.org/repository/geotools-
 snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml
 [INFO] Downloaded from nexus: https://repo.osgeo.org/repository/geotools-
 snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml (593 B at 0
 B/s)
 [INFO] Uploading to nexus: https://repo.osgeo.org/repository/geotools-
 snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom
 [INFO] Uploaded to nexus: https://repo.osgeo.org/repository/geotools-
 snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom
 (86 kB at 40 kB/s)
 }}}


 (593 B at 0 B/s) really looks like some sort of throttling to me.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:13>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 Seeing inconsistent timeouts on the geotools doc deploys:


 {{{
 + scp -oStrictHostKeyChecking=no index.zip
 [hidden email]:/home/geotools/stable
 ssh: connect to host docs.geotools.org port 22: Connection timed out
 lost connection
 }}}

 This is a second osgeo server, The interesting part is that they sometimes
 work fine (this failure did happen while geotools-master was downloading
 maven-metadata, so might be related)

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:14>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+--------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  new
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:
 Keywords:                 |
---------------------------+--------------------

Comment (by jive):

 Update, we are getting wildly different build times with some very long
 delays in europe. A wide range of folks are running builds in different
 parts of the world to try and figure out of this is a jenkins issue or a
 nexus issue.

 Examples:

 - usa: 07:26 min
 - amsterdam: 52:47 min
 - amsterdam: 06:57 min

 As shown above "something" has happened and performance has now improved,
 and we can see that improvement on the jenkins build server jobs which are
 presently running at a steady clip.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:15>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac
Reply | Threaded
Open this post in threaded view
|

Re: [OSGeo] #2411: Jenkins build server to replace build.geoserver.org

OSGeo
In reply to this post by OSGeo
#2411: Jenkins build server to replace build.geoserver.org
---------------------------+---------------------
 Reporter:  jive           |       Owner:  sac@…
     Type:  task           |      Status:  closed
 Priority:  normal         |   Milestone:
Component:  Systems Admin  |  Resolution:  fixed
 Keywords:                 |
---------------------------+---------------------
Changes (by jive):

 * status:  new => closed
 * resolution:   => fixed


Comment:

 Thanks to GeoCat for hosting!

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2411#comment:16>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.

_______________________________________________
Sac mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/sac