[Qgis-community-team] Web infrastructure

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[Qgis-community-team] Web infrastructure

pcav
Hi all.
I'm noticing frequent downs in our infrastructure (now it's hub.qgis.org, the other
day was qgis.org, etc.). I feel we're having more traffic, and our current structure
is overstretched. Where can we find server stats to check this?
In any case, I think we should make our infrastructure stronger: opinion, suggestions?
All the best.
--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

Tim Sutton
Hi

On Mon, Sep 10, 2012 at 9:48 AM, Paolo Cavallini <[hidden email]> wrote:
> Hi all.
> I'm noticing frequent downs in our infrastructure (now it's hub.qgis.org, the other
> day was qgis.org, etc.). I feel we're having more traffic, and our current structure
> is overstretched. Where can we find server stats to check this?
> In any case, I think we should make our infrastructure stronger: opinion, suggestions?
> All the best.

This is what I would like to do:

- migrate QGIS.org web site to rst so that we can a) host it on
gh-pages b) make it translatable more easily c) ease the load on the
server
- host downloads on gh
- use QGIS.org only for hosting django and doing builds

An alterior motive is that we will have a single platform for *all
documentation* in QGIS and that we can use git as a content management
and versioning system for our web site.

Regards

Tim


> --
> Paolo Cavallini - Faunalia
> www.faunalia.eu
> Full contact details at www.faunalia.eu/pc
> Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
> _______________________________________________
> Qgis-community-team mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-community-team



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

pcav
Il 10/09/2012 09:51, Tim Sutton ha scritto:

> This is what I would like to do:
>
> - migrate QGIS.org web site to rst so that we can a) host it on
> gh-pages b) make it translatable more easily c) ease the load on the
> server
> - host downloads on gh
> - use QGIS.org only for hosting django and doing builds
>
> An alterior motive is that we will have a single platform for *all
> documentation* in QGIS and that we can use git as a content management
> and versioning system for our web site.

Agreed, nice plan. Can we work out a roadmap for it? Volunteers?
All the best.
--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

pcav
In reply to this post by Tim Sutton
Il 10/09/2012 09:51, Tim Sutton ha scritto:

> - migrate QGIS.org web site to rst so that we can a) host it on
> gh-pages b) make it translatable more easily c) ease the load on the
> server
> - host downloads on gh
> - use QGIS.org only for hosting django and doing builds
>
> An alterior motive is that we will have a single platform for *all
> documentation* in QGIS and that we can use git as a content management
> and versioning system for our web site.

Any volunteer to:
- set up the structure, under the guidance of Tim (AKA rsTim)
- migrate the content (very easy task, just copy>paste>rst2html>commit).
?
Thanks.
--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

luciano.campagnolo
On Mon, Sep 10, 2012 at 7:46 AM, Paolo Cavallini <[hidden email]> wrote:

> Il 10/09/2012 09:51, Tim Sutton ha scritto:
>
>> - migrate QGIS.org web site to rst so that we can a) host it on
>> gh-pages b) make it translatable more easily c) ease the load on the
>> server
>> - host downloads on gh
>> - use QGIS.org only for hosting django and doing builds
>>
>> An alterior motive is that we will have a single platform for *all
>> documentation* in QGIS and that we can use git as a content management
>> and versioning system for our web site.
>
> Any volunteer to:
> - set up the structure, under the guidance of Tim (AKA rsTim)
> - migrate the content (very easy task, just copy>paste>rst2html>commit).
> ?
> Thanks.

Hi,
I would be happy to collaborate...

Regards
Luciano
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

Tim Sutton
Hi

On Mon, Sep 10, 2012 at 1:18 PM, Luciano Campagnolo
<[hidden email]> wrote:

> On Mon, Sep 10, 2012 at 7:46 AM, Paolo Cavallini <[hidden email]> wrote:
>> Il 10/09/2012 09:51, Tim Sutton ha scritto:
>>
>>> - migrate QGIS.org web site to rst so that we can a) host it on
>>> gh-pages b) make it translatable more easily c) ease the load on the
>>> server
>>> - host downloads on gh
>>> - use QGIS.org only for hosting django and doing builds
>>>
>>> An alterior motive is that we will have a single platform for *all
>>> documentation* in QGIS and that we can use git as a content management
>>> and versioning system for our web site.
>>
>> Any volunteer to:
>> - set up the structure, under the guidance of Tim (AKA rsTim)
>> - migrate the content (very easy task, just copy>paste>rst2html>commit).
>> ?
>> Thanks.
>
> Hi,
> I would be happy to collaborate...
>

It will be great to have your help!

regards

Tim

> Regards
> Luciano



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

Tim Sutton
Hi

On Mon, Sep 10, 2012 at 8:06 PM, Tim Sutton <[hidden email]> wrote:

> Hi
>
> On Mon, Sep 10, 2012 at 1:18 PM, Luciano Campagnolo
> <[hidden email]> wrote:
>> On Mon, Sep 10, 2012 at 7:46 AM, Paolo Cavallini <[hidden email]> wrote:
>>> Il 10/09/2012 09:51, Tim Sutton ha scritto:
>>>
>>>> - migrate QGIS.org web site to rst so that we can a) host it on
>>>> gh-pages b) make it translatable more easily c) ease the load on the
>>>> server
>>>> - host downloads on gh
>>>> - use QGIS.org only for hosting django and doing builds
>>>>
>>>> An alterior motive is that we will have a single platform for *all
>>>> documentation* in QGIS and that we can use git as a content management
>>>> and versioning system for our web site.
>>>
>>> Any volunteer to:
>>> - set up the structure, under the guidance of Tim (AKA rsTim)
>>> - migrate the content (very easy task, just copy>paste>rst2html>commit).
>>> ?
>>> Thanks.
>>
>> Hi,
>> I would be happy to collaborate...
>>
>
> It will be great to have your help!


Beginnings are here:

https://github.com/qgis/qgis.org

Yawn....off to bed...

Tim


>
> regards
>
> Tim
>
>> Regards
>> Luciano
>
>
>
> --
> Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
> ==============================================
> Visit http://linfiniti.com to find out about:
>  * QGIS programming services
>  * Mapserver and PostGIS based hosting plans
>  * FOSS Consulting Services
> Skype: timlinux Irc: timlinux on #qgis at freenode.net
> ==============================================



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] [Qgis-developer] Web infrastructure

Tim Sutton
In reply to this post by Tim Sutton
Hi

On Tue, Sep 11, 2012 at 8:17 AM, Alex Mandel <[hidden email]> wrote:

> On 09/10/2012 12:51 AM, Tim Sutton wrote:
>> Hi
>>
>> On Mon, Sep 10, 2012 at 9:48 AM, Paolo Cavallini <[hidden email]> wrote:
>>> Hi all.
>>> I'm noticing frequent downs in our infrastructure (now it's hub.qgis.org, the other
>>> day was qgis.org, etc.). I feel we're having more traffic, and our current structure
>>> is overstretched. Where can we find server stats to check this?
>>> In any case, I think we should make our infrastructure stronger: opinion, suggestions?
>>> All the best.
>>
>> This is what I would like to do:
>>
>> - migrate QGIS.org web site to rst so that we can a) host it on
>> gh-pages b) make it translatable more easily c) ease the load on the
>> server
>> - host downloads on gh
>> - use QGIS.org only for hosting django and doing builds
>>
>> An alterior motive is that we will have a single platform for *all
>> documentation* in QGIS and that we can use git as a content management
>> and versioning system for our web site.
>>
>> Regards
>>
>> Tim
>>
>>
>
> http://webextra.osgeo.osuosl.org/munin/osgeo.org/qgis.osgeo.org.html
>
> It seems to be a memory leak of some sort. Some speculate that it might
> be ruby that runs hub.qgis. I actually think finding a new home for the
> nightly builds would free up plenty of resources for the web needs, our
> web server is not miniscule (2 cores, 4 GB of ram). You can also notice
> from the charts that the issue came up several times over a couple of
> days and then has not shown up for 5 days at all.
>

Can you clarify - are you suggesting that the build processing itself
should be moved or the hosting of the built packages?

> I agree with the moving to rst but would prefer to continue to host it
> using Sphinx to build the docs (obviously the files can live in git to
> get the benefits that Tim mentions). To me this is more flexible and we
> can install caching software. This is the same route Mapserver went a
> couple of years ago, and the same setup we use for osgeo-live.
>

Sorry I was unclear - the plan is already to use sphinx.

> As for downloads, we've always had downloads.osgeo.org available and can
> use that at any time. For osgeo-live we actually pushed our downloads to
> sourceforge recently to get 20 mirrors world-wide. If we really need
> more capacity I can offer university based servers (no $ for bandwidth,
> in addition to OSUOSL which is our current host and also at a univ).
>

Ok good to know.

> Basically I like github but cautious about using it for everything.
> Enough files and it will start costing $ to keep it up.
>

Oh - I thought it was all free for FOSS projects? My main interest is
to consolidate our infrastructure - it has too many moving parts and
not enough grease monkeys to keep the cogs oiled....

Regards

Tim


> Thanks,
> Alex



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] [Qgis-developer] Web infrastructure

pcav
Il 11/09/2012 08:29, Tim Sutton ha scritto:

> Oh - I thought it was all free for FOSS projects? My main interest is
> to consolidate our infrastructure - it has too many moving parts and
> not enough grease monkeys to keep the cogs oiled....

agreed, the current structure is very complex, better KIS.
all the best.
--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

pcav
In reply to this post by Tim Sutton
Il 11/09/2012 00:38, Tim Sutton ha scritto:

> Beginnings are here:
>
> https://github.com/qgis/qgis.org
>
> Yawn....off to bed...

Hi Tim,
thanks for this. Can we set up an experimental.qgis.org website to see the progresses?
I think this will motivate helpers.
All the best.
--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

Tim Sutton
Hi

On Tue, Sep 11, 2012 at 8:52 AM, Paolo Cavallini <[hidden email]> wrote:

> Il 11/09/2012 00:38, Tim Sutton ha scritto:
>
>> Beginnings are here:
>>
>> https://github.com/qgis/qgis.org
>>
>> Yawn....off to bed...
>
> Hi Tim,
> thanks for this. Can we set up an experimental.qgis.org website to see the progresses?
> I think this will motivate helpers.
> All the best.

Yes will set up a running deployment tonight probably at some gh url for now.

Regards

Tim

> --
> Paolo Cavallini - Faunalia
> www.faunalia.eu
> Full contact details at www.faunalia.eu/pc
> Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] [Qgis-developer] Web infrastructure

Tim Sutton
In reply to this post by Tim Sutton
Hi

On Tue, Sep 11, 2012 at 8:38 AM, Radim Blazek <[hidden email]> wrote:

>
>> http://webextra.osgeo.osuosl.org/munin/osgeo.org/qgis.osgeo.org.html
>>
>> It seems to be a memory leak of some sort.
>
> WCS test using Mapserver running on wcs.qgis.org was added during the
> summer. I cannot imagine however how it could cause memory leak as it
> is running as simple CGI.
>
> I can decrease the number of WCS tests for now.
>

I can't imagine it either - though I have had issues on a clients
server where UMN Mapserver creates runaway processes that never
terminate and eat up all available cpu power. If the WCS test instance
does become an issue I am sure we can make another plan - I prefer
that reducing the number of tests we run should be the last possible
alternative, not the first.

Regards

Tim

> Radim
>
>> Some speculate that it might
>> be ruby that runs hub.qgis. I actually think finding a new home for the
>> nightly builds would free up plenty of resources for the web needs, our
>> web server is not miniscule (2 cores, 4 GB of ram). You can also notice
>> from the charts that the issue came up several times over a couple of
>> days and then has not shown up for 5 days at all.
>>
>> I agree with the moving to rst but would prefer to continue to host it
>> using Sphinx to build the docs (obviously the files can live in git to
>> get the benefits that Tim mentions). To me this is more flexible and we
>> can install caching software. This is the same route Mapserver went a
>> couple of years ago, and the same setup we use for osgeo-live.
>>
>> As for downloads, we've always had downloads.osgeo.org available and can
>> use that at any time. For osgeo-live we actually pushed our downloads to
>> sourceforge recently to get 20 mirrors world-wide. If we really need
>> more capacity I can offer university based servers (no $ for bandwidth,
>> in addition to OSUOSL which is our current host and also at a univ).
>>
>> Basically I like github but cautious about using it for everything.
>> Enough files and it will start costing $ to keep it up.
>>
>> Thanks,
>> Alex
>> _______________________________________________
>> Qgis-developer mailing list
>> [hidden email]
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Visit http://linfiniti.com to find out about:
 * QGIS programming services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] Web infrastructure

luciano.campagnolo
In reply to this post by Tim Sutton
Thanks!
I'll look into it tonight

Cheers
Luc

On Tue, Sep 11, 2012 at 4:43 AM, Tim Sutton <[hidden email]> wrote:

> Hi
>
> On Tue, Sep 11, 2012 at 8:52 AM, Paolo Cavallini <[hidden email]> wrote:
>> Il 11/09/2012 00:38, Tim Sutton ha scritto:
>>
>>> Beginnings are here:
>>>
>>> https://github.com/qgis/qgis.org
>>>
>>> Yawn....off to bed...
>>
>> Hi Tim,
>> thanks for this. Can we set up an experimental.qgis.org website to see the progresses?
>> I think this will motivate helpers.
>> All the best.
>
> Yes will set up a running deployment tonight probably at some gh url for now.
>
> Regards
>
> Tim
>
>> --
>> Paolo Cavallini - Faunalia
>> www.faunalia.eu
>> Full contact details at www.faunalia.eu/pc
>> Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
>
>
>
> --
> Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
> ==============================================
> Visit http://linfiniti.com to find out about:
>  * QGIS programming services
>  * Mapserver and PostGIS based hosting plans
>  * FOSS Consulting Services
> Skype: timlinux Irc: timlinux on #qgis at freenode.net
> ==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team
Reply | Threaded
Open this post in threaded view
|

Re: [Qgis-community-team] [Qgis-developer] Web infrastructure

Tim Sutton-4
In reply to this post by pcav
Hi

On Tue, Sep 11, 2012 at 6:08 PM, Alex Mandel <[hidden email]> wrote:

> On 09/10/2012 11:36 PM, Paolo Cavallini wrote:
>> Il 11/09/2012 08:29, Tim Sutton ha scritto:
>>
>>> Oh - I thought it was all free for FOSS projects? My main interest is
>>> to consolidate our infrastructure - it has too many moving parts and
>>> not enough grease monkeys to keep the cogs oiled....
>>
>> agreed, the current structure is very complex, better KIS.
>> all the best.
>>
>
> It's free up to a size limit, if you look in your account details or the
> org details you'll see the quota. I agree with simplifying, which to me
> means keeping it all on the qgis.org server, because we can actually
> control all of that.
>

Well we are also trying to reduce the resource usage on that server so
if we could look into one of the other available options you mentioned
that would be great.

> Moving to sphinx instead of Joomla should help reduce the complexity a
> bit. I think we were also talking about Poodle for managing the rst
> translations (a django app) and we've already moved the wiki into
> redmine. So I think we're actually started on consolidation as is.
>

Yup there is already a pootle instance running but we are on hold
until the user manual transcription is complete. We have taken an
number of other steps in this consolidation process. Bottom line is I
think we are all committed to shifting over to rst/sphinx, and where
specifically stuff gets hosted is something we can discuss and adjust
as we go along. I'm really looking for a 'push to publish' solution
for all our materials (push in git and the site / doc /etc gets
published).

Regards

Tim

> Thanks,
> Alex
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer



--
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==============================================
_______________________________________________
Qgis-community-team mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-community-team