*early* Preparation of RFP 2021

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: RFP Evaluation method

Eli Adam
Hi Jonathan,

On Thu, Jun 6, 2019 at 11:44 AM Jonathan Moules
<[hidden email]> wrote:
>
> Hi Steven,
>
> Indeed, and I would then suggest taking all of that hard earned experience, knowledge, priorities, and weightings and putting them inside an objective, measurable framework where it is less susceptible to biases, both conscious and not.
>
> This would produce a much more open process, more in line with the O in OSGeo and the "open philosophy" part of the OSGeo Mission Statement. It also means experience and lessons learnt aren't lost when people leave the voting pool as with Cameron's input for instance.
>
> Depending on the scoring metrics used, it would also allow for a more direct comparison between proposals. And as a bonus, the scoring/metrics being open means it's open to comment and feedback from everyone, meaning the process is now more "participatory community driven development" (again, straight from the one-sentence Mission Statement).
>

I do find the questions from voting and non-voting members to be the
most useful part of the process.  Generally the list seems very open
to sincere participation from everyone.

> Cheers,
>
> Jonathan
>
>
> On 2019-06-06 19:18, Steven Feldman wrote:
>
> Jonathan
>
> Each member of the committee will bring their own priorities and experiences to the voting process.
>
> So for you the environmental considerations might be paramount while for someone else delivering a highly affordable delegate price may be their priority or another might be concerned about overall financial risk to OSGeo and someone else might be very focussed on diversity. We each have a different set of criteria and we also apply different levels of importance to those criteria.
>
> The current system allows each voter to apply their own criteria and weightings and to select the proposal that they think best, the majority vote then wins. I know when I vote I usually have a good feeling for one of the proposals based on a mix of factors, you could say that was unconscious bias, I would say it was a combination of instinct and experience
>
> cheers
> ______
> Steven
>
> Unusual maps in strange places -  mappery.org
>
> Subscribe to my weekly “Maps in the Wild” newsletter
>
> On 6 Jun 2019, at 17:55, Jonathan Moules <[hidden email]> wrote:
>
> Hi List,
>
> Following the protracted discussion about voters in the parallel thread it occurs to me that it's begging the question that voting is good.
>

I'm not quite sure which thread you mean but yes, there have been many
voters/voting/etc discussions on this list and others.  A recent
thread that I started was here,
https://lists.osgeo.org/pipermail/conference_dev/2018-December/005036.html
 I didn't hear many viable suggestions for different methods from that
thread.  I'd like to hear more.  I'd particularly like to hear Paul's
thoughts that he warned would be the result of an entire day of
thinking!  (or because he has 2-3 iterations of experience of starting
conferences like this)

> Why exactly do we have voting for this? Surely the better and (far) less subjective option is to an objective scoring system by which to measure the quality of the submissions? There's still element of subjectivity of course ("is this answer a 6/10 or a 7/10?"), but it's largely objective, measurable, and transparent.
>

This raises a good question, "why are we voting?"  Historically, we
voted because selecting FOSS4G was key to OSGeo's financial future.
This is still the case (more detailed in the referenced thread) but
overwhelmingly we are selecting between multiple nearly equivalent
high quality options.

While your proposal is perhaps an improvement (to formality and
openness) to the existing voting, it is still substantially the same
thing and I don't see the point of this method any more than our
existing method.  We can vote like we do now and more or less
arbitrarily select one very good proposal over another very good
proposal.  Or we can go through a systematic open scoring method and
score one very good proposal 92/100 and the other very good proposal
93/100; that seems like an equally arbitrary selection method to me.
In the case of very good proposals, the selection method does not
matter since the result is the same (very good proposal meeting all
the criteria and leading to a very successful conference and sound
finances).  We could flip a coin or draw from a hat.


> As far as I can tell from the transparency in the current voting (i.e., none) and reading the proposals (half of which usually reads like a tourist brochure), votes could easily currently be getting cast via "I want to go on holiday there next year". And while I'm not suggesting that's actually happening intentionally, it's almost certainly going to be a subconscious bias in the current process.
>

As far as I can tell (i.e. also none), "where I want to go on
vacation" is the most valid criteria that we have currently as the
substantive criteria in both cases are so excellent and near
equivalent.

(The other) Maria wrote:

On Wed, Jun 5, 2019 at 6:50 AM María Arias de Reyna <[hidden email]> wrote:
>
> Hi,
>
> I also think this is a problem. Choosing the foss4g venue is one of the most relevant tasks of OSGeo because it decides where and how the community is going to grow that year.

Geography of past conferences is certainly one criteria that I use,
wanting to bring OSGeo and FOSS4G to new areas.  Other than
geography*, I've not found many valid criteria to vote on.  I've not
heard many proposals either.  * this is again in the case of multiple
near equivalent high quality proposals.  In the case of proposals that
don't meet the important criteria, then it is an easy decision but
that is not the situation we usually find ourselves in.  So in my
opinion, we mostly have one valid (but rather weak) criteria for
evaluation.  That's why I want to hear more criteria or even better
new selection methods entirely.

Best regards, Eli


> Cheers,
>
> Jonathan
>
>
> On 2019-06-05 08:07, Till Adams wrote:
>
> Dear CC!
>
> I had some minutes and started an *early* prepare of the call for 2021.
>
> I added this WIKI page here:
>
> https://wiki.osgeo.org/wiki/FOSS4G_2021_Bid_Process
>
> Please check carefully whether the dates fit for you and of course for
> other errors.
>
>
> I will prepapre the needed documents in the next days and send them to you.
>
> Have a great day!
>
> Till
>
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev
>
>
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev
>
>
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev
_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev
Reply | Threaded
Open this post in threaded view
|

Re: RFP Evaluation method

Till Adams-3
Dear all,

I totall agree with Steven and Eli. The fundamental idea behind the
existing voting procedure is, to make use of the knowledge of those, who
already ran a conference like this. And this - I can tell you - is a
bunch of experience ;-)

Any system we might make use of has it's strengths and weaknesses. Also,
I do not have the feeling, that we in CC did fundamentally wrong
selections in the past years, as every conference I can remember, was a
success.

Till





Am 07.06.19 um 00:18 schrieb Eli Adam:

> Hi Jonathan,
>
> On Thu, Jun 6, 2019 at 11:44 AM Jonathan Moules
> <[hidden email]> wrote:
>> Hi Steven,
>>
>> Indeed, and I would then suggest taking all of that hard earned experience, knowledge, priorities, and weightings and putting them inside an objective, measurable framework where it is less susceptible to biases, both conscious and not.
>>
>> This would produce a much more open process, more in line with the O in OSGeo and the "open philosophy" part of the OSGeo Mission Statement. It also means experience and lessons learnt aren't lost when people leave the voting pool as with Cameron's input for instance.
>>
>> Depending on the scoring metrics used, it would also allow for a more direct comparison between proposals. And as a bonus, the scoring/metrics being open means it's open to comment and feedback from everyone, meaning the process is now more "participatory community driven development" (again, straight from the one-sentence Mission Statement).
>>
> I do find the questions from voting and non-voting members to be the
> most useful part of the process.  Generally the list seems very open
> to sincere participation from everyone.
>
>> Cheers,
>>
>> Jonathan
>>
>>
>> On 2019-06-06 19:18, Steven Feldman wrote:
>>
>> Jonathan
>>
>> Each member of the committee will bring their own priorities and experiences to the voting process.
>>
>> So for you the environmental considerations might be paramount while for someone else delivering a highly affordable delegate price may be their priority or another might be concerned about overall financial risk to OSGeo and someone else might be very focussed on diversity. We each have a different set of criteria and we also apply different levels of importance to those criteria.
>>
>> The current system allows each voter to apply their own criteria and weightings and to select the proposal that they think best, the majority vote then wins. I know when I vote I usually have a good feeling for one of the proposals based on a mix of factors, you could say that was unconscious bias, I would say it was a combination of instinct and experience
>>
>> cheers
>> ______
>> Steven
>>
>> Unusual maps in strange places -  mappery.org
>>
>> Subscribe to my weekly “Maps in the Wild” newsletter
>>
>> On 6 Jun 2019, at 17:55, Jonathan Moules <[hidden email]> wrote:
>>
>> Hi List,
>>
>> Following the protracted discussion about voters in the parallel thread it occurs to me that it's begging the question that voting is good.
>>
> I'm not quite sure which thread you mean but yes, there have been many
> voters/voting/etc discussions on this list and others.  A recent
> thread that I started was here,
> https://lists.osgeo.org/pipermail/conference_dev/2018-December/005036.html
>  I didn't hear many viable suggestions for different methods from that
> thread.  I'd like to hear more.  I'd particularly like to hear Paul's
> thoughts that he warned would be the result of an entire day of
> thinking!  (or because he has 2-3 iterations of experience of starting
> conferences like this)
>
>> Why exactly do we have voting for this? Surely the better and (far) less subjective option is to an objective scoring system by which to measure the quality of the submissions? There's still element of subjectivity of course ("is this answer a 6/10 or a 7/10?"), but it's largely objective, measurable, and transparent.
>>
> This raises a good question, "why are we voting?"  Historically, we
> voted because selecting FOSS4G was key to OSGeo's financial future.
> This is still the case (more detailed in the referenced thread) but
> overwhelmingly we are selecting between multiple nearly equivalent
> high quality options.
>
> While your proposal is perhaps an improvement (to formality and
> openness) to the existing voting, it is still substantially the same
> thing and I don't see the point of this method any more than our
> existing method.  We can vote like we do now and more or less
> arbitrarily select one very good proposal over another very good
> proposal.  Or we can go through a systematic open scoring method and
> score one very good proposal 92/100 and the other very good proposal
> 93/100; that seems like an equally arbitrary selection method to me.
> In the case of very good proposals, the selection method does not
> matter since the result is the same (very good proposal meeting all
> the criteria and leading to a very successful conference and sound
> finances).  We could flip a coin or draw from a hat.
>
>
>> As far as I can tell from the transparency in the current voting (i.e., none) and reading the proposals (half of which usually reads like a tourist brochure), votes could easily currently be getting cast via "I want to go on holiday there next year". And while I'm not suggesting that's actually happening intentionally, it's almost certainly going to be a subconscious bias in the current process.
>>
> As far as I can tell (i.e. also none), "where I want to go on
> vacation" is the most valid criteria that we have currently as the
> substantive criteria in both cases are so excellent and near
> equivalent.
>
> (The other) Maria wrote:
>
> On Wed, Jun 5, 2019 at 6:50 AM María Arias de Reyna <[hidden email]> wrote:
>> Hi,
>>
>> I also think this is a problem. Choosing the foss4g venue is one of the most relevant tasks of OSGeo because it decides where and how the community is going to grow that year.
> Geography of past conferences is certainly one criteria that I use,
> wanting to bring OSGeo and FOSS4G to new areas.  Other than
> geography*, I've not found many valid criteria to vote on.  I've not
> heard many proposals either.  * this is again in the case of multiple
> near equivalent high quality proposals.  In the case of proposals that
> don't meet the important criteria, then it is an easy decision but
> that is not the situation we usually find ourselves in.  So in my
> opinion, we mostly have one valid (but rather weak) criteria for
> evaluation.  That's why I want to hear more criteria or even better
> new selection methods entirely.
>
> Best regards, Eli
>
>
>> Cheers,
>>
>> Jonathan
>>
>>
>> On 2019-06-05 08:07, Till Adams wrote:
>>
>> Dear CC!
>>
>> I had some minutes and started an *early* prepare of the call for 2021.
>>
>> I added this WIKI page here:
>>
>> https://wiki.osgeo.org/wiki/FOSS4G_2021_Bid_Process
>>
>> Please check carefully whether the dates fit for you and of course for
>> other errors.
>>
>>
>> I will prepapre the needed documents in the next days and send them to you.
>>
>> Have a great day!
>>
>> Till
>>
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>
>>
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>
>>
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev
_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev
Reply | Threaded
Open this post in threaded view
|

Re: RFP Evaluation method

Jonathan Moules-4
Hi Till, Eli, All,

 > [Till] The fundamental idea behind the existing voting procedure is,
to make use of the knowledge of those, who already ran a conference like
this

Agreed, and as I've already noted, what happens when you leave the CC?
You've already lost Cameron's expertise and that of those who don't
actually vote.


 > [Till] I do not have the feeling, that we in CC did fundamentally
wrong selections in the past years

Well, from a sustainability perspective I beg to differ. While Calgary
does do sustainability well, the Halifax one was definitely a step
above. The fact that Calgary is an excellent candidate for "most
isolated large-city in North America outside of Anchorage (Alaska)"
(it's not even on the passenger rail network!) and honestly, I'm unclear
on why it was even allowed to be a candidate given you're guaranteeing
that people will only get there by flying. For a conference like FOSS4G,
that's several tens of thousands of tons of extra CO2 added straight to
the troposphere from folks who would/could otherwise have taken public
transport (the "local" contingent of the conference). I mean, what's it
going to be in 2023, Hawaii?

This is also why I strongly disagree with Eli on this point:

 > [Eli] As far as I can tell (i.e. also none), "where I want to go on
vacation" is the most valid criteria that we have currently as the
substantive criteria in both cases are so excellent and near equivalent.

Sustainability was a strong differentiator between the two proposals and
yet the least-sustainable proposal won, despite my non-member
sustainability questions. Alas I didn't notice the lack of rail
connection before the vote although I doubt that would have made a
difference. This is why in my view the current system doesn't work -
it's only selecting for the things that the committee care about and
given two otherwise near-identical proposals, the evidence would suggest
they don't seem to care about important things like sustainability.

Cheers,

Jonathan


On 2019-06-07 08:48, Till Adams wrote:

> Dear all,
>
> I totall agree with Steven and Eli. The fundamental idea behind the
> existing voting procedure is, to make use of the knowledge of those, who
> already ran a conference like this. And this - I can tell you - is a
> bunch of experience ;-)
>
> Any system we might make use of has it's strengths and weaknesses. Also,
> I do not have the feeling, that we in CC did fundamentally wrong
> selections in the past years, as every conference I can remember, was a
> success.
>
> Till
>
>
>
>
>
> Am 07.06.19 um 00:18 schrieb Eli Adam:
>> Hi Jonathan,
>>
>> On Thu, Jun 6, 2019 at 11:44 AM Jonathan Moules
>> <[hidden email]> wrote:
>>> Hi Steven,
>>>
>>> Indeed, and I would then suggest taking all of that hard earned experience, knowledge, priorities, and weightings and putting them inside an objective, measurable framework where it is less susceptible to biases, both conscious and not.
>>>
>>> This would produce a much more open process, more in line with the O in OSGeo and the "open philosophy" part of the OSGeo Mission Statement. It also means experience and lessons learnt aren't lost when people leave the voting pool as with Cameron's input for instance.
>>>
>>> Depending on the scoring metrics used, it would also allow for a more direct comparison between proposals. And as a bonus, the scoring/metrics being open means it's open to comment and feedback from everyone, meaning the process is now more "participatory community driven development" (again, straight from the one-sentence Mission Statement).
>>>
>> I do find the questions from voting and non-voting members to be the
>> most useful part of the process.  Generally the list seems very open
>> to sincere participation from everyone.
>>
>>> Cheers,
>>>
>>> Jonathan
>>>
>>>
>>> On 2019-06-06 19:18, Steven Feldman wrote:
>>>
>>> Jonathan
>>>
>>> Each member of the committee will bring their own priorities and experiences to the voting process.
>>>
>>> So for you the environmental considerations might be paramount while for someone else delivering a highly affordable delegate price may be their priority or another might be concerned about overall financial risk to OSGeo and someone else might be very focussed on diversity. We each have a different set of criteria and we also apply different levels of importance to those criteria.
>>>
>>> The current system allows each voter to apply their own criteria and weightings and to select the proposal that they think best, the majority vote then wins. I know when I vote I usually have a good feeling for one of the proposals based on a mix of factors, you could say that was unconscious bias, I would say it was a combination of instinct and experience
>>>
>>> cheers
>>> ______
>>> Steven
>>>
>>> Unusual maps in strange places -  mappery.org
>>>
>>> Subscribe to my weekly “Maps in the Wild” newsletter
>>>
>>> On 6 Jun 2019, at 17:55, Jonathan Moules <[hidden email]> wrote:
>>>
>>> Hi List,
>>>
>>> Following the protracted discussion about voters in the parallel thread it occurs to me that it's begging the question that voting is good.
>>>
>> I'm not quite sure which thread you mean but yes, there have been many
>> voters/voting/etc discussions on this list and others.  A recent
>> thread that I started was here,
>> https://lists.osgeo.org/pipermail/conference_dev/2018-December/005036.html
>>   I didn't hear many viable suggestions for different methods from that
>> thread.  I'd like to hear more.  I'd particularly like to hear Paul's
>> thoughts that he warned would be the result of an entire day of
>> thinking!  (or because he has 2-3 iterations of experience of starting
>> conferences like this)
>>
>>> Why exactly do we have voting for this? Surely the better and (far) less subjective option is to an objective scoring system by which to measure the quality of the submissions? There's still element of subjectivity of course ("is this answer a 6/10 or a 7/10?"), but it's largely objective, measurable, and transparent.
>>>
>> This raises a good question, "why are we voting?"  Historically, we
>> voted because selecting FOSS4G was key to OSGeo's financial future.
>> This is still the case (more detailed in the referenced thread) but
>> overwhelmingly we are selecting between multiple nearly equivalent
>> high quality options.
>>
>> While your proposal is perhaps an improvement (to formality and
>> openness) to the existing voting, it is still substantially the same
>> thing and I don't see the point of this method any more than our
>> existing method.  We can vote like we do now and more or less
>> arbitrarily select one very good proposal over another very good
>> proposal.  Or we can go through a systematic open scoring method and
>> score one very good proposal 92/100 and the other very good proposal
>> 93/100; that seems like an equally arbitrary selection method to me.
>> In the case of very good proposals, the selection method does not
>> matter since the result is the same (very good proposal meeting all
>> the criteria and leading to a very successful conference and sound
>> finances).  We could flip a coin or draw from a hat.
>>
>>
>>> As far as I can tell from the transparency in the current voting (i.e., none) and reading the proposals (half of which usually reads like a tourist brochure), votes could easily currently be getting cast via "I want to go on holiday there next year". And while I'm not suggesting that's actually happening intentionally, it's almost certainly going to be a subconscious bias in the current process.
>>>
>> As far as I can tell (i.e. also none), "where I want to go on
>> vacation" is the most valid criteria that we have currently as the
>> substantive criteria in both cases are so excellent and near
>> equivalent.
>>
>> (The other) Maria wrote:
>>
>> On Wed, Jun 5, 2019 at 6:50 AM María Arias de Reyna <[hidden email]> wrote:
>>> Hi,
>>>
>>> I also think this is a problem. Choosing the foss4g venue is one of the most relevant tasks of OSGeo because it decides where and how the community is going to grow that year.
>> Geography of past conferences is certainly one criteria that I use,
>> wanting to bring OSGeo and FOSS4G to new areas.  Other than
>> geography*, I've not found many valid criteria to vote on.  I've not
>> heard many proposals either.  * this is again in the case of multiple
>> near equivalent high quality proposals.  In the case of proposals that
>> don't meet the important criteria, then it is an easy decision but
>> that is not the situation we usually find ourselves in.  So in my
>> opinion, we mostly have one valid (but rather weak) criteria for
>> evaluation.  That's why I want to hear more criteria or even better
>> new selection methods entirely.
>>
>> Best regards, Eli
>>
>>
>>> Cheers,
>>>
>>> Jonathan
>>>
>>>
>>> On 2019-06-05 08:07, Till Adams wrote:
>>>
>>> Dear CC!
>>>
>>> I had some minutes and started an *early* prepare of the call for 2021.
>>>
>>> I added this WIKI page here:
>>>
>>> https://wiki.osgeo.org/wiki/FOSS4G_2021_Bid_Process
>>>
>>> Please check carefully whether the dates fit for you and of course for
>>> other errors.
>>>
>>>
>>> I will prepapre the needed documents in the next days and send them to you.
>>>
>>> Have a great day!
>>>
>>> Till
>>>
>>> _______________________________________________
>>> Conference_dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>
>>>
>>> _______________________________________________
>>> Conference_dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>
>>>
>>> _______________________________________________
>>> Conference_dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev


_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev
Reply | Threaded
Open this post in threaded view
|

Re: RFP Evaluation method

Cameron Shorter
Jonathan,

A great question to ask and for us to consider.

Short answer: I feel we should keep our voting process.

--

Long answer: I'll rephrase your question as "Can we define a selection
process which is more trustworthy than a group of people?"

Defining a process: I think that writing an process which could
differentiate between all the subtle factors which influences a good
conference would be next to impossible, especially if trying to
anticipate future factors as well. The best we can do is list selection
criteria (as already in our RFQ docs) [1]. If we were to define a
process, the next thing to happen is that bidding cities would try to
game the system.

The trustworthiness of the conference committee: We deliberately stack
our committee with ex-chairs of prior OSGeo conferences. This is a very
difficult position to fill, and each chair learns a significant amount
about running a conference in the process, and all the subtle things
that make it great or bad. I feel these people tend to be very
qualified, intrinsically motivated, and hence trustworthy decision makers.

So I'm also in favour of keeping the voting process from our conference
committee.

However, I think there is always room to refine our selection criteria.
One think I think we should add to our list is "Security and Inclusivity
within the host country". At foss4g Dar es Salaam, we discovered there
were local laws which discriminated against our LGBT community. Till, if
you are updating the RFQ document, can you add this to your checklist of
things to add.

Counter to Eli's suggestion that we haven't made a selection mistake:
Foss4g 2012 was cancelled [2], which I feel we should consider a failure.

[1]
https://svn.osgeo.org/osgeo/foss4g/rfp/2020/FOSS4G2020-request-for-proposal.pdf

[2]
http://cameronshorter.blogspot.com/2012/08/analysing-downfall-of-foss4g-2012.html

On 8/6/19 7:16 am, Jonathan Moules wrote:

> Hi Till, Eli, All,
>
> > [Till] The fundamental idea behind the existing voting procedure is,
> to make use of the knowledge of those, who already ran a conference
> like this
>
> Agreed, and as I've already noted, what happens when you leave the CC?
> You've already lost Cameron's expertise and that of those who don't
> actually vote.
>
>
> > [Till] I do not have the feeling, that we in CC did fundamentally
> wrong selections in the past years
>
> Well, from a sustainability perspective I beg to differ. While Calgary
> does do sustainability well, the Halifax one was definitely a step
> above. The fact that Calgary is an excellent candidate for "most
> isolated large-city in North America outside of Anchorage (Alaska)"
> (it's not even on the passenger rail network!) and honestly, I'm
> unclear on why it was even allowed to be a candidate given you're
> guaranteeing that people will only get there by flying. For a
> conference like FOSS4G, that's several tens of thousands of tons of
> extra CO2 added straight to the troposphere from folks who would/could
> otherwise have taken public transport (the "local" contingent of the
> conference). I mean, what's it going to be in 2023, Hawaii?
>
> This is also why I strongly disagree with Eli on this point:
>
> > [Eli] As far as I can tell (i.e. also none), "where I want to go on
> vacation" is the most valid criteria that we have currently as the
> substantive criteria in both cases are so excellent and near equivalent.
>
> Sustainability was a strong differentiator between the two proposals
> and yet the least-sustainable proposal won, despite my non-member
> sustainability questions. Alas I didn't notice the lack of rail
> connection before the vote although I doubt that would have made a
> difference. This is why in my view the current system doesn't work -
> it's only selecting for the things that the committee care about and
> given two otherwise near-identical proposals, the evidence would
> suggest they don't seem to care about important things like
> sustainability.
>
> Cheers,
>
> Jonathan
>
>
> On 2019-06-07 08:48, Till Adams wrote:
>> Dear all,
>>
>> I totall agree with Steven and Eli. The fundamental idea behind the
>> existing voting procedure is, to make use of the knowledge of those, who
>> already ran a conference like this. And this - I can tell you - is a
>> bunch of experience ;-)
>>
>> Any system we might make use of has it's strengths and weaknesses. Also,
>> I do not have the feeling, that we in CC did fundamentally wrong
>> selections in the past years, as every conference I can remember, was a
>> success.
>>
>> Till
>>
>>
>>
>>
>>
>> Am 07.06.19 um 00:18 schrieb Eli Adam:
>>> Hi Jonathan,
>>>
>>> On Thu, Jun 6, 2019 at 11:44 AM Jonathan Moules
>>> <[hidden email]> wrote:
>>>> Hi Steven,
>>>>
>>>> Indeed, and I would then suggest taking all of that hard earned
>>>> experience, knowledge, priorities, and weightings and putting them
>>>> inside an objective, measurable framework where it is less
>>>> susceptible to biases, both conscious and not.
>>>>
>>>> This would produce a much more open process, more in line with the
>>>> O in OSGeo and the "open philosophy" part of the OSGeo Mission
>>>> Statement. It also means experience and lessons learnt aren't lost
>>>> when people leave the voting pool as with Cameron's input for
>>>> instance.
>>>>
>>>> Depending on the scoring metrics used, it would also allow for a
>>>> more direct comparison between proposals. And as a bonus, the
>>>> scoring/metrics being open means it's open to comment and feedback
>>>> from everyone, meaning the process is now more "participatory
>>>> community driven development" (again, straight from the
>>>> one-sentence Mission Statement).
>>>>
>>> I do find the questions from voting and non-voting members to be the
>>> most useful part of the process.  Generally the list seems very open
>>> to sincere participation from everyone.
>>>
>>>> Cheers,
>>>>
>>>> Jonathan
>>>>
>>>>
>>>> On 2019-06-06 19:18, Steven Feldman wrote:
>>>>
>>>> Jonathan
>>>>
>>>> Each member of the committee will bring their own priorities and
>>>> experiences to the voting process.
>>>>
>>>> So for you the environmental considerations might be paramount
>>>> while for someone else delivering a highly affordable delegate
>>>> price may be their priority or another might be concerned about
>>>> overall financial risk to OSGeo and someone else might be very
>>>> focussed on diversity. We each have a different set of criteria and
>>>> we also apply different levels of importance to those criteria.
>>>>
>>>> The current system allows each voter to apply their own criteria
>>>> and weightings and to select the proposal that they think best, the
>>>> majority vote then wins. I know when I vote I usually have a good
>>>> feeling for one of the proposals based on a mix of factors, you
>>>> could say that was unconscious bias, I would say it was a
>>>> combination of instinct and experience
>>>>
>>>> cheers
>>>> ______
>>>> Steven
>>>>
>>>> Unusual maps in strange places -  mappery.org
>>>>
>>>> Subscribe to my weekly “Maps in the Wild” newsletter
>>>>
>>>> On 6 Jun 2019, at 17:55, Jonathan Moules
>>>> <[hidden email]> wrote:
>>>>
>>>> Hi List,
>>>>
>>>> Following the protracted discussion about voters in the parallel
>>>> thread it occurs to me that it's begging the question that voting
>>>> is good.
>>>>
>>> I'm not quite sure which thread you mean but yes, there have been many
>>> voters/voting/etc discussions on this list and others.  A recent
>>> thread that I started was here,
>>> https://lists.osgeo.org/pipermail/conference_dev/2018-December/005036.html 
>>>
>>>   I didn't hear many viable suggestions for different methods from that
>>> thread.  I'd like to hear more.  I'd particularly like to hear Paul's
>>> thoughts that he warned would be the result of an entire day of
>>> thinking!  (or because he has 2-3 iterations of experience of starting
>>> conferences like this)
>>>
>>>> Why exactly do we have voting for this? Surely the better and (far)
>>>> less subjective option is to an objective scoring system by which
>>>> to measure the quality of the submissions? There's still element of
>>>> subjectivity of course ("is this answer a 6/10 or a 7/10?"), but
>>>> it's largely objective, measurable, and transparent.
>>>>
>>> This raises a good question, "why are we voting?" Historically, we
>>> voted because selecting FOSS4G was key to OSGeo's financial future.
>>> This is still the case (more detailed in the referenced thread) but
>>> overwhelmingly we are selecting between multiple nearly equivalent
>>> high quality options.
>>>
>>> While your proposal is perhaps an improvement (to formality and
>>> openness) to the existing voting, it is still substantially the same
>>> thing and I don't see the point of this method any more than our
>>> existing method.  We can vote like we do now and more or less
>>> arbitrarily select one very good proposal over another very good
>>> proposal.  Or we can go through a systematic open scoring method and
>>> score one very good proposal 92/100 and the other very good proposal
>>> 93/100; that seems like an equally arbitrary selection method to me.
>>> In the case of very good proposals, the selection method does not
>>> matter since the result is the same (very good proposal meeting all
>>> the criteria and leading to a very successful conference and sound
>>> finances).  We could flip a coin or draw from a hat.
>>>
>>>
>>>> As far as I can tell from the transparency in the current voting
>>>> (i.e., none) and reading the proposals (half of which usually reads
>>>> like a tourist brochure), votes could easily currently be getting
>>>> cast via "I want to go on holiday there next year". And while I'm
>>>> not suggesting that's actually happening intentionally, it's almost
>>>> certainly going to be a subconscious bias in the current process.
>>>>
>>> As far as I can tell (i.e. also none), "where I want to go on
>>> vacation" is the most valid criteria that we have currently as the
>>> substantive criteria in both cases are so excellent and near
>>> equivalent.
>>>
>>> (The other) Maria wrote:
>>>
>>> On Wed, Jun 5, 2019 at 6:50 AM María Arias de Reyna
>>> <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> I also think this is a problem. Choosing the foss4g venue is one of
>>>> the most relevant tasks of OSGeo because it decides where and how
>>>> the community is going to grow that year.
>>> Geography of past conferences is certainly one criteria that I use,
>>> wanting to bring OSGeo and FOSS4G to new areas.  Other than
>>> geography*, I've not found many valid criteria to vote on. I've not
>>> heard many proposals either.  * this is again in the case of multiple
>>> near equivalent high quality proposals.  In the case of proposals that
>>> don't meet the important criteria, then it is an easy decision but
>>> that is not the situation we usually find ourselves in.  So in my
>>> opinion, we mostly have one valid (but rather weak) criteria for
>>> evaluation.  That's why I want to hear more criteria or even better
>>> new selection methods entirely.
>>>
>>> Best regards, Eli
>>>
>>>
>>>> Cheers,
>>>>
>>>> Jonathan
>>>>
>>>>
>>>> On 2019-06-05 08:07, Till Adams wrote:
>>>>
>>>> Dear CC!
>>>>
>>>> I had some minutes and started an *early* prepare of the call for
>>>> 2021.
>>>>
>>>> I added this WIKI page here:
>>>>
>>>> https://wiki.osgeo.org/wiki/FOSS4G_2021_Bid_Process
>>>>
>>>> Please check carefully whether the dates fit for you and of course for
>>>> other errors.
>>>>
>>>>
>>>> I will prepapre the needed documents in the next days and send them
>>>> to you.
>>>>
>>>> Have a great day!
>>>>
>>>> Till
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>> _______________________________________________
>>> Conference_dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>
>
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev

--
Cameron Shorter
Technology Demystifier
Open Technologies and Geospatial Consultant

M +61 (0) 419 142 254

_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev
Reply | Threaded
Open this post in threaded view
|

Re: RFP Evaluation method

Eli Adam


On Fri, Jun 7, 2019 at 2:42 PM Cameron Shorter <[hidden email]> wrote:
Jonathan,

A great question to ask and for us to consider.

Short answer: I feel we should keep our voting process.

--

Long answer: I'll rephrase your question as "Can we define a selection
process which is more trustworthy than a group of people?"

Defining a process: I think that writing an process which could
differentiate between all the subtle factors which influences a good
conference would be next to impossible, especially if trying to
anticipate future factors as well. The best we can do is list selection
criteria (as already in our RFQ docs) [1]. If we were to define a
process, the next thing to happen is that bidding cities would try to
game the system.

The trustworthiness of the conference committee: We deliberately stack
our committee with ex-chairs of prior OSGeo conferences. This is a very
difficult position to fill, and each chair learns a significant amount
about running a conference in the process, and all the subtle things
that make it great or bad. I feel these people tend to be very
qualified, intrinsically motivated, and hence trustworthy decision makers.

So I'm also in favour of keeping the voting process from our conference
committee.

However, I think there is always room to refine our selection criteria.
One think I think we should add to our list is "Security and Inclusivity
within the host country". At foss4g Dar es Salaam, we discovered there
were local laws which discriminated against our LGBT community. Till, if
you are updating the RFQ document, can you add this to your checklist of
things to add.

Good idea, follow through at the right time matters.  



Counter to Eli's suggestion that we haven't made a selection mistake:
Foss4g 2012 was cancelled [2], which I feel we should consider a failure

I agree that we need to make sure any proposal exceeds a high bar on important criteria.  That’s why I usually give some disclaimer like:  * this is again in the case of multiple
near equivalent high quality proposals.  In the case of proposals that
don't meet the important criteria, then it is an easy decision but
that is not the situation we usually find ourselves in.

Base criteria I look for before going on to other criteria are:

1) High probability of success
2) low risk
3) reasonable budget
4) absence of objectionable contracts
5) LOC experience
6) FOSS4G geography and history


I think that beyond those criteria, Jonathan makes a good suggestion of considering sustainability.  

Best regards, Eli




[1]
https://svn.osgeo.org/osgeo/foss4g/rfp/2020/FOSS4G2020-request-for-proposal.pdf

[2]
http://cameronshorter.blogspot.com/2012/08/analysing-downfall-of-foss4g-2012.html

On 8/6/19 7:16 am, Jonathan Moules wrote:
> Hi Till, Eli, All,
>
> > [Till] The fundamental idea behind the existing voting procedure is,
> to make use of the knowledge of those, who already ran a conference
> like this
>
> Agreed, and as I've already noted, what happens when you leave the CC?
> You've already lost Cameron's expertise and that of those who don't
> actually vote.
>
>
> > [Till] I do not have the feeling, that we in CC did fundamentally
> wrong selections in the past years
>
> Well, from a sustainability perspective I beg to differ. While Calgary
> does do sustainability well, the Halifax one was definitely a step
> above. The fact that Calgary is an excellent candidate for "most
> isolated large-city in North America outside of Anchorage (Alaska)"
> (it's not even on the passenger rail network!) and honestly, I'm
> unclear on why it was even allowed to be a candidate given you're
> guaranteeing that people will only get there by flying. For a
> conference like FOSS4G, that's several tens of thousands of tons of
> extra CO2 added straight to the troposphere from folks who would/could
> otherwise have taken public transport (the "local" contingent of the
> conference). I mean, what's it going to be in 2023, Hawaii?
>
> This is also why I strongly disagree with Eli on this point:
>
> > [Eli] As far as I can tell (i.e. also none), "where I want to go on
> vacation" is the most valid criteria that we have currently as the
> substantive criteria in both cases are so excellent and near equivalent.
>
> Sustainability was a strong differentiator between the two proposals
> and yet the least-sustainable proposal won, despite my non-member
> sustainability questions. Alas I didn't notice the lack of rail
> connection before the vote although I doubt that would have made a
> difference. This is why in my view the current system doesn't work -
> it's only selecting for the things that the committee care about and
> given two otherwise near-identical proposals, the evidence would
> suggest they don't seem to care about important things like
> sustainability.
>
> Cheers,
>
> Jonathan
>
>
> On 2019-06-07 08:48, Till Adams wrote:
>> Dear all,
>>
>> I totall agree with Steven and Eli. The fundamental idea behind the
>> existing voting procedure is, to make use of the knowledge of those, who
>> already ran a conference like this. And this - I can tell you - is a
>> bunch of experience ;-)
>>
>> Any system we might make use of has it's strengths and weaknesses. Also,
>> I do not have the feeling, that we in CC did fundamentally wrong
>> selections in the past years, as every conference I can remember, was a
>> success.
>>
>> Till
>>
>>
>>
>>
>>
>> Am 07.06.19 um 00:18 schrieb Eli Adam:
>>> Hi Jonathan,
>>>
>>> On Thu, Jun 6, 2019 at 11:44 AM Jonathan Moules
>>> <[hidden email]> wrote:
>>>> Hi Steven,
>>>>
>>>> Indeed, and I would then suggest taking all of that hard earned
>>>> experience, knowledge, priorities, and weightings and putting them
>>>> inside an objective, measurable framework where it is less
>>>> susceptible to biases, both conscious and not.
>>>>
>>>> This would produce a much more open process, more in line with the
>>>> O in OSGeo and the "open philosophy" part of the OSGeo Mission
>>>> Statement. It also means experience and lessons learnt aren't lost
>>>> when people leave the voting pool as with Cameron's input for
>>>> instance.
>>>>
>>>> Depending on the scoring metrics used, it would also allow for a
>>>> more direct comparison between proposals. And as a bonus, the
>>>> scoring/metrics being open means it's open to comment and feedback
>>>> from everyone, meaning the process is now more "participatory
>>>> community driven development" (again, straight from the
>>>> one-sentence Mission Statement).
>>>>
>>> I do find the questions from voting and non-voting members to be the
>>> most useful part of the process.  Generally the list seems very open
>>> to sincere participation from everyone.
>>>
>>>> Cheers,
>>>>
>>>> Jonathan
>>>>
>>>>
>>>> On 2019-06-06 19:18, Steven Feldman wrote:
>>>>
>>>> Jonathan
>>>>
>>>> Each member of the committee will bring their own priorities and
>>>> experiences to the voting process.
>>>>
>>>> So for you the environmental considerations might be paramount
>>>> while for someone else delivering a highly affordable delegate
>>>> price may be their priority or another might be concerned about
>>>> overall financial risk to OSGeo and someone else might be very
>>>> focussed on diversity. We each have a different set of criteria and
>>>> we also apply different levels of importance to those criteria.
>>>>
>>>> The current system allows each voter to apply their own criteria
>>>> and weightings and to select the proposal that they think best, the
>>>> majority vote then wins. I know when I vote I usually have a good
>>>> feeling for one of the proposals based on a mix of factors, you
>>>> could say that was unconscious bias, I would say it was a
>>>> combination of instinct and experience
>>>>
>>>> cheers
>>>> ______
>>>> Steven
>>>>
>>>> Unusual maps in strange places -  mappery.org
>>>>
>>>> Subscribe to my weekly “Maps in the Wild” newsletter
>>>>
>>>> On 6 Jun 2019, at 17:55, Jonathan Moules
>>>> <[hidden email]> wrote:
>>>>
>>>> Hi List,
>>>>
>>>> Following the protracted discussion about voters in the parallel
>>>> thread it occurs to me that it's begging the question that voting
>>>> is good.
>>>>
>>> I'm not quite sure which thread you mean but yes, there have been many
>>> voters/voting/etc discussions on this list and others.  A recent
>>> thread that I started was here,
>>> https://lists.osgeo.org/pipermail/conference_dev/2018-December/005036.html
>>>
>>>   I didn't hear many viable suggestions for different methods from that
>>> thread.  I'd like to hear more.  I'd particularly like to hear Paul's
>>> thoughts that he warned would be the result of an entire day of
>>> thinking!  (or because he has 2-3 iterations of experience of starting
>>> conferences like this)
>>>
>>>> Why exactly do we have voting for this? Surely the better and (far)
>>>> less subjective option is to an objective scoring system by which
>>>> to measure the quality of the submissions? There's still element of
>>>> subjectivity of course ("is this answer a 6/10 or a 7/10?"), but
>>>> it's largely objective, measurable, and transparent.
>>>>
>>> This raises a good question, "why are we voting?" Historically, we
>>> voted because selecting FOSS4G was key to OSGeo's financial future.
>>> This is still the case (more detailed in the referenced thread) but
>>> overwhelmingly we are selecting between multiple nearly equivalent
>>> high quality options.
>>>
>>> While your proposal is perhaps an improvement (to formality and
>>> openness) to the existing voting, it is still substantially the same
>>> thing and I don't see the point of this method any more than our
>>> existing method.  We can vote like we do now and more or less
>>> arbitrarily select one very good proposal over another very good
>>> proposal.  Or we can go through a systematic open scoring method and
>>> score one very good proposal 92/100 and the other very good proposal
>>> 93/100; that seems like an equally arbitrary selection method to me.
>>> In the case of very good proposals, the selection method does not
>>> matter since the result is the same (very good proposal meeting all
>>> the criteria and leading to a very successful conference and sound
>>> finances).  We could flip a coin or draw from a hat.
>>>
>>>
>>>> As far as I can tell from the transparency in the current voting
>>>> (i.e., none) and reading the proposals (half of which usually reads
>>>> like a tourist brochure), votes could easily currently be getting
>>>> cast via "I want to go on holiday there next year". And while I'm
>>>> not suggesting that's actually happening intentionally, it's almost
>>>> certainly going to be a subconscious bias in the current process.
>>>>
>>> As far as I can tell (i.e. also none), "where I want to go on
>>> vacation" is the most valid criteria that we have currently as the
>>> substantive criteria in both cases are so excellent and near
>>> equivalent.
>>>
>>> (The other) Maria wrote:
>>>
>>> On Wed, Jun 5, 2019 at 6:50 AM María Arias de Reyna
>>> <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> I also think this is a problem. Choosing the foss4g venue is one of
>>>> the most relevant tasks of OSGeo because it decides where and how
>>>> the community is going to grow that year.
>>> Geography of past conferences is certainly one criteria that I use,
>>> wanting to bring OSGeo and FOSS4G to new areas.  Other than
>>> geography*, I've not found many valid criteria to vote on. I've not
>>> heard many proposals either.  * this is again in the case of multiple
>>> near equivalent high quality proposals.  In the case of proposals that
>>> don't meet the important criteria, then it is an easy decision but
>>> that is not the situation we usually find ourselves in.  So in my
>>> opinion, we mostly have one valid (but rather weak) criteria for
>>> evaluation.  That's why I want to hear more criteria or even better
>>> new selection methods entirely.
>>>
>>> Best regards, Eli
>>>
>>>
>>>> Cheers,
>>>>
>>>> Jonathan
>>>>
>>>>
>>>> On 2019-06-05 08:07, Till Adams wrote:
>>>>
>>>> Dear CC!
>>>>
>>>> I had some minutes and started an *early* prepare of the call for
>>>> 2021.
>>>>
>>>> I added this WIKI page here:
>>>>
>>>> https://wiki.osgeo.org/wiki/FOSS4G_2021_Bid_Process
>>>>
>>>> Please check carefully whether the dates fit for you and of course for
>>>> other errors.
>>>>
>>>>
>>>> I will prepapre the needed documents in the next days and send them
>>>> to you.
>>>>
>>>> Have a great day!
>>>>
>>>> Till
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Conference_dev mailing list
>>>> [hidden email]
>>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>>> _______________________________________________
>>> Conference_dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>> _______________________________________________
>> Conference_dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/conference_dev
>
>
> _______________________________________________
> Conference_dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/conference_dev

--
Cameron Shorter
Technology Demystifier
Open Technologies and Geospatial Consultant

M +61 (0) 419 142 254

_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev

_______________________________________________
Conference_dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/conference_dev
12