Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

pcav
Hi all,
following our discussion in Bucharest, I think we have to reply
officially to this one.
Any taker?
Cheers.

-------- Forwarded Message --------

Hi @pcav <https://github.com/pcav>,

I appreciate that you take the time to discuss a viable solution for
both sides. We do have a steady pull from customers running on a hybrid
commercial/open source landscapes who want to integrate HANA Spatial
with QGIS. We did already invest a fair amount of time in the
development of the core provider for HANA and also we have learnt our
lesson that we did not carefully study the QGIS enhancement processes
before starting our development.

However given the current (unfortunate) situation would you be able to
give an estimate by when we can discuss the way forward with regards to
HANA integration in QGIS? Please also includes us in your ongoing
discussion if you feel this would be helpful.

Many thanks in advance,
Mathias (SAP HANA Spatial Team)


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/qgis/QGIS-Enhancement-Proposals/issues/151?email_source=notifications&email_token=AAFI5ARISEF747RCAA4WUKDQEVVSPA5CNFSM4IGQ7LKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MBSWQ#issuecomment-521673050>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFI5ARH63Z36TZRZTFEJUTQEVVSPANCNFSM4IGQ7LKA>.

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Tim Sutton-6
Hi Paolo

Here is the draft email I prepared to send to them. If you are all happy with it we can send it to them...

-------

Dear xxxx

We wanted to reach out to you regarding your proposed patch for QGIS to add support for Hana. As you have probably realised, there have been some concerns raised about the integration of this code. I don’t believe these are insurmountable but we would like to ask you if you could respond on a few points as outlined below.


  1. Testing platform: Currently it is difficult for us to test the work you have done since we do not have a test / sandbox instance of the provider that we can use to try it out. Could we ask you to provide us with a perpetually available Hana database instance that we can use to test your provider both now and in the future? It would be ideal if you could set up and maintain this instance for us and simply supply the connection details since we do not have the time or expertise to manage this from within the community. We also ask that you waive any licensing fees for this test platform so that it is freely available to the project.
  2. Community participation: We would like to ask if you can send one or more of your developer team members to attend our regular QGIS Contributor Meetings (next one planned for spring 2020 in The Netherlands) - details will be added to our wiki soon: https://github.com/qgis/QGIS/wiki. We ask this because we would a) like you to be familiar with the norms and expectations of the QGIS Community and b) so that we can get to know your team members and gain a sense of confidence that you are committed to the ongoing maintenance of the Hana provider and related impacted code in QGIS.
  3. Packaging and Integration Effort: We would like to ask you if you can give us some indication of the packaging overhead expected for shipping your provider in our standard QGIS packages that we distribute for macOS, Windows and Linux. Have you been able to test on all of these platforms and is the provider fully self-contained? Or does it depend on additional libraries that need to be installed on the host operating system? If so are all of the aforementioned operating systems supported? Also, note that providers are shipped on mobile implementations of QGIS (QField and INPUT) and we would like to know that these platforms will also support your Hana provider. Finally with regard to packaging, if there is additional overhead for packaging we would like to ask you to support the QGIS project, either financially or by providing developer time, to ensure that everything is cleanly packaged for all supported platforms.
  4. Point of contact: We would like to ask you to provide a single point of contact who will be technically knowledgeable and responsive to requests such as bug reports or questions from our community concerning your provider implementation.
  5. Upcoming API changes: For QGIS 3.10 we are planning to integrate a new abstraction layer for provider connection management. We would like to ask you to work with Alessandro Pasotti to get the Hana provider in line with the other providers. This may involve subcontracting him directly if the process requires more time than he is willing to donate.

Lastly, we would like to say big “thank you” for choosing QGIS as a platform for your Hana provider implementation - we do appreciate that having it in QGIS and the new opportunities that it will open for our users in enterprise environments. We hope you enjoy working with the QGIS Community and we look forward to having a long and fruitful association together! We would also like to invite you to consider financially supporting the QGIS project through our sustaining membership programme (https://www.qgis.org/en/site/getinvolved/governance/sustaining_members/sustaining_members.html#qgis-sustaining-memberships) - as you probably realised from this email and the discussions surrounding your pull request, there is a lot of community / volunteer effort that goes into maintaining and managing the QGIS project and financial support allows us to offload the maintenance burden introduced by adding new provider such as the Hana driver to ad hoc contractors or paid community members.

Best regards

The QGIS Community


———



Regards

Tim

On 4 Sep 2019, at 08:14, Paolo Cavallini <[hidden email]> wrote:

Hi all,
following our discussion in Bucharest, I think we have to reply
officially to this one.
Any taker?
Cheers.

-------- Forwarded Message --------

Hi @pcav <https://github.com/pcav>,

I appreciate that you take the time to discuss a viable solution for
both sides. We do have a steady pull from customers running on a hybrid
commercial/open source landscapes who want to integrate HANA Spatial
with QGIS. We did already invest a fair amount of time in the
development of the core provider for HANA and also we have learnt our
lesson that we did not carefully study the QGIS enhancement processes
before starting our development.

However given the current (unfortunate) situation would you be able to
give an estimate by when we can discuss the way forward with regards to
HANA integration in QGIS? Please also includes us in your ongoing
discussion if you feel this would be helpful.

Many thanks in advance,
Mathias (SAP HANA Spatial Team)


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/qgis/QGIS-Enhancement-Proposals/issues/151?email_source=notifications&email_token=AAFI5ARISEF747RCAA4WUKDQEVVSPA5CNFSM4IGQ7LKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MBSWQ#issuecomment-521673050>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFI5ARH63Z36TZRZTFEJUTQEVVSPANCNFSM4IGQ7LKA>.

_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc









Tim Sutton

Co-founder: Kartoza
Ex Project chair: QGIS.org

Visit http://kartoza.com to find out about open source:

Desktop GIS programming services
Geospatial web development
GIS Training
Consulting Services

Skype: timlinux 
IRC: timlinux on #qgis at freenode.net

I'd love to connect. Here's my calendar link to make finding time easy.


_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

pcav
+1 from me - thanks Tim!
cheers

On 04/09/19 09:20, Tim Sutton wrote:

> Hi Paolo
>
> Here is the draft email I prepared to send to them. If you are all happy
> with it we can send it to them...
>
> -------
>
> Dear xxxx
>
> We wanted to reach out to you regarding your proposed patch for QGIS to
> add support for Hana. As you have probably realised, there have been
> some concerns raised about the integration of this code. I don’t believe
> these are insurmountable but we would like to ask you if you could
> respond on a few points as outlined below.
>
>
>  1. *Testing platform:* Currently it is difficult for us to test the
>     work you have done since we do not have a test / sandbox instance of
>     the provider that we can use to try it out. Could we ask you to
>     provide us with a perpetually available Hana database instance that
>     we can use to test your provider both now and in the future? It
>     would be ideal if you could set up and maintain this instance for us
>     and simply supply the connection details since we do not have the
>     time or expertise to manage this from within the community. We also
>     ask that you waive any licensing fees for this test platform so that
>     it is freely available to the project.
>  2. *Community participation:* We would like to ask if you can send one
>     or more of your developer team members to attend our regular QGIS
>     Contributor Meetings (next one planned for spring 2020 in The
>     Netherlands) - details will be added to our wiki soon:
>     https://github.com/qgis/QGIS/wiki. We ask this because we would a)
>     like you to be familiar with the norms and expectations of the QGIS
>     Community and b) so that we can get to know your team members and
>     gain a sense of confidence that you are committed to the ongoing
>     maintenance of the Hana provider and related impacted code in QGIS.
>  3. *Packaging and Integration Effort:* We would like to ask you if you
>     can give us some indication of the packaging overhead expected for
>     shipping your provider in our standard QGIS packages that we
>     distribute for macOS, Windows and Linux. Have you been able to test
>     on all of these platforms and is the provider fully self-contained?
>     Or does it depend on additional libraries that need to be installed
>     on the host operating system? If so are all of the aforementioned
>     operating systems supported? Also, note that providers are shipped
>     on mobile implementations of QGIS (QField and INPUT) and we would
>     like to know that these platforms will also support your Hana
>     provider. Finally with regard to packaging, if there is additional
>     overhead for packaging we would like to ask you to support the QGIS
>     project, either financially or by providing developer time, to
>     ensure that everything is cleanly packaged for all supported platforms.
>  4. *Point of contact:* We would like to ask you to provide a single
>     point of contact who will be technically knowledgeable and
>     responsive to requests such as bug reports or questions from our
>     community concerning your provider implementation.
>  5. *Upcoming API changes:* For QGIS 3.10 we are planning to integrate a
>     new abstraction layer for provider connection management. We would
>     like to ask you to work with Alessandro Pasotti to get the Hana
>     provider in line with the other providers. This may involve
>     subcontracting him directly if the process requires more time than
>     he is willing to donate.
>
>
> Lastly, we would like to say big “thank you” for choosing QGIS as a
> platform for your Hana provider implementation - we do appreciate that
> having it in QGIS and the new opportunities that it will open for our
> users in enterprise environments. We hope you enjoy working with the
> QGIS Community and we look forward to having a long and fruitful
> association together! We would also like to invite you to consider
> financially supporting the QGIS project through our sustaining
> membership programme
> (https://www.qgis.org/en/site/getinvolved/governance/sustaining_members/sustaining_members.html#qgis-sustaining-memberships)
> - as you probably realised from this email and the discussions
> surrounding your pull request, there is a lot of community / volunteer
> effort that goes into maintaining and managing the QGIS project and
> financial support allows us to offload the maintenance burden introduced
> by adding new provider such as the Hana driver to ad hoc contractors or
> paid community members.
>
> Best regards
>
> The QGIS Community
>
>
> ———
>
>
>
> Regards
>
> Tim
>
>> On 4 Sep 2019, at 08:14, Paolo Cavallini <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> Hi all,
>> following our discussion in Bucharest, I think we have to reply
>> officially to this one.
>> Any taker?
>> Cheers.
>>
>> -------- Forwarded Message --------
>>
>> Hi @pcav <https://github.com/pcav>,
>>
>> I appreciate that you take the time to discuss a viable solution for
>> both sides. We do have a steady pull from customers running on a hybrid
>> commercial/open source landscapes who want to integrate HANA Spatial
>> with QGIS. We did already invest a fair amount of time in the
>> development of the core provider for HANA and also we have learnt our
>> lesson that we did not carefully study the QGIS enhancement processes
>> before starting our development.
>>
>> However given the current (unfortunate) situation would you be able to
>> give an estimate by when we can discuss the way forward with regards to
>> HANA integration in QGIS? Please also includes us in your ongoing
>> discussion if you feel this would be helpful.
>>
>> Many thanks in advance,
>> Mathias (SAP HANA Spatial Team)
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/qgis/QGIS-Enhancement-Proposals/issues/151?email_source=notifications&email_token=AAFI5ARISEF747RCAA4WUKDQEVVSPA5CNFSM4IGQ7LKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MBSWQ#issuecomment-521673050>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AAFI5ARH63Z36TZRZTFEJUTQEVVSPANCNFSM4IGQ7LKA>.
>>
>> _______________________________________________
>> Qgis-psc mailing list
>> [hidden email] <mailto:[hidden email]>
>> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>
> —
>
>
>
>
>
>
>
>
> *Tim Sutton*
>
> *Co-founder:* Kartoza
> *Ex Project chair:* QGIS.org <http://QGIS.org>
>
> Visit http://kartoza.com <http://kartoza.com/> to find out about open
> source:
>
> Desktop GIS programming services
> Geospatial web development
> GIS Training
> Consulting Services
>
> *Skype*: timlinux 
> *IRC:* timlinux on #qgis at freenode.net <http://freenode.net>
>
> I'd love to connect. Here's my calendar link
> <https://calendly.com/timlinux> to make finding time easy.
>

--
Paolo Cavallini - www.faunalia.eu
QGIS.ORG Chair:
http://planet.qgis.org/planet/user/28/tag/qgis%20board/
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Nyall Dawson
In reply to this post by Tim Sutton-6
On Wed, 4 Sep 2019 at 17:20, Tim Sutton <[hidden email]> wrote:

>
> Testing platform: Currently it is difficult for us to test the work you have done since we do not have a test / sandbox instance of the provider that we can use to try it out. Could we ask you to provide us with a perpetually available Hana database instance that we can use to test your provider both now and in the future? It would be ideal if you could set up and maintain this instance for us and simply supply the connection details since we do not have the time or expertise to manage this from within the community. We also ask that you waive any licensing fees for this test platform so that it is freely available to the project.

Could you add here that it needs to be accessible via our CI
infrastructure, and that they'll need to include unit tests
implementing the provider conformance suite which run on our CI ?

Without this I will strongly and vocally oppose this work, because
it's effectively asking to be broken.

Nyall
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Even Rouault-2
On mercredi 4 septembre 2019 17:32:46 CEST Nyall Dawson wrote:

> On Wed, 4 Sep 2019 at 17:20, Tim Sutton <[hidden email]> wrote:
> > Testing platform: Currently it is difficult for us to test the work you
> > have done since we do not have a test / sandbox instance of the provider
> > that we can use to try it out. Could we ask you to provide us with a
> > perpetually available Hana database instance that we can use to test your
> > provider both now and in the future? It would be ideal if you could set
> > up and maintain this instance for us and simply supply the connection
> > details since we do not have the time or expertise to manage this from
> > within the community. We also ask that you waive any licensing fees for
> > this test platform so that it is freely available to the project.
> Could you add here that it needs to be accessible via our CI
> infrastructure, and that they'll need to include unit tests
> implementing the provider conformance suite which run on our CI ?
>
> Without this I will strongly and vocally oppose this work, because
> it's effectively asking to be broken.

yeah, "any untested code is broken code"

But I think they'll have a hard time communicating a public connection string
to their database as this could be abused by anybody. I was going to suggest
we could probably keep it as a Travis-CI encrypted string, but encrypted
strings don't work with pull requests. And even if it did, any QGIS developer
requesting it should also have access to it when needed. Tests against a
shared instance should also be carefuly written so that if 2 persons by chance
run the test at the same time, there's no conflict on table names that are
created or whatever at runtime.

If they have a Docker image, lightweight enough and with all setup done (it's
fine if it has limitations on database size, etc if they don't want this to be
abused for production use), that could also be a reasonable solution for both
Travis & local testing. Except that wouldn't work for Mac or Windows devs...

Cheers,

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Tim Sutton-6
Hi (replying to both Even and Nyall here)...


On 4 Sep 2019, at 09:32, Even Rouault <[hidden email]> wrote:

On mercredi 4 septembre 2019 17:32:46 CEST Nyall Dawson wrote:
On Wed, 4 Sep 2019 at 17:20, Tim Sutton <[hidden email]> wrote:
Testing platform: Currently it is difficult for us to test the work you
have done since we do not have a test / sandbox instance of the provider
that we can use to try it out. Could we ask you to provide us with a
perpetually available Hana database instance that we can use to test your
provider both now and in the future? It would be ideal if you could set
up and maintain this instance for us and simply supply the connection
details since we do not have the time or expertise to manage this from
within the community. We also ask that you waive any licensing fees for
this test platform so that it is freely available to the project.
Could you add here that it needs to be accessible via our CI
infrastructure, and that they'll need to include unit tests
implementing the provider conformance suite which run on our CI ?

Without this I will strongly and vocally oppose this work, because
it's effectively asking to be broken.

@Nyall we discussed this a bit in Bucharest. There are a few concerns - figuring out how to reasonably manage testing infrastructure as Even touches on below being one, the lack of any clear directive mandating tests (I think we should add something to our contributor guidelines), the lack of any precedent test suite (previously we only mandated libqgiscore tests) being another. Personally I wonder if we shouldn’t be thinking more broadly and have a generic test suite that works with all providers to validate general CRUD operations? Since we have a common API for providers this should be possible right?



yeah, "any untested code is broken code"

But I think they'll have a hard time communicating a public connection string 
to their database as this could be abused by anybody. I was going to suggest 
we could probably keep it as a Travis-CI encrypted string, but encrypted 
strings don't work with pull requests. And even if it did, any QGIS developer 
requesting it should also have access to it when needed. Tests against a 
shared instance should also be carefuly written so that if 2 persons by chance 
run the test at the same time, there's no conflict on table names that are 
created or whatever at runtime.

If they have a Docker image, lightweight enough and with all setup done (it's 
fine if it has limitations on database size, etc if they don't want this to be 
abused for production use), that could also be a reasonable solution for both 
Travis & local testing. Except that wouldn't work for Mac or Windows devs…

@Even yeah it seems there is one here:


Do I remember correctly from watching your GDAL summary at FOSS4G that there was also a HANA provider added in GDAL? How have you implemented tests in your situation?

If we are going to test in docker we can just spin up a HANA image in the local docker network where the tests execute and have hardcoded connection details in the test suite since it would not be publicly accessible and live only as long as the test run lives.

@nyall By the way there were quite a few people commenting on the test suite in QGIS that we should probably discuss at some point. We can pick up on that in a separate thread later.

Regards

Tim


Cheers,

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc

 




---

Tim Sutton





_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Nyall Dawson
On Thu, 5 Sep 2019 at 09:15, Tim Sutton <[hidden email]> wrote:
>
>
> @Nyall we discussed this a bit in Bucharest. There are a few concerns - figuring out how to reasonably manage testing infrastructure as Even touches on below being one, the lack of any clear directive mandating tests (I think we should add something to our contributor guidelines),

> the lack of any precedent test suite (previously we only mandated libqgiscore tests) being another.

This is far from true. EVERY vector data provider, even the infamous
db2 provider, implements the standard provider test suite. The vast
majority of these are tested using travis, including Postgres (sql
server was for a stage, but the docker image of sql server was too
fragile and it's been temporarily disabled, waiting on an updated
image for testing).

Furthermore, there's clearly a strong precedent regarding testing of
contributions. It may not be mandated, but its only very rare
exceptions (or gui heavy contributions) which are merged today without
unit tests. I think it's time we mandated this, AND also mandated the
forced use of pull requests and mandatory reviews for all
contributions. The vast majority of the QGIS developer community just
follow this as standard practice now, so it's unlikely to have any
significant effect on QGIS contributions.

> Personally I wonder if we shouldn’t be thinking more broadly and have a generic test suite that works with all providers to validate general CRUD operations?

We do already ;) It's been around since the first hackfest in Nodebo,
and has grown to many thousands of lines of code. It's also been
single-handedly responsible for the dramatic increase in provider
stability and consistency we've seen over the last few years.

See https://github.com/qgis/QGIS/blob/master/tests/src/python/providertestbase.py
and https://github.com/qgis/QGIS/blob/master/tests/src/python/featuresourcetestbase.py

Nyall
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc
Reply | Threaded
Open this post in threaded view
|

Re: [qgis/QGIS-Enhancement-Proposals] Support SAP HANA databases in QGIS (#151)

Even Rouault-2
In reply to this post by Tim Sutton-6
> @Even yeah it seems there is one here:
>
> https://developers.sap.com/tutorials/hxe-ua-install-using-docker.html
>
> Do I remember correctly from watching your GDAL summary at FOSS4G that there
> was also a HANA provider added in GDAL?

No, there's no HANA provider in GDAL.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
Qgis-psc mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/qgis-psc