geoserver tests and internet connection

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

geoserver tests and internet connection

Niels Charlier
Is there a rule about whether geoserver unit/integration tests should be
able to run without internet connection? Couldn't find anything, but it
seems that most are with only a few exceptions: one method in CSW and a
number of fairly recent WCS tests.. Because of those few exceptions you
can't build geoserver without internet, unless you turn unit tests off.
This could be a problem for some building servers that disallow remote
access.

In all cases the problem seems to be related to the downloading of XSD
schemes. The problem should be able to be resolved by either including
the right XSD's somewhere in the classpath (well, in the right place,
eh), or just remove the references to them in the XML files. I already
did this for the CSW test: https://github.com/geoserver/geoserver/pull/3414

Kind Regards

Niels



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: geoserver tests and internet connection

geowolf
On Mon, May 13, 2019 at 11:12 AM Niels Charlier <[hidden email]> wrote:
Is there a rule about whether geoserver unit/integration tests should be
able to run without internet connection?

No written rule, but the practice is to mark a test that needs to talk to a server
(remote or local, does not matter) or fetch a remote file as an OnlineTest.
However sometimes a test accesses the internet without the developer even being
aware of it, and they are hard to spot because it's uncommon to do a build in a fully
isolated environment (happens to me when I try to hack on a plane, but that's pretty much it,
and have to prepare the machine in advance to make sure maven already has all
the jars it needs).
 
Couldn't find anything, but it
seems that most are with only a few exceptions: one method in CSW and a
number of fairly recent WCS tests..

Do you have a list?
 
Because of those few exceptions you
can't build geoserver without internet, unless you turn unit tests off.
This could be a problem for some building servers that disallow remote
access.

For the case of a build server, maven itself needs internet access, e.g, to get artifacts when version change, or 
when a new dependency is added. 
Unless you're thinking about a situation where there is also a dedicated and accessible repository manager, 
which can access the internet on behalf of maven.

I guess it could be useful to have a build checking if the code can be built without
internet connection, guessing it would first need an online pass making maven load
everything it needs, and then disconnect the network and run the tests. 
Not sure if feasible though (thinking about Travis for example).
 
In all cases the problem seems to be related to the downloading of XSD
schemes. The problem should be able to be resolved by either including
the right XSD's somewhere in the classpath (well, in the right place,
eh), or just remove the references to them in the XML files. I already
did this for the CSW test: https://github.com/geoserver/geoserver/pull/3414

Yes, this is the common way to do fix that problem.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: geoserver tests and internet connection

Niels Charlier
On 13/05/2019 11:47, Andrea Aime wrote:
Do you have a list?

Apart from the CSW one from the PR, these WCS2 ones:

  testDescribeWithTimeElevation(org.geoserver.wcs2_0.ResourceAccessManagerWCSTest): Could not parse the WCS 2.0 schemas
  gridCellCenterEnforce(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVPNoWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeContinousInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testCustomNullValue(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeCustomDimensionsList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoveragesNoWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeElevationList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testAxisOrderUtm(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMetadata(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testProjectectedKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeRangeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationRangeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationContinuousInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationValuesList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoveragesLocalWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testCustomUnit(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoverages(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVPLocalWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeDiscreteInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationDiscreteInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicKVP(org.geoserver.wcs2_0.kvp.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas
  testMetadata(org.geoserver.wcs2_0.kvp.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas
  testDescribeCoverageSimple(org.geoserver.wcs2_0.xml.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeCoverageMultiband(org.geoserver.wcs2_0.xml.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicPost(org.geoserver.wcs2_0.xml.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas


Unless you're thinking about a situation where there is also a dedicated and accessible repository manager, 
which can access the internet on behalf of maven.

This is the situation I am dealing with.

Applications are built on a bamboo server with dedicated and accessible repository manager, but no internet access for tests.


Regards

Niels



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: geoserver tests and internet connection

geowolf
On Mon, May 13, 2019 at 2:22 PM Niels Charlier <[hidden email]> wrote:
  testDescribeWithTimeElevation(org.geoserver.wcs2_0.ResourceAccessManagerWCSTest): Could not parse the WCS 2.0 schemas

  gridCellCenterEnforce(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVPNoWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeContinousInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testCustomNullValue(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeCustomDimensionsList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoveragesNoWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeElevationList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testAxisOrderUtm(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMetadata(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testProjectectedKVP(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeRangeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationRangeList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationContinuousInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationValuesList(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoveragesLocalWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testCustomUnit(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultipleCoverages(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testMultiBandKVPLocalWs(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeTimeDiscreteInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeElevationDiscreteInterval(org.geoserver.wcs2_0.kvp.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicKVP(org.geoserver.wcs2_0.kvp.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas
  testMetadata(org.geoserver.wcs2_0.kvp.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas
  testDescribeCoverageSimple(org.geoserver.wcs2_0.xml.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testDescribeCoverageMultiband(org.geoserver.wcs2_0.xml.DescribeCoverageTest): Could not parse the WCS 2.0 schemas
  testBasicPost(org.geoserver.wcs2_0.xml.GetCapabilitiesTest): Could not parse the WCS 2.0 schemas


I can confirm the failure... but don't understand the " a number of fairly recent WCS tests" statement.
The tests have been around for years now, do you mean they started failing recently? If so, do you have a date and potential
commit candidates that introduced the network dependency?

As far as I can see the code is failing to get to the GML schemas (which WCS schemas do reference).
 
Unless you're thinking about a situation where there is also a dedicated and accessible repository manager, 
which can access the internet on behalf of maven.

This is the situation I am dealing with.

Applications are built on a bamboo server with dedicated and accessible repository manager, but no internet access for tests.

I see. It's an uncommon setup, but the issue is legit (if probably low priority). I'd start by opening a ticket.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: geoserver tests and internet connection

Niels Charlier


On 13/05/2019 14:38, Andrea Aime wrote:
I can confirm the failure... but don't understand the " a number of fairly recent WCS tests" statement.
The tests have been around for years now, do you mean they started failing recently? If so, do you have a date and potential
commit candidates that introduced the network dependency?

Indeed, that was incorrect. The tests were there but didn't fail in version 2.12. Just tested and they do fail in 2.13. Something must have changed in between those two versions, but I haven't narrowed it further down (yet).



As far as I can see the code is failing to get to the GML schemas (which WCS schemas do reference).
 
Unless you're thinking about a situation where there is also a dedicated and accessible repository manager, 
which can access the internet on behalf of maven.

This is the situation I am dealing with.

Applications are built on a bamboo server with dedicated and accessible repository manager, but no internet access for tests.

I see. It's an uncommon setup, but the issue is legit (if probably low priority). I'd start by opening a ticket.

ok. Once I have some spare time, I might investigate further myself and propose a solution.

Regards

Niels



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: geoserver tests and internet connection

geowolf
On Mon, May 13, 2019 at 4:05 PM Niels Charlier <[hidden email]> wrote:

ok. Once I have some spare time, I might investigate further myself and propose a solution.

Awesome, thanks for the ticket and for having a look (when you have time, of course)

Cheers
Andrea
 
==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel