Unit Test suite instability

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Unit Test suite instability

Jackie Ng
Hi All,

I wanted to double check the unit test results that Greg reported with regards to the Linux 64-bit build status query I made some time back, just to see if a 64-bit Linux build of MapGuide Open Source with limited FDO provider support was feasible in our current timeframe.

I found that my FDO unit test results were drastically different from what was reported.

To make sure my test results wasn't being tainted by possible existing unit test failures as false positives, I ran as much of the existing unit tests against the following platforms to see if we had a solid test foundation:

 * Windows 7 32-bit
 * Windows 7 64-bit
 * CentOS 6.4 32-bit
 * Ubuntu 12.04 32-bit (Tests yet to be run)
 * CentOS 6.4 64-bit (Tests yet to be run)
 * Ubuntu 12.04 64-bit (Tests yet to be run)

My unit test results so far have been ... disturbing. So much so that my initial focus of 64-bit Linux readiness has been diverted to test readiness of FDO in general.

I've shared a Google Spreadsheet of my current test results so far, with applicable notes/observations and relevant CppUnit failures listed

https://docs.google.com/spreadsheet/ccc?key=0AhBgYUE6W7xydEhmQjBIUHpUSzNGMFpLYWE4YmRSWnc&usp=drive_web#gid=0

I'd like some thoughts as to why I could be getting such results (especially on Windows where I would've expected 100% pass rate). Such widespread failure in my test runs so far is unsettling.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Karsten Winter
Hi Jackie

I can run the unit tests on a Windows 7 64-bit environment.
When they are done, which takes a while, I will let you know about my results.

Best regards
Karsten
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Karsten Winter
Hi Jackie

Some of the unit tests are finished now. Here is the result for Win 7 64-bit:
FDO Core: 1 Failure (the same you have)
SDF: OK
SHP: OK
SQLite: OK
GDAL: OK
MySQL: OK
ODBC Access: 12 Failures, 93 Errors (the same you have)
ODBC Excel: 1 Failure, 45 Errors (the same you have)
ODBC Text: 1 Failure, 1 Error (the same you have)

The rest will follow soon.

Best regards
Karsten
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Jackie Ng
My notes about the SHP test failures were correct (on Windows at least)

There were a bunch of test SHP files removed by this revision 6 years ago:

http://trac.osgeo.org/fdo/changeset/3738

These test SHP files were still being referenced in the SHP provider test suite which caused the CppUnit failures I listed on that spreadsheet.

I restored these SHP files to their expected location (I basically checked out a pre-r3738 copy of the test data) and the SHP provider test suite passes on Windows for 32-bit and 64-bit.

I gather these files still existed on your end which is why the SHP tests are OK. But as it stands, the SHP test suite will fail for anyone who does not know about this change that was made 6 years ago.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Greg Boone
>> I gather these files still existed on your end which is why the SHP tests are OK. But as it stands, the SHP test suite will fail for anyone who does not know about this change that was made 6 years ago.

These tests should be #ifdef'ed out.

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: Thursday, March 13, 2014 8:50 AM
To: [hidden email]
Subject: Re: [fdo-internals] Unit Test suite instability

My notes about the SHP test failures were correct (on Windows at least)

There were a bunch of test SHP files removed by this revision 6 years ago:

http://trac.osgeo.org/fdo/changeset/3738

These test SHP files were still being referenced in the SHP provider test suite which caused the CppUnit failures I listed on that spreadsheet.

I restored these SHP files to their expected location (I basically checked out a pre-r3738 copy of the test data) and the SHP provider test suite passes on Windows for 32-bit and 64-bit.

I gather these files still existed on your end which is why the SHP tests are OK. But as it stands, the SHP test suite will fail for anyone who does not know about this change that was made 6 years ago.

- Jackie



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Unit-Test-suite-instability-tp5128256p5128776.html
Sent from the FDO Internals mailing list archive at Nabble.com.
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Jackie Ng
I think there's some GenericRdbms test cases (eg. FdoSchemaTest::Test_Defect802102() ) that need the #ifdef treatment as well

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Jackie Ng
In reply to this post by Greg Boone
So should we use a symbol like ADSK_FDO? Just #ifdef ADSK_FDO around all tests that use commerical FDO providers or test data that's not in the open source tree. When you go to make and unit test your commercial builds of FDO, just build with ADSK_FDO defined?

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Greg Boone
Sounds OK to me.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: Friday, March 14, 2014 1:17 AM
To: [hidden email]
Subject: Re: [fdo-internals] Unit Test suite instability

So should we use a symbol like ADSK_FDO? Just #ifdef ADSK_FDO around all tests that use commerical FDO providers or test data that's not in the open source tree. When you go to make and unit test your commercial builds of FDO, just build with ADSK_FDO defined?

- Jackie



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Unit-Test-suite-instability-tp5128256p5128959.html
Sent from the FDO Internals mailing list archive at Nabble.com.
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Jackie Ng
I've made a ticket for this and attached a few patches

http://trac.osgeo.org/fdo/ticket/883

This fixes some of the failures I've identified (described in the ticket) while running the unit tests on Windows and Linux.

Please review. Thanks.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: Unit Test suite instability

Greg Boone
SHP and SDF changes are OK.

GenericRdbmS looks ok. I will need Brent to take a second look.

http://trac.osgeo.org/fdo/attachment/ticket/883/UnitTest_GenericRdbms.patch

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: Saturday, March 15, 2014 10:51 AM
To: [hidden email]
Subject: Re: [fdo-internals] Unit Test suite instability

I've made a ticket for this and attached a few patches

http://trac.osgeo.org/fdo/ticket/883

This fixes some of the failures I've identified (described in the ticket) while running the unit tests on Windows and Linux.

Please review. Thanks.

- Jackie



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Unit-Test-suite-instability-tp5128256p5129214.html
Sent from the FDO Internals mailing list archive at Nabble.com.
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Reply | Threaded
Open this post in threaded view
|

FW: Unit Test suite instability

Brent Robinson
The GenericRdbms changes look fine to me.

-----Original Message-----
From: Greg Boone
Sent: Sunday, March 16, 2014 12:35 PM
To: FDO Internals Mail List
Cc: Brent Robinson
Subject: RE: [fdo-internals] Unit Test suite instability

SHP and SDF changes are OK.

GenericRdbmS looks ok. I will need Brent to take a second look.

http://trac.osgeo.org/fdo/attachment/ticket/883/UnitTest_GenericRdbms.patch

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jackie Ng
Sent: Saturday, March 15, 2014 10:51 AM
To: [hidden email]
Subject: Re: [fdo-internals] Unit Test suite instability

I've made a ticket for this and attached a few patches

http://trac.osgeo.org/fdo/ticket/883

This fixes some of the failures I've identified (described in the ticket) while running the unit tests on Windows and Linux.

Please review. Thanks.

- Jackie



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Unit-Test-suite-instability-tp5128256p5129214.html
Sent from the FDO Internals mailing list archive at Nabble.com.
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals