GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

jason.liu
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu 7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset = (GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset = driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of 'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use their example kdu_compress to produce 4-band RGBA jp2 images. So this looks to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason
Reply | Threaded
Open this post in threaded view
|

Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

Kurt Schwehr-2
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

Kurt Schwehr-2
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Fwd: Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

boxerab
---------- Forwarded message ----------
From: "Aaron Boxer" <[hidden email]>
Date: Feb 18, 2017 8:03 AM
Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8
To: "Kurt Schwehr" <[hidden email]>
Cc:



On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

Apparently Kakadu does no regression testing before a release 😁



It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

Kurt Schwehr-2

On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:
---------- Forwarded message ----------
From: "Aaron Boxer" <[hidden email]>
Date: Feb 18, 2017 8:03 AM
Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8
To: "Kurt Schwehr" <[hidden email]>
Cc:



On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

Apparently Kakadu does no regression testing before a release 😁



It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

boxerab


On Sat, Feb 18, 2017 at 8:14 AM, Kurt Schwehr <[hidden email]> wrote:
Well,  I made a start as a part of gdal-autotest2...


That is a good start.
 


On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:
---------- Forwarded message ----------
From: "Aaron Boxer" <[hidden email]>
Date: Feb 18, 2017 8:03 AM
Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8
To: "Kurt Schwehr" <[hidden email]>
Cc:



On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

Apparently Kakadu does no regression testing before a release 😁



It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

Kurt Schwehr-2
Only 370K more LOC left to cover with proper library level unittests.

On Mon, Feb 20, 2017 at 10:01 AM, Aaron Boxer <[hidden email]> wrote:


On Sat, Feb 18, 2017 at 8:14 AM, Kurt Schwehr <[hidden email]> wrote:
Well,  I made a start as a part of gdal-autotest2...


That is a good start.
 


On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:
---------- Forwarded message ----------
From: "Aaron Boxer" <[hidden email]>
Date: Feb 18, 2017 8:03 AM
Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8
To: "Kurt Schwehr" <[hidden email]>
Cc:



On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

Apparently Kakadu does no regression testing before a release 😁



It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--




--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8

boxerab
:):)

On Mon, Feb 20, 2017 at 1:56 PM, Kurt Schwehr <[hidden email]> wrote:
Only 370K more LOC left to cover with proper library level unittests.

On Mon, Feb 20, 2017 at 10:01 AM, Aaron Boxer <[hidden email]> wrote:


On Sat, Feb 18, 2017 at 8:14 AM, Kurt Schwehr <[hidden email]> wrote:
Well,  I made a start as a part of gdal-autotest2...


That is a good start.
 


On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:
---------- Forwarded message ----------
From: "Aaron Boxer" <[hidden email]>
Date: Feb 18, 2017 8:03 AM
Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with Kakadu 7.8
To: "Kurt Schwehr" <[hidden email]>
Cc:



On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:
Jason,

Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.  From the kakadu v7.9 docs:

h. Corrected a tiny, yet very significant bug introduced into the
   handling of opacity channel descriptions in version 7.8.  The
   correction is in function `jp2_channels::set_opacity_mapping', where
   the opacity channel's component index accidentally overwrote the
   colour channel's component index.

Apparently Kakadu does no regression testing before a release 😁



It's pretty easy to backport the fix if you are currently stuck at 7.8.  It's just a change of a 0 to a 1.

--- kakadu/v7_8/apps/jp2/jp2.cpp
+++ kakadu/v7_9/apps/jp2/jp2.cpp
@@ -5976,7 +5976,7 @@
   if (lut_idx < 0)
     lut_idx = -1; // For consistency in comparisons later on.
   j2_channels::j2_channel *cp = state->channels+colour_idx;
-  cp->component_idx[0] = codestream_component;
+  cp->component_idx[1] = codestream_component; /* Was a bug in version 7.8 */
   cp->lut_idx[1] = lut_idx;
   cp->codestream_idx[1] = codestream_idx;
   cp->data_format[1] = data_format;

The 0 can trigger an error that looks just like you are seeing:

Error: GdalIO: Error in Kakadu File Format Support: Attempting to create a Component Mapping (cmap) box, one of whose channels refers to a non-existent image component or palette lookup table. (code = 1)



On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:
Hi Jason,

I've seen the same thing with an alpha channel.  I'm not sure what's up.  Using just RGB was successful.

On 7.3, I'm also seeing problems when using more than the main thread on linux.  

With 7.3, I just noticed today that kdu_get_num_processors() only returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined.  When I enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning 74 on what I thought was a 6 core (aka 12 hyperthread) box.   If I change the fall through from 0 to 2, I get all sorts of trouble from internal kakadu assertions and TSAN failures.

So I clearly have more investigating to do.

I've yet to pass along any of what I've found to Taubman and I haven't yet pushed any of my new tests to github, but I hope to do both soon.

And I've yet to see what, if any, discussion is happening in the yahoo group.

-kurt

On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]> wrote:
Hi gdal-dev,

I am having trouble using GDAL to write 4-band RGBA jp2 file with Kakadu
7.8. The following minimal code


    GDALAllRegister();

    GDALDataset* srcDataset =
(GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);
    char **papszOptions = NULL;
    papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

    GDALDriver *driver = GetGDALDriverManager()->GetDriverByName("JP2KAK");
    GDALDataset* destDataset =
driver->CreateCopy("/home/jason.liu/00000783.jp2",
                                                  srcDataset,
                                                  FALSE,
                                                  papszOptions,
                                                  NULL,
                                                  NULL);

    if( destDataset != NULL ) {
        GDALClose( (GDALDatasetH) destDataset );
    }
    GDALClose( (GDALDatasetH) srcDataset );
    CSLDestroy(papszOptions);


crashes Kakadu 7.8 with the following message in stderr:

ERROR 1: Error in Kakadu File Format Support:
Attempting to create a Component Mapping (cmap) box, one of whose channels
refers to a non-existent image component or palette lookup table.
terminate called after throwing an instance of
'kdu_cpl_error_message::JP2KAKException'


where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.


I have also done the following tests and noted the results:

3-band 8-bit RGB tif            OK
3-band 16-bit RGB tif          OK
4-band 8-bit RGBA tif          error as above
4-band 16-bit RGBA tif        error as above


Am I doing anything wrong here?

By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I can use
their example kdu_compress to produce 4-band RGBA jp2 images. So this looks
to be a problem with GDAL, or the way I am invoking it.


Any insights will be greatly appreciated.

Kind Regards
Jason




--
View this message in context: http://osgeo-org.1560.x6.nabble.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-7-8-tp5307801.html
Sent from the GDAL - Dev mailing list archive at Nabble.com.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--




--


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] gdal-autotest2

Even Rouault-2
In reply to this post by Kurt Schwehr-2

On lundi 20 février 2017 10:56:16 CET Kurt Schwehr wrote:

> Only 370K more LOC left to cover with proper library level unittests.

 

Changing the topic line.

 

Kurt

 

IHMO if your aim is to have gdal-autotest2 being merged some day, the forked approach is going to be a nightmare for you to manage, unless you have a huge workforce to put behind. I'd rather attempt at having the current autotest suite running as a hack into the cleaner approach, so it can be merged into trunk and once done, migrate things progressively.

 

Even

 

>

> On Mon, Feb 20, 2017 at 10:01 AM, Aaron Boxer <[hidden email]> wrote:

> > On Sat, Feb 18, 2017 at 8:14 AM, Kurt Schwehr <[hidden email]> wrote:

> >> Well, I made a start as a part of gdal-autotest2...

> >>

> >>

> >> https://github.com/schwehr/gdal-autotest2/blob/master/cpp/

> >> third_party/kakadu/coresys/common/kdu_arch_test.cc

> >

> > That is a good start.

> >

> >> On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:

> >>> ---------- Forwarded message ----------

> >>> From: "Aaron Boxer" <[hidden email]>

> >>> Date: Feb 18, 2017 8:03 AM

> >>> Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with

> >>> Kakadu 7.8

> >>> To: "Kurt Schwehr" <[hidden email]>

> >>> Cc:

> >>>

> >>>

> >>>

> >>> On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:

> >>>

> >>> Jason,

> >>>

> >>> Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.

> >>> From the kakadu v7.9 docs:

> >>>

> >>> h. Corrected a tiny, yet very significant bug introduced into the

> >>>

> >>> handling of opacity channel descriptions in version 7.8. The

> >>> correction is in function `jp2_channels::set_opacity_mapping', where

> >>> the opacity channel's component index accidentally overwrote the

> >>> colour channel's component index.

> >>>

> >>> Apparently Kakadu does no regression testing before a release 😁

> >>>

> >>>

> >>>

> >>> It's pretty easy to backport the fix if you are currently stuck at 7.8.

> >>> It's just a change of a 0 to a 1.

> >>>

> >>> --- kakadu/v7_8/apps/jp2/jp2.cpp

> >>> +++ kakadu/v7_9/apps/jp2/jp2.cpp

> >>> @@ -5976,7 +5976,7 @@

> >>>

> >>> if (lut_idx < 0)

> >>>

> >>> lut_idx = -1; // For consistency in comparisons later on.

> >>>

> >>> j2_channels::j2_channel *cp = state->channels+colour_idx;

> >>>

> >>> - cp->component_idx[0] = codestream_component;

> >>> + cp->component_idx[1] = codestream_component; /* Was a bug in version

> >>> 7.8 */

> >>>

> >>> cp->lut_idx[1] = lut_idx;

> >>> cp->codestream_idx[1] = codestream_idx;

> >>> cp->data_format[1] = data_format;

> >>>

> >>> The 0 can trigger an error that looks just like you are seeing:

> >>>

> >>> Error: GdalIO: Error in Kakadu File Format Support: Attempting to create

> >>> a Component Mapping (cmap) box, one of whose channels refers to a

> >>> non-existent image component or palette lookup table. (code = 1)

> >>>

> >>> On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:

> >>>> Hi Jason,

> >>>>

> >>>> I've seen the same thing with an alpha channel. I'm not sure what's

> >>>> up. Using just RGB was successful.

> >>>>

> >>>> On 7.3, I'm also seeing problems when using more than the main thread

> >>>> on linux.

> >>>>

> >>>> With 7.3, I just noticed today that kdu_get_num_processors() only

> >>>> returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so

> >>>> _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined. When I

> >>>> enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning

> >>>> 74

> >>>> on what I thought was a 6 core (aka 12 hyperthread) box. If I change

> >>>> the

> >>>> fall through from 0 to 2, I get all sorts of trouble from internal

> >>>> kakadu

> >>>> assertions and TSAN failures.

> >>>>

> >>>> So I clearly have more investigating to do.

> >>>>

> >>>> I've yet to pass along any of what I've found to Taubman and I haven't

> >>>> yet pushed any of my new tests to github, but I hope to do both soon.

> >>>>

> >>>> And I've yet to see what, if any, discussion is happening in the yahoo

> >>>> group.

> >>>>

> >>>> -kurt

> >>>>

> >>>> On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]>

> >>>>

> >>>> wrote:

> >>>>> Hi gdal-dev,

> >>>>>

> >>>>> I am having trouble using GDAL to write 4-band RGBA jp2 file with

> >>>>> Kakadu

> >>>>> 7.8. The following minimal code

> >>>>>

> >>>>> GDALAllRegister();

> >>>>>

> >>>>> GDALDataset* srcDataset =

> >>>>>

> >>>>> (GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);

> >>>>>

> >>>>> char **papszOptions = NULL;

> >>>>> papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

> >>>>>

> >>>>> GDALDriver *driver = GetGDALDriverManager()->GetDri

> >>>>>

> >>>>> verByName("JP2KAK");

> >>>>>

> >>>>> GDALDataset* destDataset =

> >>>>>

> >>>>> driver->CreateCopy("/home/jason.liu/00000783.jp2",

> >>>>>

> >>>>> srcDataset,

> >>>>> FALSE,

> >>>>> papszOptions,

> >>>>> NULL,

> >>>>> NULL);

> >>>>>

> >>>>> if( destDataset != NULL ) {

> >>>>>

> >>>>> GDALClose( (GDALDatasetH) destDataset );

> >>>>>

> >>>>> }

> >>>>> GDALClose( (GDALDatasetH) srcDataset );

> >>>>> CSLDestroy(papszOptions);

> >>>>>

> >>>>> crashes Kakadu 7.8 with the following message in stderr:

> >>>>>

> >>>>> ERROR 1: Error in Kakadu File Format Support:

> >>>>> Attempting to create a Component Mapping (cmap) box, one of whose

> >>>>> channels

> >>>>> refers to a non-existent image component or palette lookup table.

> >>>>> terminate called after throwing an instance of

> >>>>> 'kdu_cpl_error_message::JP2KAKException'

> >>>>>

> >>>>>

> >>>>> where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.

> >>>>>

> >>>>>

> >>>>> I have also done the following tests and noted the results:

> >>>>>

> >>>>> 3-band 8-bit RGB tif OK

> >>>>> 3-band 16-bit RGB tif OK

> >>>>> 4-band 8-bit RGBA tif error as above

> >>>>> 4-band 16-bit RGBA tif error as above

> >>>>>

> >>>>>

> >>>>> Am I doing anything wrong here?

> >>>>>

> >>>>> By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I

> >>>>> can use

> >>>>> their example kdu_compress to produce 4-band RGBA jp2 images. So this

> >>>>> looks

> >>>>> to be a problem with GDAL, or the way I am invoking it.

> >>>>>

> >>>>>

> >>>>> Any insights will be greatly appreciated.

> >>>>>

> >>>>> Kind Regards

> >>>>> Jason

> >>>>>

> >>>>>

> >>>>>

> >>>>>

> >>>>> --

> >>>>> View this message in context: http://osgeo-org.1560.x6.nabbl

> >>>>> e.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-

> >>>>> 7-8-tp5307801.html

> >>>>> Sent from the GDAL - Dev mailing list archive at Nabble.com.

> >>>>> _______________________________________________

> >>>>> gdal-dev mailing list

> >>>>> [hidden email]

> >>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>>>

> >>>> --

> >>>> --

> >>>> http://schwehr.org

> >>>

> >>> --

> >>> --

> >>> http://schwehr.org

> >>>

> >>> _______________________________________________

> >>> gdal-dev mailing list

> >>> [hidden email]

> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>>

> >>>

> >>>

> >>> _______________________________________________

> >>> gdal-dev mailing list

> >>> [hidden email]

> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>

> >> --

> >> --

> >> http://schwehr.org

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Kurt Schwehr-2
Well, it would be great if the GDAL community wants to merge that code.  I wanted to get a big enough body of code that people could get a sense of it before suggesting any kind of merge.

But anyway it goes, it's a win for me. It's what I run my continuous integration against in the custom environment I'm responsible for.  And it's the code that makes merging from svn manageable.  It's running for all dependent lib and tool chain changes many times an hour and also runs as hardened, asan, msan, and tsan.

On Mon, Feb 20, 2017 at 11:26 AM, Even Rouault <[hidden email]> wrote:

On lundi 20 février 2017 10:56:16 CET Kurt Schwehr wrote:

> Only 370K more LOC left to cover with proper library level unittests.

 

Changing the topic line.

 

Kurt

 

IHMO if your aim is to have gdal-autotest2 being merged some day, the forked approach is going to be a nightmare for you to manage, unless you have a huge workforce to put behind. I'd rather attempt at having the current autotest suite running as a hack into the cleaner approach, so it can be merged into trunk and once done, migrate things progressively.

 

Even

 

>

> On Mon, Feb 20, 2017 at 10:01 AM, Aaron Boxer <[hidden email]> wrote:

> > On Sat, Feb 18, 2017 at 8:14 AM, Kurt Schwehr <[hidden email]> wrote:

> >> Well, I made a start as a part of gdal-autotest2...

> >>

> >>

> >> https://github.com/schwehr/gdal-autotest2/blob/master/cpp/

> >> third_party/kakadu/coresys/common/kdu_arch_test.cc

> >

> > That is a good start.

> >

> >> On Sat, Feb 18, 2017 at 5:03 AM, Aaron Boxer <[hidden email]> wrote:

> >>> ---------- Forwarded message ----------

> >>> From: "Aaron Boxer" <[hidden email]>

> >>> Date: Feb 18, 2017 8:03 AM

> >>> Subject: Re: [gdal-dev] GDAL unable to write 4-band RGBA jp2 file with

> >>> Kakadu 7.8

> >>> To: "Kurt Schwehr" <[hidden email]>

> >>> Cc:

> >>>

> >>>

> >>>

> >>> On Feb 18, 2017 7:26 AM, "Kurt Schwehr" <[hidden email]> wrote:

> >>>

> >>> Jason,

> >>>

> >>> Turns out you are hitting a known bug in v7.8 that is fixed in v7.9.

> >>> From the kakadu v7.9 docs:

> >>>

> >>> h. Corrected a tiny, yet very significant bug introduced into the

> >>>

> >>> handling of opacity channel descriptions in version 7.8. The

> >>> correction is in function `jp2_channels::set_opacity_mapping', where

> >>> the opacity channel's component index accidentally overwrote the

> >>> colour channel's component index.

> >>>

> >>> Apparently Kakadu does no regression testing before a release 😁

> >>>

> >>>

> >>>

> >>> It's pretty easy to backport the fix if you are currently stuck at 7.8.

> >>> It's just a change of a 0 to a 1.

> >>>

> >>> --- kakadu/v7_8/apps/jp2/jp2.cpp

> >>> +++ kakadu/v7_9/apps/jp2/jp2.cpp

> >>> @@ -5976,7 +5976,7 @@

> >>>

> >>> if (lut_idx < 0)

> >>>

> >>> lut_idx = -1; // For consistency in comparisons later on.

> >>>

> >>> j2_channels::j2_channel *cp = state->channels+colour_idx;

> >>>

> >>> - cp->component_idx[0] = codestream_component;

> >>> + cp->component_idx[1] = codestream_component; /* Was a bug in version

> >>> 7.8 */

> >>>

> >>> cp->lut_idx[1] = lut_idx;

> >>> cp->codestream_idx[1] = codestream_idx;

> >>> cp->data_format[1] = data_format;

> >>>

> >>> The 0 can trigger an error that looks just like you are seeing:

> >>>

> >>> Error: GdalIO: Error in Kakadu File Format Support: Attempting to create

> >>> a Component Mapping (cmap) box, one of whose channels refers to a

> >>> non-existent image component or palette lookup table. (code = 1)

> >>>

> >>> On Mon, Feb 13, 2017 at 9:28 PM, Kurt Schwehr <[hidden email]> wrote:

> >>>> Hi Jason,

> >>>>

> >>>> I've seen the same thing with an alpha channel. I'm not sure what's

> >>>> up. Using just RGB was successful.

> >>>>

> >>>> On 7.3, I'm also seeing problems when using more than the main thread

> >>>> on linux.

> >>>>

> >>>> With 7.3, I just noticed today that kdu_get_num_processors() only

> >>>> returns 0 on linux as kdu_arch.cpp is missing #include <unistd.h> so

> >>>> _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF are undefined. When I

> >>>> enabled _SC_NPROCESSORS_ONLN, I got kdu_get_num_processors() returning

> >>>> 74

> >>>> on what I thought was a 6 core (aka 12 hyperthread) box. If I change

> >>>> the

> >>>> fall through from 0 to 2, I get all sorts of trouble from internal

> >>>> kakadu

> >>>> assertions and TSAN failures.

> >>>>

> >>>> So I clearly have more investigating to do.

> >>>>

> >>>> I've yet to pass along any of what I've found to Taubman and I haven't

> >>>> yet pushed any of my new tests to github, but I hope to do both soon.

> >>>>

> >>>> And I've yet to see what, if any, discussion is happening in the yahoo

> >>>> group.

> >>>>

> >>>> -kurt

> >>>>

> >>>> On Mon, Feb 13, 2017 at 8:14 PM, jason.liu <[hidden email]>

> >>>>

> >>>> wrote:

> >>>>> Hi gdal-dev,

> >>>>>

> >>>>> I am having trouble using GDAL to write 4-band RGBA jp2 file with

> >>>>> Kakadu

> >>>>> 7.8. The following minimal code

> >>>>>

> >>>>> GDALAllRegister();

> >>>>>

> >>>>> GDALDataset* srcDataset =

> >>>>>

> >>>>> (GDALDataset*)GDALOpen("/home/jason.liu/00000783.tif", GA_ReadOnly);

> >>>>>

> >>>>> char **papszOptions = NULL;

> >>>>> papszOptions = CSLSetNameValue( papszOptions, "QUALITY", "20" );

> >>>>>

> >>>>> GDALDriver *driver = GetGDALDriverManager()->GetDri

> >>>>>

> >>>>> verByName("JP2KAK");

> >>>>>

> >>>>> GDALDataset* destDataset =

> >>>>>

> >>>>> driver->CreateCopy("/home/jason.liu/00000783.jp2",

> >>>>>

> >>>>> srcDataset,

> >>>>> FALSE,

> >>>>> papszOptions,

> >>>>> NULL,

> >>>>> NULL);

> >>>>>

> >>>>> if( destDataset != NULL ) {

> >>>>>

> >>>>> GDALClose( (GDALDatasetH) destDataset );

> >>>>>

> >>>>> }

> >>>>> GDALClose( (GDALDatasetH) srcDataset );

> >>>>> CSLDestroy(papszOptions);

> >>>>>

> >>>>> crashes Kakadu 7.8 with the following message in stderr:

> >>>>>

> >>>>> ERROR 1: Error in Kakadu File Format Support:

> >>>>> Attempting to create a Component Mapping (cmap) box, one of whose

> >>>>> channels

> >>>>> refers to a non-existent image component or palette lookup table.

> >>>>> terminate called after throwing an instance of

> >>>>> 'kdu_cpl_error_message::JP2KAKException'

> >>>>>

> >>>>>

> >>>>> where /home/jason.liu/00000783.tif is a 4-band 16-bit RGBA tif.

> >>>>>

> >>>>>

> >>>>> I have also done the following tests and noted the results:

> >>>>>

> >>>>> 3-band 8-bit RGB tif OK

> >>>>> 3-band 16-bit RGB tif OK

> >>>>> 4-band 8-bit RGBA tif error as above

> >>>>> 4-band 16-bit RGBA tif error as above

> >>>>>

> >>>>>

> >>>>> Am I doing anything wrong here?

> >>>>>

> >>>>> By the way, I am quite certain Kakadu 7.8 supports 4-band RGBA as I

> >>>>> can use

> >>>>> their example kdu_compress to produce 4-band RGBA jp2 images. So this

> >>>>> looks

> >>>>> to be a problem with GDAL, or the way I am invoking it.

> >>>>>

> >>>>>

> >>>>> Any insights will be greatly appreciated.

> >>>>>

> >>>>> Kind Regards

> >>>>> Jason

> >>>>>

> >>>>>

> >>>>>

> >>>>>

> >>>>> --

> >>>>> View this message in context: http://osgeo-org.1560.x6.nabbl

> >>>>> e.com/GDAL-unable-to-write-4-band-RGBA-jp2-file-with-Kakadu-

> >>>>> 7-8-tp5307801.html

> >>>>> Sent from the GDAL - Dev mailing list archive at Nabble.com.

> >>>>> _______________________________________________

> >>>>> gdal-dev mailing list

> >>>>> [hidden email]

> >>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>>>

> >>>> --

> >>>> --

> >>>> http://schwehr.org

> >>>

> >>> --

> >>> --

> >>> http://schwehr.org

> >>>

> >>> _______________________________________________

> >>> gdal-dev mailing list

> >>> [hidden email]

> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>>

> >>>

> >>>

> >>> _______________________________________________

> >>> gdal-dev mailing list

> >>> [hidden email]

> >>> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> >>

> >> --

> >> --

> >> http://schwehr.org

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Mateusz Loskot
Kurt Schwehr-2 wrote
Well, it would be great if the GDAL community wants to merge that code.
+1 - I'd welcome it

Perhaps it would make sense to name autotest2 simply as tests
and place it inside gdal folder.

Mateusz
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Mateusz Loskot
On 21 February 2017 at 14:01, Mateusz Loskot <[hidden email]> wrote:
> Kurt Schwehr-2 wrote
>> Well, it would be great if the GDAL community wants to merge that code.
>
> +1 - I'd welcome it

Is re-writing tests of GDAL dependencies part of the big plan?
https://github.com/schwehr/gdal-autotest2/commit/e65e6d5aa03753c4b9a67e2cec6a8bdbe0997a80)

AFAIK, GEOS is covered with tests pretty well.

Best regards.
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Kurt Schwehr-2
The geos coverage is okay, but I have a hard time working with it.  I've mostly been putting tests for libs that GDAL can be dependent on for lack of another place to put them as I can push the code.

On Tue, Feb 21, 2017 at 10:34 AM, Mateusz Loskot <[hidden email]> wrote:
On 21 February 2017 at 14:01, Mateusz Loskot <[hidden email]> wrote:
> Kurt Schwehr-2 wrote
>> Well, it would be great if the GDAL community wants to merge that code.
>
> +1 - I'd welcome it

Is re-writing tests of GDAL dependencies part of the big plan?
https://github.com/schwehr/gdal-autotest2/commit/e65e6d5aa03753c4b9a67e2cec6a8bdbe0997a80)

AFAIK, GEOS is covered with tests pretty well.

Best regards.
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Mateusz Loskot
I always thought GEOS tests are dead simple based on lightweight runner.

If something is missing, just add it and submit to GEOS.

Rewriting tests just for the sake of making them based on GTest.
Well...

Mateusz

On 21 Feb 2017 8:10 p.m., "Kurt Schwehr" <[hidden email]> wrote:
The geos coverage is okay, but I have a hard time working with it.  I've mostly been putting tests for libs that GDAL can be dependent on for lack of another place to put them as I can push the code.

On Tue, Feb 21, 2017 at 10:34 AM, Mateusz Loskot <[hidden email]> wrote:
On 21 February 2017 at 14:01, Mateusz Loskot <[hidden email]> wrote:
> Kurt Schwehr-2 wrote
>> Well, it would be great if the GDAL community wants to merge that code.
>
> +1 - I'd welcome it

Is re-writing tests of GDAL dependencies part of the big plan?
https://github.com/schwehr/gdal-autotest2/commit/e65e6d5aa03753c4b9a67e2cec6a8bdbe0997a80)

AFAIK, GEOS is covered with tests pretty well.

Best regards.
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal-autotest2

Kurt Schwehr-2
It's often easier to port tests to get the breakout I need that is hard to do by chaining test runners.  Plus I need to get in there and setup some fuzzers, which is very not like tut.

On Tue, Feb 21, 2017 at 11:21 AM, Mateusz Loskot <[hidden email]> wrote:
I always thought GEOS tests are dead simple based on lightweight runner.

If something is missing, just add it and submit to GEOS.

Rewriting tests just for the sake of making them based on GTest.
Well...

Mateusz

On 21 Feb 2017 8:10 p.m., "Kurt Schwehr" <[hidden email]> wrote:
The geos coverage is okay, but I have a hard time working with it.  I've mostly been putting tests for libs that GDAL can be dependent on for lack of another place to put them as I can push the code.

On Tue, Feb 21, 2017 at 10:34 AM, Mateusz Loskot <[hidden email]> wrote:
On 21 February 2017 at 14:01, Mateusz Loskot <[hidden email]> wrote:
> Kurt Schwehr-2 wrote
>> Well, it would be great if the GDAL community wants to merge that code.
>
> +1 - I'd welcome it

Is re-writing tests of GDAL dependencies part of the big plan?
https://github.com/schwehr/gdal-autotest2/commit/e65e6d5aa03753c4b9a67e2cec6a8bdbe0997a80)

AFAIK, GEOS is covered with tests pretty well.

Best regards.
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



--



--

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev