ImageMosaic custom dimension internal to NetCDF files

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ImageMosaic custom dimension internal to NetCDF files

Ben Caradoc-Davies-2
I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x)  --ImageMosaic-NetCDF-out-->  foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, y, x)

Not, as I had hoped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Daniele Romagnoli-3
I Ben.
I'm not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with inner dimension referencing to times and elevations/depths/heights/ (everything is a Z).

Please, let us know.
Cheers,
Daniele




On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <[hidden email]> wrote:
I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x)  --ImageMosaic-NetCDF-out-->  foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, y, x)

Not, as I had hoped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users



--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

 

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Adam Steer
Hi all

I have a similar (I think) issue I’d like to solve. Like Ben, I have netCDF files with bands, and in each band are timeslices. Here’s an openDAP look inside a sample file:

http://dapds01.nci.org.au/thredds/dodsC/uc0/rs0_dev/alberscube/LANDSAT_5_TM_NBAR_ALB_15_-41_1990-03-02T23-11-39.000000.nc.das

If I publish each band as a layer (eg band1, band30, …), I can see the time slices inside each ‘band’. What I really *want* to do is publish the netCDF file as one unit - so that an end user can subset the data with all the bands and time slices intact, or potentially do some band maths at the server side (using WPS).

On a similar topic (without wanting to derail this conversation) - does geoserver have any /plan to have any openDAP style services? It’s great to be able to subset netCDFs, but being able to query a file (like the link above) before downloading it would be great!

Thanks

Adam


> On 30 May 2016, at 5:40 PM, Daniele Romagnoli <[hidden email]> wrote:
>
> I Ben.
> I'm not sure to have fully understood your problem/question.
> Can you give us an example of a ncdump output (a shorter form of the interesting part) for 2 involved files?
>
> As far as I remember, the current NetCDF low level readers only deal with inner dimension referencing to times and elevations/depths/heights/ (everything is a Z).
>
> Please, let us know.
> Cheers,
> Daniele
>
>
>
>
> On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <[hidden email]> wrote:
> I have several NetCDF files that comprise an ImageMosaic time series.
> This works fine for variables with only spatial dimensions, and I can
> get NetCDF output from WCS:
>
> foo(y, x)  --ImageMosaic-NetCDF-out-->  foo(t, y, x)
>
> However, when there is an extra dimension in the source NetCDF files
> (same values in all files), it is silently dropped:
>
> bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, y, x)
>
> Not, as I had hoped:
>
> bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, s, y, x)
>
> The documentation describes mosaicking over a custom dimension which
> varies between files and whose value is stored in the granule index. How
> to I deliver a variable with a dimension that is internal to each source
> NetCDF file? Is this supported? Is extra configuration is required?
>
> Kind regards,
>
> --
> Ben Caradoc-Davies <[hidden email]>
> Director
> Transient Software Limited <http://transient.nz/>
> New Zealand
>
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> _______________________________________________
> Geoserver-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Daniele Romagnoli
> Senior Software Engineer
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax:      +39 0584 1660272
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
>
>  
> The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
>
>
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e_______________________________________________
> Geoserver-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Ben Caradoc-Davies-2
In reply to this post by Daniele Romagnoli-3
Daniele,

these dimensions are in effect a Z dimension. The data is sourced from
GRIB2 but by the time it is ingested by ImageMosaic, it is equivalent to
NetCDF (CDM API). Below I have included a small but self-consistent
subset from the CDL generated with NetCDF-Java ToolsUI (current master
4.6.6-SNAPSHOT with rotated pole GRIB2 support). There are multiple
files that vary in time only, and ImageMosaic aggregates them into a
time series. I have included two variables as they appear in one file
(granule):

Temperature_surface(time=1, rlat=834, rlon=953) works fine and I can use
ImageMosaic to obtain a correct WCS time series. Panoply animations here:
https://github.com/bencaradocdavies/geoserver/wiki/RAP-Native-Grid-ImageMosaic

u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953) appears
to work but the output contains no sigma dimension. If I add a sigma
dimension in indexer.xml, the ImageMosaic refuses to load. Without a
sigma dimension in indexer.xml, the request appears to work, takes 50
times longer (as expected), but the output file contains no sigma
dimension or coordinate variable and is the wrong size (same as
Temperature_surface).

In a nutshell: should elevation variables that are *internal* to each
granule be indexed? Is this supported? If so, is indexer configuration
required? Why would the granule index need to know?

The indexer domain I tried unsuccessfully was this (no entry in the
granule schema, ImageMosaic would not load):

<domain name="sigma">
<attributes>
<attribute>sigma</attribute>
</attributes>
</domain>

All the examples that I have seen with an elevation or custom dimension
have been where this dimension has a single value for each granule and
varies across them. I have not seen an example with extra dimensions
internal to each NetCDF file (granule).

Kind regards,
Ben.


netcdf forecast {
   dimensions:
     rlat = 834;
     rlon = 953;
     time = 1;
     sigma = 50;
   variables:
     int RotatedLatLon32769_Projection;
       :grid_mapping_name = "rotated_latitude_longitude";
       :grid_north_pole_latitude = 36.0; // double
       :grid_north_pole_longitude = 74.0; // double
     float rlat(rlat=834);
       :standard_name = "grid_latitude";
       :units = "degrees";
     float rlon(rlon=953);
       :standard_name = "grid_longitude";
       :units = "degrees";
     double reftime;
       :units = "Hour since 2016-04-25T22:00:00Z";
       :standard_name = "forecast_reference_time";
       :long_name = "GRIB reference time";
       :calendar = "proleptic_gregorian";
     double time(time=1);
       :units = "Hour since 2016-04-25T22:00:00Z";
       :standard_name = "time";
       :long_name = "GRIB forecast or observation time";
       :calendar = "proleptic_gregorian";
     float sigma(sigma=50);
       :units = "sigma";
       :long_name = "Sigma level";
       :positive = "down";
       :Grib_level_type = 104; // int
     float Temperature_surface(time=1, rlat=834, rlon=953);
       :long_name = "Temperature @ Ground or water surface";
       :units = "K";
       :abbreviation = "TMP";
       :missing_value = NaNf; // float
       :grid_mapping = "RotatedLatLon32769_Projection";
       :coordinates = "reftime time rlat rlon ";
       :Grib_Variable_Id = "VAR_0-0-0_L1";
       :Grib2_Parameter = 0, 0, 0; // int
       :Grib2_Parameter_Discipline = "Meteorological products";
       :Grib2_Parameter_Category = "Temperature";
       :Grib2_Parameter_Name = "Temperature";
       :Grib2_Level_Type = "Ground or water surface";
       :Grib2_Generating_Process_Type = "Forecast";
     float u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953);
       :long_name = "u-component of wind @ Sigma level";
       :units = "m/s";
       :abbreviation = "UGRD";
       :missing_value = NaNf; // float
       :grid_mapping = "RotatedLatLon32769_Projection";
       :coordinates = "reftime time sigma rlat rlon ";
       :Grib_Variable_Id = "VAR_0-2-2_L104";
       :Grib2_Parameter = 0, 2, 2; // int
       :Grib2_Parameter_Discipline = "Meteorological products";
       :Grib2_Parameter_Category = "Momentum";
       :Grib2_Parameter_Name = "u-component of wind";
       :Grib2_Level_Type = "Sigma level";
       :Grib2_Generating_Process_Type = "Forecast";
   // global attributes:
   :Originating_or_generating_Center = "The NOAA Forecast Systems
Laboratory, Boulder, CO, United States";
   :Originating_or_generating_Subcenter = "0";
   :GRIB_table_version = "2,1";
   :Type_of_generating_process = "Forecast";
 
:Analysis_or_forecast_generating_process_identifier_defined_by_originating_centre
= "High-Resolution Rapid Refresh";
   :file_format = "GRIB-2";
   :Conventions = "CF-1.6";
   :history = "Read using CDM IOSP GribCollection v3";
   :featureType = "GRID";
}


On 30/05/16 19:40, Daniele Romagnoli wrote:

> I Ben.
> I'm not sure to have fully understood your problem/question.
> Can you give us an example of a ncdump output (a shorter form of the
> interesting part) for 2 involved files?
>
> As far as I remember, the current NetCDF low level readers only deal with
> inner dimension referencing to times and elevations/depths/heights/
> (everything is a Z).
>
> Please, let us know.
> Cheers,
> Daniele
>
>
>
>
> On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <[hidden email]>
> wrote:
>
>> I have several NetCDF files that comprise an ImageMosaic time series.
>> This works fine for variables with only spatial dimensions, and I can
>> get NetCDF output from WCS:
>>
>> foo(y, x)  --ImageMosaic-NetCDF-out-->  foo(t, y, x)
>>
>> However, when there is an extra dimension in the source NetCDF files
>> (same values in all files), it is silently dropped:
>>
>> bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, y, x)
>>
>> Not, as I had hoped:
>>
>> bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, s, y, x)
>>
>> The documentation describes mosaicking over a custom dimension which
>> varies between files and whose value is stored in the granule index. How
>> to I deliver a variable with a dimension that is internal to each source
>> NetCDF file? Is this supported? Is extra configuration is required?
>>
>> Kind regards,
>>
>> --
>> Ben Caradoc-Davies <[hidden email]>
>> Director
>> Transient Software Limited <http://transient.nz/>
>> New Zealand
>>
>>
>> ------------------------------------------------------------------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and
>> traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols
>> are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity
>> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
>> _______________________________________________
>> Geoserver-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>
>
>
>

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Daniele Romagnoli-3
Hi Ben,

On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <[hidden email]> wrote:
Daniele,

these dimensions are in effect a Z dimension. The data is sourced from GRIB2 but by the time it is ingested by ImageMosaic, it is equivalent to NetCDF (CDM API). Below I have included a small but self-consistent subset from the CDL generated with NetCDF-Java ToolsUI (current master 4.6.6-SNAPSHOT with rotated pole GRIB2 support). There are multiple files that vary in time only, and ImageMosaic aggregates them into a time series. I have included two variables as they appear in one file (granule):

Temperature_surface(time=1, rlat=834, rlon=953) works fine and I can use ImageMosaic to obtain a correct WCS time series. Panoply animations here:
https://github.com/bencaradocdavies/geoserver/wiki/RAP-Native-Grid-ImageMosaic

u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953) appears to work but the output contains no sigma dimension. If I add a sigma dimension in indexer.xml, the ImageMosaic refuses to load. Without a sigma dimension in indexer.xml, the request appears to work, takes 50 times longer (as expected), but the output file contains no sigma dimension or coordinate variable and is the wrong size (same as Temperature_surface).

In a nutshell: should elevation variables that are *internal* to each granule be indexed? Is this supported? If so, is indexer configuration required? Why would the granule index need to know?

I would say yes. In the past we have worked with NetCDF containing sigma levels, exposing them as sigma dimension, similarly to what you did.
If my memory serves me right, when having issues with custom dimensions the problem was related to the axisType not being declared but I should double check.
Would this help?

Daniele
 
The indexer domain I tried unsuccessfully was this (no entry in the granule schema, ImageMosaic would not load):

<domain name="sigma">
<attributes>
<attribute>sigma</attribute>
</attributes>
</domain>

All the examples that I have seen with an elevation or custom dimension have been where this dimension has a single value for each granule and varies across them. I have not seen an example with extra dimensions internal to each NetCDF file (granule).

Kind regards,
Ben.


netcdf forecast {
  dimensions:
    rlat = 834;
    rlon = 953;
    time = 1;
    sigma = 50;
  variables:
    int RotatedLatLon32769_Projection;
      :grid_mapping_name = "rotated_latitude_longitude";
      :grid_north_pole_latitude = 36.0; // double
      :grid_north_pole_longitude = 74.0; // double
    float rlat(rlat=834);
      :standard_name = "grid_latitude";
      :units = "degrees";
    float rlon(rlon=953);
      :standard_name = "grid_longitude";
      :units = "degrees";
    double reftime;
      :units = "Hour since 2016-04-25T22:00:00Z";
      :standard_name = "forecast_reference_time";
      :long_name = "GRIB reference time";
      :calendar = "proleptic_gregorian";
    double time(time=1);
      :units = "Hour since 2016-04-25T22:00:00Z";
      :standard_name = "time";
      :long_name = "GRIB forecast or observation time";
      :calendar = "proleptic_gregorian";
    float sigma(sigma=50);
      :units = "sigma";
      :long_name = "Sigma level";
      :positive = "down";
      :Grib_level_type = 104; // int
    float Temperature_surface(time=1, rlat=834, rlon=953);
      :long_name = "Temperature @ Ground or water surface";
      :units = "K";
      :abbreviation = "TMP";
      :missing_value = NaNf; // float
      :grid_mapping = "RotatedLatLon32769_Projection";
      :coordinates = "reftime time rlat rlon ";
      :Grib_Variable_Id = "VAR_0-0-0_L1";
      :Grib2_Parameter = 0, 0, 0; // int
      :Grib2_Parameter_Discipline = "Meteorological products";
      :Grib2_Parameter_Category = "Temperature";
      :Grib2_Parameter_Name = "Temperature";
      :Grib2_Level_Type = "Ground or water surface";
      :Grib2_Generating_Process_Type = "Forecast";
    float u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953);
      :long_name = "u-component of wind @ Sigma level";
      :units = "m/s";
      :abbreviation = "UGRD";
      :missing_value = NaNf; // float
      :grid_mapping = "RotatedLatLon32769_Projection";
      :coordinates = "reftime time sigma rlat rlon ";
      :Grib_Variable_Id = "VAR_0-2-2_L104";
      :Grib2_Parameter = 0, 2, 2; // int
      :Grib2_Parameter_Discipline = "Meteorological products";
      :Grib2_Parameter_Category = "Momentum";
      :Grib2_Parameter_Name = "u-component of wind";
      :Grib2_Level_Type = "Sigma level";
      :Grib2_Generating_Process_Type = "Forecast";
  // global attributes:
  :Originating_or_generating_Center = "The NOAA Forecast Systems Laboratory, Boulder, CO, United States";
  :Originating_or_generating_Subcenter = "0";
  :GRIB_table_version = "2,1";
  :Type_of_generating_process = "Forecast";

:Analysis_or_forecast_generating_process_identifier_defined_by_originating_centre = "High-Resolution Rapid Refresh";
  :file_format = "GRIB-2";
  :Conventions = "CF-1.6";
  :history = "Read using CDM IOSP GribCollection v3";
  :featureType = "GRID";

}


On 30/05/16 19:40, Daniele Romagnoli wrote:
I Ben.
I'm not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the
interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with
inner dimension referencing to times and elevations/depths/heights/
(everything is a Z).

Please, let us know.
Cheers,
Daniele




On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <[hidden email]>
wrote:

I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x)  --ImageMosaic-NetCDF-out-->  foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, y, x)

Not, as I had hoped:

bar(s, y, x)  --ImageMosaic-NetCDF-out-->  bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users





--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand



--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

 

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Ben Caradoc-Davies-2
On 01/06/16 00:26, Daniele Romagnoli wrote:

> On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <[hidden email]>
> wrote:
>> In a nutshell: should elevation variables that are *internal* to each
>> granule be indexed? Is this supported? If so, is indexer configuration
>> required? Why would the granule index need to know?
> I would say yes. In the past we have worked with NetCDF containing sigma
> levels, exposing them as sigma dimension, similarly to what you did.
> If my memory serves me right, when having issues with custom dimensions the
> problem was related to the axisType not being declared but I should double
> check.
> Would this help?

Yes, please.

- When using a custom dimension like sigma, should it be specified in
the indexer.xml?

- If an indexer dimension for sigma is required, what form should it take?

- Does sigma need an entry in the granule schema in indexer.xml?

- How did you overcome sigma not having a recognised axisType?

Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Daniele Romagnoli-3


On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <[hidden email]> wrote:
On 01/06/16 00:26, Daniele Romagnoli wrote:
On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <[hidden email]>
wrote:
In a nutshell: should elevation variables that are *internal* to each
granule be indexed? Is this supported? If so, is indexer configuration
required? Why would the granule index need to know?
I would say yes. In the past we have worked with NetCDF containing sigma
levels, exposing them as sigma dimension, similarly to what you did.
If my memory serves me right, when having issues with custom dimensions the
problem was related to the axisType not being declared but I should double
check.
Would this help?

Yes, please.

- When using a custom dimension like sigma, should it be specified in the indexer.xml?
Yes. each domain/dimension involved in your mosaic should be declared.

- If an indexer dimension for sigma is required, what form should it take?

You should declare it in the <domains> section.
As an instance
<domains>
   <domain name="sigma">  <!-- See Note1 -->
      <attributes><attribute>sigma</attribute></attributes> <!-- See Note2 -->
   </domain>

Note1: you can put any name you want here. It will represent the dimension exposed in GeoServer.
Note2: the attribute here should match the real dimension name on the underlying NetCDF dataset.
 

- Does sigma need an entry in the granule schema in indexer.xml?
Yes.
Your <coverage> should refer that sigma domain as well as include a column in the schema
<coverage>
   <name>u-component_of_wind_sigma</name>
   <domains>
      ....  time?
      <domain ref="sigma" />
   </domains>
    <schema name="u-component_of_wind_sigma" >
        <attributes>.........., sigma: java.lang.Float</attributes> <!-- See Note 3 -->
    </schema>
 
Note 3: the attribute in the schema should match the attribute of the previously declared domain.

Note that imageMosaic has its own indexer. NetCDFs are multidimensional datasets so they also have a similar indexer to catalog NetCDF sections into a (by default) H2 hidden DB. That inner indexer is generated on the fly OR you can define it once (in case all NetCDF has same structure) and declare it in the imageMosaic indexer.xml
http://geoserver.geo-solutions.it/multidim/en/netcdf/netcdf_indexer.html#netcdf-indexer-file

How your NetCDF indexer file looks like?
Some months ago I have developed an experimental tool which may help in setting up the indexers:

java -jar CreateIndexer.jar /path/to/a/sample.nc -p  /path/to/GEOSERVER_DATA_DIR/user_projections/
netcdf.projections.properties

It will create both the ImageMosaic indexer.xml and the NetCDF indexer for a sample.nc file of a collection of NetCDF files with same structure. You can take a look on the produced results and customize them if needed.
As I said, that tool is still experimental and it can be surely improved or may throws errors on some untested datasets.


- How did you overcome sigma not having a recognised axisType?
If my memory serves me right the data producer fixed their coordinate variables but I'm not pretty sure about this since it was the past year. Let's see if the previous suggestions allows you to get them work. Otherwise, you may consider sharing a sample or do some debug on the low level readers to check if that's the case.

Cheers,
Daniele


Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand



--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

 

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Ben Caradoc-Davies-2
On 01/06/16 20:12, Daniele Romagnoli wrote:

> On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <[hidden email]>
>> - When using a custom dimension like sigma, should it be specified in the
>> indexer.xml?
> Yes. each domain/dimension involved in your mosaic should be declared.
>> - If an indexer dimension for sigma is required, what form should it take?
> You should declare it in the <domains> section.
> As an instance
> <domains>
>    <domain name="sigma">  <!-- See Note1 -->
>       <attributes><attribute>sigma</attribute></attributes> <!-- See Note2
> -->
>    </domain>
> Note1: you can put any name you want here. It will represent the dimension
> exposed in GeoServer.
> Note2: the attribute here should match the real dimension name on the
> underlying NetCDF dataset.
>> - Does sigma need an entry in the granule schema in indexer.xml?
> Yes.
> Your <coverage> should refer that sigma domain as well as include a column
> in the schema
> <coverage>
>    <name>u-component_of_wind_sigma</name>
>    <domains>
>       ....  time?
>       <domain ref="sigma" />
>    </domains>
>     <schema name="u-component_of_wind_sigma" >
>         <attributes>.........., sigma: java.lang.Float</attributes> <!--
> See Note 3 -->
>     </schema>
> Note 3: the attribute in the schema should match the attribute of the
> previously declared domain.

Thank you so much, Daniele. That works almost perfectly.

The one problem is that Float is not supported, only times, Integer,
Double, and String (and unsupported types like Float are treated as
String and fail because String cannot be a coordinate variable type):
https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/utilities/NetCDFUtilities.java#L852

If I add a sigma domain as you described above and a schema attribute
"sigma:Double", then enable both time and sigma in the Dimensions tab in
the web UI, the coverage is returned with both time and sigma dimensions
as expected. I also had to increase my heap to -Xmx2048m. I am now
looking at a 647MB NetCDF-4 file with 4 time values and 50 sigma layers
in Panoply.  :-)

Thanks also for your other suggestions. I have not yet tried them.

Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImageMosaic custom dimension internal to NetCDF files

Daniele Romagnoli-3


On Thu, Jun 2, 2016 at 3:28 AM, Ben Caradoc-Davies <[hidden email]> wrote:
On 01/06/16 20:12, Daniele Romagnoli wrote:
On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <[hidden email]>
- When using a custom dimension like sigma, should it be specified in the
indexer.xml?
Yes. each domain/dimension involved in your mosaic should be declared.
- If an indexer dimension for sigma is required, what form should it take?
You should declare it in the <domains> section.
As an instance
<domains>
   <domain name="sigma">  <!-- See Note1 -->
      <attributes><attribute>sigma</attribute></attributes> <!-- See Note2
-->
   </domain>
Note1: you can put any name you want here. It will represent the dimension
exposed in GeoServer.
Note2: the attribute here should match the real dimension name on the
underlying NetCDF dataset.
- Does sigma need an entry in the granule schema in indexer.xml?
Yes.
Your <coverage> should refer that sigma domain as well as include a column
in the schema
<coverage>
   <name>u-component_of_wind_sigma</name>
   <domains>
      ....  time?
      <domain ref="sigma" />
   </domains>
    <schema name="u-component_of_wind_sigma" >
        <attributes>.........., sigma: java.lang.Float</attributes> <!--
See Note 3 -->
    </schema>
Note 3: the attribute in the schema should match the attribute of the
previously declared domain.

Thank you so much, Daniele. That works almost perfectly.

The one problem is that Float is not supported, only times, Integer, Double, and String (and unsupported types like Float are treated as String and fail because String cannot be a coordinate variable type):
https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/utilities/NetCDFUtilities.java#L852

Mmm... I should already have fixed that several months ago. I'll double check it again.
 
If I add a sigma domain as you described above and a schema attribute "sigma:Double", then enable both time and sigma in the Dimensions tab in the web UI, the coverage is returned with both time and sigma dimensions as expected. I also had to increase my heap to -Xmx2048m. I am now looking at a 647MB NetCDF-4 file with 4 time values and 50 sigma layers in Panoply.  :-)

Good news :-)
Cheers,
Daniele
 
Thanks also for your other suggestions. I have not yet tried them.


Kind regards,

--
Ben Caradoc-Davies <[hidden email]>
Director
Transient Software Limited <http://transient.nz/>
New Zealand



--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

 

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users