Fwd: [Geoserver-users] netcdf in non latlon grid

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: [Geoserver-users] netcdf in non latlon grid

Richard Duivenvoorde
Hi,

Because no response on user list, I'm forwarding my (too long?) question
to here hoping to meet someone with deeper netcdf implementation knowledge.

tldr; Should it be able to server a netcdf with data with elevation AND
time in a normal/known epsg crs (non latlon, but like a national epsg).
I fail to load it in Geoserver. A 'simple' netcdf in epsg:xxxx without
time or elevation loads fine.

Regards,

Richard Duivenvoorde

PS Example data + python to create it here: http://duif.net/ncgeoserver.zip


-------- Forwarded Message --------
Subject: [Geoserver-users] netcdf in non latlon grid
Date: Wed, 9 Jan 2019 18:15:20 +0100
From: Richard Duivenvoorde <[hidden email]>
Reply-To: [hidden email]
To: GeoServer Mailing List List <[hidden email]>

Hi List,

I've succesfully created a simple minimal netcdf file (using Python)
with time-range-based data in a latlon grid:
like: float fakedata(time=3, z=3, lat=3, lon=4)

I can load the file (with several layers) both in QGIS (as mesh) and in
Geoserver (2.14.1 with netcdf extension) (with both time and elevation
dimensions): NICE!

BUT: we also want to project the data to our national grid: epsg:28992

If I understand docs correct, this should be possible (so: a known epsg,
no custom crs)?

Using gdal somebody provided me with a simple example in which instead
of lat lon the coordinates are x and y, and a CRS definition is defined:
"oblique_stereographic" and used as 'grid_mapping'.
But this is only 2D: (y=3, x=4)

With my python-netcdf-creation script I'm able to do that, and Geoserver
loads the data as one layer.
But trying to add time and/or z I fail:

Caused by: org.geotools.feature.IllegalAttributeException:Unknown
attribute time:null value:null
        at
org.geotools.feature.simple.SimpleFeatureImpl.setAttribute(SimpleFeatureImpl.java:238)
        at
org.geotools.imageio.netcdf.VariableAdapter.setFeatureTime(VariableAdapter.java:1327)
        at
org.geotools.imageio.netcdf.VariableAdapter.createFeature(VariableAdapter.java:1280)
        at
org.geotools.imageio.netcdf.VariableAdapter.getFeatures(VariableAdapter.java:1232)
        at
org.geotools.imageio.netcdf.NetCDFImageReader.initIndex(NetCDFImageReader.java:363)

or other silent issues.

I tried to google and look into the code to find the magic combination
of attributes/variables needed to be defined, but after hours of
fiddling, I tend to give up and ask for help :-)

So Question: should Geoserver be able to serve
fakedata(time=3, z=3, y=3, x=4)
with given crs, just as it is able to serve:
fakedata(time=3, z=3, lat=3, lon=4) ?

If anybody has a working example of such a grid I'm also happy, as that
proves that my feeling is right that it should be possible and I miss
something...

The python code and working latlon and not-working epsg28992 output
netcdf's are here http://duif.net/ncgeoserver.zip if somebody is willing
to have a look.

Regards,

Richard Duivenvoorde




_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to
this list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

geowolf
Richard,
if you get no answer on the user list, it means that nobody knows the answer, has time to answer, 
or has time to investigate. Please use the devel list only if you are participating to the actual development.

Thanks for your understanding.

Cheers
Andrea


On Fri, Jan 11, 2019 at 9:37 AM Richard Duivenvoorde <[hidden email]> wrote:
Hi,

Because no response on user list, I'm forwarding my (too long?) question
to here hoping to meet someone with deeper netcdf implementation knowledge.

tldr; Should it be able to server a netcdf with data with elevation AND
time in a normal/known epsg crs (non latlon, but like a national epsg).
I fail to load it in Geoserver. A 'simple' netcdf in epsg:xxxx without
time or elevation loads fine.

Regards,

Richard Duivenvoorde

PS Example data + python to create it here: http://duif.net/ncgeoserver.zip


-------- Forwarded Message --------
Subject: [Geoserver-users] netcdf in non latlon grid
Date: Wed, 9 Jan 2019 18:15:20 +0100
From: Richard Duivenvoorde <[hidden email]>
Reply-To: [hidden email]
To: GeoServer Mailing List List <[hidden email]>

Hi List,

I've succesfully created a simple minimal netcdf file (using Python)
with time-range-based data in a latlon grid:
like: float fakedata(time=3, z=3, lat=3, lon=4)

I can load the file (with several layers) both in QGIS (as mesh) and in
Geoserver (2.14.1 with netcdf extension) (with both time and elevation
dimensions): NICE!

BUT: we also want to project the data to our national grid: epsg:28992

If I understand docs correct, this should be possible (so: a known epsg,
no custom crs)?

Using gdal somebody provided me with a simple example in which instead
of lat lon the coordinates are x and y, and a CRS definition is defined:
"oblique_stereographic" and used as 'grid_mapping'.
But this is only 2D: (y=3, x=4)

With my python-netcdf-creation script I'm able to do that, and Geoserver
loads the data as one layer.
But trying to add time and/or z I fail:

Caused by: org.geotools.feature.IllegalAttributeException:Unknown
attribute time:null value:null
        at
org.geotools.feature.simple.SimpleFeatureImpl.setAttribute(SimpleFeatureImpl.java:238)
        at
org.geotools.imageio.netcdf.VariableAdapter.setFeatureTime(VariableAdapter.java:1327)
        at
org.geotools.imageio.netcdf.VariableAdapter.createFeature(VariableAdapter.java:1280)
        at
org.geotools.imageio.netcdf.VariableAdapter.getFeatures(VariableAdapter.java:1232)
        at
org.geotools.imageio.netcdf.NetCDFImageReader.initIndex(NetCDFImageReader.java:363)

or other silent issues.

I tried to google and look into the code to find the magic combination
of attributes/variables needed to be defined, but after hours of
fiddling, I tend to give up and ask for help :-)

So Question: should Geoserver be able to serve
fakedata(time=3, z=3, y=3, x=4)
with given crs, just as it is able to serve:
fakedata(time=3, z=3, lat=3, lon=4) ?

If anybody has a working example of such a grid I'm also happy, as that
proves that my feeling is right that it should be possible and I miss
something...

The python code and working latlon and not-working epsg28992 output
netcdf's are here http://duif.net/ncgeoserver.zip if somebody is willing
to have a look.

Regards,

Richard Duivenvoorde




_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to
this list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Richard Duivenvoorde
On 1/11/19 10:00 AM, Andrea Aime wrote:
> Richard,
> if you get no answer on the user list, it means that nobody knows the
> answer, has time to answer, 
> or has time to investigate. Please use the devel list only if you are
> participating to the actual development.

Ok, sorry.

It's just that I've invested some days already to get it going, and

https://docs.geoserver.org/stable/en/user/extensions/netcdf/netcdf.html#supporting-custom-netcdf-coordinate-reference-systems


gives me the feeling it should (and I just fail to call the magic
spells). I was hoping a dev with implementation knowledge could confirm
it SHOULD work. A 'NYI" would do too...

Regards,

Richard


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Daniele Romagnoli-3
Hi Richard.
Today I have downloaded GeoServer 2.14.1, together with NetCDF extension and I have configured your sample data.
I have simply created a NetCDF store, pointing to your test4dz28992.nc file.
I was able to configure the layer and related dimensions. I have setup a fake style so you can see the 4 different values as 4 different colors.

This is the result of the openLayers preview (getMap request with default values).
fakedata-default.jpg
Next one, is same getMap request with elevation=100.
fakedata-elevation100.jpg

So, I'm not sure why you are having issues in configuring it.
Could you retry from a clean situation?
Wondering if you successfully configured the dataset without t/z and then you replaced the file without reconfiguring it or something similar so that the configuration doesn't reflect the new data.
As a quick check, you may want to move the sample file to another location and reconfigure it, in order to make sure it isn't using any preconfigured/partially configured ancillary file.
Are you configuring it as a simple NetCDF store? (I mean... not an ImageMosaic)

Please, let us know.
Cheers,
Daniele





On Fri, Jan 11, 2019 at 10:17 AM Richard Duivenvoorde <[hidden email]> wrote:
On 1/11/19 10:00 AM, Andrea Aime wrote:
> Richard,
> if you get no answer on the user list, it means that nobody knows the
> answer, has time to answer, 
> or has time to investigate. Please use the devel list only if you are
> participating to the actual development.

Ok, sorry.

It's just that I've invested some days already to get it going, and

https://docs.geoserver.org/stable/en/user/extensions/netcdf/netcdf.html#supporting-custom-netcdf-coordinate-reference-systems


gives me the feeling it should (and I just fail to call the magic
spells). I was hoping a dev with implementation knowledge could confirm
it SHOULD work. A 'NYI" would do too...

Regards,

Richard


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


--
Regards,
Daniele Romagnoli
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272


-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Richard Duivenvoorde
On 1/14/19 12:23 PM, Daniele Romagnoli wrote:

> Hi Richard.
> Today I have downloaded GeoServer 2.14.1, together with NetCDF extension
> and I have configured your sample data.
> I have simply created a NetCDF store, pointing to your test4dz28992.nc
> <http://test4dz28992.nc> file.
> I was able to configure the layer and related dimensions. I have setup a
> fake style so you can see the 4 different values as 4 different colors.
>
> This is the result of the openLayers preview (getMap request with
> default values).
> fakedata-default.jpg
> Next one, is same getMap request with elevation=100.
> fakedata-elevation100.jpg
>
> So, I'm not sure why you are having issues in configuring it.
> Could you retry from a clean situation?
> Wondering if you successfully configured the dataset without t/z and
> then you replaced the file without reconfiguring it or something similar
> so that the configuration doesn't reflect the new data.
> As a quick check, you may want to move the sample file to another
> location and reconfigure it, in order to make sure it isn't using any
> preconfigured/partially configured ancillary file.
> Are you configuring it as a simple NetCDF store? (I mean... not an
> ImageMosaic)

Hi Daniele,

THANKS! For trying and testing, and letting me know it was working,
you've made my day ( see below :-) )!

I've already invested a lot of hours to try all kind of different
netcdf-configs creations with attributes/variables etc etc, and 2d was
working but 3d or 4d (that is with t/z) was NOT. I just was not sure if
my self created netcdf's were crippled, or I hit a geoserver problem.

I just downloaded a fresh 2.14.1 + netcdf extension, but had the same
error again...

BUT... because you are showing results, I figured out the netcdf is ok,
so it has to do with something else...
So I moved the test4dz28992.nc from my /home/richard/dev/python/netcdf
dir to the [geoserver]/datadir/coverages.. and THEN it works!!! Halleluya

So the crux is: while latlon-netcdfs work outside the datadir,
NON-latlon-netcdfs fail outside the datadir (though you can browse in
the store creation page to you home dir....).

Worth an issue I think?

Regards & thanks again, now I can proceed with my life \o/

Richard


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Daniele Romagnoli-3
Hi Richard
any chance that you are maybe using a different user running GeoServer which doesn't have write access to that /home/richard folder (and subfolders)?
The NetCDF format creates/writes a couple of ancillary files to index the ND variables stored on the nc file. There might be the case that it was unable to create them due to permissions issues.

Daniele

On Mon, Jan 14, 2019 at 1:11 PM Richard Duivenvoorde <[hidden email]> wrote:
On 1/14/19 12:23 PM, Daniele Romagnoli wrote:
> Hi Richard.
> Today I have downloaded GeoServer 2.14.1, together with NetCDF extension
> and I have configured your sample data.
> I have simply created a NetCDF store, pointing to your test4dz28992.nc
> <http://test4dz28992.nc> file.
> I was able to configure the layer and related dimensions. I have setup a
> fake style so you can see the 4 different values as 4 different colors.
>
> This is the result of the openLayers preview (getMap request with
> default values).
> fakedata-default.jpg
> Next one, is same getMap request with elevation=100.
> fakedata-elevation100.jpg
>
> So, I'm not sure why you are having issues in configuring it.
> Could you retry from a clean situation?
> Wondering if you successfully configured the dataset without t/z and
> then you replaced the file without reconfiguring it or something similar
> so that the configuration doesn't reflect the new data.
> As a quick check, you may want to move the sample file to another
> location and reconfigure it, in order to make sure it isn't using any
> preconfigured/partially configured ancillary file.
> Are you configuring it as a simple NetCDF store? (I mean... not an
> ImageMosaic)

Hi Daniele,

THANKS! For trying and testing, and letting me know it was working,
you've made my day ( see below :-) )!

I've already invested a lot of hours to try all kind of different
netcdf-configs creations with attributes/variables etc etc, and 2d was
working but 3d or 4d (that is with t/z) was NOT. I just was not sure if
my self created netcdf's were crippled, or I hit a geoserver problem.

I just downloaded a fresh 2.14.1 + netcdf extension, but had the same
error again...

BUT... because you are showing results, I figured out the netcdf is ok,
so it has to do with something else...
So I moved the test4dz28992.nc from my /home/richard/dev/python/netcdf
dir to the [geoserver]/datadir/coverages.. and THEN it works!!! Halleluya

So the crux is: while latlon-netcdfs work outside the datadir,
NON-latlon-netcdfs fail outside the datadir (though you can browse in
the store creation page to you home dir....).

Worth an issue I think?

Regards & thanks again, now I can proceed with my life \o/

Richard


--
Regards,
Daniele Romagnoli
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272


-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Richard Duivenvoorde
On 1/14/19 2:39 PM, Daniele Romagnoli wrote:
> Hi Richard
> any chance that you are maybe using a different user running GeoServer
> which doesn't have write access to that /home/richard folder (and
> subfolders)?
> The NetCDF format creates/writes a couple of ancillary files to index
> the ND variables stored on the nc file. There might be the case that it
> was unable to create them due to permissions issues.

Nope permissions should not be a problem, it's all done in my home dir,
both the writing/creation of the netcdf (via python), as running (as
user richard) the standalone Geoserver bin 2.14.1 etc

But did you test to put the nc file outside of the datadir? Like put it
in /tmp and 'browse' to it when you define the datastore?
With me that fails with a big red block of errors there (IF I do not
have the nc in the datadir/coverages)?

I could try to do this in a windows virt machine at home...

Regards,

Richard Duivenvoorde


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Daniele Romagnoli-3


On Mon, Jan 14, 2019 at 5:06 PM Richard Duivenvoorde <[hidden email]> wrote:
On 1/14/19 2:39 PM, Daniele Romagnoli wrote:
> Hi Richard
> any chance that you are maybe using a different user running GeoServer
> which doesn't have write access to that /home/richard folder (and
> subfolders)?
> The NetCDF format creates/writes a couple of ancillary files to index
> the ND variables stored on the nc file. There might be the case that it
> was unable to create them due to permissions issues.

Nope permissions should not be a problem, it's all done in my home dir,
both the writing/creation of the netcdf (via python), as running (as
user richard) the standalone Geoserver bin 2.14.1 etc

But did you test to put the nc file outside of the datadir? Like put it
in /tmp and 'browse' to it when you define the datastore?
With me that fails with a big red block of errors there (IF I do not
have the nc in the datadir/coverages)?
Mmm. I only did the test on my Windows workstation. I didn't test on Linux yet.
sorry...
 
Daniele

I could try to do this in a windows virt machine at home...

Regards,

Richard Duivenvoorde


--
Regards,
Daniele Romagnoli
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272


-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [Geoserver-users] netcdf in non latlon grid

Richard Duivenvoorde
On 1/14/19 5:08 PM, Daniele Romagnoli wrote:

> Mmm. I only did the test on my Windows workstation. I didn't test on
> Linux yet.

Just installed 2.14.1+netcdf on a pretty fresh Win10 machine, and
confirmed my earlier finding: outside of the data directory creation of
a 4D nonlatlon netcdf store just fails. When you put exact the same nc
file inside the data dir, geoserver succeeds to create and serve the data.

Created an issue for it:
https://osgeo-org.atlassian.net/projects/GEOS/issues/GEOS-9098

Regards,

Richard Duivenvoorde

ps I'm not able to start 2.15-M0 with the netcdf extension (GEOS-9099)


_______________________________________________
Geoserver-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel