Encoding error on upload zipped shapefile on centos 7

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Encoding error on upload zipped shapefile on centos 7

xbartolone
All,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168

It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea, how to overcome it, is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work
_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users
Reply | Threaded
Open this post in threaded view
|

Re: Encoding error on upload zipped shapefile on centos 7

mentaer

perhaps not helpful, but would it not be easier to load the shape in QGIS and save it as a new dataset with another encoding there?

..just wondering,

stefan


On 07/12/2017 04:34 PM, Francesco Bartoli wrote:
All,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168

It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea, how to overcome it, is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work

_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users


_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users
Reply | Threaded
Open this post in threaded view
|

Re: Encoding error on upload zipped shapefile on centos 7

xbartolone
The problem isn't the shapefile itself, it rather means any archive with non ascii character would break the upload.

Sent from Nylas Mail, the best free email app for work

On lug 12 2017, at 10:53 pm, Stefan Steiniger <[hidden email]> wrote:

perhaps not helpful, but would it not be easier to load the shape in QGIS and save it as a new dataset with another encoding there?

..just wondering,

stefan


On 07/12/2017 04:34 PM, Francesco Bartoli wrote:
All,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168

It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea, how to overcome it, is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work

_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users

_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users

_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users
Reply | Threaded
Open this post in threaded view
|

Re: Encoding error on upload zipped shapefile on centos 7

xbartolone
In reply to this post by xbartolone
Just for the record, the error described below is from a false cause. Despite the message from the traceback which confused me the problem was due to a lack of communication with postgresql. The missing configuration was the md5 authentication rather than ident

host    all             all             127.0.0.1/32           md5

has been added to the PR.

Hope this helps somebody for the future.

Francesco

Il giorno 12/lug/2017, alle ore 22:34, francesco bartoli <[hidden email]> ha scritto:

All,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168

It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea, how to overcome it, is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work


_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users
Reply | Threaded
Open this post in threaded view
|

Re: Encoding error on upload zipped shapefile on centos 7

Alessio Fabiani-2
+1

Regards,

Alessio Fabiani

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead


GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:     +39 0584 1660272
mob:   +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Mon, Jul 17, 2017 at 10:16 AM, Francesco Bartoli <[hidden email]> wrote:
Just for the record, the error described below is from a false cause. Despite the message from the traceback which confused me the problem was due to a lack of communication with postgresql. The missing configuration was the md5 authentication rather than ident

host    all             all             127.0.0.1/32           md5

has been added to the PR.

Hope this helps somebody for the future.

Francesco

Il giorno 12/lug/2017, alle ore 22:34, francesco bartoli <[hidden email]> ha scritto:

All,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168

It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea, how to overcome it, is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work


_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users



_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users