Encoding error on upload zipped shapefile on centos 7

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Encoding error on upload zipped shapefile on centos 7

xbartolone
Hi devs,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168.
It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea how to represent it to our users is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work

_______________________________________________
geonode-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-devel
Reply | Threaded
Open this post in threaded view
|

Re: Encoding error on upload zipped shapefile on centos 7

xbartolone
Following my previous message I have opened the ticket below in the Django documentation:


Sent from Nylas Mail, the best free email app for work

On lug 11 2017, at 11:48 pm, Francesco Bartoli <[hidden email]> wrote:
Hi devs,

I'm facing a common error while uploading an archived shapefile. The file is the commonly used ne_50m_admin_0_countries.
The error is well known also in django documentation and due to non ascii characters:


Unfortunately the recommended solution from the django docs for deployments with apache and mod_wsgi 


doesn't solve the issue in centos 7. The box has been created while reviewing the installation on that environment as per this PR https://github.com/GeoNode/geonode/pull/3168.
It can be easily reproduced by adding this view and route within geonode app:

#my_views.py
import locale
import sys
from django.http import HttpResponse

def view_locale(request):
   loc_info = "getlocale: " + str(locale.getlocale()) + \
       "<br/>getdefaultlocale(): " + str(locale.getdefaultlocale()) + \
       "<br/>fs_encoding: " + str(sys.getfilesystemencoding()) + \
       "<br/>sys default encoding: " + str(sys.getdefaultencoding())
   return HttpResponse(loc_info)

#urls.py
...
url(r'^locale/$', 'geonode.my_views.view_locale'),
...

Result is always this:
getlocale: (None, None)
getdefaultlocale(): ('en_US', 'UTF-8')
fs_encoding: UTF-8
sys default encoding: ascii

Setting the following variables in /etc/sysconfig/httpd and restart as suggested

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"

does not change anything in particular the system default encoding which is the reason why the upload is failing.

Adding such variables to WSGIDaemonProcess:

WSGIDaemonProcess geonode lang='en_US.UTF-8' locale='en_US.UTF-8' python-path=/home/geonode/geonode:/home/geonode/.venvs/geonode/lib/python2.7/site-packages user=apache threads=15 processes=2

just affects the getlocale() function while default encoding is unaltered. The only way to affect it would be to add and enforce it:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

but this is strongly not recommended and discouraged. More here https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

Did anyone already encounter this kind of error? Actually I don't have a solution but for sure it's not a bug in geonode code base despite I did not figure out the real source.

Any help and idea how to represent it to our users is welcome :)

thx
Francesco

Sent from Nylas Mail, the best free email app for work

_______________________________________________
geonode-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-devel