Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 Not sure if this is the right ticket. Attached a GeoJSON geometry covering
 most of the world (digitized in a Web GIS in Web Mercator in a lousy way
 which is reflecting reality and reprojected to Latlong with ogr2ogr).

 It takes "forever" on the "Breaking boundaries..." step while the geometry
 is a single polygon.
 I wonder why?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:24>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------
Changes (by neteler):

 * Attachment "world_AOI_latlong.geojson" added.

 Single polygon geometry in Latlong

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by mmetz):

 Replying to [comment:24 neteler]:
 > Not sure if this is the right ticket. Attached a GeoJSON geometry
 covering most of the world (digitized in a Web GIS in Web Mercator in a
 lousy way which is reflecting reality and reprojected to Latlong with
 ogr2ogr).
 >
 > It takes "forever" on the "Breaking boundaries..." step while the
 geometry is a single polygon.
 > I wonder why?

 With trunk, import happens instantly (0.08 seconds). What GRASS version
 are you using? Do you have changes in your local copy?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:25>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 Since it is a production system, I'm on 7.2.svn

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:26>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by mmetz):

 Replying to [comment:26 neteler]:
 > Since it is a production system, I'm on 7.2.svn

 With 7.2, I get the same instant (0.08 seconds) import with

 {{{
 v.in.ogr in=world_AOI_latlong.geojson out=world_AOI_latlong
 }}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:27>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 Strange, here it is not fast at all:


 {{{
 GRASS 7.2.1svn (latlong_wgs84):~ > time -p v.in.ogr
 in=world_AOI_latlong.geojson out=world_AOI_latlong
 Check if OGR layer <OGRGeoJSON> contains polygons...
  100%
 WARNING: Width for column properties set to 255 (was not specified by
 OGR),
          some strings may be truncated!
 Importing 1 features (OGR layer <OGRGeoJSON>)...
  100%
 -----------------------------------------------------
 Registering primitives...
 One primitive registered
 7 vertices registered
 Number of nodes: 1
 Number of primitives: 1
 Number of points: 0
 Number of lines: 0
 Number of boundaries: 1
 Number of centroids: 0
 Number of areas: -
 Number of isles: -
 -----------------------------------------------------
 Cleaning polygons
 -----------------------------------------------------
 Breaking polygons...
 Breaking polygons (pass 1: select break points)...
  100%
 Breaking polygons (pass 2: break at selected points)...
  100%
 -----------------------------------------------------
 Removing duplicates...
  100%
 -----------------------------------------------------
 Breaking boundaries...
 98% ^C
 real 373.04
 user 371.13
 sys 0.97

 GRASS 7.2.1svn (latlong_wgs84):~ > g.version -g
 version=7.2.1svn
 date=2017
 revision=r70769
 build_date=2017-03-20
 build_platform=x86_64-pc-linux-gnu
 build_off_t_size=8

 GRASS 7.2.1svn (latlong_wgs84):~ > cs2cs
 Rel. 4.9.2, 08 September 2015
 usage: cs2cs [ -eEfIlrstvwW [args] ] [ +opts[=arg] ]
                    [+to [+opts[=arg] [ files ]

 GRASS 7.2.1svn (latlong_wgs84):~ > gdal-config --version
 2.1.2

 db.connect -p
 driver: sqlite
 database: /home/mundialis/grassdata/latlong_wgs84/user1/sqlite/sqlite.db
 ...
 }}}

 "svn status" does not show a single local modification.

 The disk is a SSD drive. The system is a Fedora 25 installation:

 {{{
 rpm -qa | sort | grep 'geos\|gdal\|proj'
 gdal-2.1.2-5.fc25.x86_64
 gdal-devel-2.1.2-5.fc25.x86_64
 gdal-grass-2.1.2-1.fc25.x86_64
 gdal-libs-2.1.2-5.fc25.x86_64
 gdal-python-2.1.2-5.fc25.x86_64
 geos-3.5.0-3.fc25.x86_64
 geos-devel-3.5.0-3.fc25.x86_64
 proj-4.9.2-2.fc24.x86_64
 proj-devel-4.9.2-2.fc24.x86_64
 proj-epsg-4.9.2-2.fc24.x86_64
 proj-nad-4.9.2-2.fc24.x86_64
 pyproj-1.9.5.1-3.fc25.x86_64
 }}}

 strace shows a lot of lseek() if that matters:

 {{{
 ...
 lseek(6, 814250, SEEK_SET)              = 814250
 lseek(6, 814250, SEEK_SET)              = 814250
 lseek(6, 814250, SEEK_SET)              = 814250
 lseek(6, 814250, SEEK_SET)              = 814250
 lseek(6, -138, SEEK_CUR)                = 814112
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814250, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3242) = 3242
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=814351, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3343) = 3343
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3205) = 3205
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 4096) = 175
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, 814388, SEEK_SET)              = 814388
 lseek(6, -138, SEEK_CUR)                = 814250
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814388, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3380) = 3380
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=814489, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3481) = 3481
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3343) = 3343
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 4096) = 175
 lseek(6, 814526, SEEK_SET)              = 814526
 lseek(6, 814526, SEEK_SET)              = 814526
 lseek(6, 814526, SEEK_SET)              = 814526
 lseek(6, 814526, SEEK_SET)              = 814526
 lseek(6, -138, SEEK_CUR)                = 814388
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814526, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3518) = 3518
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=814627, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3619) = 3619
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3481) = 3481
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 4096) = 175
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, 814664, SEEK_SET)              = 814664
 lseek(6, -138, SEEK_CUR)                = 814526
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814664, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3656) = 3656
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=814765, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3757) = 3757
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3619) = 3619
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 4096) = 175
 lseek(6, 814802, SEEK_SET)              = 814802
 lseek(6, 814802, SEEK_SET)              = 814802
 lseek(6, 814802, SEEK_SET)              = 814802
 lseek(6, 814802, SEEK_SET)              = 814802
 lseek(6, -138, SEEK_CUR)                = 814664
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814802, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3794) = 3794
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=814903, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3895) = 3895
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3757) = 3757
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 4096) = 175
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, 814940, SEEK_SET)              = 814940
 lseek(6, -138, SEEK_CUR)                = 814802
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=814940, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3932) = 3932
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=815041, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 4033) = 4033
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 3895) = 3895
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 4096) = 175
 lseek(6, 815078, SEEK_SET)              = 815078
 lseek(6, 815078, SEEK_SET)              = 815078
 lseek(6, 815078, SEEK_SET)              = 815078
 lseek(6, 815078, SEEK_SET)              = 815078
 lseek(6, -138, SEEK_CUR)                = 814940
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=815078, ...}) = 0
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 4070) = 4070
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1",
 26) = 26
 write(6,
 "ve@\357B\276I\231$e@\305\240g\372\204\344e\300\246n\300\212l]e\300\23_\363\0\254"...,
 75) = 75
 fstat(6, {st_mode=S_IFREG|0664, st_size=815179, ...}) = 0
 lseek(6, 815104, SEEK_SET)              = 815104
 read(6,
 "ve@\357B\276I\231$e@\305\240g\372\204\344e\300\246n\300\212l]e\300\23_\363\0\254"...,
 75) = 75
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 37) = 37
 lseek(6, 811008, SEEK_SET)              = 811008
 read(6,
 "l\fK\300&U@\304p\327\354y1U\300L\205\334i\322\27U\300\23_\363\0\254\350T@\r"...,
 4033) = 4033
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 4096) = 175
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, 815216, SEEK_SET)              = 815216
 lseek(6, -138, SEEK_CUR)                = 815078
 write(6, "\f", 1)                       = 1
 fstat(6, {st_mode=S_IFREG|0664, st_size=815216, ...}) = 0
 lseek(6, 815104, SEEK_SET)              = 815104
 read(6,
 "ve@\357B\276I\231$e@\305\240g\372\204\344e\300\246n\300\212l]e\300\23_\363\0\254"...,
 112) = 112
 write(6,
 "\r\6\0\0\0\246n\300\212l]e\300\305\240g\372\204\344e\300d\216\25\215\1ve@\357B\276"...,
 101) = 101
 fstat(6, {st_mode=S_IFREG|0664, st_size=815317, ...}) = 0
 lseek(6, 815104, SEEK_SET)              = 815104
 read(6,
 "ve@\357B\276I\231$e@\305\240g\372\204\344e\300\246n\300\212l]e\300\23_\363\0\254"...,
 213) = 213
 write(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\24_\363\0\254\350T@\23_\363"...,
 37) = 37
 lseek(6, 815104, SEEK_SET)              = 815104
 read(6,
 "ve@\357B\276I\231$e@\305\240g\372\204\344e\300\246n\300\212l]e\300\23_\363\0\254"...,
 75) = 75
 read(6,
 "\r\2\0\0\0\246n\300\212l]e\300\246n\300\212l]e\300\23_\363\0\254\350T@\24_\363"...,
 4096) = 175
 ...
 }}}

 I'm a bit clueless... do you have uncommitted improvements? :-)

 Can anyone else please test this tiny GeoJSON in a latlong location?
 thanks.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:28>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by hcho):

 Markus, I'm having exactly the same issue. Takes forever at that stage...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:29>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by pvanbosgeo):

 Takes a long time on my computer as well, gets stuck at breaking
 boundaries. I am running 7.3.svn r70733 on Ubuntu 16.04. I don't normally
 have problems with importing much larger shapefiles.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:30>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------
Changes (by hcho):

 * Attachment "intersect.c.patch" added.

 lib/vector/Vlib/intersect.c.patch

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by hcho):

 Can you try this patch? The issue was that Vect_line_intersection would
 create breaks on first and last line vertices because there was no
 tolerance for comparing points. There are more such comparisons (potential
 issue?)...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:31>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by hcho):

 Not sure if this patch would address the original issue though.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:32>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 In addition, here my compiler settings, maybe that makes a difference?

 {{{
 sh -x conf_grass7.sh
 + INTEL='-march=native -std=gnu99 -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
 -fstack-protector -m64'
 + MYGCC=-fdiagnostics-color
 + MYCFLAGS='-Wall -fopenmp -lgomp -Ofast -fno-fast-math -march=core-avx-i
 -fno-common -fexceptions -march=native -std=gnu99 -Wp,-D_FORTIFY_SOURCE=2
 -fexceptions -fstack-protector -m64 MYGCC'
 + MYLDFLAGS='-Wl,--no-undefined -fopenmp -lgomp'
 + MYCXXFLAGS=-Ofast
 + CC=clang
 + CXX=clang++
 + CFLAGS=-O2
 + CXXFLAGS=-O2
 + LDFLAGS=-s
 + ./configure --with-cxx --enable-largefile --with-proj --with-proj-
 share=/usr/share/proj --with-gdal=/usr/bin/gdal-config --with-python
 --with-geos --with-liblas --with-sqlite --with-nls --with-blas --with-
 blas-includes=/usr/include/atlas-x86_64-base/ --with-lapack --with-lapack-
 includes=/usr/include/atlas-x86_64-base/ --with-cairo --with-cairo-
 ldflags=-lfontconfig --with-freetype --with-freetype-
 includes=/usr/include/freetype2 --with-wxwidgets=/usr/bin/wx-config
 --with-fftw --with-motif --with-postgres --with-netcdf --without-mysql
 --without-odbc --without-openmp --without-ffmpeg
 + tee config_log.txt
 checking host system type... x86_64-pc-linux-gnu
 checking for gcc... clang
 checking whether the C compiler (clang -O2 -s) works... yes
 checking whether the C compiler (clang -O2 -s) is a cross-compiler... no
 checking whether we are using GNU C... yes
 ...

 # CPU:
 cat /proc/cpuinfo | grep 'model name'
 model name      : Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
 ...
 }}}

 I'll switch to gcc and test again.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:33>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 Wow. After full recompilation, with gcc it does "hang":

 {{{
 ## gcc production
 INTEL="-march=native"
 MYGCC="-fdiagnostics-color"
 MYCFLAGS="-O2 $INTEL $MYGCC"
 MYCXXFLAGS="-O2"
 MYLDFLAGS="-Wl,--no-undefined"
 MYLDFLAGS="-s $MYLDFLAGS"

 # clang
 #CC=clang CXX=clang++ CFLAGS="-O2" CXXFLAGS="-O2" LDFLAGS="-s" ./configure
 \

 # gcc
 LDFLAGS="$MYLDFLAGS" CFLAGS="$MYCFLAGS" CXXFLAGS="$MYCXXFLAGS" ./configure
 \
   --with-cxx \
   --enable-largefile \
 ...
 }}}

 Now:

 {{{
 GRASS 7.2.1svn (latlong_wgs84):~ > time -p v.in.ogr
 in=world_AOI_latlong.geojson out=world_AOI_latlong --o
 ...
 -----------------------------------------------------
 1 input polygons
 Total area: 2.37675E+13 (2 areas)
 Area without category: 2.04916E+11 (1 areas)
 -----------------------------------------------------
 Copying features...
  100%
 Building topology for vector map <world_AOI_latlong@user1>...
 Registering primitives...
 3 primitives registered
 19 vertices registered
 Building areas...
  100%
 2 areas built
 One isle built
 Attaching islands...
  100%
 Attaching centroids...
  100%
 Number of nodes: 1
 Number of primitives: 3
 Number of points: 0
 Number of lines: 0
 Number of boundaries: 2
 Number of centroids: 1
 Number of areas: 2
 Number of isles: 1
 real 0.09
 user 0.04
 sys 0.01
 }}}

 So, unfortunate clang compiler settings before or gcc-is-better-than-clang
 here?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:34>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

Markus Neteler
On Tue, Mar 21, 2017 at 12:24 PM, GRASS GIS <[hidden email]> wrote:
> #2185: Painfully Slow 'v.in.ogr' Vector Import
...
> Comment (by neteler):
>
>  Wow. After full recompilation, with gcc it does "hang":

typo argh! (ticket already edited)

it does _NOT_ hang with gcc!

More in the ticket.
_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by hcho):

 Markus, you mean it does not hang with or without my patch?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:35>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by mmetz):

 Replying to [comment:34 neteler]:
 > Wow. After full recompilation, with gcc it does  NOT "hang":
 [...]
 >
 > So, unfortunate clang compiler settings before or gcc-is-better-than-
 clang here?

 No, the difference is -O2 which you used with gcc but not with clang. I
 could reproduce the problem by not using any optimization.

 The problem is that Vect_break_lines() enters an infinite loop because it
 continuously generates two new lines that are again intersecting each
 other, always at the same locations.

 The attached patch ([comment:31 hcho]) helps, but as mentioned in
 comment:31 there are more comparisons and it does not seem to be a good
 idea to allow tolerance at some places but not at other places when
 comparing a cross with a vertex.

 What also helps is using Vect_line_intersection2() instead of
 Vect_line_intersection(). Interestingly, Vect_line_intersection2() does
 not need the patch, even though that part of the code is identical.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:36>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by hcho):

 Yes, that's what I found too. Vect_line_intersection2 doesn't have this
 issue, but it still creates a single point intersection at the first
 vertex of the second new line (line ID 3) in the geojson example.

 Line ID 1: original unbroken line

 1st iteration
 Line ID 2-4: new broken lines

 2nd iteration
 Line ID 5: identical to line 3
 Line ID 6: start node of line 3

 Lines 5 & 6 shouldn't be returned at all from the intersection routine, I
 think. The patch fixes this.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:37>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------
Changes (by mmetz):

 * Attachment "intersect.c.patch2" added.

 patch for Vect_segment_intersection()

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by mmetz):

 Replying to [comment:37 hcho]:
 > Yes, that's what I found too. Vect_line_intersection2 doesn't have this
 issue, but it still creates a single point intersection at the first
 vertex of the second new line (line ID 3) in the geojson example.
 >
 > Line ID 1: original unbroken line
 >
 > 1st iteration
 > Line ID 2-4: new broken lines
 >
 > 2nd iteration
 > Line ID 5: identical to line 3
 > Line ID 6: start node of line 3
 >
 > Lines 5 & 6 shouldn't be returned at all from the intersection routine,
 I think. The patch fixes this.

 I found another issue in Vect_segment_intersection(): the order of the
 segments matters, i.e. the intersection point of a with b can be slightly
 different from the intersection point of b with a. The second attached
 patch fixes that and also avoids that infinite loop. I opt to apply both
 patches.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:38>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GRASS GIS
In reply to this post by GRASS GIS
#2185: Painfully Slow 'v.in.ogr' Vector Import
-------------------------+------------------------------------------------
  Reporter:  justinzane  |      Owner:  grass-dev@…
      Type:  defect      |     Status:  new
  Priority:  normal      |  Milestone:  7.0.6
 Component:  Vector      |    Version:  svn-trunk
Resolution:              |   Keywords:  import, OGR, performance, v.in.ogr
       CPU:  x86-64      |   Platform:  Linux
-------------------------+------------------------------------------------

Comment (by neteler):

 Replying to [comment:35 hcho]:
 > Markus, you mean it does not hang with or without my patch?

 Due to lack of time I tried without your patch so far.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:39>
GRASS GIS <https://grass.osgeo.org>


_______________________________________________
grass-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-dev
12