Error saving/exporting data with many attributes

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Error saving/exporting data with many attributes

b.j.kobben

Hi all,

 

I have a problem that (I think) comes down to a limit in the amount of attributes per object the QGIS exporting tools can handle.  Case in hand:

 

1) Import OSM data Points with many attributes (OSM tags), using QuickOSM – works

 

2) Temporary layer in QGIS is displayed -  seems fine

 

3) "Make Permanent" or "Export As..." fails on all file formats (except GeoJSON!). Error message is (for the GeoPackage case):

 

Export to vector file failed.

Error: Feature write errors:

Feature creation error (OGR error: sqlite3_exec(CREATE TABLE "test" ( "fid" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "geom" POINT, "full_id" TEXT(255), "osm_id" TEXT(255), "osm_type" TEXT(255), "highway" TEXT(255),

 

... many more attributes...

 

"safe_place:school" TEXT(255), "safe_place:mosque" TEXT(255))) failed: duplicate column name: Operational_status)

 

Feature creation error (OGR error: failed to prepare SQL: INSERT INTO "test" ( "geom", "full_id", "osm_id", "osm_type", "highway", "name",

 

... again the same many more attributes...

 

"safe_place:school", "safe_place:mosque") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))

 

This happens even when I have only 2 features (see attached test file). The workaraound is to use GeoJSON, bacuse for some reason that one works, but in normal case I have 1000s of points instead of two, and then a GeoJSON file becomes very large and unwieldy...

 

Any ideas if this is a bug or a limit in the actual formats such as GeoPackage...?

yours,

 

--

Barend Köbben

Senior Lecturer – ITC-University of Twente & ATLAS

PO Box 217, 7500 AE Enschede (Netherlands)

ITC Building (room 1-065)

@barendkobben

+31-(0)53 4874 253

 

 

 

 

 

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

test_small.geojson (22K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Error saving/exporting data with many attributes

b.j.kobben

To answer my own question: the problem is actually not the amount of attributes, but the fact that there are (2x) attributes columns with the same name (although they are different if you consider case sensitivity). Apparently GeoJSON is the only format that allows differentiating column names by upper/lowercase....

 

yours,

 

--

Barend Köbben

 

 

On 11/03/2019, 11:55, "Qgis-user on behalf of [hidden email]" <[hidden email] on behalf of [hidden email]> wrote:

 

Hi all,

 

I have a problem that (I think) comes down to a limit in the amount of attributes per object the QGIS exporting tools can handle.  Case in hand:

 

1) Import OSM data Points with many attributes (OSM tags), using QuickOSM – works

 

2) Temporary layer in QGIS is displayed -  seems fine

 

3) "Make Permanent" or "Export As..." fails on all file formats (except GeoJSON!). Error message is (for the GeoPackage case):

 

Export to vector file failed.

Error: Feature write errors:

Feature creation error (OGR error: sqlite3_exec(CREATE TABLE "test" ( "fid" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "geom" POINT, "full_id" TEXT(255), "osm_id" TEXT(255), "osm_type" TEXT(255), "highway" TEXT(255),

 

... many more attributes...

 

"safe_place:school" TEXT(255), "safe_place:mosque" TEXT(255))) failed: duplicate column name: Operational_status)

 

Feature creation error (OGR error: failed to prepare SQL: INSERT INTO "test" ( "geom", "full_id", "osm_id", "osm_type", "highway", "name",

 

... again the same many more attributes...

 

"safe_place:school", "safe_place:mosque") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))

 

This happens even when I have only 2 features (see attached test file). The workaraound is to use GeoJSON, bacuse for some reason that one works, but in normal case I have 1000s of points instead of two, and then a GeoJSON file becomes very large and unwieldy...

 

Any ideas if this is a bug or a limit in the actual formats such as GeoPackage...?

yours,

 

--

Barend Köbben

Senior Lecturer – ITC-University of Twente & ATLAS

PO Box 217, 7500 AE Enschede (Netherlands)

ITC Building (room 1-065)

@barendkobben

+31-(0)53 4874 253

 

 

 

 

 

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Error saving/exporting data with many attributes

Patrick Dunford-2

Case sensitive names (from experience) are handled inconsistently within the Qgis software. It would be unwise to have two different columns where the names are only distinguished by case, unless the developers choose to address these issues in future.

On 12/03/19 12:20 AM, [hidden email] wrote:

To answer my own question: the problem is actually not the amount of attributes, but the fact that there are (2x) attributes columns with the same name (although they are different if you consider case sensitivity). Apparently GeoJSON is the only format that allows differentiating column names by upper/lowercase....

 

yours,

 

--

Barend Köbben

 

 

On 11/03/2019, 11:55, "Qgis-user on behalf of [hidden email]" <[hidden email] on behalf of [hidden email]> wrote:

 

Hi all,

 

I have a problem that (I think) comes down to a limit in the amount of attributes per object the QGIS exporting tools can handle.  Case in hand:

 

1) Import OSM data Points with many attributes (OSM tags), using QuickOSM – works

 

2) Temporary layer in QGIS is displayed -  seems fine

 

3) "Make Permanent" or "Export As..." fails on all file formats (except GeoJSON!). Error message is (for the GeoPackage case):

 

Export to vector file failed.

Error: Feature write errors:

Feature creation error (OGR error: sqlite3_exec(CREATE TABLE "test" ( "fid" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "geom" POINT, "full_id" TEXT(255), "osm_id" TEXT(255), "osm_type" TEXT(255), "highway" TEXT(255),

 

... many more attributes...

 

"safe_place:school" TEXT(255), "safe_place:mosque" TEXT(255))) failed: duplicate column name: Operational_status)

 

Feature creation error (OGR error: failed to prepare SQL: INSERT INTO "test" ( "geom", "full_id", "osm_id", "osm_type", "highway", "name",

 

... again the same many more attributes...

 

"safe_place:school", "safe_place:mosque") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))

 

This happens even when I have only 2 features (see attached test file). The workaraound is to use GeoJSON, bacuse for some reason that one works, but in normal case I have 1000s of points instead of two, and then a GeoJSON file becomes very large and unwieldy...

 

Any ideas if this is a bug or a limit in the actual formats such as GeoPackage...?

yours,

 

--

Barend Köbben

Senior Lecturer – ITC-University of Twente & ATLAS

PO Box 217, 7500 AE Enschede (Netherlands)

ITC Building (room 1-065)

@barendkobben

+31-(0)53 4874 253

 

 

 

 

 

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Error saving/exporting data with many attributes

b.j.kobben

Thanks for the info Patrick. I totally agree this is unwise, but the OSM (OpenStreetMap) dataset I am working with unfortunately comes with some of these (unwisely chosen) attribute tags :-(

So indeed inconsistency in how that 's handled is annoying. In defence of QGIS programmers, I think it's because the different import/export formats have different rules and defaults...

 

yours,

--

Barend Köbben

 

 

On 12/03/2019, 11:45, "Qgis-user on behalf of Patrick Dunford" <[hidden email] on behalf of [hidden email]> wrote:

 

Case sensitive names (from experience) are handled inconsistently within the Qgis software. It would be unwise to have two different columns where the names are only distinguished by case, unless the developers choose to address these issues in future.

On 12/03/19 12:20 AM, [hidden email] wrote:

To answer my own question: the problem is actually not the amount of attributes, but the fact that there are (2x) attributes columns with the same name (although they are different if you consider case sensitivity). Apparently GeoJSON is the only format that allows differentiating column names by upper/lowercase....

 

yours,

 

--

Barend Köbben

 

 

On 11/03/2019, 11:55, "Qgis-user on behalf of [hidden email]" <[hidden email] on behalf of [hidden email]> wrote:

 

Hi all,

 

I have a problem that (I think) comes down to a limit in the amount of attributes per object the QGIS exporting tools can handle.  Case in hand:

 

1) Import OSM data Points with many attributes (OSM tags), using QuickOSM – works

 

2) Temporary layer in QGIS is displayed -  seems fine

 

3) "Make Permanent" or "Export As..." fails on all file formats (except GeoJSON!). Error message is (for the GeoPackage case):

 

Export to vector file failed.

Error: Feature write errors:

Feature creation error (OGR error: sqlite3_exec(CREATE TABLE "test" ( "fid" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "geom" POINT, "full_id" TEXT(255), "osm_id" TEXT(255), "osm_type" TEXT(255), "highway" TEXT(255),

 

... many more attributes...

 

"safe_place:school" TEXT(255), "safe_place:mosque" TEXT(255))) failed: duplicate column name: Operational_status)

 

Feature creation error (OGR error: failed to prepare SQL: INSERT INTO "test" ( "geom", "full_id", "osm_id", "osm_type", "highway", "name",

 

... again the same many more attributes...

 

"safe_place:school", "safe_place:mosque") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))

 

This happens even when I have only 2 features (see attached test file). The workaraound is to use GeoJSON, bacuse for some reason that one works, but in normal case I have 1000s of points instead of two, and then a GeoJSON file becomes very large and unwieldy...

 

Any ideas if this is a bug or a limit in the actual formats such as GeoPackage...?

yours,

 

--

Barend Köbben

Senior Lecturer – ITC-University of Twente & ATLAS

PO Box 217, 7500 AE Enschede (Netherlands)

ITC Building (room 1-065)

@barendkobben

+31-(0)53 4874 253

 

 

 

 

 

 



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Error saving/exporting data with many attributes

Stefan Keller
Can't you delete such attributes with same (lower case) name - and
their contents - before making the layer permanent / exporting it?

:Stefan

Am Di., 12. März 2019 um 16:36 Uhr schrieb <[hidden email]>:

>
> Thanks for the info Patrick. I totally agree this is unwise, but the OSM (OpenStreetMap) dataset I am working with unfortunately comes with some of these (unwisely chosen) attribute tags :-(
>
> So indeed inconsistency in how that 's handled is annoying. In defence of QGIS programmers, I think it's because the different import/export formats have different rules and defaults...
>
>
>
> yours,
>
> --
>
> Barend Köbben
>
>
>
>
>
> On 12/03/2019, 11:45, "Qgis-user on behalf of Patrick Dunford" <[hidden email] on behalf of [hidden email]> wrote:
>
>
>
> Case sensitive names (from experience) are handled inconsistently within the Qgis software. It would be unwise to have two different columns where the names are only distinguished by case, unless the developers choose to address these issues in future.
>
> On 12/03/19 12:20 AM, [hidden email] wrote:
>
> To answer my own question: the problem is actually not the amount of attributes, but the fact that there are (2x) attributes columns with the same name (although they are different if you consider case sensitivity). Apparently GeoJSON is the only format that allows differentiating column names by upper/lowercase....
>
>
>
> yours,
>
>
>
> --
>
> Barend Köbben
>
>
>
>
>
> On 11/03/2019, 11:55, "Qgis-user on behalf of [hidden email]" <[hidden email] on behalf of [hidden email]> wrote:
>
>
>
> Hi all,
>
>
>
> I have a problem that (I think) comes down to a limit in the amount of attributes per object the QGIS exporting tools can handle.  Case in hand:
>
>
>
> 1) Import OSM data Points with many attributes (OSM tags), using QuickOSM – works
>
>
>
> 2) Temporary layer in QGIS is displayed -  seems fine
>
>
>
> 3) "Make Permanent" or "Export As..." fails on all file formats (except GeoJSON!). Error message is (for the GeoPackage case):
>
>
>
> Export to vector file failed.
>
> Error: Feature write errors:
>
> Feature creation error (OGR error: sqlite3_exec(CREATE TABLE "test" ( "fid" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "geom" POINT, "full_id" TEXT(255), "osm_id" TEXT(255), "osm_type" TEXT(255), "highway" TEXT(255),
>
>
>
> ... many more attributes...
>
>
>
> "safe_place:school" TEXT(255), "safe_place:mosque" TEXT(255))) failed: duplicate column name: Operational_status)
>
>
>
> Feature creation error (OGR error: failed to prepare SQL: INSERT INTO "test" ( "geom", "full_id", "osm_id", "osm_type", "highway", "name",
>
>
>
> ... again the same many more attributes...
>
>
>
> "safe_place:school", "safe_place:mosque") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?))
>
>
>
> This happens even when I have only 2 features (see attached test file). The workaraound is to use GeoJSON, bacuse for some reason that one works, but in normal case I have 1000s of points instead of two, and then a GeoJSON file becomes very large and unwieldy...
>
>
>
> Any ideas if this is a bug or a limit in the actual formats such as GeoPackage...?
>
> yours,
>
>
>
> --
>
> Barend Köbben
>
> Senior Lecturer – ITC-University of Twente & ATLAS
>
> PO Box 217, 7500 AE Enschede (Netherlands)
>
> ITC Building (room 1-065)
>
> @barendkobben
>
> +31-(0)53 4874 253
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Qgis-user mailing list
>
> [hidden email]
>
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user