Import of Large CSV File

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Import of Large CSV File

Grant Boxer

I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records. QGIS does not seem to be able to import this using the “Delimited Text” option. Any suggestions?

 

Regards Grant

 

Grant Boxer

Perth, Western Australia

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Import of Large CSV File

Even Rouault-2
On mardi 16 avril 2019 16:06:36 CEST Grant Boxer wrote:
> I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records.
> QGIS does not seem to be able to import this using the "Delimited Text"
> option. Any suggestions?

Try opening it with the "Add vector layer" functionality. This will go through
the OGR CSV driver.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Import of Large CSV File

Nicolas Cadieux
In reply to this post by Grant Boxer
Hi,

If Even’s solution does not work, the file is probably problematic. You probably have missing or corrupted line.  I have seen crashes were comma separated lines are mixed with space separated lines, or with files or different row length. Did the file work in the past?

Solution would would be to look at each line and count the field with a python script.  I  could help if the file is really important.  If it was downloaded, try downloading the file again.  What are the symptoms? Crashes?  Can’t see all the line?  These are points I imagine?

Nicolas


Le 16 avr. 2019 à 04:06, Grant Boxer <[hidden email]> a écrit :

I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records. QGIS does not seem to be able to import this using the “Delimited Text” option. Any suggestions?

 

Regards Grant

 

Grant Boxer

Perth, Western Australia

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Import of Large CSV File

b.j.kobben

simple test is also to check if the file can be imported in Excel or OpenOffice spreadsheet...

 

 

--

Barend Köbben

 

 

On 16/04/2019, 14:11, "Qgis-user on behalf of Nicolas Cadieux" <[hidden email] on behalf of [hidden email]> wrote:

 

Hi,

 

If Even’s solution does not work, the file is probably problematic. You probably have missing or corrupted line.  I have seen crashes were comma separated lines are mixed with space separated lines, or with files or different row length. Did the file work in the past?

 

Solution would would be to look at each line and count the field with a python script.  I  could help if the file is really important.  If it was downloaded, try downloading the file again.  What are the symptoms? Crashes?  Can’t see all the line?  These are points I imagine?

 

Nicolas

 


Le 16 avr. 2019 à 04:06, Grant Boxer <[hidden email]> a écrit :

I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records. QGIS does not seem to be able to import this using the “Delimited Text” option. Any suggestions?

 

Regards Grant

 

Grant Boxer

Perth, Western Australia

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Import of Large CSV File

Nicolas Cadieux
Hi,

That can work up well to a million lines.  That does work well for smaller files. If the file come from Excel, it will have a tendency of adding extra empty columns in cells that have been erased. Notepad++ will get you farther and GVim even more but these are simple texte editor.  If the error something like a empty row or commas in a texte field that add row, you will most likely never find it.  That’s how I got into Python.  The simple ideas in Python is to count the row of each line in a comma separated or space sep file and to make sure each x and y fields are actually floats and to print out the first few problem line.That will find most errors In a LiDAR file anyway.

Nicolas

Le 16 avr. 2019 à 08:43, <[hidden email]> <[hidden email]> a écrit :

simple test is also to check if the file can be imported in Excel or OpenOffice spreadsheet...

 

 

--

Barend Köbben

 

 

On 16/04/2019, 14:11, "Qgis-user on behalf of Nicolas Cadieux" <[hidden email] on behalf of [hidden email]> wrote:

 

Hi,

 

If Even’s solution does not work, the file is probably problematic. You probably have missing or corrupted line.  I have seen crashes were comma separated lines are mixed with space separated lines, or with files or different row length. Did the file work in the past?

 

Solution would would be to look at each line and count the field with a python script.  I  could help if the file is really important.  If it was downloaded, try downloading the file again.  What are the symptoms? Crashes?  Can’t see all the line?  These are points I imagine?

 

Nicolas

 


Le 16 avr. 2019 à 04:06, Grant Boxer <[hidden email]> a écrit :

I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records. QGIS does not seem to be able to import this using the “Delimited Text” option. Any suggestions?

 

Regards Grant

 

Grant Boxer

Perth, Western Australia

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Import of Large CSV File

Grant Boxer
In reply to this post by Grant Boxer
Hi Chris,

That worked! I used the command line as you suggested and I then needed to
create a table from points to get the locations registered. Thanks.

There is probably a way to do that on the import - I will need to do a
little research.

Cheers Grant
Perth WA

-----Original Message-----
From: Chris Crook <[hidden email]>
Sent: Wednesday, 17 April 2019 2:26 AM
To: Grant Boxer <[hidden email]>
Subject: RE: [Qgis-user] Import of Large CSV File

Hi Grant

That is probably pushing the delimited text provider ... it doesn't import
the CSV file, it reads it directly.  So even if it did load it would be very
slow to use for any sort of searching, zooming, feature selection etc.  I'd
suggest turning into some sort of database format, for example geopackage
(which is just another file - it doesn't need a database server).  You could
do that with the ogr2ogr.  The ogr2ogr command line would be something like

ogr2ogr -f GPKG mydata.gpkg mydata.csv

It would probably need some other options depending on the content of the
CSV file and the coordinate reference system.

Cheers
Chris
________________________________________
From: Qgis-user [[hidden email]] on behalf of Grant Boxer
[[hidden email]]
Sent: 16 April 2019 20:07
To: [hidden email]
Subject: [Qgis-user] Import of Large CSV File

I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records.
QGIS does not seem to be able to import this using the "Delimited Text"
option. Any suggestions?

Regards Grant

Grant Boxer
Perth, Western Australia


________________________________

This message contains information, which may be in confidence and may be
subject to legal privilege. If you are not the intended recipient, you must
not peruse, use, disseminate, distribute or copy this message. If you have
received this message in error, please notify us immediately (Phone 0800 665
463 or [hidden email]) and destroy the original message. LINZ accepts no
responsibility for changes to this email, or for any attachments, after its
transmission from LINZ. Thank You.


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user