ctrl-c selection issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

ctrl-c selection issue

Jeroen Hovens

Hi,

 

Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).

 

What could be the problem? Could the data be corrupted? How can I check what is wrong?

 

Thank you

Gr.

Jeroen

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Ramon Andinach
I’m not sure what the problem is, but if I were doing that repeatedly, I’d probably use Layer -> Save As, tick the Save Only Selected Features box and save it to CSV.  Then muck about with excel later.

This way ought to be a bit lighter on the memory. 


Ramon.
On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]> wrote:

Hi,
 
Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).
 
What could be the problem? Could the data be corrupted? How can I check what is wrong?
 
Thank you
Gr.
Jeroen 
 
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Kirk Schmidt
After a few selections, you may have to close and reopen qgis to purge the memory as well. Importing a csv is memory intensive so another approach is to try saving as a spatial database (SQLite) or even a shapefile.  Once written, you can load the spatial file and you should have lots of memory to keep selecting points.

Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:
I’m not sure what the problem is, but if I were doing that repeatedly, I’d probably use Layer -> Save As, tick the Save Only Selected Features box and save it to CSV.  Then muck about with excel later.

This way ought to be a bit lighter on the memory. 


Ramon.
On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]> wrote:

Hi,
 
Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).
 
What could be the problem? Could the data be corrupted? How can I check what is wrong?
 
Thank you
Gr.
Jeroen 
 
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: [hidden email]
Web: www.nortekreources.com

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Jeroen Hovens

Thanks for the ideas.

 

The option to save the selection as new csv file as Ramon suggested, doesn’t work; qgis goes into ‘not responding’ and no csv file is added to the layer list. However, a csv file is made, lacking a decent number of objects (first test 15095 of 18582 objects and second test is 15179 of 18582 objects, so missing about 3,5k objects). This csv file can be opened manually in qgis.

And I checked: this options does work for selections where the ctrl-c option also worked. Then the new cvs layer is added to the layer list in no time.

 

Is there some sort of log file in qgis that might give clues to the ‘not responding?

 

@Kirk

I did close and reopen qgis multiple times, even shut down the pc entirely, but to no effect. Selections that did allow for copying earlier, still work: I can make that selection again and ctrl-c works fine. Seems to me this rules out memory-issues.

 

Gr.

Jeroen

 

 

Van: Qgis-user [mailto:[hidden email]] Namens Kirk Schmidt
Verzonden: dinsdag 9 januari 2018 22:20
Aan: [hidden email]
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

After a few selections, you may have to close and reopen qgis to purge the memory as well. Importing a csv is memory intensive so another approach is to try saving as a spatial database (SQLite) or even a shapefile.  Once written, you can load the spatial file and you should have lots of memory to keep selecting points.


Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:

I’m not sure what the problem is, but if I were doing that repeatedly, I’d probably use Layer -> Save As, tick the Save Only Selected Features box and save it to CSV.  Then muck about with excel later.

 

This way ought to be a bit lighter on the memory. 

 

 

Ramon.

On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]> wrote:

 

Hi,

 

Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).

 

What could be the problem? Could the data be corrupted? How can I check what is wrong?

 

Thank you

Gr.

Jeroen 

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

 




_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: [hidden email]
Web: www.nortekreources.com

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Ramon Andinach
Just so we’ve got just about everything covered, does this happen for “Edit-> Paste As”?

Also looking at the size of your original CSV, I’d go along with Jeroen, and try and import that csv into some other format first. 
If you can find a way of making the selection without going near the attribute table that will help too.


Regards,
Ramon.
On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]> wrote:

Thanks for the ideas.
 
The option to save the selection as new csv file as Ramon suggested, doesn’t work; qgis goes into ‘not responding’ and no csv file is added to the layer list. However, a csv file is made, lacking a decent number of objects (first test 15095 of 18582 objects and second test is 15179 of 18582 objects, so missing about 3,5k objects). This csv file can be opened manually in qgis.
And I checked: this options does work for selections where the ctrl-c option also worked. Then the new cvs layer is added to the layer list in no time.
 
Is there some sort of log file in qgis that might give clues to the ‘not responding?
 
@Kirk
I did close and reopen qgis multiple times, even shut down the pc entirely, but to no effect. Selections that did allow for copying earlier, still work: I can make that selection again and ctrl-c works fine. Seems to me this rules out memory-issues.
 
Gr.
Jeroen
 
 
Van: Qgis-user [[hidden email]] Namens Kirk Schmidt
Verzonden: dinsdag 9 januari 2018 22:20
Aan: [hidden email]
Onderwerp: Re: [Qgis-user] ctrl-c selection issue
 

After a few selections, you may have to close and reopen qgis to purge the memory as well. Importing a csv is memory intensive so another approach is to try saving as a spatial database (SQLite) or even a shapefile.  Once written, you can load the spatial file and you should have lots of memory to keep selecting points. 


Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:
I’m not sure what the problem is, but if I were doing that repeatedly, I’d probably use Layer -> Save As, tick the Save Only Selected Features box and save it to CSV.  Then muck about with excel later. 
 
This way ought to be a bit lighter on the memory. 
 
 
Ramon.
On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]> wrote:
 
Hi,
 
Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).
 
What could be the problem? Could the data be corrupted? How can I check what is wrong?
 
Thank you
Gr.
Jeroen 
 
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
 



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: [hidden email]
Web: www.nortekreources.com
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Jeroen Hovens

Hmm, I can’t use Edit > Paste As, because qgis does not finish the Edit > Copy command.

 

The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no x-y data.

If you want to recreate the issue, the file can be found here:

https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-voor-postcode-huisnummer

I run into this issue when selecting all objects with Attribute Gem217 = 310

355 also has this issue. 321 and 356 work without a problem.

 

I am using qgis here just as a means to chop the csv file into smaller selections which I can then use to edit, add attributes and data to and make maps of them.

 

Gr.

Jeroen

 

Van: Qgis-user [mailto:[hidden email]] Namens Ramon Andinach
Verzonden: dinsdag 9 januari 2018 23:35
Aan: qgis-user <[hidden email]>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

Just so we’ve got just about everything covered, does this happen for “Edit-> Paste As”?

 

Also looking at the size of your original CSV, I’d go along with Jeroen, and try and import that csv into some other format first. 

If you can find a way of making the selection without going near the attribute table that will help too.

 

 

Regards,

Ramon.

On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]> wrote:

 

Thanks for the ideas.

 

The option to save the selection as new csv file as Ramon suggested, doesn’t work; qgis goes into ‘not responding’ and no csv file is added to the layer list. However, a csv file is made, lacking a decent number of objects (first test 15095 of 18582 objects and second test is 15179 of 18582 objects, so missing about 3,5k objects). This csv file can be opened manually in qgis.

And I checked: this options does work for selections where the ctrl-c option also worked. Then the new cvs layer is added to the layer list in no time.

 

Is there some sort of log file in qgis that might give clues to the ‘not responding?

 

@Kirk

I did close and reopen qgis multiple times, even shut down the pc entirely, but to no effect. Selections that did allow for copying earlier, still work: I can make that selection again and ctrl-c works fine. Seems to me this rules out memory-issues.

 

Gr.

Jeroen

 

 

Van: Qgis-user [[hidden email]] Namens Kirk Schmidt
Verzonden: dinsdag 9 januari 2018 22:20
Aan: [hidden email]
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

After a few selections, you may have to close and reopen qgis to purge the memory as well. Importing a csv is memory intensive so another approach is to try saving as a spatial database (SQLite) or even a shapefile.  Once written, you can load the spatial file and you should have lots of memory to keep selecting points. 



Kirk

On 1/9/2018 4:58 PM, Ramon Andinach wrote:

I’m not sure what the problem is, but if I were doing that repeatedly, I’d probably use Layer -> Save As, tick the Save Only Selected Features box and save it to CSV.  Then muck about with excel later. 

 

This way ought to be a bit lighter on the memory. 

 

 

Ramon.

On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]> wrote:

 

Hi,

 

Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln objects). I just need selections of this database which I can then modify in excel. So I opened the attribute table and made a selection based on a specific value for an attribute, and copied that selection (about 30.000 objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then saved the excel file and repeated the process for another selection. This worked perfect for some time. However, some selections won’t copy: when I press the copy button the programs turn to ‘not responding’ and won’t snap out of it anymore.  Selection size when this happens is about the same (about 20k to 35k objects).

 

What could be the problem? Could the data be corrupted? How can I check what is wrong?

 

Thank you

Gr.

Jeroen 

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

 





_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user




-- 
Kirk Schmidt, MScF, BScF, RPF
General Manager
Nortek Resource Solutions Inc.
RR # 1
Thorburn, NS
B0K 1W0
Tel (902) 922.3607
Email: [hidden email]
Web: www.nortekreources.com

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Richard Duivenvoorde

Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an
index, so when you create a small selection, QGIS has to run over all
those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as
GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think
tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:

> Hmm, I can’t use Edit > Paste As, because qgis does not finish the Edit
>> Copy command.
>
>  
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no x-y
> data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217 = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>  
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to and
> make maps of them.
>
>  
>
> Gr.
>
> Jeroen
>
>  
>
> *Van:* Qgis-user [mailto:[hidden email]] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <[hidden email]>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>  
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>  
>
> Also looking at the size of your original CSV, I’d go along with Jeroen,
> and try and import that csv into some other format first. 
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>  
>
>  
>
> Regards,
>
> Ramon.
>
>     On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>      
>
>     Thanks for the ideas.
>
>      
>
>     The option to save the selection as new csv file as Ramon suggested,
>     doesn’t work; qgis goes into ‘not responding’ and no csv file is
>     added to the layer list. However, a csv file is made, lacking a
>     decent number of objects (first test 15095 of 18582 objects and
>     second test is 15179 of 18582 objects, so missing about 3,5k
>     objects). This csv file can be opened manually in qgis.
>
>     And I checked: this options does work for selections where the
>     ctrl-c option also worked. Then the new cvs layer is added to the
>     layer list in no time.
>
>      
>
>     Is there some sort of log file in qgis that might give clues to the
>     ‘not responding?
>
>      
>
>     @Kirk
>
>     I did close and reopen qgis multiple times, even shut down the pc
>     entirely, but to no effect. Selections that did allow for copying
>     earlier, still work: I can make that selection again and ctrl-c
>     works fine. Seems to me this rules out memory-issues.
>
>      
>
>     Gr.
>
>     Jeroen
>
>      
>
>      
>
>     *Van:* Qgis-user
>     [mailto:[hidden email]] *Namens *Kirk Schmidt
>     *Verzonden:* dinsdag 9 januari 2018 22:20
>     *Aan:* [hidden email] <mailto:[hidden email]>
>     *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>      
>
>     After a few selections, you may have to close and reopen qgis to
>     purge the memory as well. Importing a csv is memory intensive so
>     another approach is to try saving as a spatial database (SQLite) or
>     even a shapefile.  Once written, you can load the spatial file and
>     you should have lots of memory to keep selecting points. 
>
>
>
>     Kirk
>
>     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>
>         I’m not sure what the problem is, but if I were doing that
>         repeatedly, I’d probably use Layer -> Save As, tick the Save
>         Only Selected Features box and save it to CSV.  Then muck about
>         with excel later. 
>
>          
>
>         This way ought to be a bit lighter on the memory. 
>
>          
>
>          
>
>         Ramon.
>
>             On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>              
>
>             Hi,
>
>              
>
>             Using qgis 2.14.18 I have been working today with a large
>             csv file (7.5 mln objects). I just need selections of this
>             database which I can then modify in excel. So I opened the
>             attribute table and made a selection based on a specific
>             value for an attribute, and copied that selection (about
>             30.000 objects) via the clipboard (ctrl-c) to a new
>             Microsoft Excel file, then saved the excel file and repeated
>             the process for another selection. This worked perfect for
>             some time. However, some selections won’t copy: when I press
>             the copy button the programs turn to ‘not responding’ and
>             won’t snap out of it anymore.  Selection size when this
>             happens is about the same (about 20k to 35k objects).
>
>              
>
>             What could be the problem? Could the data be corrupted? How
>             can I check what is wrong?
>
>              
>
>             Thank you
>
>             Gr.
>
>             Jeroen 
>
>              
>
>             _______________________________________________
>             Qgis-user mailing list
>             [hidden email] <mailto:[hidden email]>
>             List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>             Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>          
>
>
>
>
>
>         _______________________________________________
>
>         Qgis-user mailing list
>
>         [hidden email] <mailto:[hidden email]>
>
>         List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>         Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>     --
>
>     Kirk Schmidt, MScF, BScF, RPF
>
>     General Manager
>
>     Nortek Resource Solutions Inc.
>
>     RR # 1
>
>     Thorburn, NS
>
>     B0K 1W0
>
>     Tel (902) 922.3607
>
>     Email: [hidden email] <mailto:[hidden email]>
>
>     Web: www.nortekreources.com <http://www.nortekreources.com/>
>
>     _______________________________________________
>     Qgis-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>  
>
>
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Jeroen Hovens
At the moment I make the selection using the Select by expression tool. I then choose one of the  Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever

I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused ☹




-----Oorspronkelijk bericht-----
Van: Richard Duivenvoorde [mailto:[hidden email]]
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <[hidden email]>; 'Ramon Andinach' <[hidden email]>; 'qgis-user' <[hidden email]>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue


Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:

> Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> Edit
>> Copy command.
>
>  
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> x-y data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217
> = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>  
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to
> and make maps of them.
>
>  
>
> Gr.
>
> Jeroen
>
>  
>
> *Van:* Qgis-user [mailto:[hidden email]] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <[hidden email]>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>  
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>  
>
> Also looking at the size of your original CSV, I’d go along with
> Jeroen, and try and import that csv into some other format first.
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>  
>
>  
>
> Regards,
>
> Ramon.
>
>     On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>      
>
>     Thanks for the ideas.
>
>      
>
>     The option to save the selection as new csv file as Ramon suggested,
>     doesn’t work; qgis goes into ‘not responding’ and no csv file is
>     added to the layer list. However, a csv file is made, lacking a
>     decent number of objects (first test 15095 of 18582 objects and
>     second test is 15179 of 18582 objects, so missing about 3,5k
>     objects). This csv file can be opened manually in qgis.
>
>     And I checked: this options does work for selections where the
>     ctrl-c option also worked. Then the new cvs layer is added to the
>     layer list in no time.
>
>      
>
>     Is there some sort of log file in qgis that might give clues to the
>     ‘not responding?
>
>      
>
>     @Kirk
>
>     I did close and reopen qgis multiple times, even shut down the pc
>     entirely, but to no effect. Selections that did allow for copying
>     earlier, still work: I can make that selection again and ctrl-c
>     works fine. Seems to me this rules out memory-issues.
>
>      
>
>     Gr.
>
>     Jeroen
>
>      
>
>      
>
>     *Van:* Qgis-user
>     [mailto:[hidden email]] *Namens *Kirk Schmidt
>     *Verzonden:* dinsdag 9 januari 2018 22:20
>     *Aan:* [hidden email] <mailto:[hidden email]>
>     *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>      
>
>     After a few selections, you may have to close and reopen qgis to
>     purge the memory as well. Importing a csv is memory intensive so
>     another approach is to try saving as a spatial database (SQLite) or
>     even a shapefile.  Once written, you can load the spatial file and
>     you should have lots of memory to keep selecting points.
>
>
>
>     Kirk
>
>     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>
>         I’m not sure what the problem is, but if I were doing that
>         repeatedly, I’d probably use Layer -> Save As, tick the Save
>         Only Selected Features box and save it to CSV.  Then muck about
>         with excel later.
>
>          
>
>         This way ought to be a bit lighter on the memory.
>
>          
>
>          
>
>         Ramon.
>
>             On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>              
>
>             Hi,
>
>              
>
>             Using qgis 2.14.18 I have been working today with a large
>             csv file (7.5 mln objects). I just need selections of this
>             database which I can then modify in excel. So I opened the
>             attribute table and made a selection based on a specific
>             value for an attribute, and copied that selection (about
>             30.000 objects) via the clipboard (ctrl-c) to a new
>             Microsoft Excel file, then saved the excel file and repeated
>             the process for another selection. This worked perfect for
>             some time. However, some selections won’t copy: when I press
>             the copy button the programs turn to ‘not responding’ and
>             won’t snap out of it anymore.  Selection size when this
>             happens is about the same (about 20k to 35k objects).
>
>              
>
>             What could be the problem? Could the data be corrupted? How
>             can I check what is wrong?
>
>              
>
>             Thank you
>
>             Gr.
>
>             Jeroen
>
>              
>
>             _______________________________________________
>             Qgis-user mailing list
>             [hidden email] <mailto:[hidden email]>
>             List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>             Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>          
>
>
>
>
>
>         _______________________________________________
>
>         Qgis-user mailing list
>
>         [hidden email] <mailto:[hidden email]>
>
>         List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>         Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>     --
>
>     Kirk Schmidt, MScF, BScF, RPF
>
>     General Manager
>
>     Nortek Resource Solutions Inc.
>
>     RR # 1
>
>     Thorburn, NS
>
>     B0K 1W0
>
>     Tel (902) 922.3607
>
>     Email: [hidden email] <mailto:[hidden email]>
>
>     Web: www.nortekreources.com <http://www.nortekreources.com/>
>
>     _______________________________________________
>     Qgis-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>  
>
>
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Saber Razmjooei
Drag-n-drop the file on QGIS canvas!

Regards
Saber

On 10 January 2018 at 11:26, Jeroen Hovens <[hidden email]> wrote:
At the moment I make the selection using the Select by expression tool. I then choose one of the  Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever

I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused ☹




-----Oorspronkelijk bericht-----
Van: Richard Duivenvoorde [mailto:[hidden email]]
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <[hidden email]>; 'Ramon Andinach' <[hidden email]>; 'qgis-user' <[hidden email]>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue


Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> Edit
>> Copy command.
>
>
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> x-y data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217
> = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to
> and make maps of them.
>
>
>
> Gr.
>
> Jeroen
>
>
>
> *Van:* Qgis-user [mailto:[hidden email]] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <[hidden email]>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>
>
> Also looking at the size of your original CSV, I’d go along with
> Jeroen, and try and import that csv into some other format first.
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>
>
>
>
> Regards,
>
> Ramon.
>
>     On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>     Thanks for the ideas.
>
>
>
>     The option to save the selection as new csv file as Ramon suggested,
>     doesn’t work; qgis goes into ‘not responding’ and no csv file is
>     added to the layer list. However, a csv file is made, lacking a
>     decent number of objects (first test 15095 of 18582 objects and
>     second test is 15179 of 18582 objects, so missing about 3,5k
>     objects). This csv file can be opened manually in qgis.
>
>     And I checked: this options does work for selections where the
>     ctrl-c option also worked. Then the new cvs layer is added to the
>     layer list in no time.
>
>
>
>     Is there some sort of log file in qgis that might give clues to the
>     ‘not responding?
>
>
>
>     @Kirk
>
>     I did close and reopen qgis multiple times, even shut down the pc
>     entirely, but to no effect. Selections that did allow for copying
>     earlier, still work: I can make that selection again and ctrl-c
>     works fine. Seems to me this rules out memory-issues.
>
>
>
>     Gr.
>
>     Jeroen
>
>
>
>
>
>     *Van:* Qgis-user
>     [mailto:[hidden email]] *Namens *Kirk Schmidt
>     *Verzonden:* dinsdag 9 januari 2018 22:20
>     *Aan:* [hidden email] <mailto:[hidden email]>
>     *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
>     After a few selections, you may have to close and reopen qgis to
>     purge the memory as well. Importing a csv is memory intensive so
>     another approach is to try saving as a spatial database (SQLite) or
>     even a shapefile.  Once written, you can load the spatial file and
>     you should have lots of memory to keep selecting points.
>
>
>
>     Kirk
>
>     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>
>         I’m not sure what the problem is, but if I were doing that
>         repeatedly, I’d probably use Layer -> Save As, tick the Save
>         Only Selected Features box and save it to CSV.  Then muck about
>         with excel later.
>
>
>
>         This way ought to be a bit lighter on the memory.
>
>
>
>
>
>         Ramon.
>
>             On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>
>
>             Hi,
>
>
>
>             Using qgis 2.14.18 I have been working today with a large
>             csv file (7.5 mln objects). I just need selections of this
>             database which I can then modify in excel. So I opened the
>             attribute table and made a selection based on a specific
>             value for an attribute, and copied that selection (about
>             30.000 objects) via the clipboard (ctrl-c) to a new
>             Microsoft Excel file, then saved the excel file and repeated
>             the process for another selection. This worked perfect for
>             some time. However, some selections won’t copy: when I press
>             the copy button the programs turn to ‘not responding’ and
>             won’t snap out of it anymore.  Selection size when this
>             happens is about the same (about 20k to 35k objects).
>
>
>
>             What could be the problem? Could the data be corrupted? How
>             can I check what is wrong?
>
>
>
>             Thank you
>
>             Gr.
>
>             Jeroen
>
>
>
>             _______________________________________________
>             Qgis-user mailing list
>             [hidden email] <mailto:[hidden email]>
>             List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>             Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
>
>
>         _______________________________________________
>
>         Qgis-user mailing list
>
>         [hidden email] <mailto:[hidden email]>
>
>         List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>         Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>     --
>
>     Kirk Schmidt, MScF, BScF, RPF
>
>     General Manager
>
>     Nortek Resource Solutions Inc.
>
>     RR # 1
>
>     Thorburn, NS
>
>     B0K 1W0
>
>     Tel (902) 922.3607
>
>     Email: [hidden email] <mailto:[hidden email]>
>
>     Web: www.nortekreources.com <http://www.nortekreources.com/>
>
>     _______________________________________________
>     Qgis-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



--
Saber Razmjooei
www.lutraconsulting.co.uk
+44 (0)7568 129733

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Jeroen Hovens

Update: Solution.

After saving the csv file as a shape-file and then loading the .dbf file into qgis everything worked as intended.

Although Qgis couldn’t create the .shp file and the .dbf file actually ‘grew’ from 227 MB (.cvs) to 9.4 GB (.dbf), I can now copy any selection without any problem.

 

And along the way I discovered the option Split Vector Layer, which is what I was doing, however I was doing it one at the time, instead of creating 388 new layers at once (because I need just a few).

And Split Vector Layer doesn’t work fort the .csv file or the .dbf by the way, it is probably missing some information.

 

So, still no idea what is going wrong in the .cvs file, but I have a workable solution.

Thank you all for your thoughts.

 

Gr.

Jeroen

 

Van: Saber Razmjooei [mailto:[hidden email]]
Verzonden: woensdag 10 januari 2018 12:33
Aan: Jeroen Hovens <[hidden email]>
CC: Richard 🌍 Duivenvoorde <[hidden email]>; Ramon Andinach <[hidden email]>; qgis-user <[hidden email]>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue

 

Drag-n-drop the file on QGIS canvas!

 

Regards

Saber

 

On 10 January 2018 at 11:26, Jeroen Hovens <[hidden email]> wrote:

At the moment I make the selection using the Select by expression tool. I then choose one of the  Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever

I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused




-----Oorspronkelijk bericht-----
Van: Richard Duivenvoorde [mailto:[hidden email]]
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <[hidden email]>; 'Ramon Andinach' <[hidden email]>; 'qgis-user' <[hidden email]>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue



Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:


> Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> Edit
>> Copy command.
>
>
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> x-y data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217
> = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to
> and make maps of them.
>
>
>
> Gr.
>
> Jeroen
>
>
>
> *Van:* Qgis-user [mailto:[hidden email]] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <[hidden email]>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>
>
> Also looking at the size of your original CSV, I’d go along with
> Jeroen, and try and import that csv into some other format first.
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>
>
>
>
> Regards,
>
> Ramon.
>
>     On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>     Thanks for the ideas.
>
>
>
>     The option to save the selection as new csv file as Ramon suggested,
>     doesn’t work; qgis goes into ‘not responding’ and no csv file is
>     added to the layer list. However, a csv file is made, lacking a
>     decent number of objects (first test 15095 of 18582 objects and
>     second test is 15179 of 18582 objects, so missing about 3,5k
>     objects). This csv file can be opened manually in qgis.
>
>     And I checked: this options does work for selections where the
>     ctrl-c option also worked. Then the new cvs layer is added to the
>     layer list in no time.
>
>
>
>     Is there some sort of log file in qgis that might give clues to the
>     ‘not responding?
>
>
>
>     @Kirk
>
>     I did close and reopen qgis multiple times, even shut down the pc
>     entirely, but to no effect. Selections that did allow for copying
>     earlier, still work: I can make that selection again and ctrl-c
>     works fine. Seems to me this rules out memory-issues.
>
>
>
>     Gr.
>
>     Jeroen
>
>
>
>
>
>     *Van:* Qgis-user
>     [mailto:[hidden email]] *Namens *Kirk Schmidt
>     *Verzonden:* dinsdag 9 januari 2018 22:20
>     *Aan:* [hidden email] <mailto:[hidden email]>
>     *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
>     After a few selections, you may have to close and reopen qgis to
>     purge the memory as well. Importing a csv is memory intensive so
>     another approach is to try saving as a spatial database (SQLite) or
>     even a shapefile.  Once written, you can load the spatial file and
>     you should have lots of memory to keep selecting points.
>
>
>
>     Kirk
>
>     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>
>         I’m not sure what the problem is, but if I were doing that
>         repeatedly, I’d probably use Layer -> Save As, tick the Save
>         Only Selected Features box and save it to CSV.  Then muck about
>         with excel later.
>
>
>
>         This way ought to be a bit lighter on the memory.
>
>
>
>
>
>         Ramon.
>
>             On 10 Jan 2018, at 04:25, Jeroen Hovens <[hidden email]
>             <mailto:[hidden email]>> wrote:
>
>
>
>             Hi,
>
>
>
>             Using qgis 2.14.18 I have been working today with a large
>             csv file (7.5 mln objects). I just need selections of this
>             database which I can then modify in excel. So I opened the
>             attribute table and made a selection based on a specific
>             value for an attribute, and copied that selection (about
>             30.000 objects) via the clipboard (ctrl-c) to a new
>             Microsoft Excel file, then saved the excel file and repeated
>             the process for another selection. This worked perfect for
>             some time. However, some selections won’t copy: when I press
>             the copy button the programs turn to ‘not responding’ and
>             won’t snap out of it anymore.  Selection size when this
>             happens is about the same (about 20k to 35k objects).
>
>
>
>             What could be the problem? Could the data be corrupted? How
>             can I check what is wrong?
>
>
>
>             Thank you
>
>             Gr.
>
>             Jeroen
>
>
>
>             _______________________________________________
>             Qgis-user mailing list
>             [hidden email] <mailto:[hidden email]>
>             List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>             Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
>
>
>         _______________________________________________
>
>         Qgis-user mailing list
>
>         [hidden email] <mailto:[hidden email]>
>
>         List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>         Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>     --
>
>     Kirk Schmidt, MScF, BScF, RPF
>
>     General Manager
>
>     Nortek Resource Solutions Inc.
>
>     RR # 1
>
>     Thorburn, NS
>
>     B0K 1W0
>
>     Tel (902) 922.3607
>
>     Email: [hidden email] <mailto:[hidden email]>
>
>     Web: www.nortekreources.com <http://www.nortekreources.com/>
>
>     _______________________________________________
>     Qgis-user mailing list
>     [hidden email] <mailto:[hidden email]>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user



 

--

Saber Razmjooei

www.lutraconsulting.co.uk
+44 (0)7568 129733


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Fernando M. Roxo da Motta
In reply to this post by Jeroen Hovens
On Wed, 10 Jan 2018 12:26:42 +0100, "Jeroen Hovens" <[hidden email]>
wrote:

    Hi Jeroen,

  If you allow, I will suggest you two possible approach.  You didn't
specify your OS platform.  I tested on Linux but the first approach
should work anywhere.

  First one using qgis only.   Load your CSV file in qgis as a table
only, no geom.  At least I did not found one.   After loading save it
as a SQLite database (save as...).   It should sabe and load the
database as a table.   Unload the CSV file.

  At this point double click the sqlite layer and got to general tab.
There click on Query Builder button and use an expression like:

    "Gem2017" = 310

You can just apply this selection or, if you wish, test it to get the
number of lines selected.   When you apply a selection like this your
layer is composed of only those selected one.   Just save it.  For
example save as a "MS Office Open XML spreadsheet", that is one of the
offered formats in my Qgis.

  After that you just change the desired value and so on.

  Another approach is pure shell approach.

  I am attaching to the message a shell script that will:

     1- Convert the CVS from DOS (CRLF) to Linux (LF) format
     2- Sort it by the 5th column (the Gem2017)
     3- Get the unique values for that column
     4- Iterate over the CSV file extracting all lines with
        each unique Gem2017, saving it with the name:
        Gem2017-<GemValue>.csv

  I think it is possible to bach convert them to Excel or LibreOffice
calc format, just did not looked at it.

  If you are using Windows, you can try this in a Cygwin or even the
Linux subsystem for Windows 10 (I never saw this last one, it is just a
guess).

  Hope this helps.


P.S. - If you prefer to deal with database it self, a SQL like:

   select * from pc6hnr20170801_gwb where gem2017 = 310;

   Will get you all entries with that value of gem2017.

> At the moment I make the selection using the Select by expression
> tool. I then choose one of the  Gem2017 values (each object is an
> address, and has one of 388 unique Gem217 values). I want to select
> all objects bwith one unique Gem2017 value. Making the selection is
> no problem, I can make any selection I want. Making the selection
> usually takes less then a minute. The problem is in copying specific
> selections to the clipboard or saving a selection as a new file.
> Copying a selection of about 20.000 objects from the 7,5 mln takes
> about a second. Unless the issue arises, then the copy process takes
> forever
>
> I have no experience with Postgis or using databases. I tried saving
> the csv file as a sqlite, but then had no idea how to re-open that
> file. The dialogbox got me all confused ☹
>
>
>
>
> -----Oorspronkelijk bericht-----
> Van: Richard Duivenvoorde [mailto:[hidden email]]
> Verzonden: woensdag 10 januari 2018 10:21
> Aan: Jeroen Hovens <[hidden email]>; 'Ramon Andinach'
> <[hidden email]>; 'qgis-user' <[hidden email]>
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
>
>
> Hi Jeroen,
>
> Loading a csv in QGIS is not optimal. I think csv does not have an
> index, so when you create a small selection, QGIS has to run over all
> those 7,5 mln object every time.
>
> Not sure how you do your 'selecting'?
> Is it spatial or based on zip/postcode/buurtnr/wijknr?
>
> I think I would either load it in Postgis (if available) or save as
> GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
> shapefile for it. AND then create indexes on desired columns.
>
> IF you are searching for strings like zipcodes in that list, I think
> tools like grep (available in OSGeo4W too!) work best:
>
> Eg searching fro 3 zip codes is milliseconds work:
>
> grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
>
> If all fails, give me a call.
>
> Regards,
>
> Richard Duivenvoorde
>
> On 10-01-18 00:33, Jeroen Hovens wrote:
> > Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> > Edit  
> >> Copy command.  
> >
> >  
> >
> > The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> > x-y data.
> >
> > If you want to recreate the issue, the file can be found here:
> >
> > https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> > voor-postcode-huisnummer
> >
> > I run into this issue when selecting all objects with Attribute
> > Gem217 = 310
> >
> > 355 also has this issue. 321 and 356 work without a problem.
> >
> >  
> >
> > I am using qgis here just as a means to chop the csv file into
> > smaller selections which I can then use to edit, add attributes and
> > data to and make maps of them.
> >
> >  
> >
> > Gr.
> >
> > Jeroen
> >
> >  
> >
> > *Van:* Qgis-user [mailto:[hidden email]] *Namens
> > *Ramon Andinach
> > *Verzonden:* dinsdag 9 januari 2018 23:35
> > *Aan:* qgis-user <[hidden email]>
> > *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> >
> >  
> >
> > Just so we’ve got just about everything covered, does this happen
> > for “Edit-> Paste As”?
> >
> >  
> >
> > Also looking at the size of your original CSV, I’d go along with
> > Jeroen, and try and import that csv into some other format first.
> >
> > If you can find a way of making the selection without going near
> > the attribute table that will help too.
> >
> >  
> >
> >  
> >
> > Regards,
> >
> > Ramon.
> >
> >     On 10 Jan 2018, at 06:26, Jeroen Hovens <[hidden email]
> >     <mailto:[hidden email]>> wrote:
> >
> >      
> >
> >     Thanks for the ideas.
> >
> >      
> >
> >     The option to save the selection as new csv file as Ramon
> > suggested, doesn’t work; qgis goes into ‘not responding’ and no csv
> > file is added to the layer list. However, a csv file is made,
> > lacking a decent number of objects (first test 15095 of 18582
> > objects and second test is 15179 of 18582 objects, so missing about
> > 3,5k objects). This csv file can be opened manually in qgis.
> >
> >     And I checked: this options does work for selections where the
> >     ctrl-c option also worked. Then the new cvs layer is added to
> > the layer list in no time.
> >
> >      
> >
> >     Is there some sort of log file in qgis that might give clues to
> > the ‘not responding?
> >
> >      
> >
> >     @Kirk
> >
> >     I did close and reopen qgis multiple times, even shut down the
> > pc entirely, but to no effect. Selections that did allow for copying
> >     earlier, still work: I can make that selection again and ctrl-c
> >     works fine. Seems to me this rules out memory-issues.
> >
> >      
> >
> >     Gr.
> >
> >     Jeroen
> >
> >      
> >
> >      
> >
> >     *Van:* Qgis-user
> >     [mailto:[hidden email]] *Namens *Kirk Schmidt
> >     *Verzonden:* dinsdag 9 januari 2018 22:20
> >     *Aan:* [hidden email]
> > <mailto:[hidden email]> *Onderwerp:* Re: [Qgis-user]
> > ctrl-c selection issue
> >
> >      
> >
> >     After a few selections, you may have to close and reopen qgis to
> >     purge the memory as well. Importing a csv is memory intensive so
> >     another approach is to try saving as a spatial database
> > (SQLite) or even a shapefile.  Once written, you can load the
> > spatial file and you should have lots of memory to keep selecting
> > points.
> >
> >
> >
> >     Kirk
> >
> >     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
> >
> >         I’m not sure what the problem is, but if I were doing that
> >         repeatedly, I’d probably use Layer -> Save As, tick the Save
> >         Only Selected Features box and save it to CSV.  Then muck
> > about with excel later.
> >
> >          
> >
> >         This way ought to be a bit lighter on the memory.
> >
> >          
> >
> >          
> >
> >         Ramon.
> >
> >             On 10 Jan 2018, at 04:25, Jeroen Hovens
> > <[hidden email] <mailto:[hidden email]>> wrote:
> >
> >              
> >
> >             Hi,
> >
> >              
> >
> >             Using qgis 2.14.18 I have been working today with a
> > large csv file (7.5 mln objects). I just need selections of this
> >             database which I can then modify in excel. So I opened
> > the attribute table and made a selection based on a specific
> >             value for an attribute, and copied that selection (about
> >             30.000 objects) via the clipboard (ctrl-c) to a new
> >             Microsoft Excel file, then saved the excel file and
> > repeated the process for another selection. This worked perfect for
> >             some time. However, some selections won’t copy: when I
> > press the copy button the programs turn to ‘not responding’ and
> >             won’t snap out of it anymore.  Selection size when this
> >             happens is about the same (about 20k to 35k objects).
> >
> >              
> >
> >             What could be the problem? Could the data be corrupted?
> > How can I check what is wrong?
> >
> >              
> >
> >             Thank you
> >
> >             Gr.
> >
> >             Jeroen
> >
> >              
> >
> >             _______________________________________________
> >             Qgis-user mailing list
> >             [hidden email]
> > <mailto:[hidden email]> List info:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >          
> >
> >
> >
> >
> >
> >         _______________________________________________
> >
> >         Qgis-user mailing list
> >
> >         [hidden email] <mailto:[hidden email]>
> >
> >         List info:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >         Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> >     --
> >
> >     Kirk Schmidt, MScF, BScF, RPF
> >
> >     General Manager
> >
> >     Nortek Resource Solutions Inc.
> >
> >     RR # 1
> >
> >     Thorburn, NS
> >
> >     B0K 1W0
> >
> >     Tel (902) 922.3607
> >
> >     Email: [hidden email]
> > <mailto:[hidden email]>
> >
> >     Web: www.nortekreources.com <http://www.nortekreources.com/>
> >
> >     _______________________________________________
> >     Qgis-user mailing list
> >     [hidden email] <mailto:[hidden email]>
> >     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >  
> >
> >
> >
> > _______________________________________________
> > Qgis-user mailing list
> > [hidden email]
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >  
>
>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user






  Roxo

--
---------------- Non luctari, ludare -------------------+ WYSIWYG
Fernando M. Roxo da Motta <[hidden email]>              | Editor?
Except where explicitly stated I speak on my own behalf.|  VI !!
      ( Usuário Linux registrado #39505 )               | I see text,
------------ Quis custodiet ipsos custodes?-------------+ I get text!
 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

doit.sh (864 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Mike Flannigan
In reply to this post by Jeroen Hovens

Hi,

I would recommend not using Excel to modify CSV files.  That
program has a reputation for changing numbers to dates and
performing other modifications to the file contents that may
not be wanted.

Personally I use the Text::CSV module in Perl, but there are
many other methods to modify a CSV file without the changes that
Excel imposes on you.


Mike


On 1/9/2018 4:37 PM, [hidden email] wrote:

> Hi,
>  
>
> Using qgis 2.14.18 I have been working today with a large csv file (7.5 mln
> objects). I just need selections of this database which I can then modify in
> excel. So I opened the attribute table and made a selection based on a
> specific value for an attribute, and copied that selection (about 30.000
> objects) via the clipboard (ctrl-c) to a new Microsoft Excel file, then
> saved the excel file and repeated the process for another selection. This
> worked perfect for some time. However, some selections won't copy: when I
> press the copy button the programs turn to 'not responding' and won't snap
> out of it anymore.  Selection size when this happens is about the same
> (about 20k to 35k objects).
>
>  
>
> What could be the problem? Could the data be corrupted? How can I check what
> is wrong?
>
>  
>
> Thank you
>
> Gr.
>
> Jeroen

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

KOSZA Antal-2
In reply to this post by Jeroen Hovens
Jeroen Hovens a következőt írta ekkor: 2018. január 10., szerda 12:26:42 EET

> At the moment I make the selection using the Select by expression tool. I
> then choose one of the  Gem2017 values (each object is an address, and has
> one of 388 unique Gem217 values). I want to select all objects bwith one
> unique Gem2017 value. Making the selection is no problem, I can make any
> selection I want. Making the selection usually takes less then a minute.
> The problem is in copying specific selections to the clipboard or saving a
> selection as a new file. Copying a selection of about 20.000 objects from
> the 7,5 mln takes about a second. Unless the issue arises, then the copy
> process takes forever
>

You can try with csvgrep from csvkit (https://pypi.python.org/pypi/csvkit)

csvgrep -d ';' -c Gem2017  -m 363 pc6hnr20170801_gwb.csv > new.csv
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: ctrl-c selection issue

Tobias Wendorff
In reply to this post by Mike Flannigan
Hi,

Am So, 14.01.2018, 02:23 schrieb Mike Flannigan:
>
> I would recommend not using Excel to modify CSV files.  That
> program has a reputation for changing numbers to dates and
> performing other modifications to the file contents that may
> not be wanted.

By the way: I'm wondering, why native Excel support didn't make
it into QGIS3. I think, FreeXL is integrated into GDAL/OGR, so
we could support it.

There was a plugin for QGIS2, I hope it will get ported to QGIS3.

Best regards,
Tobias

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user