Re: grass-user Digest, Vol 157, Issue 33

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: grass-user Digest, Vol 157, Issue 33

MehrdadVaredi
Subject: Re: [GRASS-user] v.what.vect never ends, SQLite warning  

Hi Everyone,
I think I figured out the problem.
I found that the problem was only the size or number of attributes in the point layer when I dropped and reduced the attributes it worked very fast and with no problem.
I am interested to know the limitations in GRASS GIS in terms of the number of features or attributes, if there is any reference, please let me know.

Thanks,

Mehrdad


On Wed, May 22, 2019 at 7:07 PM <[hidden email]> wrote:
Send grass-user mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.osgeo.org/mailman/listinfo/grass-user
or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of grass-user digest..."


Today's Topics:

   1. Re: v.what.vect never ends, SQLite warning (Mehrdad Varedi)
   2. Average height over catchment in grass gis (Rengifo Ortega)
   3. Re: importing and cleaning overlapping polygons that are
      supposed to overlap (Moritz Lennert)
   4. Re: importing and cleaning overlapping polygons that are
      supposed to overlap (Veronica Andreo)


----------------------------------------------------------------------

Message: 1
Date: Wed, 22 May 2019 16:20:21 -0400
From: Mehrdad Varedi <[hidden email]>
To: grass-user <[hidden email]>
Subject: Re: [GRASS-user] v.what.vect never ends, SQLite warning
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi Everyone,
When I try v.what.vect on a point layer and the output of a v.voronoi,
after around an hour I begin getting the warnings that apparently are
because of an SQLite database lock.

This is not what happens with v.what.vect only.
When exporting the layer to shapefile or any other format it happens too.
The point layer is not very big. it has 275,000 records and each has 350
features (they are mostly double precision). The same happens with v.color
on this layer.

I have read feedback like, the SQLite doesn't like concurrent access on the
same table, although I have restarted the computer, run only grass or tried
to run it from R within GRASS and no other connection to that database
existed, although the process takes forever and I begin getting the warning
"Waiting for XXX seconds"

Do you know how can I accelerate the processing or make it work?

FYI, the CPU is not very busy more than 15-20% of its total capacity and
the memory of 16GB is usually half free. The writing on disk is very
minimal when it is working before the SQLite lock warning.

Kind regards,

Mehrdad


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20190522/a2ed69a6/attachment-0001.html>

------------------------------

Message: 2
Date: Wed, 22 May 2019 21:29:38 +0000 (UTC)
From: Rengifo Ortega <[hidden email]>
To: [hidden email]
Subject: [GRASS-user] Average height over catchment in grass gis
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Dear users  I am writting you here with the hope of getting some ideas about how to caculate what Kirsten Hennrich et al., refers to as average catchment height (Ha) I found this infomation in the proceedings of an international conference called regionalisation of hydrology. 
In page 184, he says that it is possible to calculate  Ha  using r.statistics in GRASS GIS (IAHS publication no. 254). 
I have been using sofar SAGA GIS to do the job.   But recently  I am  experiencing some memory issues with SAGA GIS. In the old versions of  SAGA GIS  this calculation was named catchment height.  The new versions of SAGA GIS require that the user calculate  the mean over catchment  (MoC) first. Thereafter is MoC substrated from the original DTM. The product of this is the average catchment height or simply catchment height. 
Have someone in this group used GRASS GIS to calculate Catchment height in GRASS GIS. Any  hint will be appreciated.
Best regardsRengifo Ortega
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20190522/49e1a001/attachment-0001.html>

------------------------------

Message: 3
Date: Thu, 23 May 2019 00:15:16 +0200
From: Moritz Lennert <[hidden email]>
To: Veronica Andreo <[hidden email]>
Cc: Markus Metz <[hidden email]>, grass-user
        <[hidden email]>, Micha Silver <[hidden email]>
Subject: Re: [GRASS-user] importing and cleaning overlapping polygons
        that are supposed to overlap
Message-ID: <20190523001516.07cd6c77@moritz-ulb>
Content-Type: text/plain; charset=UTF-8

Le Wed, 22 May 2019 19:25:19 +0200,
Veronica Andreo <[hidden email]> a écrit :

> Hi Markus,
>
> Thanks for the explanation. It is possible to import topologically
> incorrect vectors, but not create them within GRASS. So, as a
> solution to my problem, I rather create (potentially overlapping)
> buffers outside GRASS and import them with -c in v.in.ogr to use
> v.rast.stats with no loss of areas, or follow the procedure that you
> described earlier in the thread.

You can also have a look at the v.rast.bufferstats addon.

Moritz

>
> Cheers,
> Vero
>
> El mié., 22 may. 2019 a las 18:48, Markus Metz (<
> [hidden email]>) escribió: 
>
> >
> >
> > On Wed, May 22, 2019 at 3:11 PM Veronica Andreo
> > <[hidden email]> wrote: 
> > >
> > > Dear all,
> > >
> > > thanks again for your answers. I found an easier way, use -c flag
> > > in 
> > v.in.ogr seems to not build topology of overlapping areas and then
> > v.rast.stats does not complain. 
> > >
> > > However, if one builds buffer areas for points within GRASS,
> > > i.e., using 
> > v.buffer, the problem appears again when buffer areas overlap,
> > which it's pretty common when creating buffers around neighbouring
> > points. Then, to get zonal stats for those areas the approach
> > suggested by Markus Metz should be followed. I believe it is a bit
> > of an overkill for such a common task in GIS. Couldn't v.buffer
> > have a -c flag as v.in.ogr so when overlapping of buffer areas is
> > fine the topology is not built and one would get just one area per
> > input point? Would that be possible?
> >
> > In short, no.
> >
> > The reason is that with a topological vector model, areas are
> > constructed from boundaries and centroids. If you use the -c flag
> > of v.in.ogr and there are polygons that overlap to a large degree
> > or completely, it is not possible to find a centroid for each
> > topological area, i.e. the overlapping input polygons can not be
> > properly represented with a topological model when using the -c
> > flag. As soon as you get incorrect boundaries and/or duplicate
> > centroids when building topology, results are wrong.
> >
> > Markus M
> > 
> > >
> > > cheers,
> > > Vero
> > >
> > >
> > >
> > > El jue., 16 may. 2019 a las 11:27, Moritz Lennert (<
> > [hidden email]>) escribió: 
> > >>
> > >> On 16/05/19 03:11, Veronica Andreo wrote: 
> > >> > Dear all,
> > >> >
> > >> > thanks for the answers...
> > >> >
> > >> > @Madi, I know, but that's how I got the data from a colleague
> > >> > using SaTScan to get cluster sizes.
> > >> >
> > >> > So, these "clusters" are 3, they are represented by circular
> > >> > areas 
> > and 2 
> > >> > of them overlap, and it is just fine that they overlap. I just
> > >> > want 
> > one 
> > >> > centroid per area to be able to get one value per original
> > >> > area.
> > >> >
> > >> > I tested your solution, @Micha (thanks much for your time!),
> > >> > but it gives me 4 values, and I need 3. Moreover, I can no
> > >> > longer recognize which polygon is which since v.db.select for
> > >> > layer 3 reports cats 
> > from 1 
> > >> > to 4, but d.vect shows something different (I'd assume 2/3 has
> > >> > become 4?). See screenshot below. 
> > >>
> > >> To see the cat values of layer 3 you have to indicate that layer
> > >> in the label_layer parameter (Labels tab).
> > >>
> > >> You can also see the correspondance between the two layers with
> > >>
> > >> v.category polys_3layers op=print layer=1,3
> > >> 
> > >> >
> > >> > So, is it possible somehow to keep my 3 original polygons? And
> > >> > how 
> > can I 
> > >> > get raster values for my original 3 polygons in GRASS? Or is
> > >> > it that this is not allowed by topology? 
> > >>
> > >> This depends on how you want to get raster values. If
> > >> v.what.rast at the location of the centroid is all you need than
> > >> Micha's solution should 
> > work. 
> > >>
> > >> If you need v.rast.stats then you either have to use Markus'
> > >> suggestion, or you use Micha's approach running v.rast.stats on
> > >> layer three and then use some magic to transform the output of
> > >> the above v.category call to a correspondance table between
> > >> layer. Something like this
> > >>
> > >> import grass.script as g
> > >> for feature in g.read_command('v.category',
> > >>                               input_='polys_3layers',
> > >>                               op='print',
> > >>                                layer=[3,1]).splitlines():
> > >>       cat1 = feature.split('|')[0]
> > >>       cats2 = feature.split('|')[1].split('/')
> > >>       for cat2 in cats2:
> > >>           print cat1, cat2
> > >>
> > >> This correspondance table could then be used to aggregate the
> > >> raster stats per (original) polygon in SQL. This could be the
> > >> easily put into an addon module.
> > >>
> > >> Moritz
> > >> 
> > > _______________________________________________
> > > grass-user mailing list
> > > [hidden email]
> > > https://lists.osgeo.org/mailman/listinfo/grass-user 
> > 



--
Département Géosciences, Environnement et Société Université Libre de
Bruxelles Bureau: S.DB.6.138
CP 130/03
Av. F.D. Roosevelt 50
1050 Bruxelles
Belgique

tél. + 32 2 650.68.12 / 68.11 (secr.)
fax  + 32 2 650.68.30


------------------------------

Message: 4
Date: Wed, 22 May 2019 20:06:22 -0300
From: Veronica Andreo <[hidden email]>
To: Moritz Lennert <[hidden email]>
Cc: Markus Metz <[hidden email]>, grass-user
        <[hidden email]>, Micha Silver <[hidden email]>
Subject: Re: [GRASS-user] importing and cleaning overlapping polygons
        that are supposed to overlap
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi Moritz

Thanks for the heads-up! Yes, for overlapping buffers v.rast.bufferstats
seems just perfect! IIUC, it does exactly what MM was explaining. Great!

Thanks again! And thanks Stefan for the add-on!

Vero


El mié., 22 may. 2019 19:30, Moritz Lennert <[hidden email]>
escribió:

> Le Wed, 22 May 2019 19:25:19 +0200,
> Veronica Andreo <[hidden email]> a écrit :
>
> > Hi Markus,
> >
> > Thanks for the explanation. It is possible to import topologically
> > incorrect vectors, but not create them within GRASS. So, as a
> > solution to my problem, I rather create (potentially overlapping)
> > buffers outside GRASS and import them with -c in v.in.ogr to use
> > v.rast.stats with no loss of areas, or follow the procedure that you
> > described earlier in the thread.
>
> You can also have a look at the v.rast.bufferstats addon.
>
> Moritz
>
> >
> > Cheers,
> > Vero
> >
> > El mié., 22 may. 2019 a las 18:48, Markus Metz (<
> > [hidden email]>) escribió:
> >
> > >
> > >
> > > On Wed, May 22, 2019 at 3:11 PM Veronica Andreo
> > > <[hidden email]> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > thanks again for your answers. I found an easier way, use -c flag
> > > > in
> > > v.in.ogr seems to not build topology of overlapping areas and then
> > > v.rast.stats does not complain.
> > > >
> > > > However, if one builds buffer areas for points within GRASS,
> > > > i.e., using
> > > v.buffer, the problem appears again when buffer areas overlap,
> > > which it's pretty common when creating buffers around neighbouring
> > > points. Then, to get zonal stats for those areas the approach
> > > suggested by Markus Metz should be followed. I believe it is a bit
> > > of an overkill for such a common task in GIS. Couldn't v.buffer
> > > have a -c flag as v.in.ogr so when overlapping of buffer areas is
> > > fine the topology is not built and one would get just one area per
> > > input point? Would that be possible?
> > >
> > > In short, no.
> > >
> > > The reason is that with a topological vector model, areas are
> > > constructed from boundaries and centroids. If you use the -c flag
> > > of v.in.ogr and there are polygons that overlap to a large degree
> > > or completely, it is not possible to find a centroid for each
> > > topological area, i.e. the overlapping input polygons can not be
> > > properly represented with a topological model when using the -c
> > > flag. As soon as you get incorrect boundaries and/or duplicate
> > > centroids when building topology, results are wrong.
> > >
> > > Markus M
> > >
> > > >
> > > > cheers,
> > > > Vero
> > > >
> > > >
> > > >
> > > > El jue., 16 may. 2019 a las 11:27, Moritz Lennert (<
> > > [hidden email]>) escribió:
> > > >>
> > > >> On 16/05/19 03:11, Veronica Andreo wrote:
> > > >> > Dear all,
> > > >> >
> > > >> > thanks for the answers...
> > > >> >
> > > >> > @Madi, I know, but that's how I got the data from a colleague
> > > >> > using SaTScan to get cluster sizes.
> > > >> >
> > > >> > So, these "clusters" are 3, they are represented by circular
> > > >> > areas
> > > and 2
> > > >> > of them overlap, and it is just fine that they overlap. I just
> > > >> > want
> > > one
> > > >> > centroid per area to be able to get one value per original
> > > >> > area.
> > > >> >
> > > >> > I tested your solution, @Micha (thanks much for your time!),
> > > >> > but it gives me 4 values, and I need 3. Moreover, I can no
> > > >> > longer recognize which polygon is which since v.db.select for
> > > >> > layer 3 reports cats
> > > from 1
> > > >> > to 4, but d.vect shows something different (I'd assume 2/3 has
> > > >> > become 4?). See screenshot below.
> > > >>
> > > >> To see the cat values of layer 3 you have to indicate that layer
> > > >> in the label_layer parameter (Labels tab).
> > > >>
> > > >> You can also see the correspondance between the two layers with
> > > >>
> > > >> v.category polys_3layers op=print layer=1,3
> > > >>
> > > >> >
> > > >> > So, is it possible somehow to keep my 3 original polygons? And
> > > >> > how
> > > can I
> > > >> > get raster values for my original 3 polygons in GRASS? Or is
> > > >> > it that this is not allowed by topology?
> > > >>
> > > >> This depends on how you want to get raster values. If
> > > >> v.what.rast at the location of the centroid is all you need than
> > > >> Micha's solution should
> > > work.
> > > >>
> > > >> If you need v.rast.stats then you either have to use Markus'
> > > >> suggestion, or you use Micha's approach running v.rast.stats on
> > > >> layer three and then use some magic to transform the output of
> > > >> the above v.category call to a correspondance table between
> > > >> layer. Something like this
> > > >>
> > > >> import grass.script as g
> > > >> for feature in g.read_command('v.category',
> > > >>                               input_='polys_3layers',
> > > >>                               op='print',
> > > >>                                layer=[3,1]).splitlines():
> > > >>       cat1 = feature.split('|')[0]
> > > >>       cats2 = feature.split('|')[1].split('/')
> > > >>       for cat2 in cats2:
> > > >>           print cat1, cat2
> > > >>
> > > >> This correspondance table could then be used to aggregate the
> > > >> raster stats per (original) polygon in SQL. This could be the
> > > >> easily put into an addon module.
> > > >>
> > > >> Moritz
> > > >>
> > > > _______________________________________________
> > > > grass-user mailing list
> > > > [hidden email]
> > > > https://lists.osgeo.org/mailman/listinfo/grass-user
> > >
>
>
>
> --
> Département Géosciences, Environnement et Société Université Libre de
> Bruxelles Bureau: S.DB.6.138
> CP 130/03
> Av. F.D. Roosevelt 50
> 1050 Bruxelles
> Belgique
>
> tél. + 32 2 650.68.12 / 68.11 (secr.)
> fax  + 32 2 650.68.30
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20190522/6f32132a/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user

------------------------------

End of grass-user Digest, Vol 157, Issue 33
*******************************************


--
Mehrdad Varedi

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.what.vect never ends, SQLite warning

Markus Neteler
Hi

Mehrdad Varedi <[hidden email]> schrieb am Di., 28. Mai 2019, 19:38:
Subject: Re: [GRASS-user] v.what.vect never ends, SQLite warning  

Hi Everyone,
I think I figured out the problem.
I found that the problem was only the size or number of attributes in the point layer when I dropped and reduced the attributes it worked very fast and with no problem.
I am interested to know the limitations in GRASS GIS in terms of the number of features or attributes, if there is any reference, please let me know.

Yes, please see


Best
Markus


_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user
Reply | Threaded
Open this post in threaded view
|

Re: v.what.vect never ends, SQLite warning

MehrdadVaredi
Thank you, Markus, for sharing.

The link you send indicates that the SQLite supports up to 2000 columns and the size can be 140TB. In my case, the number of columns was 60 that majority of them had REAL types. It had around 250K rows. My guess is that it is not the SQLite issue and something in between causes the problem. Maybe something on my computer or the way GRASS interacts with SQLite (probably some setting such as buffer size, on my computer etc. might be an issue)

If you are aware of any settings that I could play with, please let me know to speed up (read/write or Import / export) of layers in GRASS.

Kind regards,

Mehrdad
 

On Tue, May 28, 2019 at 1:51 PM Markus Neteler <[hidden email]> wrote:
Hi

Mehrdad Varedi <[hidden email]> schrieb am Di., 28. Mai 2019, 19:38:
Subject: Re: [GRASS-user] v.what.vect never ends, SQLite warning  

Hi Everyone,
I think I figured out the problem.
I found that the problem was only the size or number of attributes in the point layer when I dropped and reduced the attributes it worked very fast and with no problem.
I am interested to know the limitations in GRASS GIS in terms of the number of features or attributes, if there is any reference, please let me know.

Yes, please see


Best
Markus



--
Mehrdad Varedi

_______________________________________________
grass-user mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/grass-user