t.list slow

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

t.list slow

Radim Blazek-2
Hi,
I found t.list module to be quite slow, about 500ms. Simple "select *
from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
If I want to get list of more types for more mapsets (for example
mapsets which are not in SEARCH_PATH) it takes seconds.

Is there a chance to make it faster?

Radim
_______________________________________________
grass-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: t.list slow

Sören Gebbert
Hi Radim,
the reason why t.list and many other t.* modules are "slow" is the
initialization of the temporal framework. Several background processes
are started and the temporal databases are checked for each mapset in
the mapset searchpath.

You have to use the GRASS GIS temporal framework directly to make it
faster, so that the temporal framework is only initiated once and not
every time a t.* module is called.
I have implemented such a functionality here:
 https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739

The function from the temporal framework is: tgis.get_dataset_list()
This should be almost as fast as the direct sqlite interface.

Best regards
Soeren

2015-10-18 13:15 GMT+02:00 Radim Blazek <[hidden email]>:
> Hi,
> I found t.list module to be quite slow, about 500ms. Simple "select *
> from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
> If I want to get list of more types for more mapsets (for example
> mapsets which are not in SEARCH_PATH) it takes seconds.
>
> Is there a chance to make it faster?
>
> Radim
_______________________________________________
grass-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: t.list slow

Radim Blazek-2
On Sun, Oct 18, 2015 at 1:42 PM, Sören Gebbert
<[hidden email]> wrote:
> Hi Radim,
> the reason why t.list and many other t.* modules are "slow" is the
> initialization of the temporal framework. Several background processes
> are started and the temporal databases are checked for each mapset in
> the mapset searchpath.

The time is almost constant regardless how many mapsets are in
SEARCH_PATH and still about 500ms if SEARCH_PATH is empty so it must
be something else.

> You have to use the GRASS GIS temporal framework directly to make it
> faster, so that the temporal framework is only initiated once and not
> every time a t.* module is called.

I would prefer to use it directly but I need a C API. Could be some
basic functions like get_dataset_list() ported to C library? If not,
is it some trick, some environment variable or something like that to
avoid unnecessary initialization, which could make it faster?

Radim

> I have implemented such a functionality here:
>  https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739
>
> The function from the temporal framework is: tgis.get_dataset_list()
> This should be almost as fast as the direct sqlite interface.
>
> Best regards
> Soeren
>
> 2015-10-18 13:15 GMT+02:00 Radim Blazek <[hidden email]>:
>> Hi,
>> I found t.list module to be quite slow, about 500ms. Simple "select *
>> from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
>> If I want to get list of more types for more mapsets (for example
>> mapsets which are not in SEARCH_PATH) it takes seconds.
>>
>> Is there a chance to make it faster?
>>
>> Radim
_______________________________________________
grass-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-dev
Reply | Threaded
Open this post in threaded view
|

Re: t.list slow

Sören Gebbert
Hi Radim,


2015-10-18 14:08 GMT+02:00 Radim Blazek <[hidden email]>:

> On Sun, Oct 18, 2015 at 1:42 PM, Sören Gebbert
> <[hidden email]> wrote:
>> Hi Radim,
>> the reason why t.list and many other t.* modules are "slow" is the
>> initialization of the temporal framework. Several background processes
>> are started and the temporal databases are checked for each mapset in
>> the mapset searchpath.
>
> The time is almost constant regardless how many mapsets are in
> SEARCH_PATH and still about 500ms if SEARCH_PATH is empty so it must
> be something else.

My Python profiler shows me that the most time is spend by calling the
C-library access subprocess, some GRASS modules and in the temporal
database check.

>
>> You have to use the GRASS GIS temporal framework directly to make it
>> faster, so that the temporal framework is only initiated once and not
>> every time a t.* module is called.
>
> I would prefer to use it directly but I need a C API. Could be some
> basic functions like get_dataset_list() ported to C library? If not,
> is it some trick, some environment variable or something like that to
> avoid unnecessary initialization, which could make it faster?

Unfortunately not. The initialization is important and cant be
skipped, since a lot of functionality depends on it.
Almost 95% of the GRASS GIS temporal framework is implemented in
Python, hence all of the functionality.
There is no C/C++ API available to access the Python implementation.

A work around would be to implement a Python script that calls all the
Python API parts to generate the desired text output, so substitute
several t.list and t.rast.list calls.

Best regards
Soeren

> Radim
>
>> I have implemented such a functionality here:
>>  https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739
>>
>> The function from the temporal framework is: tgis.get_dataset_list()
>> This should be almost as fast as the direct sqlite interface.
>>
>> Best regards
>> Soeren
>>
>> 2015-10-18 13:15 GMT+02:00 Radim Blazek <[hidden email]>:
>>> Hi,
>>> I found t.list module to be quite slow, about 500ms. Simple "select *
>>> from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
>>> If I want to get list of more types for more mapsets (for example
>>> mapsets which are not in SEARCH_PATH) it takes seconds.
>>>
>>> Is there a chance to make it faster?
>>>
>>> Radim
_______________________________________________
grass-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/grass-dev