Quantcast

Move to a new search engine

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Move to a new search engine

Francois Prunayre
Hi all, so far we've been working and investigating a move from Lucene to Solr (cf. https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR, https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).

More recently, I've been checking that the work done on Solr and WFS indexing was also feasible with Elasticsearch (ES) (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new projects having dashboard requirements, I'm about to start moving some GeoNetwork features (ie. search statistics in the admin) to Elasticsearch+Kibana.

Having more and more projects and people interested on ES, it might be more relevant to move from Lucene to ES. So I would like to hear from you if there is any ongoing or coming projects which might help on this move from Lucene to a new search engine in order to try to find synergies on that topic for the next major GeoNetwork release.

Looking forward your feedback.

Francois
 

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

María Arias de Reyna-2
Hi,

To me, both SOLR and ES are good options:

http://solr-vs-elasticsearch.com/

I would say that the community in general is moving to ES, but that
may be only my perception, SOLR has (had?) a lot of people using it.
In both cases it can be integrated with GeoNetwork so we don't need an
external installation, right?

If that's the case, you have good feelings for ES and already funded
steps towards it, I have no objection.


On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
<[hidden email]> wrote:

> Hi all, so far we've been working and investigating a move from Lucene to
> Solr (cf.
> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>
> More recently, I've been checking that the work done on Solr and WFS
> indexing was also feasible with Elasticsearch (ES)
> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
> projects having dashboard requirements, I'm about to start moving some
> GeoNetwork features (ie. search statistics in the admin) to
> Elasticsearch+Kibana.
>
> Having more and more projects and people interested on ES, it might be more
> relevant to move from Lucene to ES. So I would like to hear from you if
> there is any ongoing or coming projects which might help on this move from
> Lucene to a new search engine in order to try to find synergies on that
> topic for the next major GeoNetwork release.
>
> Looking forward your feedback.
>
> Francois
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Jeroen Ticheler - GeoCat
Hi François and Maria,
From talks I had in Boston last year with Steve McDonald who knows the communities well, my impression was that SOLR was actually the way to go coming from Lucene. It would be good if you could discuss with him to hear his reasoning/ knowledge on this topic.
Cheers,
Jeroen

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer, MapServer, PostGIS and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jeroen Ticheler
GeoCat bv
Veenderweg 13
6721 WD Bennekom
Tel: +31 (0)6 81286572
http://geocat.net

> Op 17 nov. 2016 om 16:06 heeft María Arias de Reyna <[hidden email]> het volgende geschreven:
>
> Hi,
>
> To me, both SOLR and ES are good options:
>
> http://solr-vs-elasticsearch.com/
>
> I would say that the community in general is moving to ES, but that
> may be only my perception, SOLR has (had?) a lot of people using it.
> In both cases it can be integrated with GeoNetwork so we don't need an
> external installation, right?
>
> If that's the case, you have good feelings for ES and already funded
> steps towards it, I have no objection.
>
>
> On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
> <[hidden email]> wrote:
>> Hi all, so far we've been working and investigating a move from Lucene to
>> Solr (cf.
>> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
>> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>>
>> More recently, I've been checking that the work done on Solr and WFS
>> indexing was also feasible with Elasticsearch (ES)
>> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
>> projects having dashboard requirements, I'm about to start moving some
>> GeoNetwork features (ie. search statistics in the admin) to
>> Elasticsearch+Kibana.
>>
>> Having more and more projects and people interested on ES, it might be more
>> relevant to move from Lucene to ES. So I would like to hear from you if
>> there is any ongoing or coming projects which might help on this move from
>> Lucene to a new search engine in order to try to find synergies on that
>> topic for the next major GeoNetwork release.
>>
>> Looking forward your feedback.
>>
>> Francois
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
> ------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Francois Prunayre
Thanks for the feedback on this Steve.

2016-11-18 2:32 GMT+01:00 Steve McDonald <[hidden email]>:

Good Evening,

I don't know the communities well but I have two observations.  I went to the Lucene Revolution conference in Boston a couple months ago.  I think the ES is about 3 times bigger. 

Regarding dashboards, LucidWorks has developed Fusion as open source (https://doc.lucidworks.com/fusion/2.4/Dashboards/Dashboards-Getting-Started.html) however it doesn't connect to Solr/Lucene.  You have to run LucidWork's commercial solution built on Solr/Lucene.  I talked to the Fusion lead about the lagging dashboard solutions in S/L (kibana) and suggested Fusion could fill that space.  He said there's no plans to support directly connecting Fusion to S/L. 

Contributing to Banana & looking after silk in the last months, it looks like there is not much (open) progress in that area. 

 
I know when David Smiley extends spatial search in Lucene it also extends the Solr API.  Is someone else extending the ES API to make the new features visible?

On the spatial side, we have in both solutions the minimum required for having standard CSW spatial filter capability. The main issue I've in that area for now is in indexing performance mainly on line geometry type depending on the spatial resolution set in the index (and this happens in both ES & Solr).

Francois



 

Steve

On Thu, Nov 17, 2016 at 12:13 PM, Jeroen Ticheler <[hidden email]> wrote:
Hi François and Maria,
>From talks I had in Boston last year with Steve McDonald who knows the communities well, my impression was that SOLR was actually the way to go coming from Lucene. It would be good if you could discuss with him to hear his reasoning/ knowledge on this topic.
Cheers,
Jeroen

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer, MapServer, PostGIS and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jeroen Ticheler
GeoCat bv
Veenderweg 13
6721 WD Bennekom
Tel: <a href="tel:%2B31%20%280%296%2081286572" value="+31681286572" target="_blank">+31 (0)6 81286572
http://geocat.net

> Op 17 nov. 2016 om 16:06 heeft María Arias de Reyna <[hidden email]> het volgende geschreven:
>
> Hi,
>
> To me, both SOLR and ES are good options:
>
> http://solr-vs-elasticsearch.com/
>
> I would say that the community in general is moving to ES, but that
> may be only my perception, SOLR has (had?) a lot of people using it.
> In both cases it can be integrated with GeoNetwork so we don't need an
> external installation, right?
>
> If that's the case, you have good feelings for ES and already funded
> steps towards it, I have no objection.
>
>
> On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
> <[hidden email]> wrote:
>> Hi all, so far we've been working and investigating a move from Lucene to
>> Solr (cf.
>> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
>> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>>
>> More recently, I've been checking that the work done on Solr and WFS
>> indexing was also feasible with Elasticsearch (ES)
>> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
>> projects having dashboard requirements, I'm about to start moving some
>> GeoNetwork features (ie. search statistics in the admin) to
>> Elasticsearch+Kibana.
>>
>> Having more and more projects and people interested on ES, it might be more
>> relevant to move from Lucene to ES. So I would like to hear from you if
>> there is any ongoing or coming projects which might help on this move from
>> Lucene to a new search engine in order to try to find synergies on that
>> topic for the next major GeoNetwork release.
>>
>> Looking forward your feedback.
>>
>> Francois
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
> ------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Jose Garcia
Hi Francois

For me any of the options look fine. You have work more with Banana and know the status of the project, so if you consider that ES is better to get integrated a dashboard for statistics, that is maintained (like seem Kibana) thats fair enough to me. 

As indicates Jeroen, seem Lucene/SOLR communities joined efforts to go in the same direction, but if related projects like Banana are not properly maintained, I think we should choose what we think is better and for that I trust in your experience.

About joining efforts, I don't know any running project in my company that is able to join. I think we can check between our customers, but probably would be more efficient if we create a list of clear benefits that such a change will offer to customers. I mean no customer will want to join just for changing the search/indexing engine from Lucene to another stuff if there're no obvious benefits they can see about this. I know there're potentially quite, starting with steps towards horizontal scaling, integrating of dashboards for statistics, etc. But maybe we can work a bit more to elaborate this a bit more.

But even in the case that in the end we would not have projects to join to this effort, at least personally I would like to be involved in the design of this feature. I think should be designed properly defining properly the interfaces so a new search/index engine can be added later without having to re-code all the core code (like happens now with the current Lucene code). Maybe the SOLR branch you created some months ago for testing SOLR change manages this nicely (I haven't got a chance to check yet), so in that case, forget these comments.

Regards,
Jose García


On Fri, Nov 18, 2016 at 8:17 AM, Francois Prunayre <[hidden email]> wrote:
Thanks for the feedback on this Steve.

2016-11-18 2:32 GMT+01:00 Steve McDonald <[hidden email]>:

Good Evening,

I don't know the communities well but I have two observations.  I went to the Lucene Revolution conference in Boston a couple months ago.  I think the ES is about 3 times bigger. 

Regarding dashboards, LucidWorks has developed Fusion as open source (https://doc.lucidworks.com/fusion/2.4/Dashboards/Dashboards-Getting-Started.html) however it doesn't connect to Solr/Lucene.  You have to run LucidWork's commercial solution built on Solr/Lucene.  I talked to the Fusion lead about the lagging dashboard solutions in S/L (kibana) and suggested Fusion could fill that space.  He said there's no plans to support directly connecting Fusion to S/L. 

Contributing to Banana & looking after silk in the last months, it looks like there is not much (open) progress in that area. 

 
I know when David Smiley extends spatial search in Lucene it also extends the Solr API.  Is someone else extending the ES API to make the new features visible?

On the spatial side, we have in both solutions the minimum required for having standard CSW spatial filter capability. The main issue I've in that area for now is in indexing performance mainly on line geometry type depending on the spatial resolution set in the index (and this happens in both ES & Solr).

Francois



 

Steve

On Thu, Nov 17, 2016 at 12:13 PM, Jeroen Ticheler <[hidden email]> wrote:
Hi François and Maria,
>From talks I had in Boston last year with Steve McDonald who knows the communities well, my impression was that SOLR was actually the way to go coming from Lucene. It would be good if you could discuss with him to hear his reasoning/ knowledge on this topic.
Cheers,
Jeroen

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer, MapServer, PostGIS and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jeroen Ticheler
GeoCat bv
Veenderweg 13
6721 WD Bennekom
Tel: <a href="tel:%2B31%20%280%296%2081286572" value="+31681286572" target="_blank">+31 (0)6 81286572
http://geocat.net

> Op 17 nov. 2016 om 16:06 heeft María Arias de Reyna <[hidden email]> het volgende geschreven:
>
> Hi,
>
> To me, both SOLR and ES are good options:
>
> http://solr-vs-elasticsearch.com/
>
> I would say that the community in general is moving to ES, but that
> may be only my perception, SOLR has (had?) a lot of people using it.
> In both cases it can be integrated with GeoNetwork so we don't need an
> external installation, right?
>
> If that's the case, you have good feelings for ES and already funded
> steps towards it, I have no objection.
>
>
> On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
> <[hidden email]> wrote:
>> Hi all, so far we've been working and investigating a move from Lucene to
>> Solr (cf.
>> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
>> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>>
>> More recently, I've been checking that the work done on Solr and WFS
>> indexing was also feasible with Elasticsearch (ES)
>> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
>> projects having dashboard requirements, I'm about to start moving some
>> GeoNetwork features (ie. search statistics in the admin) to
>> Elasticsearch+Kibana.
>>
>> Having more and more projects and people interested on ES, it might be more
>> relevant to move from Lucene to ES. So I would like to hear from you if
>> there is any ongoing or coming projects which might help on this move from
>> Lucene to a new search engine in order to try to find synergies on that
>> topic for the next major GeoNetwork release.
>>
>> Looking forward your feedback.
>>
>> Francois
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
> ------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



--
Vriendelijke groeten / Kind regards,

Jose García


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="font-family:helvetica,arial,sans-serif" target="_blank">+31 (0)318 416664

  

Please consider the environment before printing this email.

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Francois Prunayre
Hola Jose,

2016-11-18 8:37 GMT+01:00 Jose Garcia <[hidden email]>:
Hi Francois

For me any of the options look fine. You have work more with Banana and know the status of the project, so if you consider that ES is better to get integrated a dashboard for statistics, that is maintained (like seem Kibana) thats fair enough to me. 

As indicates Jeroen, seem Lucene/SOLR communities joined efforts to go in the same direction, but if related projects like Banana are not properly maintained, I think we should choose what we think is better and for that I trust in your experience. 

About joining efforts, I don't know any running project in my company that is able to join. I think we can check between our customers, but probably would be more efficient if we create a list of clear benefits that such a change will offer to customers. I mean no customer will want to join just for changing the search/indexing engine from Lucene to another stuff if there're no obvious benefits they can see about this. I know there're potentially quite, starting with steps towards horizontal scaling, integrating of dashboards for statistics, etc. But maybe we can work a bit more to elaborate this a bit more.

The POC we made with Patrick highlights some of the advantages
 
Thanks for the feedback.

Francois


But even in the case that in the end we would not have projects to join to this effort, at least personally I would like to be involved in the design of this feature. I think should be designed properly defining properly the interfaces so a new search/index engine can be added later without having to re-code all the core code (like happens now with the current Lucene code). Maybe the SOLR branch you created some months ago for testing SOLR change manages this nicely (I haven't got a chance to check yet), so in that case, forget these comments.

Regards,
Jose García


On Fri, Nov 18, 2016 at 8:17 AM, Francois Prunayre <[hidden email]> wrote:
Thanks for the feedback on this Steve.

2016-11-18 2:32 GMT+01:00 Steve McDonald <[hidden email]>:

Good Evening,

I don't know the communities well but I have two observations.  I went to the Lucene Revolution conference in Boston a couple months ago.  I think the ES is about 3 times bigger. 

Regarding dashboards, LucidWorks has developed Fusion as open source (https://doc.lucidworks.com/fusion/2.4/Dashboards/Dashboards-Getting-Started.html) however it doesn't connect to Solr/Lucene.  You have to run LucidWork's commercial solution built on Solr/Lucene.  I talked to the Fusion lead about the lagging dashboard solutions in S/L (kibana) and suggested Fusion could fill that space.  He said there's no plans to support directly connecting Fusion to S/L. 

Contributing to Banana & looking after silk in the last months, it looks like there is not much (open) progress in that area. 

 
I know when David Smiley extends spatial search in Lucene it also extends the Solr API.  Is someone else extending the ES API to make the new features visible?

On the spatial side, we have in both solutions the minimum required for having standard CSW spatial filter capability. The main issue I've in that area for now is in indexing performance mainly on line geometry type depending on the spatial resolution set in the index (and this happens in both ES & Solr).

Francois



 

Steve

On Thu, Nov 17, 2016 at 12:13 PM, Jeroen Ticheler <[hidden email]> wrote:
Hi François and Maria,
>From talks I had in Boston last year with Steve McDonald who knows the communities well, my impression was that SOLR was actually the way to go coming from Lucene. It would be good if you could discuss with him to hear his reasoning/ knowledge on this topic.
Cheers,
Jeroen

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer, MapServer, PostGIS and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jeroen Ticheler
GeoCat bv
Veenderweg 13
6721 WD Bennekom
Tel: <a href="tel:%2B31%20%280%296%2081286572" value="+31681286572" target="_blank">+31 (0)6 81286572
http://geocat.net

> Op 17 nov. 2016 om 16:06 heeft María Arias de Reyna <[hidden email]> het volgende geschreven:
>
> Hi,
>
> To me, both SOLR and ES are good options:
>
> http://solr-vs-elasticsearch.com/
>
> I would say that the community in general is moving to ES, but that
> may be only my perception, SOLR has (had?) a lot of people using it.
> In both cases it can be integrated with GeoNetwork so we don't need an
> external installation, right?
>
> If that's the case, you have good feelings for ES and already funded
> steps towards it, I have no objection.
>
>
> On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
> <[hidden email]> wrote:
>> Hi all, so far we've been working and investigating a move from Lucene to
>> Solr (cf.
>> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
>> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>>
>> More recently, I've been checking that the work done on Solr and WFS
>> indexing was also feasible with Elasticsearch (ES)
>> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
>> projects having dashboard requirements, I'm about to start moving some
>> GeoNetwork features (ie. search statistics in the admin) to
>> Elasticsearch+Kibana.
>>
>> Having more and more projects and people interested on ES, it might be more
>> relevant to move from Lucene to ES. So I would like to hear from you if
>> there is any ongoing or coming projects which might help on this move from
>> Lucene to a new search engine in order to try to find synergies on that
>> topic for the next major GeoNetwork release.
>>
>> Looking forward your feedback.
>>
>> Francois
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
> ------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



--
Vriendelijke groeten / Kind regards,

Jose García


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="font-family:helvetica,arial,sans-serif" target="_blank">+31 (0)318 416664

  

Please consider the environment before printing this email.


------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Jose Garcia
Hi Francois

Thats great, thanks for pointing the document.

Regards,
Jose García

On Fri, Nov 18, 2016 at 8:48 AM, Francois Prunayre <[hidden email]> wrote:
Hola Jose,

2016-11-18 8:37 GMT+01:00 Jose Garcia <[hidden email]>:
Hi Francois

For me any of the options look fine. You have work more with Banana and know the status of the project, so if you consider that ES is better to get integrated a dashboard for statistics, that is maintained (like seem Kibana) thats fair enough to me. 

As indicates Jeroen, seem Lucene/SOLR communities joined efforts to go in the same direction, but if related projects like Banana are not properly maintained, I think we should choose what we think is better and for that I trust in your experience. 

About joining efforts, I don't know any running project in my company that is able to join. I think we can check between our customers, but probably would be more efficient if we create a list of clear benefits that such a change will offer to customers. I mean no customer will want to join just for changing the search/indexing engine from Lucene to another stuff if there're no obvious benefits they can see about this. I know there're potentially quite, starting with steps towards horizontal scaling, integrating of dashboards for statistics, etc. But maybe we can work a bit more to elaborate this a bit more.

The POC we made with Patrick highlights some of the advantages
 
Thanks for the feedback.

Francois


But even in the case that in the end we would not have projects to join to this effort, at least personally I would like to be involved in the design of this feature. I think should be designed properly defining properly the interfaces so a new search/index engine can be added later without having to re-code all the core code (like happens now with the current Lucene code). Maybe the SOLR branch you created some months ago for testing SOLR change manages this nicely (I haven't got a chance to check yet), so in that case, forget these comments.

Regards,
Jose García


On Fri, Nov 18, 2016 at 8:17 AM, Francois Prunayre <[hidden email]> wrote:
Thanks for the feedback on this Steve.

2016-11-18 2:32 GMT+01:00 Steve McDonald <[hidden email]>:

Good Evening,

I don't know the communities well but I have two observations.  I went to the Lucene Revolution conference in Boston a couple months ago.  I think the ES is about 3 times bigger. 

Regarding dashboards, LucidWorks has developed Fusion as open source (https://doc.lucidworks.com/fusion/2.4/Dashboards/Dashboards-Getting-Started.html) however it doesn't connect to Solr/Lucene.  You have to run LucidWork's commercial solution built on Solr/Lucene.  I talked to the Fusion lead about the lagging dashboard solutions in S/L (kibana) and suggested Fusion could fill that space.  He said there's no plans to support directly connecting Fusion to S/L. 

Contributing to Banana & looking after silk in the last months, it looks like there is not much (open) progress in that area. 

 
I know when David Smiley extends spatial search in Lucene it also extends the Solr API.  Is someone else extending the ES API to make the new features visible?

On the spatial side, we have in both solutions the minimum required for having standard CSW spatial filter capability. The main issue I've in that area for now is in indexing performance mainly on line geometry type depending on the spatial resolution set in the index (and this happens in both ES & Solr).

Francois



 

Steve

On Thu, Nov 17, 2016 at 12:13 PM, Jeroen Ticheler <[hidden email]> wrote:
Hi François and Maria,
>From talks I had in Boston last year with Steve McDonald who knows the communities well, my impression was that SOLR was actually the way to go coming from Lucene. It would be good if you could discuss with him to hear his reasoning/ knowledge on this topic.
Cheers,
Jeroen

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer, MapServer, PostGIS and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jeroen Ticheler
GeoCat bv
Veenderweg 13
6721 WD Bennekom
Tel: <a href="tel:%2B31%20%280%296%2081286572" value="+31681286572" target="_blank">+31 (0)6 81286572
http://geocat.net

> Op 17 nov. 2016 om 16:06 heeft María Arias de Reyna <[hidden email]> het volgende geschreven:
>
> Hi,
>
> To me, both SOLR and ES are good options:
>
> http://solr-vs-elasticsearch.com/
>
> I would say that the community in general is moving to ES, but that
> may be only my perception, SOLR has (had?) a lot of people using it.
> In both cases it can be integrated with GeoNetwork so we don't need an
> external installation, right?
>
> If that's the case, you have good feelings for ES and already funded
> steps towards it, I have no objection.
>
>
> On Thu, Nov 17, 2016 at 3:56 PM, Francois Prunayre
> <[hidden email]> wrote:
>> Hi all, so far we've been working and investigating a move from Lucene to
>> Solr (cf.
>> https://github.com/geonetwork/core-geonetwork/wiki/WFS-Filters-based-on-WFS-indexing-with-SOLR,
>> https://github.com/geonetwork/core-geonetwork/wiki/Moving-to-a-search-server:-Why-&-how-%3F).
>>
>> More recently, I've been checking that the work done on Solr and WFS
>> indexing was also feasible with Elasticsearch (ES)
>> (https://github.com/geonetwork/core-geonetwork/pull/1745). Also on some new
>> projects having dashboard requirements, I'm about to start moving some
>> GeoNetwork features (ie. search statistics in the admin) to
>> Elasticsearch+Kibana.
>>
>> Having more and more projects and people interested on ES, it might be more
>> relevant to move from Lucene to ES. So I would like to hear from you if
>> there is any ongoing or coming projects which might help on this move from
>> Lucene to a new search engine in order to try to find synergies on that
>> topic for the next major GeoNetwork release.
>>
>> Looking forward your feedback.
>>
>> Francois
>>
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
> ------------------------------------------------------------------------------
> _______________________________________________
> GeoNetwork-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



--
Vriendelijke groeten / Kind regards,

Jose García


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="font-family:helvetica,arial,sans-serif" target="_blank">+31 (0)318 416664

  

Please consider the environment before printing this email.




--
Vriendelijke groeten / Kind regards,

Jose García


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="font-family:Helvetica,Arial,sans-serif" target="_blank">+31 (0)318 416664

  

Please consider the environment before printing this email.

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move to a new search engine

Paul van Genuchten
In reply to this post by Francois Prunayre
hi, everywhere around me i hear people whisper Elastic Search, so seems a very valuable asset to consider. However when reading through http://solr-vs-elasticsearch.com/ this paragraph stroke me:

"7. Overall from working with clients as a Solr/Elasticsearch consultant, I've found that developer preferences tend to end up along language party lines: if you're a Java/c# developer, you'll be pretty happy with Solr. If you live in Javascript or Ruby, you'll probably love Elasticsearch. If you're on Python or PHP, you'll probably be fine with either.”

in combination with:

3. I find Elasticsearch's documentation to be pretty awful. It doesn't help that some examples in the documentation are written in YAML and others in JSON. I wrote a ES code parser once to auto-generate documentation from Elasticsearch's source and found a number of discrepancies between code and what's documented on the website, not to mention a number of undocumented/alternative ways to specify the same config key.

We have grown to be more javascript oriented in recent years, with the switch to angular and swagger api. However we are also quite java/xslt oriented. For me a switch to ES would be a choice in the line of more javascript oriented and Solr keep more on the java/xslt.

my 2 cents, Paul

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Loading...