Upgrade dependencies

[lewismc]

Hi folks, I came across this project right now. I just release Nutch 2.3.1 a few weeks back and also Gora 0.7 is about to be released so you will be able to take advantage of some good updates and big fixes. What in particular have you been doing with Nutch? This looks like a cool search stack you have going on.

[etriviere]

Hi Lewis, thanks for your message.
We have used Nutch and Gora in the context of a research project named LEADS, with the goal of building a proof-of-concept shared search/processing stack for public web data. Our work is described on our website http://www.leads-project.eu ; the project ended a few months ago actually.
One of our main goals was to support our stack on a collection of small data centers rather than on a single larger one; one of our partners is a German company that builds exactly this kind of so-called 'micro-clouds' (https://www.cloudandheat.com/en/index.html). One of our key layers is called Infinispan Ensemble. It unifies multiple instances of the Infinispan KVS in different sites as a single store with possibilities for explicit data placement, replication across sites, etc. Unicrawl uses Nutch to crawl from multiple sites simultaneously rather than from a single one, and leverages Ensemble for this--this is one of the reasons we built support for Gora in Infinispan (https://github.com/leads-project/gora-infinispan). If you are interested in the details, we published a paper in the IEEE Cloud conference last year (http://members.unine.ch/etienne.riviere/publications/LeFeScSuRiFe-CLOUD-15.pdf).
We will definitely check the new versions of Nutch and Gora and see if we can update easily; but since we do not have funding at the moment on this I cannot guarantee when this will happen but we will make sure to keep you and the Nutch community in the loop.
Etienne -- lead of LEADS.

[lewismc]

This is great. I asked over on the infinite span gora module if you guys
would be interested in bringing the code over and into Gora master.
It would have a better chance of being maintained k would think.
Thanks for te link to your paper I'll read it at my lunch.

On Thursday, February 11, 2016, Etienne Riviere notifications@github.com
wrote:

Hi Lewis, thanks for your message.
We have used Nutch and Gora in the context of a research project named
LEADS, with the goal of building a proof-of-concept shared
search/processing stack for public web data. Our work is described on our
website http://www.leads-project.eu ; the project ended a few months ago
actually.
One of our main goals was to support our stack on a collection of small
data centers rather than on a single larger one; one of our partners is a
German company that builds exactly this kind of so-called 'micro-clouds' (
https://www.cloudandheat.com/en/index.html). One of our key layers is
called Infinispan Ensemble. It unifies multiple instances of the Infinispan
KVS in different sites as a single store with possibilities for explicit
data placement, replication across sites, etc. Unicrawl uses Nutch to crawl
from multiple sites simultaneously rather than from a single one, and
leverages Ensemble for this--this is one of the reasons we built support
for Gora in Infinispan (https://github.com/leads-project/gora-infinispan).
If you are interested in the details, we published a paper in the IEEE
Cloud conference last year (
http://members.unine.ch/etienne.riviere/publications/LeFeScSuRiFe-CLOUD-15.pdf
http://members.unine.ch/etien%20ne.rivie%20re/publications/LeFeScSuRiFe-CLOUD-15.pdf
).
We will definitely check the new versions of Nutch and Gora and see if we
can update easily; but since we do not have funding at the moment on this I
cannot guarantee when this will happen but we will make sure to keep you
and the Nutch community in the loop.
Etienne -- lead of LEADS.

—
Reply to this email directly or view it on GitHub
#2 (comment)
.

Lewis