elasticsearch-carrot2

The clustering plugin attempts to automatically group together similar "documents" and assign human-readable labels to these groups. The clusters can be thought-of as "dynamic facets" generated for each unique query and set of search result hits. Take a look at the Carrot² demo page to glimpse at how this can be used in practice.

Each document passed for clustering is composed of several logical parts: the document's identifier, its origin URL, title and the main content. Only the identifier field is mandatory, everything else is optional but at least one of these fields will be required to make the clustering process reasonable.

Documents indexed in ElasticSearch do not have to follow any predefined schema so actual fields of a JSON document need to be mapped to the logical layout required by the clustering plugin. An example mapping can look as illustrated in the figure below:

Note that two document fields are mapped to the title. This is not an error, any number of fields can be mapped to either the title or the content—the content of those fields will be concatenated and used for clustering.

The logical fields can also be filled with generated content, for example by applying the highlighter to the document's fields. This can be useful to decrease the amount of text passed to the clustering algorithm (improves performance) or to make the clustered content more query-specific (this typically clusters better). REST examples below demonstrate the API for field mapping in details.

The Java API for clustering search results is fully functional and works the magic behind all REST requests described in the subsequent part of this document. For API examples, see the source code of the plugin at github, especially the unit and integration tests.

The clustering plugin binds itself to the following URI schemes (GET or POST):

• /_search_with_clusters
• /{index}/_search_with_clusters
• /{index}/{type}/_search_with_clusters

The index and type URI segments implicitly bind the search request part of the message to a given index and document type, exactly as in the search request API.

A clustering request is a JSON document composed of the following sections:

search_request

required The search request to fetch documents to be clustered. This section follows exactly what the search DSL specifies, including all optional bells and whistles such as sorting, filtering, query DSL, highlighter, etc.

query_hint

required This is a string attribute specifying query terms which were used to fetch the matching documents. This hint helps the clustering algorithm to avoid trivial clusters around the query terms. Typically the query terms hint will be identical to what the user typed in the search box. If possible, it should be pruned from any boolean or search-engine specific operators which could affect the clustering process. The query hint is obligatory but may be an empty string.

field_mapping

required Defines how to map actual fields of the documents matching the search_request to logical fields of the documents to be clustered. The value should be a hash where keys indicate logical document fields and values are arrays with field source specifications (content of fields defined by these specifications is concatenated). For example this is a valid field mapping specification:

{
  "url":     [_source.urlSource],
  "title":   [fields.subject],
  "content": [_source.abstract, highlight.main]
}

Any of the following logical field names are valid:

url

The URL of the document.

title

The title of the document.

content

The main body (content) of the document.

A field source specification defines where the value is taken from: the search hit's fields, stored document's content, or from the highlighter's output. The syntax of field source specification is as follows:

fields.{fieldname}

Defines a search hit's field (stored field or field reparsed from source document but requested and returned in the search request).

highlight.{fieldname}

Defines a search hit's highlighted field. The highlighter output must also be configured properly in the search request (see field mapping example).

_source.{fieldname}

Defines a source document's field (top-level property of the json document). This will reparse the source document and fetch the appropriate value from there.

algorithm

optional Defines which clustering component (algorithm) should be used for clustering. Names of all built-in clustering algorithms are logged at startup. If not present, the default algorithm is used.

attributes

optional A map of key-value attributes overriding the default algorithm settings per-query (runtime attributes in Carrot² parlance). Typically the default settings are overridden using init-time XML configuration files.

The response format is identical to a plain search request response, with extra properties presented in the schematic output below.

{
  /* Typical search response fields. */
  "hits": { /* ... */ },

  /* Clustering response fields. */
  "clusters": [
    /* Each cluster is defined by the following. */
    {
      "id":    /* identifier */,
      "score": /* numeric score */,
      "label": /* primary cluster label */,
      "other_topics": /* if present, and true, this cluster groups
                         unrelated documents (no related topics) */,
      "phrases": [
        /* cluster label array, will include primary. */
      ],
      "documents": [
        /* This cluster's document ID references.
           May be undefined if this cluster holds sub-clusters only. */
      ],
      "clusters": [
        /* This cluster's subclusters (recursive objects of the same
           structure). May be undefined if this cluster holds documents only. */
      ],
    },
    /* ...more clusters */
  ],
  "info": {
    /* Additional information about the clustering: execution times,
       the algorithm used, etc. */
  }
}

Given the following function that recursively dumps clusters:

window.dumpClusters = function(arr, clusters, indent) {
  indent = indent ? indent : "";
  clusters.forEach(function(cluster) {
    arr.push(
        indent + cluster.label
        + (cluster.documents ? " [" + cluster.documents.length + " documents]"   : "")
        + (cluster.clusters  ? " [" + cluster.clusters.length  + " subclusters]" : ""));
    if (cluster.clusters) {
      dumpClusters(arr, cluster.clusters, indent + "  ");
    }
  });
  return arr;
}

We can dump all cluster labels of a clustering request with the following snippet of javascript:

var request = {
  "search_request": {
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "title": ["_source.title"],
    "content": ["_source.content"]
  }
};

$.post("/test/test/_search_with_clusters",
  JSON.stringify(request),
  function(response) {
    $("#cluster-list-result").text(
      dumpClusters([], response.clusters).join("\n"));
});
            

The output will vary depending on the choice of clustering algorithm (and particular documents that made it to the hit list if search is not deterministic). The following example shows a pseudo-clustering algorithm that uses the logical url field to produce clusters based on the components of each document's domain.

var request = {
  "search_request": {
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "url": ["_source.url"]
  },
  "algorithm": "byurl"
};

$.post("/test/test/_search_with_clusters",
  JSON.stringify(request), function(response) {
    $("#cluster-list-result2").text(
      dumpClusters([], response.clusters).join("\n"));
});
            

A full response for a clustering request can look as shown below (note the difference in field mapping in this example).

var request = {
  "search_request": {
    "fields": [ "title", "content" ],
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "title": ["fields.title"],
    "content": ["fields.content"]
  }
};

$.post("/test/test/_search_with_clusters",
  JSON.stringify(request),
  function(response) {
    $("#simple-request-result").text(
      JSON.stringify(response, false, "  "));
});
            

The field mapping section provides a connection between actual data and logical data to cluster on. The different field mapping sources (_source.*, highlight.* and fields.*) can be used to tune the amount of data returned in the request and the amount of text passed to the clustering engine (and in result the required processing cost).

• The _source.* mapping takes data directly from the source document, if _source is available as part of the search hit. The content pointed to by this mapping is not returned as part of the request, it is only used internally for clustering.

  Warning! The _source may not be published by ElasticSearch's internal search infrastructure, in particular, when only selected fields are filtered, the source will not be available. This issue should be addressed in the future (ES API constraint).

• The fields.* mapping must be accompanied by appropriate fields declaration in the search request. The content of those fields is returned back with the request and thus can be used for display purposes (for example to show each document's title).

• The highlight.* mapping also must be accompanied by appropriate highlight declaration in the search request. The highlighting request specification can be used to tune the amount of content passed to the clustering engine (the number of fragments, their width, boundary, etc.). This is of particular importance when the documents are long (full content is stored): it is typical that clustering algorithms run perceptually "better" when focused on the context surrounding the query, rather than when presented with full content of all documents. Any highlighted content will also be returned as part of the request.

Compare the output for the following requests and note the differences outlined above.

var request = {
  "search_request": {
    "fields": ["url", "title", "content"],
    "query": {"match" : { "_all": "computer" }},
    "size": 100
  },

  "query_hint": "computer",
  "field_mapping": {
    "url":     ["fields.url"],
    "title":   ["fields.title"],
    "content": ["fields.content"]
  }
};

$.post("/test/test/_search_with_clusters", JSON.stringify(request), function(response) {
  $("#fields-request").text(JSON.stringify(response, false, "  "));
});
            

var request = {
  "search_request": {
    "fields": ["url", "title"],
    "query": {"match" : { "_all": "computer" }},
    "size": 100,
    "highlight" : {
      "pre_tags" :  ["", ""],
      "post_tags" : ["", ""],
      "fields" : {
        "content" : { "fragment_size" : 100, "number_of_fragments" : 2 }
      }
    },
  },

  "query_hint": "computer",
  "field_mapping": {
    "url":     ["fields.url"],
    "title":   ["fields.title"],
    "content": ["highlight.content"]
  }
};

$.post("/test/test/_search_with_clusters", JSON.stringify(request), function(response) {
  $("#highlight-request").text(JSON.stringify(response, false, "  "));
});
            

The clustering plugin comes with several open-source algorithms from the Carrot² project and has a built-in support for the commercial Lingo3G clustering algorithm.

The question of which algorithm to choose depends on the traffic (Lingo is more expensive than STC, Lingo3G is yet faster), expected result (Lingo3G provides hierarchical clusters, Lingo and STC provide flat clusters), and the input data (each algorithm will cluster the input sligtly differently). There is no one answer which algorithm is "the best".

Compare the clusters dumped for the following identical search request.

var request = {
  "search_request": {
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "title":   ["_source.title"],
    "content": ["_source.content"]
  },
  "algorithm": "lingo"
};

$.post("/test/test/_search_with_clusters", JSON.stringify(request), function(response) {
  $("#request-algorithm1").text(dumpClusters([], response.clusters).join("\n"));
});
            

var request = {
  "search_request": {
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "title":   ["_source.title"],
    "content": ["_source.content"]
  },
  "algorithm": "stc"
};

$.post("/test/test/_search_with_clusters", JSON.stringify(request), function(response) {
  $("#request-algorithm2").text(dumpClusters([], response.clusters).join("\n"));
});
            

Every clustering algorithm comes with (tons) of attributes that modify its behavior (the Carrot² Workbench can be used for tuning these). If desired, certain attributes can be modified per-request, as the following example shows by modifying the number of desired clusters randomly (execute the example a few times to see the difference).

var request = {
  "search_request": {
    "query": {"match" : { "_all": "data mining" }},
    "size": 100
  },

  "query_hint": "data mining",
  "field_mapping": {
    "title":   ["_source.title"],
    "content": ["_source.content"]
  },
  "algorithm": "lingo",
  "attributes": {
     "LingoClusteringAlgorithm.desiredClusterCountBase": Math.round(5 + Math.random() * 5)
  }
};

$.post("/test/test/_search_with_clusters", JSON.stringify(request), function(response) {
  $("#request-attributes").text(dumpClusters([], response.clusters).join("\n"));
});
            

The plugin comes with sensible defaults out of the box and should require no additional configuration. Customize only if really necessary.

The following configuration files and properties can be used to modify plugin defaults.

{path.conf}/elasticsearch.yml,
{path.conf}/elasticsearch.json,
{path.conf}/elasticsearch.properties

Main ES configuration file with plugin-specific settings.

carrot2.enabled

If set to false disables the plugin, even if it is installed.

{path.conf}/carrot2.yml,
{path.conf}/carrot2.json,
{path.conf}/carrot2.properties

An optional configuration file with plugin-specific settings.

suite

Algorithm suite XML. The resource is looked up in path.conf and in classpath. The default suite resource name is carrot2.suite.xml, it contains the defaults for all open-source algorithms and attempts to load Lingo3G.

resources

Resource lookup path for loading Carrot² lexical resources, Lingo3G's lexical resources and algorithm descriptor files (including any initialization-time attributes). Relative paths are resolved from ElasticSearch's path.conf variable (typically config folder). The value can be an absolute path.

Any resources not present in this location will be loaded from classpath (defaults).

The default algorithm suite contains empty stubs for all initialization attributes of every included algorithm. These files follow a naming convention of: {algorithm-name}-attributes.xml and are resolved relative to the current setting of the resources configuration setting (see plugin configuration).

For example to override the default attributes for all requests to the lingo algorithm, one would create a {es.home}/config/lingo-attributes.xml and place any overridden attributes in there, as in:

<attribute-sets default="overridden-attributes">
  <attribute-set id="overridden-attributes">
    <value-set>
      <label>overridden-attributes</label>

      <attribute key="LingoClusteringAlgorithm.desiredClusterCountBase">
        <value type="java.lang.Integer" value="5"/>
      </attribute>
    </value-set>
  </attribute-set>
</attribute-sets>

It is perhaps most convenient to export the configuration XMLs directly from the Carrot² Workbench.