Understanding and Resolving Elasticsearch Score Changes after Document Updates

5 min readApr 18, 2023

I answered a question on StackOverflow a while ago and want to share the details and my answer in an article here. So let’s start with the question and try to understand the problem.

Understanding the Problem

The questionnaire has documents like the one below, and the developer wants to search for these documents with match query to use the score.

POST sample-index-test/_doc/1
{
    "first_name": "James",
    "last_name" : "Osaka"
}

Here is a sample query for the document above:

GET sample-index-test/_explain/1
{
  "query": {
    "match": {
      "first_name": "James"
    }
  }
}

As you know, Elasticsearch scores the documents according to relevance. Let’s search the index now after indexing this document. So yes, we have only one document on this index for now.

GET sample-index-test/_search
{
  "query": {
    "match": {
      "first_name": "James"
    }
  }
}

After this search will see the results below :

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "sample-index-test",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "first_name": "James",
          "last_name": "Osaka"
        }
      }
    ]
  }
}

The point I want to draw your attention to _score fields of the result. As you can see, _score value of our document is 0.2876821 . When you update the document several times, for example, let’s say we updated the record 10 times with the following requests:

POST sample-index-test/_update/1
{
  "script" : "ctx._source.first_name = 'James'; ctx._source.last_name = 'Cena';"
}

OR 

POST sample-index-test/_doc/1
{
    "first_name": "James",
    "last_name" : "Cena"
}

There will be no addition to the index. We have one document again, and there is no more. We just updated the last_name field of the document. Let’s do the exact search again and try to see what the will result:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.042559613,
    "hits": [
      {
        "_index": "sample-index-test",
        "_id": "1",
        "_score": 0.042559613,
        "_source": {
          "first_name": "James",
          "last_name": "Cena"
        }
      }
    ]
  }
}

As you can see here, the score changed. The score for the document now 0.042559613 . But according to the TF/IDF calculation, we need to see the same score as our first search response. Because nothing changed when we compared it with the first state of the document. Even, though I did not change the first name field, I just changed last_name fields and continued searching on first_name . Let’s dig a little bit more with _explain endpoint.

GET sample-index-test/_explain/1
{
  "query": {
    "match": {
      "first_name": "James"
    }
  }
}

Explain API endpoint will compute a score explanation for a query and a specific document. The result of the request above will be something like below :

{
  "_index": "sample-index-test",
  "_id": "1",
  "matched": true,
  "explanation": {
    "value": 0.042559613,
    "description": "weight(first_name:james in 0) [PerFieldSimilarity], result of:",
    "details": [
      {
        "value": 0.042559613,
        "description": "score(freq=1.0), computed as boost * idf * tf from:",
        "details": [
          ...
          {
            "value": 0.042559616,
            "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
            "details": [
              {
                "value": 11,
                "description": "n, number of documents containing term",
                "details": []
              },
              {
                "value": 11,
                "description": "N, total number of documents with field",
                "details": []
              }
            ]
          },
          {
            "value": 0.45454544,
            "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
            "details": [
              ...
            ]
          }
        ]
      }
    ]
  }
}

I cut some parts which are optional for us. Now let’s pay attention to IDF calculation. As you know, Inverse Document Frequency looks at how common (or uncommon) a word is amongst the corpus. This means we will use the count of documents in the index to calculate the IDF.

idf, computed as log(1 + (N - n + 0.5) / (n + 0.5))

As you can see above, we are using the total number of documents, but the problem is we had one document in the index, but it is showing 11.

{
    "value": 11,
    "description": "n, number of documents containing term",
    "details": []
},
{
    "value": 11,
    "description": "N, total number of documents with field",
    "details": []
}

So, this is the problem if you are using this score to calculate something for the other services.

Why does this happen?

Elasticsearch using Lucene and all the documents stored in segments. And the segments are immutable, and the document update action has a 2-step process. When a document is updated, a new document is created, and the old document is marked as deleted. So, when you create the first document in Elasticsearch index, Elasticsearch will save it in a segment, and there will be only one document. Then you update the same document 10 times; in any update operation, Elasticsearch will create another document in a segment and flag the older one as deleted. But when you search the index, you will reach the latest document state from the segments. For the time being, the number of deleted documents will be 10. You will reach the latest state of the document again with the search, but Elasticsearch will continue counting them internally for IDF calculation. For this reason, “the number of documents with field” and “number of documents containing term” change after every update.

Solution

If you have some information about what a segment is, as you know, this problem will resolve itself after a while. So, if you want to do this yourself without waiting, you need to use _forcemerge. I need to put here an explanation from Elasticsearch documentation.

Merging reduces the number of segments in each shard by merging some of them together, and also frees up the space used by deleted documents. Merging normally happens automatically, but sometimes it is useful to trigger a merge manually.
We recommend only force merging a read-only index (meaning the index is no longer receiving writes).

To execute _forcemerge to our index, we used the following request:

POST sample-index-test/_forcemerge

This request can take some time according to your index size, and you can follow the tasks with the following request on Kibana:

GET _tasks?actions=*forcemerge*&detailed

Another way is just waiting. Elasticsearch also has a scheduler and merge policy to merge the segments automatically. Before using force merge, I recommend reading the documentation mentioned below carefully.

Lastly, there is an index life cycle action to execute the force merge operation with a policy. According to your logic, you can use different solutions to get better results on your search scoring.

Elasticsearch re-indexing same document causing score changes

We have created an index with the document POST sample-index-test/_doc/1 { "first_name": "James", "last_name" : "Osaka"…

stackoverflow.com

References :