The analyzer keeps the analysis of former analyzed tokens in the results

[dpappas]

Hi everyone!

I am trying to use the greeklish analyzer in ElasticSearch 6.7.1 but the analyzer keeps appending the analysis of former tokens.

For example i use the token "αυγο" and i get a first analysis.
Then is use the token "παιχνιδι" and i get the analysis of "παιχνιδι" but then appended in the results i see the analysis of "αυγό".

Can you help me?
Thank you in advance

[cmantas]

Hello. Could you illustrate the problem using the analyze API?

[dpappas]

Ok!

We can reproduce this in kibana as well

DELETE /my_index

PUT /my_index
{
  "settings": {
            "analysis": {
                "analyzer": {
                    "simple_greeklish": {
                        "tokenizer": "standard",
                        "filter":[
                          "lowercase","greeklish"
                          ]
                    }
                },
                "filter": {
                    "greeklish": {
                        "type": "skroutz_greeklish",
                        "max_expansions" 	: 5,
                        "greek_variants" 	: "false"
                    }
                }
            }
  }
}

When i do the following request

GET /my_index/_analyze
{
	"analyzer" : "simple_greeklish",
	"text"     : "αυγο"
}

i get a perfect analysis:

{
  "tokens" : [
    {
      "token" : "αυγο",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "aygo",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "avgo",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "afgo",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "augo",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "greeklish_word",
      "position" : 0
    }
  ]
}

But then when i do the following request:

GET /my_index/_analyze
{
	"analyzer" : "simple_greeklish",
	"text"     : "παιχνιδι"
}

I get the new analysis but the former as well!

{
  "tokens" : [
    {
      "token" : "παιχνιδι",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "pehnidi",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "paichnidi",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "paihnidi",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "pexnidi",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "paixnidi",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "aygo",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "avgo",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "afgo",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    },
    {
      "token" : "augo",
      "start_offset" : 0,
      "end_offset" : 8,
      "type" : "greeklish_word",
      "position" : 0
    }
  ]
}

I have tried resetting the:
tokenStream in GreeklishTokenFilterFactory
or the
greeklishWords in GreeklishTokenFilter
or the
greeklishList and the perWordGreeklish in GreeklishGenerator
but nothing happened!

[cmantas]

Very strange.
Thankfully this bug does not manifest itself in ElasticSearch v5.4.2 (Naturally one can use that version if downgrading is an option).

Unfortunatelly development for this plugin hasn't started yet for newer versions, so the dev team cannot offer an timeline/ETA for fixing this.
Seems like a critical issue for v6.x.x though.

[petsoukos]

This issue still persists up to ES7.7. Anyone found a solution/workaround?