记一次ElasticSearch冷热分离索引无法正确分配的问题

2020-04-10 阅读量 | 文章字数:903

前记

1.问题

当给ElasticSearch配置了ILM后,ElasticSearch自动执行,按照配置把索引进行处理迁移,但是运行一段时间后发现,之前用logstash传过来的且没有使用mapping的数据都无法正常执行ILM,而其他数据却能正常处理.
由于发现这个问题时,对于Es不是十分了解,只能copy错误去搜索,搜索后发现几乎没有一个答案是满意的,最多说的是升级ElasticSearch版本解决,所以就又水了一文

2.查看索引ilm运行情况

1	GET so1n_index/_ilm/explain

首先使用ilm/explain查看索引的ilm运行情况,这里使用so1n_index代替索引名(下同).

{
  "indices" : {
    "so1n_index" : {
      "index" : "so1n_index",
      "managed" : true,
      "policy" : "so1n_index_policy",
      "lifecycle_date_millis" : 1585008002962,
      "age" : "35.41d",
      "phase" : "cold",
      "phase_time_millis" : 1587686411329,
      "action" : "allocate",
      "action_time_millis" : 1587686419823,
      "step" : "check-allocation",
      "step_time_millis" : 1587686428607,
      "step_info" : {
        "message" : "Waiting for [2] shards to be allocated to nodes matching the given filters",
        "shards_left_to_allocate" : 2,
        "all_shards_active" : true,
        "actual_replicas" : 1
      },
      "phase_execution" : {
        "policy" : "so1n_index_policy",
        "phase_definition" : {
          "min_age" : "31d",
          "actions" : {
            "freeze" : { },
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "cold"
              }
            },
            "set_priority" : {
              "priority" : 10
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1587663954354
      }
    }
  }
}

看输出可以发现,我在ilm配置索引min_age超过31天就执行so1n_index_policy中的一个cold phase,而在执行其中allocate的check-allocation步骤时报错Waiting for [2] shards to be allocated to nodes matching the given filters,也就是ilm执行到so1n_index节点分配时就已经出问题了,而且还是在check-allocation阶段.所以我们只要查明为什么check-allocation不通过就好了.

3.查看索引allocation情况

直接使用explain查看allocation的错误原因

GET /_cluster/allocation/explain
{
  "index": "so1n_index",
  "shard": 0,
  "primary": true
}

{
  "index" : "so1n_index",
  "shard" : 0,
  "primary" : true,
  "current_state" : "started",
  "current_node" : {
    "id" : "so1n_id_1",
    "name" : "so1n-elastic-node-1",
    "transport_address" : "10.142.0.1:9300",
    "attributes" : {
      "ml.machine_memory" : "7839637504",
      "xpack.installed" : "true",
      "box_type" : "hot",
      "ml.max_open_jobs" : "20"
    }
  },
  "can_remain_on_current_node" : "no",
  "can_remain_decisions" : [
    {
      "decider" : "filter",
      "decision" : "NO",
      "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box_type:"cold",_id:"so1n_id_3"]"""
    }
  ],
  "can_move_to_other_node" : "no",
  "move_explanation" : "cannot move shard to another node, even though it is not allowed to remain on its current node",
  "node_allocation_decisions" : [
    {
      "node_id" : "so1n_id_2",
      "node_name" : "so1n-elastic-node-2",
      "transport_address" : "10.142.0.2:9300",
      "node_attributes" : {
        "ml.machine_memory" : "7839653888",
        "ml.max_open_jobs" : "20",
        "box_type" : "cold",
        "xpack.installed" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : """node does not match index setting [index.routing.allocation.require] filters [box_type:"cold",_id:"so1n_id_3"]"""
        }
      ]
    }
  ]
}

这里可以发现,原来是索引多了一段配置,导致在分配索引时,必须要确保节点的id是:so1n_id_3,可是这一台的box_type是cold,具体可以看索引的设置

{
  "index.blocks.read_only_allow_delete": "false",
  "index.query.default_field": [
    "*"
  ],
  "index.refresh_interval": "1s",
  "index.write.wait_for_active_shards": "1",
  "index.lifecycle.name": "so1n_index_policy",
  "index.routing.allocation.require._id": "so1n_id_3",
  "index.routing.allocation.require.box_type": "cold",
  "index.blocks.write": "true",
  "index.priority": "10",
  "index.number_of_replicas": "1"
}

可以发现,这里多了一段

1	"index.routing.allocation.require._id": "so1n_id_3"

这一段从来没配置过,估计是没使用mapping时,当数据发送到so1n_id_3节点时,Es会自动给索引加上index.routing.allocation.require._id或者我的mapping指定有5个分片,所以Es才不会自动给索引加上index.routing.allocation.require._id.

4.解决

可确定Es上面的所有索引都可以不需要index.routing.allocation.require._id,那可以通过以下进行修改,如果有部分索引需要index.routing.allocation.require._id,则需要通配符修改

PUT */_settings
{
   "index.routing.allocation.require._id": null
}

然后再调用explain查看

GET /_cluster/allocation/explain
{
  "index": "so1n_index",
  "shard": 0,
  "primary": true
}

发现Es已经在后台执行索引迁移了

{
  "index" : "so1n_index",
  "shard" : 0,
  "primary" : true,
  "current_state" : "relocating",
  "current_node" : {
    "id" : "so1n_id_1",
    "name" : "so1n-elastic-node-1",
    "transport_address" : "10.142.0.1:9300",
    "attributes" : {
      "ml.machine_memory" : "7839637504",
      "xpack.installed" : "true",
      "box_type" : "hot",
      "ml.max_open_jobs" : "20"
    }
  },
  "explanation" : "the shard is in the process of relocating from node [so1n-elastic-node-1] to node [so1n-elastic-node-4], wait until relocation has completed"
}

迁移完成后再去查看可以发现字段rebalance_explanation会显示:cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance
则此,索引冷热分离规则全部适配运行完成.

本文作者： So1n
本文链接： http://so1n.me/2020/04/10/记一次ElasticSearch冷热分离索引无法正确分配的问题/
版权声明： 本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可。转载请注明出处！