Elasticsearch问题定位

Scroll Down

1.报错:es_rejected_execution_exception

1.1.现象一

以bulk方式往Elasticsearch写入数据,写入失败,返回es_rejected_execution_exception错误,原因为rejected execution of coordinating operation

{"error":{"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=10140072, replica_bytes=0, all_bytes=10140072, coordinating_operation_bytes=98550072, max_coordinating_and_primary_bytes=107374182]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=10140072, replica_bytes=0, all_bytes=10140072, coordinating_operation_bytes=98550072, max_coordinating_and_primary_bytes=107374182]"},"status":429}

原因

数据写入ES的速度超过了ES索引的速度,当未索引的数据超过ES堆内存的10%,ES就会拒绝写入线程并产生该错误。
通过max_coordinating_and_primary_bytes可以知道ES堆内存为1g,该值一般为ES堆内存的10%

查看内存情况

curl -X GET "ip:9200/_cat/nodes?v&h=name,ram.percent,ram.max,heap.current,heap.max,disk.used,disk.total"

解决方法

扩大ES内存,或数据分多次写入。

1.2.现象二

"type": "es_rejected_execution_exception",
[Running, pool size = 40, active threads = 40, queued tasks = 199, completed tasks = 5678440244]]"     },     "status": 429 }

原因

threadpool.write.queue_size默认太小
查看线程可以看到写线程堆积

curl -X GET "ip:9200/_cat/thread_pool?v"

解决方法

修改elasticsearch.yml,增加threadpool.write.queue_size

thread_pool:
    write:
        queue_size: 1000