When querying large amounts of data, Kibana may run into a client-side timeout when querying Elasticsearch. This leads to a UI error indicating a timeout of 30000ms has been exceeded. This can be found in the elasticsearch-kubeaddons-client-0 pod's logs:
{"type": "server", "timestamp": "", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-kubeaddons-client-0", "message": "path: /kubernetes_cluster-*/_search, params: {rest_total_hits_as_int=true, ignore_unavailable=true, preference=<>, ignore_throttled=true, index=kubernetes_cluster-*, timeout=30000ms}", "cluster.uuid": "", "node.id": "" ,
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: "
...
This timeout can happen for many reasons, such as network slowness, Elastic being unavailable, or high query times when pulling large data sets. If this is the case, a simple way to verify the underlying issue is to open a shell for one of your Elastic containers and run a query similar to the below:
kubectl exec -n kubeaddons elasticsearch-kubeaddons-client-1 -it --tty -- /bin/bash
time curl -X GET "localhost:9200/kubernetes_cluster-*/_search?pretty" -H 'Content-Type: application/json' -d' { "query": { "range": { "age": { "gte": "now-1d/d", "lt": "now/d" } } } } '
The query above will check the target kubernetes_cluster-* for the last 24 hours and time the results; if your queries are timing out in a larger time frame, increase the 'GTE' field to reflect that. After getting the result, note the time it takes to query your target as this value, which we will use to base our client-side timeout. Once we have this value, we can modify our cluster.yaml to add the parameter elasticsearch.requestTimeout, please note that there is an upper limit of 120000ms for this value:
- name: kibana
enabled: true
values: |
files:
kibana.yml:
elasticsearch.requestTimeout: 90000
After making this change, you will need to run ./konvoy deploy addons -y to push the value to your cluster. After the command has been completed, you can validate the pod has been updated by checking the configuration file within the pod:
kubectl exec -it --tty -n kubeaddons kibana-kubeaddons-74b96fd969-l2zlq -- /bin/bash
bash-4.2$ cat config/kibana.yml
elasticsearch.hosts:
- http://elasticsearch-kubeaddons-client:9200
elasticsearch.requestTimeout: 90000
server.basePath: /ops/portal/kibana
server.host: 0.0.0.0
server.port: 5601
server.rewriteBasePath: false