ES-Curator to clear old logs/indices
Curator is a tool from Elastic (the company behind Elasticsearch) to help manage your Elasticsearch cluster. You can create, backup, and delete some indices, Curator helps make this process automated and repeatable. Curator is written in Python, so it is well supported by almost all operating systems. It can easily manage the huge number of logs that are written to the Elasticsearch cluster periodically by deleting them and thus helps you to save the disk space.
A very elegant way to configure and automate Elasticsearch Curator execution is using a YAML configuration. The ‘es-curator-infra-values.yaml’ file
# Common Labels
labels:
group: "es-curator-infra"
cron:
schedule: "45 18 * * *"
# Container Configs
namespace: es-cluster-infra
image:
repository: "bobrik/curator"
tag: 5.6.0
logs-cleanup-enabled: true
jaeger-cleanup-enabled: true
logs-to-retain-in-days: 7
memory_limits: 512Mi
args:
- --config
- /etc/config/config.yml
- /etc/config/action_file.yml
# Additional Container Envs
env: |
- name: SERVER_PORT
value: "8080"
- name: JAVA_OPTS
value: {{ index .Values "heap" | quote }}
- name: ES_CLIENT_HOST
valueFrom:
configMapKeyRef:
name: egov-config
key: es-infra-host
- name: ES_CLIENT_PORT
value: "9200"
- name: LOG_LEVEL
value: "DEBUG"
{{- if index .Values "logs-cleanup-enabled" }}
- name: LOGS_CLEANUP_DISABLED
value: "False"
- name: RETAIN_LOGS_IN_DAYS
value: {{ index .Values "logs-to-retain-in-days" | quote }}
{{- end }}
{{- if index .Values "jaeger-cleanup-enabled" }}
- name: JAEGER_CLEANUP_DISABLED
value: "False"
- name: RETAIN_JAEGER_DATA_IN_DAYS
value: "14"
{{- end }}
extraVolumes: |
- name: config-volume
configMap:
name: {{ template "name" . }}-config
extraVolumeMounts: |
- mountPath: /etc/config
name: config-volume
resources: |
requests:
memory: {{ .Values.memory_limits | quote }}
limits:
memory: {{ .Values.memory_limits | quote }}
You can modify the above es-curator-infra-values.yaml according to the requirements, some modifications are suggested below:
schedule: "45 18 * * *"
represents all the possible numbers for that position
- Schedule Cron Job: In the above code, at line number 6, the Cron Job is Scheduled to run at 18:45 PM every day. You can schedule your Cron Job accordingly.
- RETAIN_LOGS_IN_DAYS: You can specify how old the logs should be, to get deleted. At line number 14 of the above code, logs-to-retain-in-days specifies the logs that are older than 7 days will be deleted.
Last modified 9mo ago