I have a single monolithic loki instance installed from the loki-stack chart. This is running on a kubernetes cluster, taking logs from 3 other clusters too. The log volume is pretty high and needed. I have s3 configured to store these logs.
However, when I try to fetch these logs in grafana ui from explore, and when I click on label in query builder, it takes forever to load the labels. Is it because of something am I doing wrong? Some queries are evening timing out and trying to consuming CPU upto 1500m.
here is my loki config:
loki:
enabled: true
isDefault: true
url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
datasource:
jsonData: "{}"
uid: ""
config:
schema_config:
configs:
- from: 2021-05-12
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
s3: <s3 buckets>
s3forcepathstyle: true
bucketnames: <s3 bucket name>
region: <region>
insecure: false
sse_encryption: false
boltdb_shipper:
shared_store: s3
cache_ttl: 24h
limits_config:
ingestion_rate_mb: 24
ingestion_burst_size_mb: 24
per_stream_rate_limit: "6MB"
per_stream_rate_limit_burst: "15MB"
- Should I consider Loki-distributed if I query requirements and log ingestions are higher?
- Will it help if I add more replicas of current monolithic setup?