Skip to content

KEDA error establishing rabbitmq connection in EKS cluster with Calico networking #252

Closed

Description

@visla-xugeng

Hi
I installed EKS cluster with Calico networking(xvlan). In this env, I have rabbitmq installed through helm chart. A KEDA setup through helm chart as well. Before Calico network, I was using aws vpc cni network by default. The KEDA scaling worked well. After switched to the Calico network (overlay), I got below errors:

btw: I already updated the KEDA to use "ClusterFirstWithHostNet", since I am using overlay network.

I got below error. As a result, the KEDA cannot scaling more consumers(pods) to process messages.

E0222 03:51:12.400833       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error when getting scalers error getting scaler for trigger #0: error establishing rabbitmq connection: dial tcp: lookup rabbitmq.default.svc.cluster.local on 10.64.0.2:53: no such host"}
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   4h18m
kubectl logs --follow -n kube-system --selector 'k8s-app=kube-dns'
[INFO] 192.168.206.133:47082 - 16294 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 192 0.002322684s
[INFO] 192.168.206.133:50325 - 13295 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000217494s
[INFO] 192.168.206.133:54154 - 16701 "A IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000200126s
[INFO] 192.168.206.133:59313 - 30695 "AAAA IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000163252s
[INFO] 192.168.206.133:50486 - 25237 "A IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000179662s
[INFO] 192.168.206.133:39588 - 12812 "AAAA IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000321733s
[INFO] 192.168.206.133:37375 - 53129 "AAAA IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000156612s
[INFO] 192.168.206.133:47697 - 37961 "A IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000164842s
[INFO] 192.168.206.133:36139 - 35017 "AAAA IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 79 0.000721258s
[INFO] 192.168.206.133:35654 - 52322 "AAAA IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.0008279s
[INFO] 192.168.206.133:50002 - 12758 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000239663s
[INFO] 192.168.206.133:60410 - 22924 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,aa,rd,ra 192 0.000127624s
[INFO] 192.168.206.133:46533 - 44750 "A IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.000227403s
[INFO] 192.168.206.133:43526 - 35349 "AAAA IN rabbitmq.default.svc.cluster.local.keda.svc.cluster.local. udp 75 false 512" NXDOMAIN qr,aa,rd 168 0.00022409s
[INFO] 192.168.206.133:56380 - 29165 "A IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000213602s
[INFO] 192.168.206.133:54311 - 57911 "AAAA IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000151722s
[INFO] 192.168.206.133:54528 - 1582 "A IN rabbitmq.default.svc.cluster.local.cluster.local. udp 66 false 512" NXDOMAIN qr,aa,rd 159 0.000304043s
[INFO] 192.168.206.133:38030 - 25405 "AAAA IN rabbitmq.default.svc.cluster.local.svc.cluster.local. udp 70 false 512" NXDOMAIN qr,aa,rd 163 0.000210289s
[INFO] 192.168.206.133:45711 - 25891 "AAAA IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,rd,ra 79 0.000494616s
[INFO] 192.168.206.133:54905 - 60151 "A IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 102 0.000232012s
[INFO] 192.168.206.133:43114 - 51847 "A IN rabbitmq.default.svc.cluster.local.us-west-2.compute.internal. udp 79 false 512" NXDOMAIN qr,aa,rd,ra 192 0.00021606s
[INFO] 192.168.206.133:40021 - 3288 "AAAA IN rabbitmq.default.svc.cluster.local. udp 52 false 512" NOERROR qr,aa,rd 145 0.00027382s

any idea will be appreciated.

Activity

  1. locked and limited conversation to collaborators on Feb 23, 2022
  2. converted this issue into a discussion #253 on Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions