Kubernetes informers log errors on GKE

[jpeach]

What steps did you take and what happened:

Running Contour tests on GKE.

What did you expect to happen:

They should pass (they do on kind).

Anything else you would like to add:

It looks like something goes wrong when we may the GroupVersionResource and start an informer.

TAP version 13
# reading document from ./httpproxy/002-header-condition-match.yaml
# decoding document with 14 parts from ./httpproxy/002-header-condition-match.yaml
# decoded part 0 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:1-22)
# decoded part 1 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:24-32)
# decoded part 2 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:34-42)
# decoded part 3 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:44-52)
# decoded part 4 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:54-62)
# decoded part 5 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:64-72)
# decoded part 6 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:74-82)
# decoded part 7 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:84-92)
# decoded part 8 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:94-102)
# decoded part 9 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:104-112)
# decoded part 10 as Kubernetes (lines ./httpproxy/002-header-condition-match.yaml:114-158)
# decoded part 11 as Rego (lines ./httpproxy/002-header-condition-match.yaml:160-172)
# decoded part 12 as Rego (lines ./httpproxy/002-header-condition-match.yaml:174-192)
# decoded part 13 as Rego (lines ./httpproxy/002-header-condition-match.yaml:194-272)
ok 1 - validating document "./httpproxy/002-header-condition-match.yaml"
E1006 14:55:00.197755   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
# test run ID is 7c80c72e-a5cb-4182-bee2-2ce0b3a28f7c
ok 2 - compiling test document
# hydrated apps/v1:Deployment object 'default/echo-header-present'
ok 3 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:1-22
ok 4 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-present'
ok 5 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-present'
ok 6 - running object update check
# hydrated v1:Service object 'default/echo-header-present'
ok 7 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:24-32
ok 8 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-present'
ok 9 - updating Kubernetes object
# checking update of Service 'default/echo-header-present'
ok 10 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-contains'
ok 11 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:34-42
ok 12 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-contains'
E1006 14:55:01.372665   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 13 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-contains'
ok 14 - running object update check
# hydrated v1:Service object 'default/echo-header-contains'
ok 15 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:44-52
ok 16 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-contains'
ok 17 - updating Kubernetes object
# checking update of Service 'default/echo-header-contains'
ok 18 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-notcontains'
ok 19 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:54-62
ok 20 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-notcontains'
ok 21 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-notcontains'
ok 22 - running object update check
# hydrated v1:Service object 'default/echo-header-notcontains'
ok 23 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:64-72
ok 24 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-notcontains'
ok 25 - updating Kubernetes object
# checking update of Service 'default/echo-header-notcontains'
ok 26 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-exact'
ok 27 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:74-82
ok 28 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-exact'
ok 29 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-exact'
ok 30 - running object update check
# hydrated v1:Service object 'default/echo-header-exact'
ok 31 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:84-92
ok 32 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-exact'
E1006 14:55:02.572764   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 33 - updating Kubernetes object
# checking update of Service 'default/echo-header-exact'
ok 34 - running object update check
# hydrated apps/v1:Deployment object 'default/echo-header-notexact'
ok 35 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:94-102
ok 36 - matching anonymous Kubernetes object
# performing update operation on Deployment 'default/echo-header-notexact'
ok 37 - updating Kubernetes object
# checking update of Deployment 'default/echo-header-notexact'
ok 38 - running object update check
# hydrated v1:Service object 'default/echo-header-notexact'
ok 39 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:104-112
ok 40 - matching anonymous Kubernetes object
# performing update operation on Service 'default/echo-header-notexact'
ok 41 - updating Kubernetes object
# checking update of Service 'default/echo-header-notexact'
ok 42 - running object update check
# hydrated projectcontour.io/v1:HTTPProxy object 'default/header-conditions'
ok 43 - hydrating Kubernetes object lines ./httpproxy/002-header-condition-match.yaml:114-158
ok 44 - matching anonymous Kubernetes object
# performing update operation on HTTPProxy 'default/header-conditions'
ok 45 - updating Kubernetes object
# checking update of HTTPProxy 'default/header-conditions'
ok 46 - running object update check
ok 47 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:160-172
E1006 14:55:04.091891   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 48 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:174-192
E1006 14:55:05.140502   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:06.208101   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:07.258947   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:08.311610   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:09.362543   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:10.409447   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:11.457482   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:12.504765   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:13.554265   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:14.611691   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:15.708549   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:16.759884   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:17.805455   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:18.850454   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:20.008406   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:21.057146   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 49 - running Rego check lines ./httpproxy/002-header-condition-match.yaml:194-272
E1006 14:55:22.104704   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:23.234507   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:24.284648   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:25.387649   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:26.507108   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:27.559167   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:28.611710   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:29.666883   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
E1006 14:55:30.718077   48944 reflector.go:330] pkg/driver/object.go:218: Failed to watch *unstructured.Unstructured: the server does not allow this method on the requested resource
ok 50 - deleting test objects
1..50

[erwbgy]

In my case these errors were caused by multiple "pods" resources:

$ kubectl api-resources | grep -w pods
pods                              po                                                                  true         Pod
pods                                                                   metrics.k8s.io                 true         PodMetrics

I "fixed" this by stopping after the first one:

--- a/pkg/driver/kubernetes.go
+++ b/pkg/driver/kubernetes.go
@@ -138,6 +138,7 @@ func (k *KubeClient) ResourcesForName(name string) ([]schema.GroupVersionResourc
                        Version:  r.Version,
                        Resource: r.Name,
                })
+               break
        }

        return matched, nil

[jpeach]

Ah I see. I don't think it's right to just break there (it would not work if we happened to list the resources in the reverse order for example). We probably want to deal with this at a slightly higher layer - need some way to know that by "pods" we mean the core not the metrics.

[erwbgy]

No it definitely isn't right, hence the "fixed" in quotes :-)

I'd suggest that the group needs to be explicit and defaults to empty if not specified, so:

• "pods" means group "" and resource "pods"
• "pods.metrics.k8s.io" means group "metrics.k8s.io" and resource "pods"

which is the same way "kubectl get" works.

[jpeach]

No it definitely isn't right, hence the "fixed" in quotes :-)

:)

I'd suggest that the group needs to be explicit and defaults to empty if not specified, so:

• "pods" means group "" and resource "pods"
• "pods.metrics.k8s.io" means group "metrics.k8s.io" and resource "pods"

which is the same way "kubectl get" works.

Yeh that makes sense. Does Kubectl have some programmatic disambiguation?