Show subscriptions to event types in Backstage (#24)

* Build subscribers list Signed-off-by: Ali Ok <[email protected]> * Embed the consumedBy info in eventTypes Signed-off-by: Ali Ok <[email protected]> * Make the tests working Signed-off-by: Ali Ok <[email protected]> * Get dynamic client from the context Signed-off-by: Ali Ok <[email protected]> * go mod tidy && go mod vendor Signed-off-by: Ali Ok <[email protected]> * Link event mesh plugin locally Signed-off-by: Ali Ok <[email protected]> * Store the consumer relation in a custom annotation Signed-off-by: Ali Ok <[email protected]> * Process the custom annotation to emit relations Signed-off-by: Ali Ok <[email protected]> * Does not work - extended the ApiEntity and added a root level field Signed-off-by: Ali Ok <[email protected]> * Put the consumer information in the metadata as is Signed-off-by: Ali Ok <[email protected]> * Use Backstage Kube id annotation to fetch the consumer Signed-off-by: Ali Ok <[email protected]> * Improve code, add logging Signed-off-by: Ali Ok <[email protected]> * Add some test cases in the backend code Signed-off-by: Ali Ok <[email protected]> * Add more test cases in the backend code Signed-off-by: Ali Ok <[email protected]> * Use UnsafeGuessKindToResource Signed-off-by: Ali Ok <[email protected]> * Add some docs Signed-off-by: Ali Ok <[email protected]> * Rename `etNameMap` to `etByNamespacedName` Signed-off-by: Ali Ok <[email protected]> * Simplify trigger processing Signed-off-by: Ali Ok <[email protected]> * Add code comments Signed-off-by: Ali Ok <[email protected]> * Update architecture diagrams Signed-off-by: Ali Ok <[email protected]> * Address comments Signed-off-by: Ali Ok <[email protected]> * Fix provider unit tests Signed-off-by: Ali Ok <[email protected]> * Created separate issues for TODOs Signed-off-by: Ali Ok <[email protected]> * Tests for the processor Signed-off-by: Ali Ok <[email protected]> * Remove trailing whitespace Signed-off-by: Ali Ok <[email protected]> --------- Signed-off-by: Ali Ok <[email protected]>
knative-extensions · Feb 7, 2024 · 408beed · 408beed
1 parent 413359b
commit 408beed
Show file tree

Hide file tree

Showing 94 changed files with 5,827 additions and 91 deletions.
diff --git a/README.md b/README.md
@@ -14,30 +14,130 @@ See [Event Mesh plugin README file](./backstage/plugins/knative-event-mesh-backe
 
 The architecture of the plugin is as follows:
 ```
-      Kubernetes                        Backstage
-┌────────────────────┐           ┌─────────────────────┐
-│                    │           │                     │
-│ ┌───────────────┐  │           │       Plugin        │
-│ │               │  │           │ ┌─────────────────┐ │
-│ │    Backend    ◄──┼────┐      │ │                 │ │
-│ │               │  │    │      │ │ ┌─────────────┐ │ │
-│ └───────┬───────┘  │    │      │ │ │             │ │ │
-│         │          │    └──────┼─┼─┤  Provider   │ │ │
-│         │          │           │ │ │             │ │ │
-│         │          │           │ │ └─────────────┘ │ │
-│ ┌───────▼───────┐  │           │ │                 │ │
-│ │               │  │           │ └─────────────────┘ │
-│ │   API Server  │  │           │                     │
-│ │               │  │           └─────────────────────┘
-│ └───────────────┘  │
-│                    │
-└────────────────────┘
+      Kubernetes                                     Backstage
+┌────────────────────┐           ┌───────────────────────────────────────────────┐
+│                    │           │       Plugin                                  │
+│ ┌───────────────┐  │           │ ┌─────────────────┐        ┌───────────────┐  │
+│ │               │  │           │ │                 │        │               │  │
+│ │    Backend    ◄──┼────┐      │ │ ┌─────────────┐ │        │               │  │
+│ │               │  │    │      │ │ │             │ │        │               │  │
+│ └───────┬───────┘  │    └──────┼─┼─┤  Provider   ├─┼────────►               │  │
+│         │          │           │ │ │             │ │        │               │  │
+│         │          │           │ │ └─────────────┘ │        │               │  │
+│         │          │           │ │                 │        │               │  │
+│ ┌───────▼───────┐  │           │ │                 │        │               │  │
+│ │               │  │           │ │        ┌────────┼────────┤   Catalog     │  │
+│ │   API Server  │  │           │ │        │        │        │               │  │
+│ │               │  │           │ │        │        │        │               │  │
+│ └───────────────┘  │           │ │ ┌──────▼──────┐ │        │               │  │
+│                    │           │ │ │             │ │        │               │  │
+└────────────────────┘           │ │ │ Processor   ├─┼────────►               │  │
+                                 │ │ │             │ │        │               │  │
+                                 │ │ └─────────────┘ │        │               │  │
+                                 │ │                 │        │               │  │
+                                 │ └─────────────────┘        └───────────────┘  │
+                                 └───────────────────────────────────────────────┘
 ```
 
 The plugin use providers (and possibly other mechanisms) to communicate with a special backend-for-frontend.
 
 This backend talks to the Kubernetes API server to get information about the resources in the cluster.
 
+```mermaid
+---
+title: Overall
+---
+flowchart TD
+    Start --> FetchBrokers
+    FetchBrokers --> ProcessBrokers
+    ProcessBrokers --> FetchEventTypes
+    FetchEventTypes --> ProcessEventTypes
+    ProcessEventTypes --> FetchTriggers
+    FetchTriggers --> ProcessTriggers
+```
+
+## Processing the brokers
+
+```mermaid
+---
+title: ProcessBrokers
+---
+flowchart LR
+    GetNextBroker --> CreateDTO
+```
+
+## Processing the event types
+
+```mermaid
+---
+title: ProcessEventTypes
+---
+flowchart TD
+    GetEventType[Get next event type]
+    CheckRef{spec.ref exists?}
+    RefIsABrokerInTheBrokerMap{ref is a broker in the previously <br> built broker map?}
+    RegisterEventType[Add event type to broker DTO's `providedEventTypes` list]
+    DontRegisterEventType[Don't relate the event type to any broker]
+    Done[Done]
+
+    GetEventType --> CheckRef
+    CheckRef --> |Yes| RefIsABrokerInTheBrokerMap
+    RefIsABrokerInTheBrokerMap --> |Yes| RegisterEventType
+
+    CheckRef --> |No| DontRegisterEventType
+    RefIsABrokerInTheBrokerMap --> |No| DontRegisterEventType
+    RegisterEventType --> Done
+    DontRegisterEventType --> Done
+```
+
+## Processing the triggers
+
+```mermaid
+---
+title: ProcessTriggers
+---
+flowchart TD
+    GetTrigger[Get next trigger]
+    CheckSubscriberRef{spec.subscriber.ref <br> exists?}
+    FetchSubscriberRef[Fetch subscriber resource]
+    CheckSubscriberLabel{Subscriber has the <br> Backstage label}
+    CheckEventType{Trigger has an <br> event type}
+    RegisterSingleRelation[Register `ConsumedBy` relation <br> for eventType and subscriber]
+    RegisterRelation[Register `ConsumedBy` relation <br> for eventType and subscriber]
+
+
+    Ignore[Ignore trigger]
+
+    Done[Done]
+
+    GetTrigger --> CheckSubscriberRef
+    CheckSubscriberRef --> |Yes| FetchSubscriberRef
+    FetchSubscriberRef --> CheckSubscriberLabel
+    CheckSubscriberLabel --> |Yes| CheckEventType
+    CheckEventType --> |Yes| RegisterSingleRelation
+    CheckEventType --> |No| FetchAllEventTypesForBroker
+    FetchAllEventTypesForBroker --> ForEachEventType --> RegisterRelation
+
+
+    RegisterSingleRelation --> Done
+    RegisterRelation --> Done
+
+
+    CheckSubscriberLabel --> |No| Ignore
+    CheckSubscriberRef --> |No| Ignore
+
+    Ignore --> Done
+
+    CheckSubscriberRef -.- CheckSubscriberRefNote["We can't collect subscriber information using the URL. <br> So, let's simply check the subsciber ref."]
+    CheckSubscriberLabel -.- CheckSubscriberLabelNote["The target is to show what resource is using what event types. <br> However, Backstage will only show the resource if it has a special label. <br> So, if that label is missing, simply ignore the subscriber."]
+    CheckEventType -.- CheckEventTypeNote["If the trigger has an event type filter, <br> that means the subscriber is subscribed to that event. <br> If not, the subscriber is subscribed to all events from this trigger. <br> Please note that we ignore other filtering mechanisms such as 'source'."]
+
+    CheckSubscriberRefNote:::note
+    CheckSubscriberLabelNote:::note
+    CheckEventTypeNote:::note
+    classDef note fill:yellow
+```
+
 #### Running the backend
 
 The backend is a Go project that runs in a Kubernetes cluster.

diff --git a/backends/config/100-eventmesh/100-clusterrole.yaml b/backends/config/100-eventmesh/100-clusterrole.yaml
@@ -25,6 +25,7 @@ rules:
     resources:
       - brokers
       - eventtypes
+      - triggers
     verbs:
       - get
       - list
@@ -43,3 +44,14 @@ rules:
       - delete
       - patch
       - watch
+
+
+  # permissions to get subscribers for triggers
+  # as subscribers can be any resource, we need to give access to all resources
+  # we fetch subscribers one by one, we only need `get` verb
+  - apiGroups:
+      - "*"
+    resources:
+      - "*"
+    verbs:
+      - get
diff --git a/backends/pkg/reconciler/eventmesh/annotations.go b/backends/pkg/reconciler/eventmesh/annotations.go
@@ -4,6 +4,8 @@ var ExcludedAnnotations = map[string]struct{}{
 	"kubectl.kubernetes.io/last-applied-configuration": {},
 }
 
+// FilterAnnotations filters out annotations that are not interesting to provide to the Backstage plugin.
+// Specifically, it filters out the annotations in ExcludedAnnotations.
 func FilterAnnotations(annotations map[string]string) map[string]string {
 	if annotations == nil {
 		return nil

diff --git a/backends/pkg/reconciler/eventmesh/broker.go b/backends/pkg/reconciler/eventmesh/broker.go
@@ -4,28 +4,43 @@ import (
 	eventingv1 "knative.dev/eventing/pkg/apis/eventing/v1"
 )
 
+// Broker is a simplified representation of a Knative Eventing Broker that is easier to consume by the Backstage plugin.
 type Broker struct {
-	Name        string            `json:"name"`
-	Namespace   string            `json:"namespace"`
-	UID         string            `json:"uid"`
-	Labels      map[string]string `json:"labels,omitempty"`
+	// Namespace of the broker
+	Namespace string `json:"namespace"`
+
+	// Name of the broker
+	Name string `json:"name"`
+
+	// UID of the broker
+	UID string `json:"uid"`
+
+	// Labels of the broker. These are passed as is.
+	Labels map[string]string `json:"labels,omitempty"`
+
+	// Annotations of the broker. These are passed as is, except that are filtered out by the FilterAnnotations function.
 	Annotations map[string]string `json:"annotations,omitempty"`
-	//
+
+	// ProvidedEventTypes is a list of event types that the broker provides.
+	// This is a list of strings, where each string is a "<namespace>/<name>" of the event type.
 	ProvidedEventTypes []string `json:"providedEventTypes,omitempty"`
 }
 
-func (b Broker) GetNameAndNamespace() string {
-	return NameAndNamespace(b.Namespace, b.Name)
+// GetNamespacedName returns the name and namespace of the broker in the format "<namespace>/<name>"
+func (b Broker) GetNamespacedName() string {
+	return NamespacedName(b.Namespace, b.Name)
 }
 
+// convertBroker converts a Knative Eventing Broker to a simplified representation that is easier to consume by the Backstage plugin.
+// see Broker.
 func convertBroker(br *eventingv1.Broker) Broker {
 	return Broker{
 		Name:        br.Name,
 		Namespace:   br.Namespace,
 		UID:         string(br.UID),
 		Labels:      br.Labels,
 		Annotations: FilterAnnotations(br.Annotations),
-		// to be filled later
+		// this field will be populated later on, when we have the list of event types
 		ProvidedEventTypes: []string{},
 	}
 }
diff --git a/backends/pkg/reconciler/eventmesh/controller.go b/backends/pkg/reconciler/eventmesh/controller.go
@@ -13,6 +13,7 @@ import (
 	eventtypereconciler "knative.dev/eventing/pkg/client/injection/reconciler/eventing/v1beta2/eventtype"
 
 	brokerinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1/broker"
+	triggerinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1/trigger"
 	eventtypeinformer "knative.dev/eventing/pkg/client/injection/informers/eventing/v1beta2/eventtype"
 
 	eventinglistersv1 "knative.dev/eventing/pkg/client/listers/eventing/v1"
@@ -22,6 +23,7 @@ import (
 type Listers struct {
 	EventTypeLister eventinglistersv1beta2.EventTypeLister
 	BrokerLister    eventinglistersv1.BrokerLister
+	TriggerLister   eventinglistersv1.TriggerLister
 }
 
 func NewController(ctx context.Context) *controller.Impl {
@@ -40,6 +42,7 @@ func NewController(ctx context.Context) *controller.Impl {
 	listers := Listers{
 		EventTypeLister: eventtypeinformer.Get(ctx).Lister(),
 		BrokerLister:    brokerinformer.Get(ctx).Lister(),
+		TriggerLister:   triggerinformer.Get(ctx).Lister(),
 	}
 
 	go startWebServer(ctx, listers)
@@ -56,7 +59,7 @@ func startWebServer(ctx context.Context, listers Listers) {
 	r := mux.NewRouter()
 	r.Use(commonMiddleware)
 
-	r.HandleFunc("/", EventMeshHandler(ctx, listers)).Methods("GET")
+	r.HandleFunc("/", HttpHandler(ctx, listers)).Methods("GET")
 	http.Handle("/", r)
 
 	log.Fatal(http.ListenAndServe(":8080", r))

diff --git a/backends/pkg/reconciler/eventmesh/eventmesh.go b/backends/pkg/reconciler/eventmesh/eventmesh.go
@@ -8,6 +8,7 @@ import (
 	eventingv1beta2 "knative.dev/eventing/pkg/apis/eventing/v1beta2"
 )
 
+// Reconciler is a stub reconciler for getting the informers injected by sharedmain.
 type Reconciler struct {
 }
 

diff --git a/backends/pkg/reconciler/eventmesh/eventtype.go b/backends/pkg/reconciler/eventmesh/eventtype.go
@@ -4,23 +4,37 @@ import (
 	"knative.dev/eventing/pkg/apis/eventing/v1beta2"
 )
 
+// EventType is a simplified representation of a Knative Eventing EventType that is easier to consume by the Backstage plugin.
 type EventType struct {
-	Name        string            `json:"name"`
-	Namespace   string            `json:"namespace"`
-	Type        string            `json:"type"`
-	UID         string            `json:"uid"`
-	Description string            `json:"description,omitempty"`
-	SchemaData  string            `json:"schemaData,omitempty"`
-	SchemaURL   string            `json:"schemaURL,omitempty"`
-	Labels      map[string]string `json:"labels,omitempty"`
+	Namespace   string `json:"namespace"`
+	Name        string `json:"name"`
+	Type        string `json:"type"`
+	UID         string `json:"uid"`
+	Description string `json:"description,omitempty"`
+	SchemaData  string `json:"schemaData,omitempty"`
+	SchemaURL   string `json:"schemaURL,omitempty"`
+	// Labels of the event type. These are passed as is.
+	Labels map[string]string `json:"labels,omitempty"`
+	// Annotations of the event type. These are passed as is, except that are filtered out by the FilterAnnotations function.
 	Annotations map[string]string `json:"annotations,omitempty"`
-	Reference   string            `json:"reference,omitempty"`
+	// Reference is the ET's reference to a resource like a broker or a channel. It is in the format "<namespace>/<name>".
+	Reference string `json:"reference,omitempty"`
+	// ConsumedBy is a <namespace/name> list of the consumers of the event type.
+	ConsumedBy []string `json:"consumedBy,omitempty"`
 }
 
-func (et EventType) NameAndNamespace() string {
-	return NameAndNamespace(et.Namespace, et.Name)
+// NamespacedName returns the name and namespace of the event type in the format "<namespace>/<name>"
+func (et EventType) NamespacedName() string {
+	return NamespacedName(et.Namespace, et.Name)
 }
 
+// NamespacedType returns the type and namespace of the event type in the format "<namespace>/<type>"
+func (et EventType) NamespacedType() string {
+	return NamespacedName(et.Namespace, et.Type)
+}
+
+// convertEventType converts a Knative Eventing EventType to a simplified representation that is easier to consume by the Backstage plugin.
+// see EventType.
 func convertEventType(et *v1beta2.EventType) EventType {
 	return EventType{
 		Name:        et.Name,
@@ -32,6 +46,8 @@ func convertEventType(et *v1beta2.EventType) EventType {
 		SchemaURL:   et.Spec.Schema.String(),
 		Labels:      et.Labels,
 		Annotations: FilterAnnotations(et.Annotations),
-		Reference:   RefNameAndNamespace(et.Spec.Reference),
+		Reference:   NamespacedRefName(et.Spec.Reference),
+		// this field will be populated later on, when we have process the triggers
+		ConsumedBy: make([]string, 0),
 	}
 }