What events are triggered for PV/PVC and from where?

Question

kubectl get events list the events for the K8s objects. From where the events are triggered for PV/PVC actually ? There is a list of volume events https://docs.openshift.com/container-platform/4.5/nodes/clusters/nodes-containers-events.html but it does not identifies that which events are for which resource ?

Answer 1

Let`s start from what exactly is an Kubernetes event. Those are object that provide insight into what is happening inside a cluster, such as what decisions were made by scheduler or why some pods were evicted from the node. Those API objects are persisted in etcd.

You can read more about them here and here. There is also an excellent tutorial about Kubernetes events which you may find here.

---

There are couple of ways to view/fetch more detailed events from Kubernetes:

Use kubectl get events -o wide. This will give you information about object, subobject and source of the event. Here`s an example:

LAST SEEN   TYPE      REASON                  OBJECT                            SUBOBJECT                         SOURCE                                                          MESSAGE

<unknown>   Warning   FailedScheduling        pod/web-1                                                           default-scheduler                                               running "VolumeBinding" filter plugin for pod "web-1": pod has unbound immediate PersistentVolumeClaims

6m2s        Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-1                                     k8s.io/minikube-hostpath 2481b4d6-0d2c-11eb-899d-02423db39261   Successfully provisioned volume pvc-a56b3f35-e7ac-4370-8fda-27342894908d

6m2s        Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-1                                     k8s.io/minikube-hostpath 2481b4d6-0d2c-11eb-899d-02423db39261   Successfully provisioned volume pvc-a56b3f35-e7ac-4370-8fda-27342894908d

Use kubectl get events --output json will give you list of the event in json format containing other details such as selflink.

        ---
                "apiVersion": "v1",
        "count": 1,
        "eventTime": null,
        "firstTimestamp": "2020-10-13T12:07:17Z",
        "involvedObject": {
        ---  
        "kind": "Event",
        "lastTimestamp": "2020-10-13T12:07:17Z",
        "message": "Created container nginx",
        "metadata": {       
        ---    

Selflink can be used to determine the the API location from where the data is being fetched.

We can take as an example /api/v1/namespaces/default/events/ and fetch the data from API server using kubectl proxy:

kubectl proxy --port=8080 & curl http://localhost:8080/api/v1/namespaces/default/events/

Using all those information you can narrow down to a specific details from the underlying object using field-selector:

    kubectl get events --field-selector type=!Normal
    or
    kubectl get events --field-selector involvedObject.kind=PersistentVolumeClaim
    
    LAST SEEN   TYPE     REASON                  OBJECT                            MESSAGE
    44m         Normal   ExternalProvisioning    persistentvolumeclaim/www-web-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    44m         Normal   Provisioning            persistentvolumeclaim/www-web-0   External provisioner is provisioning volume for claim "default/www-web-0"
    44m         Normal   ProvisioningSucceeded   persistentvolumeclaim/www-web-0   Successfully provisioned volume pvc-815beb0a-b5f9-4b27-94ce-d21f2be728d5

Please also remember that all the information provided by kubectl events are the same ones from the kubectl describe <ojbect>.

Lastly, if you look carefully into the event.go code you may see all the events reference for volumes. If you compare those with the Table 13. Volumes you can see that they are almost the same (execpt for WaitForPodScheduled and ExternalExpanding)

This means that Openshift provided an aggregated set of information about possible events from different kubernetes that may occur in the cluster.