sdlc.dev

Kubernetes path · The cluster's brain · Topic 3 of 13

The scheduler, nodes and the kubelet

Not started

A new pod has no node. The scheduler watches for such pods, chooses the best node for each, and records the choice in the pod object. The kubelet on that node sees a pod assigned to it and makes it real: it pulls the images, asks the container runtime to create the containers, mounts volumes, runs probes and reports status. The scheduler decides where; the kubelet decides nothing and does everything.

Why it matters: Pods stuck in Pending, hot spots on one node and replicas that all die with one zone are scheduling problems, and the fixes are a few fields in the pod spec.

  1. 1Placement happens once

    The scheduler picks a node for each new pod and writes it into the pod (binding). After that the pod stays there until it ends.

  2. 2Filter, then score

    Nodes that cannot run the pod (not enough requested CPU or memory, taints, selectors) are filtered out; the rest are scored and the best one wins.

  3. 3Requests, not usage

    The scheduler adds up resource requests, not actual consumption. A node can look idle and still be full.

  4. 4The kubelet does the work

    The kubelet on the chosen node pulls images, starts containers through the container runtime, runs probes and reports status.

1

How it works

The explanation, with examples you can run.

Filtering and scoring

The scheduler works in two steps for each pod:

  1. Filtering removes nodes where the pod cannot run: not enough unrequested CPU or memory for the pod's requests, a nodeSelector or required affinity that does not match, a taint the pod does not tolerate, a volume in another zone, a port already used on the host.
  2. Scoring ranks the remaining nodes: spreading pods of the same app, preferred affinities, image already present, balanced resource use.

The pod is bound to the highest-scoring node. If no node passes filtering, the pod stays Pending, and kubectl describe pod shows the reason in its events, for example 0/5 nodes are available: 5 Insufficient memory.

Telling the scheduler what you need

spec:  nodeSelector:    kubernetes.io/arch: arm64          # only arm64 nodes  tolerations:    - key: dedicated                   # allowed onto nodes tainted dedicated=gpu:NoSchedule      operator: Equal      value: gpu      effect: NoSchedule  topologySpreadConstraints:    - maxSkew: 1                       # keep zones within one pod of each other      topologyKey: topology.kubernetes.io/zone      whenUnsatisfiable: ScheduleAnyway      labelSelector:        matchLabels:          app: trainer  containers:    - name: trainer      image: ghcr.io/acme/trainer:2.0.0      resources:        requests:          cpu: "2"          memory: 4Gi
  • Requests are the most important input: the scheduler only places a pod where the sum of requests fits the node's allocatable capacity. It never looks at live usage.
  • nodeSelector / node affinity attract pods to labelled nodes (architecture, instance type, zone).
  • Taints and tolerations work the other way: a tainted node repels pods unless they tolerate the taint. Dedicated or special nodes (GPUs, system pools) are tainted so ordinary pods stay off.
  • Topology spread and pod anti-affinity keep replicas apart, so one node or one zone failing does not take them all.

Nodes and the kubelet

Each node runs a kubelet and a container runtime (containerd or CRI-O, through the Container Runtime Interface; Docker Engine itself has not been a built-in runtime since dockershim was removed in 1.24, though images built with Docker run unchanged). The kubelet reports the node's capacity and conditions. If it stops reporting, the node becomes NotReady, and after a timeout its pods are evicted and replaced elsewhere by their controllers.

Useful views:

kubectl get nodes -o widekubectl describe node <name>     # allocatable, requests already placed, conditions, taintskubectl get pods -o wide         # which node each pod landed on

When a node must be emptied for maintenance, kubectl drain evicts its pods (respecting PodDisruptionBudgets) and marks it unschedulable, so the scheduler places the replacements elsewhere.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A pod stays Pending with the event '0/5 nodes are available: 5 Insufficient memory', yet node dashboards show only 40% memory used. Why?
  2. 2.A node has the taint dedicated=gpu:NoSchedule. Which pods can be scheduled there?
  3. 3.You want the three replicas of a Deployment spread across availability zones. What do you add?

0 of 3 right so far. Get all three to finish this topic.

Go deeper