Run components as non-root

[maxbrunet]

Thanos components run as root by default:

$ docker run --rm -ti --entrypoint= quay.io/thanos/thanos:v0.14.0 id
uid=0(root) gid=0(root) groups=10(wheel)

Pods should probably have a restricted security context, I currently run them with the following (except receive and rule that I do not have):

securityContext:
  fsGroup: 2000
  runAsNonRoot: true
  runAsUser: 1000

Do you have an opinion on the uid/gid to use? 65534/nobody/nogroup seems popular too, but not everyone thinks this what they should be used for.

[metalmatze]

That seems like something we should totally do indeed. Which specific user we choose, I'm not too concerned about.

[maxbrunet]

I'm wondering if we should choose a UID in the main project first, and set it the Dockerfile (would it break something? Sidecars?), and then just re-use it in the security context here.

How breaking is the change? User might have to change the ownership of the volumes data, right? Store and Compactor data can be discarded, what about rule and receive?

[metalmatze]

Receive might be the actual tricky one - yes. Ruler too. If we make it clear it should be fine to change.
@bwplotka do you know if we have any deployments using some different UID already?

[brancz]

Actually the kubelet chowns the entire volume recursively before mounting to the fsgroup specified in the pod, so this should not break anything (unless the respective CSI driver is broken and doesn't do this, which we have seen with NFS sometimes but I don't think we should limit ourselves by incorrect implementations of CSI).