runsc doesn't work with rootless podman

[sdeoras]

I am trying to evaluate the use of gVisor via [podman](https://github.com/containers/libpod) that allows container creation in rootless mode. gVisor works fine via sudo but panics when in rootless mode. Below is stack trace and other relevant info.

system info:

uname -a
Linux 4.18.0-21-generic #22~18.04.1-Ubuntu SMP Thu May 16 15:07:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

runsc --version
runsc version 90a116890fce
spec: 1.0.1-dev

permissions on runsc

ls -la `which runsc`
-rwxr-xr-x 1 root root 20123510 Jun  4 01:20 /usr/bin/runsc

podman --version
podman version 1.3.2-dev

works fine when sudo
sudo podman --runtime=runsc run --rm -it docker.io/library/ubuntu:latest bash

runc works fine in both root and rootless modes

sudo podman --runtime=runc run --rm -it docker.io/library/ubuntu:latest bash
podman --runtime=runc run --rm -it docker.io/library/ubuntu:latest bash

panics when running in podman/rootless mode
stack trace:

podman --runtime=runsc run --rm -it docker.io/library/ubuntu:latest bash
I0604 20:47:56.621539   21217 x:0] ***************************
I0604 20:47:56.621611   21217 x:0] Args: [/usr/bin/runsc start fb6738612f208a2786470ab33803763b976290531168c3716ea72b30ae74f310]
I0604 20:47:56.621673   21217 x:0] Version 90a116890fce
I0604 20:47:56.621686   21217 x:0] PID: 21217
I0604 20:47:56.621698   21217 x:0] UID: 0, GID: 0
I0604 20:47:56.621706   21217 x:0] Configuration:
I0604 20:47:56.621712   21217 x:0] 		RootDir: /run/user/1000/runsc
I0604 20:47:56.621720   21217 x:0] 		Platform: ptrace
I0604 20:47:56.621734   21217 x:0] 		FileAccess: exclusive, overlay: false
I0604 20:47:56.621744   21217 x:0] 		Network: sandbox, logging: false
I0604 20:47:56.621755   21217 x:0] 		Strace: false, max size: 1024, syscalls: []
I0604 20:47:56.621762   21217 x:0] ***************************
I0604 20:47:56.625479   21217 x:0] Setting up network
I0604 20:47:56.625961   21217 x:0] Applying namespace network at path "/proc/21187/ns/net"
I0604 20:47:56.626170   21217 x:0] Skipping down interface: {Index:1 MTU:65536 Name:lo HardwareAddr: Flags:loopback}
W0604 20:47:56.626272   21217 x:0] IPv6 is not supported, skipping: fe80::40b5:4cff:fe3c:9d9/64
W0604 20:47:56.649254   21217 x:0] IPv6 is not supported, skipping route: {Ifindex: 2 Dst: fe80::/64 Src: <nil> Gw: <nil> Flags: [] Table: 254}
I0604 20:47:56.649925   21217 x:0] Restoring namespace network
panic: error restoring namespace: of type network: operation not permitted

goroutine 1 [running, locked to thread]:
gvisor.googlesource.com/gvisor/runsc/specutils.ApplyNS.func1()
runsc/specutils/namespace.go:146 +0x29d
gvisor.googlesource.com/gvisor/runsc/sandbox.joinNetNS.func1()
runsc/sandbox/network.go:119 +0x24
gvisor.googlesource.com/gvisor/runsc/sandbox.createInterfacesAndRoutesFromNS(0xc00019eb60, 0xc0001d6160, 0x12, 0xc0001d6101, 0xe27480, 0xc00019ec40)
runsc/sandbox/network.go:274 +0x10d0
gvisor.googlesource.com/gvisor/runsc/sandbox.setupNetwork(0xc00019eb60, 0x52c3, 0xc000097420, 0xc000178000, 0x2, 0xc000068080)
runsc/sandbox/network.go:71 +0x380
gvisor.googlesource.com/gvisor/runsc/sandbox.(*Sandbox).StartRoot(0xc0001558c0, 0xc000097420, 0xc000178000, 0x0, 0x0)
runsc/sandbox/sandbox.go:139 +0x192
gvisor.googlesource.com/gvisor/runsc/container.(*Container).Start(0xc0000d23c0, 0xc000178000, 0x0, 0x0)
runsc/container/container.go:397 +0x288
gvisor.googlesource.com/gvisor/runsc/cmd.(*Start).Execute(0x14e48c0, 0xe38480, 0xc000044008, 0xc0001684e0, 0xc000136780, 0x2, 0x2, 0x7fcc5f2b4008)
runsc/cmd/start.go:61 +0x139
github.com/google/subcommands.(*Commander).Execute(0xc000096000, 0xe38480, 0xc000044008, 0xc000136780, 0x2, 0x2, 0x13)
external/com_github_google_subcommands/subcommands.go:141 +0x2fb
github.com/google/subcommands.Execute(...)
external/com_github_google_subcommands/subcommands.go:371
main.main()
runsc/main.go:245 +0x1452
Killed

[fvoznika]

I don't have experience with podman, so I don't really know what podman rootless is doing. The code above joined the network namespace to configure the network and it's trying to restore back to the original namespace, which should be allowed. Not sure why it's failing.

Having said that, runsc requires the caller to be root right now. It would be nice to make runsc work rootless under a flag, especially for runsc do, but we don't have immediate plans to do that. If you are interested in working on it, I can help you getting started...

@avagin has poked around this recently.

[amscanne]

I don't think we require the caller to be root, we just create new namespaces by default.

I think there's an explicit test for this behavior with runsc do:
https://github.com/google/gvisor/blob/master/tools/run_tests.sh#L212

Maybe there would be a way to detect sufficient namespaces that we skip it? Versus having to pass --netns=none.

[sdeoras]

@fvoznika, thanks for taking a look at the bug. If there is nothing fundamentally blocking runsc from running in rootless mode I would be interested in helping out resolving this bug. Let me spend some time with the code so I have relevant questions to ask you. Thanks again!

[fvoznika]

Ideally we would be able to run runsc --rootless <cmd> without the need to be root or unshare. However, this is difficult because many of the defense in depth steps that runsc takes require CAP_SYS_ADMIN.

In summary, runsc enters/creates namespaces, maps user/groups in the new namespace, calls pivot_root and chroot, and mounts /proc inside the new roots. Many of these operations require CAP_SYS_ADMIN, which unshare -Ur solves. However, mounting /proc requires being the root, not only root inside a user namespace. This is why the test @amscanne pointed to is using --TESTONLY-unsafe-nonroot. This flag makes the sandbox run as the same user that called runsc create (in this case, root inside the user namespace unshare created), and will not chroot the sandbox process to an empty directory.

I think we can remove /proc usage so that --TESTONLY-unsafe-nonroot is not needed anymore, only CAP_SYS_ADMIN, CAP_SYS_CHROOT, etc that are acquired via unshare. I'm not sure what kind of configuration controls you have with podman. Is there a way to configure it (or add a plugin) that will create a new user namespace and execute runsc as root inside this namespace? If not, maybe creating a wrapper script that intercepts calls to runsc create ... and calls unshare -Ur runsc create ...?

[prattmic]

@fvoznika 356d1be added support for --rootless in 'runsc do'. What's missing to make it work for 'runsc create'?

[avagin]

@prattmic runsc create requires to set uid and gid mappings which has to be set via newuidmap.

[Liloyoung]

Am 13.09.2019 22:56 schrieb Andrei Vagin <notifications@github.com>:@prattmic runsc create requires to set uid and gid mappings which has to be set via newuidmap. —You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

[rhatdan]

@giuseppe PTAL
Podman already does the user namespace setup and configuration.

[giuseppe]

AFAICS, the issue seems to be in the function https://github.com/google/gvisor/blob/master/runsc/specutils/namespace.go#L143-L149

oldNS points to a namespace not owned by the rootless user (in this case the network namespace on the host), so gVisor fails to re-join it when running in a user namespace.

A possible solution is to let the code run in a goroutine and on error, keep the OS thread locked, so that the Go Runtime will destroy the underlying thread when the go routine ends. From: https://golang.org/pkg/runtime/#LockOSThread

 LockOSThread wires the calling goroutine to its current operating system thread.
The calling goroutine will always execute in that thread, and no other goroutine will
execute in it, until the calling goroutine has made as many calls to UnlockOSThread
as to LockOSThread. If the calling goroutine exits without unlocking the thread, the
thread will be terminated. 

so there is no risk another goroutine will run in the wrong namespace