tetragon: Add bench application #261

olsajiri · 2022-07-26T09:36:05Z

Adding benchmark application to measure tracing workloads.

The tetragon-bench application starts tetragon (or not for -baseline option)
for given CRD file and runs workload.

At the end it displays some stats, like:

Benchmark summary
-----------------
Test started:      2022-09-08 15:04:28.04513503 +0200 CEST m=+0.016539427
Test ended:        2022-09-08 15:04:38.268072909 +0200 CEST m=+10.239477312
Workload start:    2022-09-08 15:04:38.268072909 +0200 CEST m=+10.239477312
Arguments:         
Total duration:    10.222937885s
Setup duration:    1.743823005s
Workload duration: 8.479094472s
Test duration:     10.222937885s
Export duration:   0s
Export stats:      bytes=0, writes=0
FGS cpu usage:     system=15.170223s, user=17.522276s, rss=230504, ctxsw=1231909
Ring buffer:       received=3958406, lost=41842, errors=0
BPF statistics:
  event_execve                   [tracepoint/212]   :   8.89µs (2)
  event_wake_up_new_task         [kprobe/211]       :   1.41µs (6)
  generic_kprobe_event           [kprobe/205]       :   1.54µs (4001580)
  restrict_filesystems           [lsm/58]           :   0.13µs (1569)
  event_exit                     [tracepoint/210]   :   0.49µs (32)

It's possible to run predefined workloads with builtin CRD files,
or run specific custom benchmark.

The predefined benchmarks are rw and open and can be run with:

# ./tetragon-bench -trace rw
# ./tetragon-bench -trace open

They do and monitor some simple read/write open operations and they are there
mainly to show the possibility of builtin benchmark if we think it's good idea ;-)

The custom benchmark is what I'm using most at the moment and
it allows to pass CRD file to start tetragon with and specify workload, like:

# ./tetragon-bench -crd crds/examples/names.yaml  -- perf bench sched pipe

I'm doing perf profiles to check on CRD stuff with:

# perf record -g -e cycles:k ./tetragon-bench -crd ./read_write.yaml -json-encode -- perf bench sched pipe

Signed-off-by: Jiri Olsa [email protected]

tixxdz · 2022-09-08T16:31:03Z

pkg/bench/bench.go

+
+		if _, err := os.Stat(option.Config.HubbleLib); err != nil {
+			// Running outside the source tree, fall back to default location.
+			option.Config.HubbleLib = "/var/lib/hubble-fgs"


Maybe this one: https://github.com/cilium/tetragon/blob/main/pkg/defaults/defaults.go#L44 ?

ok, will use that one, thanks

tixxdz · 2022-09-08T16:31:08Z

pkg/bench/bpfstats.go

+)
+
+func EnableBpfStats() {
+	err := os.WriteFile("/proc/sys/kernel/bpf_stats_enabled", []byte("1"), 0666)


Is the perm mode ok? maybe restrict it a bit

that permission will be used only if the file does not exist and needs to be created, so we are fine here

tixxdz · 2022-09-08T16:40:29Z

pkg/bench/cpuusage.go

+func CPUUsageFromCPUAcct(containerID string) CPUUsage {
+
+	rss := int64(0)
+	memStatFilename := fmt.Sprintf("/sys/fs/cgroup/memory/docker/%s/memory.stat", containerID)


So this seems cgroupv1 hierarchy, I assume that this could also run in cgroupv2 the path would be /sys/fs/cgroup/$path/%s/memory.stat in that case where $path does not container the 'memory' directory or controller as it is unified.

You could use this new helper:
https://github.com/cilium/tetragon/blob/main/pkg/cgroups/cgroups.go#L567 if it returns CGROUP_UNIFIED then you are in pure cgroupv2 and you have to fix that path without 'memory' directory, for everything else you can assume cgroupv1 that is the one that is used to track resources. Hybrid cgroupv2 is only used to organize processes together but resources tracking at cgroupv1.

It would be cool if we can standardize on those ;-) !

btw where this is called?

The full cgroup path may not container docker directory at all or another container runtime, I'm missing context, but if you could read /proc/self/cgroups and adapt according to cgroup version and the controllers you are interested in to get right working paths without hardcoding it here. There is some logic in that cgroups.go package

right, I'll remove it for now, because it's not used and check on that more, thanks

kkourt

Thanks! Some minor requests / suggestions.

pkg/bench/summary.go

pkg/bench/bench.go

pkg/bench/cpuusage.go

pkg/bench/trace.go

Makefile

Leftover changes from 'make generate'. Signed-off-by: Jiri Olsa <[email protected]>

Adding summary object to carry all the measured data and doing the results output. Signed-off-by: Jiri Olsa <[email protected]>

Adding base of the bench package, that allows to run observer with custom configuration and provides generic arguments. Signed-off-by: Jiri Olsa <[email protected]>

Adding interface to quesry bpfstats data from bpf programs. Signed-off-by: Jiri Olsa <[email protected]>

Adding util object that carries some other helper objects. Signed-off-by: Jiri Olsa <[email protected]>

Adding tetragon-bench application to meassure tracing workloads. It runs benchmark specified with -trace option. At the moment there is no benchmark defined, they are coming in following changes. Signed-off-by: Jiri Olsa <[email protected]>

Adding benchmark that does many read and writes in multiple threads and monitors all this data in tetragon. Could be run with: # ./tetragon-bench -trace rw And configured with extra options: -bench-rw-count uint bench rw read/write count (default 100) -bench-rw-loops uint bench rw number of loops (default 100) -bench-rw-size uint bench rw buffer size (default 1024) -bench-rw-threads uint bench rw number of threads (default 4) -bench-rw-sleep uint bench rw sleep in ms (default 1) Signed-off-by: Jiri Olsa <[email protected]>

Adding benchmark that does many failed open syscalls and monitors them in tetragon. Could be run with: # ./tetragon-bench -trace open And configured with extra options: -bench-open-loops uint bench open number of loops (default 1000000) -bench-open-sleep uint bench open sleep between open syscalls (us) (default 1) -bench-open-threads uint bench open number of threads (default 4) Signed-off-by: Jiri Olsa <[email protected]>

Allows to run custom benchmark with specific crd file, like: # ./tetragon-bench -crd crds/examples/names.yaml -- perf bench sched pipe Or with perf sampling: # perf record -g -e cycles:k ./tetragon-bench -crd ./read_write.yaml -json-encode -- perf bench sched pipe Arguments after '--' will be executed as workload. When workload is finished the bench will exit. Signed-off-by: Jiri Olsa <[email protected]>

kkourt

Thanks!

olsajiri force-pushed the bench branch from 4264682 to 26a3900 Compare July 28, 2022 13:26

olsajiri force-pushed the bench branch from 26a3900 to 0e9e62f Compare August 22, 2022 12:54

olsajiri force-pushed the bench branch 4 times, most recently from 1730ea3 to c4394ab Compare September 8, 2022 13:05

olsajiri marked this pull request as ready for review September 8, 2022 13:55

olsajiri requested a review from a team as a code owner September 8, 2022 13:55

olsajiri requested a review from jrfastab September 8, 2022 13:55

tixxdz reviewed Sep 8, 2022

View reviewed changes

olsajiri force-pushed the bench branch from c4394ab to 9eba6c3 Compare September 9, 2022 08:10

kkourt requested changes Sep 12, 2022

View reviewed changes

olsajiri force-pushed the bench branch from 9eba6c3 to 1c64aa5 Compare September 16, 2022 08:37

olsajiri requested a review from kkourt September 19, 2022 21:08

olsajiri added 9 commits September 20, 2022 12:15

tetragon: go.mod update

79d6dd6

Leftover changes from 'make generate'. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add summary to bench package

6211a87

Adding summary object to carry all the measured data and doing the results output. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add runTetragon to bench package

0cd10c9

Adding base of the bench package, that allows to run observer with custom configuration and provides generic arguments. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add bpfstats to bench package

e261423

Adding interface to quesry bpfstats data from bpf programs. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add util to bench package

24fa1c5

Adding util object that carries some other helper objects. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the bench branch from 1c64aa5 to 9c7b429 Compare September 20, 2022 10:17

kkourt approved these changes Sep 20, 2022

View reviewed changes

kkourt merged commit 0d408db into cilium:main Sep 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tetragon: Add bench application #261

tetragon: Add bench application #261

Uh oh!

olsajiri commented Jul 26, 2022 •

edited

Loading

Uh oh!

tixxdz Sep 8, 2022

Uh oh!

olsajiri Sep 9, 2022

Uh oh!

tixxdz Sep 8, 2022

Uh oh!

olsajiri Sep 9, 2022

Uh oh!

tixxdz Sep 8, 2022 •

edited

Loading

Uh oh!

tixxdz Sep 8, 2022

Uh oh!

tixxdz Sep 8, 2022

Uh oh!

olsajiri Sep 9, 2022

Uh oh!

kkourt left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkourt left a comment

Uh oh!

Uh oh!

tetragon: Add bench application #261

tetragon: Add bench application #261

Uh oh!

Conversation

olsajiri commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tixxdz Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

olsajiri Sep 9, 2022

Choose a reason for hiding this comment

Uh oh!

tixxdz Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

olsajiri Sep 9, 2022

Choose a reason for hiding this comment

Uh oh!

tixxdz Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tixxdz Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

tixxdz Sep 8, 2022

Choose a reason for hiding this comment

Uh oh!

olsajiri Sep 9, 2022

Choose a reason for hiding this comment

Uh oh!

kkourt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkourt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

olsajiri commented Jul 26, 2022 •

edited

Loading

tixxdz Sep 8, 2022 •

edited

Loading