inconsistent label cardinality when trying to report metrics

[ddseapy]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened:
When enabling metrics config on the workflow controller, the controller crashes trying to create metrics:

    enabled: true
    path: /metrics
    port: 8080

time="2019-10-02T15:32:00Z" level=info msg="Processing workflow" namespace=ddseapy workflow=hello-world-hflln
panic: inconsistent label cardinality

goroutine 303 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(0xc0001f97a0, 0x2, 0x0, 0xc0008ead40, 0x3, 0x4, 0x1, 0x1)
	/go/src/github.com/prometheus/client_golang/prometheus/value.go:172 +0xb0
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow.func1(0xc0001f97a0, 0x2, 0x0, 0xc000842760, 0x1, 0x1)
	/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:100 +0x16d
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow.func2(0xc0001f97a0, 0x0, 0xc000842760, 0x1, 0x1)
	/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:103 +0x62
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow(0xc0007f6ff0, 0xc00094c540, 0xc000792638, 0x8, 0xc000176440, 0x14, 0xc000176560, 0x11, 0xc000792670, 0xc, ...)
	/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:115 +0x248
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).Collect(0xc0007f6ff0, 0xc00094c540)
	/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:93 +0xc3
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc0006aa0b0, 0xc00094c540, 0x1891f40, 0xc0007f6ff0)
	/go/src/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/go/src/github.com/prometheus/client_golang/prometheus/registry.go:431 +0x302

What you expected to happen:
No error should occur when metrics are enabled.

How to reproduce it (as minimally and precisely as possible):
argo submit the helloworld example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

Environment:

• Argo version: latest
• Kubernetes version : v1.13.1

[dguendisch]

Just upgraded to argo v2.4.0 and seeing the same issue and error msg, metrics doesn't work anymore.

[schrodit]

we also saw this with argo 2.4.1
@sarabala1979 are there any changes for the metrics in the new release

[schrodit]

looks like this is the PR causing this issue (its the only PR on the metrics) #1550

[jessesuen]

Fixed by #1695