[Clone from Snapshot] CloneCommand never send

[PrivatePuffin]

What steps did you take and what happened:
When creating a pvc as a clone from a snapshot of another PVC, the clone command never gets send

ZFS command log:

2024-05-19.17:46:29 zfs create -o quota=274877906944 -o recordsize=64k -o dedup=off -o compression=zstd-6 -o encryption=off -o mountpoint=legacy NVME/app-pvc/pvc-b44a3c8a-4cae-4d63-aed1-fafe853f8e8b
2024-05-19.17:46:29 zfs create -o quota=1073741824 -o recordsize=64k -o dedup=off -o compression=zstd-6 -o encryption=off -o mountpoint=legacy NVME/app-pvc/pvc-176eff42-3b84-4ce6-8fe9-a06192320869
2024-05-19.17:46:52 zfs snapshot NVME/app-pvc/pvc-b44a3c8a-4cae-4d63-aed1-fafe853f8e8b@snapshot-08240c13-7fcb-4425-838d-79eacfc27407
2024-05-19.18:57:04 zfs clone NVME/app-pvc/pvc-b44a3c8a-4cae-4d63-aed1-fafe853f8e8b@snapshot-08240c13-7fcb-4425-838d-79eacfc27407 NVME/app-pvc/testclone

Notice "testclone" which was ran manually, the log never showed any attempt at running the clone command, successfull or otherwise.

What did you expect to happen:

It should at least send the clone command, failure or success regardless.

The output of the following commands will help us better understand what's going on:

• All OpenZFS pods and containers are running, healthy, latest and able to create normal PVC defined datasets.
• zfsvolumes cr for origin PVC is present
• zfssnapshot cr for originPVC is present
• zfs snapshot itself for origin PVC is present
• zfs snapshot itself for origin PVC can be cloned manually
• zfsvolume for targetPVC is present
• dataset for targetPVC is missing
• clonecommand never send

zfsvolume cr for targetPVC

Name:         pvc-095889bb-2da5-4b40-838e-d54c499b2ab0
Annotations:  <none>
API Version:  zfs.openebs.io/v1
Kind:         ZFSVolume
Spec:
  Capacity:        274877906944
  Compression:     zstd-6
  Dedup:           off
  Encryption:      off
  Fs Type:         zfs
  Owner Node ID:   ix-truenas
  Pool Name:       NVME/app-pvc
  Recordsize:      64k
  Shared:          yes
  Snapname:        pvc-b44a3c8a-4cae-4d63-aed1-fafe853f8e8b@snapshot-08240c13-7fcb-4425-838d-79eacfc27407
  Thin Provision:  yes
  Volume Type:     DATASET
Status:
  State:  Failed
Events:   <none>

volume for origin is all correct.
target and origin share the same (known-good) storageClass

• kubectl logs -f openebs-zfs-controller-f78f7467c-blr7q -n openebs -c openebs-zfs-plugin
  Gist logs are to be added, but highlights.
  But error is just failing on the Dataset existence check, after creation of the ZFSvolume CR and no descriptive error is output in any of them. Just the fact volume creation failed, which we can also already see on the zfsvolume cr.

• kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin
  https://gist.github.com/Ornias1993/9cb23dc0df026233e8d64c74d70bd39a
  https://gist.github.com/Ornias1993/991e63fbc41fd68e71a35c2f8f2e3a62

Anything else you would like to add:

Everything works perfectly fine and the PVC is also created fine.
Other PVC creation works fine.
All OpenEBS pods and components are fine
volumesnapshot objects are present and fine, as-well-as ZFS snapshots.

But regardless of this, at least the clone command should've been send.

Environment:

• Verified Latest
• Verified Latest
• Platform independent (tested multiple)
• Debian/TrueNAS/and-others

[PrivatePuffin]

I've also noticed that the clone feature support is not tested in any way either, or at least I don't see it being ran.

That being said, with the lack of logging it's REALLY hard to bugtrace.

[tiagolobocastro]

I've also noticed that the clone feature support is not tested in any way either, or at least I don't see it being ran.

I took a few notes the other day of what should be done about lacking test for the local engines. I'll write it up on a ticket so we can track progress on addressing it.
Some e2e-test infra is also lost since archival, so that will have to be replaced somehow.

[PrivatePuffin]

I took a few notes the other day of what should be done about lacking test for the local engines.

Ahh good to soo activity on that!

I'll write it up on a ticket so we can track progress on addressing it.

Yes that would be great as well!

Some e2e-test infra is also lost since archival, so that will have to be replaced somehow.

I noticed, I think it might be best to fix the tests first and go from there?
If this issue is correct, it would show itself when tests get implemented anyway...

---

Edit/Addition

On another note is the lack of verbosity though...
Lot of errors come down to "no worky", it would be nice to at least more easily trace it back to the step in the provisioning/creation process thats failing.

[tiagolobocastro]

Yes that would be great as well!

I've raised openebs/openebs#3726
Hopefully more folks will pitch in, correct anything I've said wrong and provide more details.

I noticed, I think it might be best to fix the tests first and go from there? If this issue is correct, it would show itself when tests get implemented anyway...

Yes that makes sense. I'd also like to see how much of these local engines (lvm, zfs...) test we can "unify". Basic CSI etc functionality should work exactly the same, so presumably the same basic tests should apply to all?

On another note is the lack of verbosity though... Lot of errors come down to "no worky", it would be nice to at least more easily trace it back to the step in the provisioning/creation process thats failing.

So you're saying even if you increase verbosity there's nothing else being added? Maybe we can track this as an enhancement too. CC @sinhaashish

EDIT: sorry mouse slip...

[PrivatePuffin]

On another note is the lack of verbosity though... Lot of errors come down to "no worky", it would be nice to at least more easily trace it back to the step in the provisioning/creation process thats failing.

So you're saying even if you increase verbosity there's nothing else being added? Maybe we can track this as an enhancement too. CC @sinhaashish

EDIT: sorry mouse slip...

Is there even a verbosity option?

[PrivatePuffin]

Doubt verbosity would've helped much in this case btw, as I manually tracked the log messages through the code and it never even hit the function that should've send the zfs-clone command.

[tiagolobocastro]

Is there even a verbosity option?

sigh, sorry my mistake for making this assumption, seems there's not..

[PrivatePuffin]

@jnels124 Can you confirm on here that you also see this behavior as well?

[w3aman]

Hey @Ornias1993, I tried to reproduce this issue but it seems it is working for me. I tried on Debian 12, k8s 1.26.15, with zfs-2.1.11-1. I am able to successfully created clone out of a snapshot which was created on a thinProvisioned, and shared mount volume.

Can i get more specific details so that i can see how to reproduce this issue (what exact steps you performed, with yaml if possible)?

Logs doesn't seems to be useful here, i just see bunch of repeating following lines for different PVCs
Started PVC processing "ix-authelia/authelia-cnpg-main-1-wal No need to resize PVC "ix-wg-easy/wg-easy-config"

[PrivatePuffin]

Logs doesn't seems to be useful here

I've not added all logs yet, but the issue is also that there is no verbosity at all.
So the logs are, sorry to say, completely useless crap.

Those "this no worky" errors are not useable log output anyway.

i just see bunch of repeating following lines for different PVCs

Obviously this is a running system so the logs do contain things that aren't related.

Can i get more specific details so that i can see how to reproduce this issue

Primarily the use is through VolSync, so I cannot provide YAML for you.
However, I verified everything on VolSync is green, it's just waiting on the clone-from-snapshot to be created by OpenEBS. Which, obviously, never happens as it errors out

However, everything works perfectly fine, except that the PVC volsync creates throws a vague error and OpenEBS never executes the clone command. But due to the insane lack of any way of bugtracing it (because there is no verbosity anywhere), I've no way of even trying to fix it myself.

[PrivatePuffin]

@w3aman Can you share your way of testing this in yaml format?

[PrivatePuffin]

More logs:
https://gist.github.com/Ornias1993/2a9f9b32d82b5720a3ae3917486c48ea

https://gist.github.com/Ornias1993/903229b76a44da17208753519e9b755c