graph : use fill instead of scale_bias in grouped expert selection

[CISC]

Since all major backends (CUDA, Metal and Vulkan) support the fill op now use that instead of scale_bias.

[ggerganov]

Instead of using the explicit ggml_fill_inplace, leave this to the allocator by adding the op to ggml-alloc.c::ggml_op_can_inplace. Would also need to remove the restrict classifier in the CUDA implementation.

[CISC]

Instead of using the explicit ggml_fill_inplace, leave this to the allocator by adding the op to ggml-alloc.c::ggml_op_can_inplace. Would also need to remove the restrict classifier in the CUDA implementation.

Ah, didn't know about that, I'll change it to just ggml_fill.

I'll look into the rest for another PR where/how is the decision to do it in place done?

[ggerganov]

Generally, avoid using explicit _inplace ops unless you need to modify some buffer permanently (e.g. a KV cache).

I'll look into the rest for another PR where/how is the decision to do it in place done?

The logic for that is in ggml_gallocr_allocate_node. If one of the parents of the op is suitable, the result will reuse it's memory making the op effectively inplace.

[CISC]

@ggerganov Hmmm, it won't reuse it for FILL, it's skipping it here:

not reusing parent ffn_moe_probs_biased-19 (reshaped) for node_1556 as (nil) is external

[ggerganov]

I think this means that the parent buffer is on a different backend buffer - this is expected to not be able to inplace.

[CISC]

I think this means that the parent buffer is on a different backend buffer - this is expected to not be able to inplace.

Ah, this might be why:

not reusing parent blk.19.exp_probs_b.bias for ffn_moe_probs_biased-19 as 0x7e10eba4b800 is external

[CISC]

It's a little odd that ggml_fill_inplace works fine then, but I guess it's an ordering issue, I tried forcing ggml_gallocr_allocate_node to do it for FILL and then it just outputs ?.