Skip to content

[OSX] Implemented IOSurface/MTLSharedEvent interop APIs #18791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

kekekeks
Copy link
Member

@kekekeks kekekeks commented May 5, 2025

Added interop APIs for IOSurface/MTLSharedEvent.

MTLSharedEvent behaves differently to regular binary semaphores we've used previously, so new API for dealing with timeline semaphores was added (MoltenVK maps MTLSharedEvent to timeline semaphores and we are using Vulkan terminology anyway).

Public API additions:

class KnownPlatformGraphicsExternalImageHandleTypes
{
    /// <summary>
    /// A reference to IOSurface
    /// </summary>
    public const string IOSurfaceRef = nameof(IOSurfaceRef);
}


class KnownPlatformGraphicsExternalSemaphoreHandleTypes
    /// <summary>
    /// A pointer to MTLSharedEvent object
    /// </summary>
    public const string MetalSharedEvent = nameof(MetalSharedEvent);
}

class CompositionDrawingSurface
{

    /// <summary>
    /// Updates the surface contents using an imported memory image using a semaphore pair as the means of synchronization
    /// </summary>
    /// <param name="image">GPU image with new surface contents</param>
    /// <param name="waitForSemaphore">The semaphore to wait for before accessing the image</param>
    /// <param name="signalSemaphore">The semaphore to signal after accessing the image</param>
    /// <returns>A task that completes when update operation is completed and user code is free to destroy or dispose the image</returns>
    public Task UpdateWithTimelineSemaphoresAsync(ICompositionImportedGpuImage image,
        ICompositionImportedGpuSemaphore waitForSemaphore, ulong waitForValue,
        ICompositionImportedGpuSemaphore signalSemaphore, ulong signalValue)
    {
        var img = (CompositionImportedGpuImage)image;
        var wait = (CompositionImportedGpuSemaphore)waitForSemaphore;
        var signal = (CompositionImportedGpuSemaphore)signalSemaphore;
        return Compositor.InvokeServerJobAsync(() => Server.UpdateWithTimelineSemaphores(img, wait, waitForValue, signal, signalValue));
    }
}

enum CompositionGpuImportedImageSynchronizationCapabilities
    /// <summary>
    /// Pre-render and after-render timeline semaphores must be provided alongside with the image
    /// </summary>
    TimelineSemaphores = 8
}

Will address complaints about API "breaking changes" later today

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 12.0.999-cibuild0056386-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@kekekeks kekekeks requested review from MrJul and maxkatz6 and removed request for MrJul June 26, 2025 09:32
@kekekeks kekekeks marked this pull request as ready for review July 24, 2025 14:30
Comment on lines +29 to +30
//TODO12: Make private and expose IBitmapImpl-based import API for composition visuals
[NotClientImplementable]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create an issue with brief API idea?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general idea is to have an interface similar to ICompositionGpuInterop that's can be consumed from the render thread by CustomCompositionVisualHandler (and other API that we may add in the future). Instead of returning an asynchronously imported object the import operation would be completed immediately and will provide in some way a render-thread bitmap object suitable for consumption with ImmediateDrawingContext


public IGlExportableExternalImageTexture CreateSemaphore(string type) => throw new NotSupportedException();

public IGlExternalImageTexture ImportImage(IPlatformHandle handle, PlatformGraphicsExternalImageProperties properties)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we read these properties (format, width, height) from the surface directly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those aren't always readable depending on the platform (it's an implementation of an xplat interface)

Comment on lines +78 to +82
ImportCompleted = Compositor.InvokeServerJobAsync(() =>
{
using var _ = handle as IExternalObjectsWrappedGpuHandle;
Import();
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is not clear from the code. This handle feels like something to be disposed in the DisposeAsync method rather than here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea of handle wrappers is to have something that extends the lifetime of the object passed to the import operation.

e. g. consider the following scenario:

  1. a Metal API wrapper (such as MoltenVK) or a 3rd party rendering engine provides an API to access the backing resource (IOSurface/MtlTexture/etc), most notably this API does NOT increase the reference counter of said backing resource and user code has no means of manipulating such reference counter
  2. Avalonia API is used to import such resource asynchronously
  3. While operation is still pending the backing resource gets destroyed

So instead of sending the raw resource pointer/handle we instead use a handle wrapper that has internal knowledge of how to manipulate the reference counter, duplicate handle, etc. The wrapper is created immediately on the UI thread when Avalonia import API is called and then disposed on the render thread after the import operation

Comment on lines +22 to +24
extern uint64_t GetRetainCountForNSObject(void* obj)
{
return [(NSObject*)obj retainCount];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetRetainCountForNSObject doesn't seem to be used. And as per apple "This method is of no value in debugging memory management issues" (and in my observations, it's true, returned value often is unreliable)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and in my observations, it's true, returned value often is unreliable

It was very much usable while debugging this PR (I needed to observe when the value in fact changes), but we can delete it if you insist.

Copy link
Member

@maxkatz6 maxkatz6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MrJul
Copy link
Member

MrJul commented Aug 4, 2025

This doesn't build, the SkiaMetalApi class is missing.
(It's used only for CreateBackendRenderTarget so it might be worth just having that method exposed instead.)

@kekekeks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants