Camera Panel for iOS.

How-to Discussions
1

Hi,

I’m trying to show the live feed from the camera in a Panel on iOS. I feel I’m pretty close to a PoC, but the picture is not showing up on the screen. At first I was afraid I was not getting any picture, so I made an effort to base64-encode it and log it, and that picture shows just fine in my browser. I’ve tried a lot of things, with a little help from Olle, but still no luck.

The textureFromSampleBuffer is almost double what it needs to be, but I’ve not taken the time to update it yet.

If someone could take a look at it, and push me in the right direction I would appreciate it.

I’ve cleaned up a bit now:

https://github.com/bolav/fuse-camerapanel

If you look at the commits you can see some of the things I’ve tried.

Here is where I’m logging base64 strings:

https://github.com/bolav/fuse-camerapanel/tree/base64

2

Hi

I have looked into streaming video from Camera on iOS and I think there is a simpler approach. Take a look at this example from Apple https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Introduction/Intro.html

In RippleViewController.mm they use CVOpenGLESTextureCache which can be used to create a GL texture and draw it in fuse. This way you avoid copies of data.

After getting the texture into fuse you can draw it with draw statements. Fuse.Video uses the exact same approach with CVOpenGLESTextureCache for playing video on iOS, I think you more or less can copy paste that code and use it for drawing from Camera

3

I have put together some code from how I would approach this problem

This should be possible to achieve with the iOS camera API using ObjC and UXL

extern(iOS) class Camera
{
    void Start();
    void Stop();
    int2 Size { get; }
    event EventHandler FrameAvailable;
    GLTextureHandle Texture { get; }
    void Update();
}

And then using it in fuse:

public class CameraStream : Panel { }

public extern (iOS) class iOSCameraVisual : ControlVisual<CameraStream>
{

    readonly Camera _camera = new Camera();
    readonly SizingContainer _sizing = new SizingContainer();

    protected override void Attach()
    {
        _camera.FrameAvailable += OnFrameAvailable;
        Fuse.UpdateManager.AddAction(Update);
    }

    protected override void Detach()
    {
        _camera.FrameAvailable -= OnFrameAvailable;
        Fuse.UpdateManager.RemoveAction(Update);
    }

    public sealed override float2 GetMarginSize( float2 fillSize, SizeFlags fillSet)
    {
        _sizing.snapToPixels = Control.SnapToPixels;
        _sizing.absoluteZoom = Control.AbsoluteZoom;
        return _sizing.ExpandFillSize(GetSize(), fillSize, fillSet);
    }

    void Update()
    {
        _camera.Update();
    }

    int2 _sizeCache = int2(0,0);
    void OnFrameAvailable(object sender, EventArgs args)
    {
        if (_camera.Size != _sizeCache)
        {
            _sizeCache = _camera.Size;
            InvalidateLayout();
        }
        InvalidateVisual();
    }


    float2 GetSize()
    {
        return (float2)_camera.Size;
    }

    float2 _origin;
    float2 _scale;
    float2 _drawOrigin;
    float2 _drawSize;
    float4 _uvClip;
    protected sealed override float2 OnArrangeMarginBox(float2 position, float2 availableSize, SizeFlags fillSet)
    {
        var size = base.OnArrangeMarginBox(position, availableSize, fillSet);

        _sizing.snapToPixels = Control.SnapToPixels;
        _sizing.absoluteZoom = Control.AbsoluteZoom;

        var contentDesiredSize = GetSize();

        _scale = _sizing.CalcScale( size, contentDesiredSize );
        _origin = _sizing.CalcScale( size, contentDesiredSize * _scale );

        _drawOrigin = _origin;
        _drawSize = contentDesiredSize * _scale;
        _uvClip = _sizing.CalcClip( size, ref _drawOrigin, ref _drawSize );

        return size;
    }

    protected sealed override void OnDraw(DrawContext dc)
    {
        var texture = _camera.VideoTexture;
        if (texture == null)
            return;

        if (Control.StretchMode == StretchMode.Scale9)
            {
                // Not implemented
            }

        else
            VideoDrawElement.Impl.
                Draw(dc, this, _drawOrigin, _drawSize, _uvClip.XY, _uvClip.ZW - _uvClip.XY, texture);
    }

    class VideoDrawElement
    {
        static public VideoDrawElement Impl = new VideoDrawElement();

        public void Draw(DrawContext dc, Node element, float2 offset, float2 size,
            float2 uvPosition, float2 uvSize, VideoTexture tex)
        {
            draw
            {
                apply Fuse.Drawing.Planar.Rectangle;

                DrawContext: dc;
                Node: element;
                Size: size;
                Position: offset;

                TexCoord: VertexData * uvSize + uvPosition;

                PixelColor: float4(sample(tex, TexCoord).XYZ, 1.0f);
            };
        }
    }

    protected override void OnHitTest(HitTestContext htc)
    {
        //must be in the actual video part shown
        var lp = htc.LocalPoint;
        if (lp.X >= _drawOrigin.X && lp.X <= (_drawOrigin.X + _drawSize.X) &&
            lp.Y >= _drawOrigin.Y && lp.Y <= (_drawOrigin.Y + _drawSize.Y) )
            htc.Hit(this);

        base.OnHitTest(htc);
    }
}
4

Thanks! It’s working!