Taking pictures (especially access to iOS EXIF data)

Feature Suggestions
1

We need the ability to take pictures on Android and iOS. Currently this is in place on Android, including rotation of bitmap in response to EXIF data. On iOS we have a sort of camera hack thing, which allows us to successfully capture images, but not rotate them when needed. The result is that images will have the rotation it feels like. Not the one the user wants.

So the feature request is something like:

Ideally: A robust an flexible way of taking pictures provided by the Fuse framework which works on Android and iOS with a single and simple interface.

Or, failing (or postponing) that: A way to query images on iOS for EXIF data, so we can do the rotation ourselves.

Thanks.

2

Adobe AIR has a nice way of handling this… Native Extensions. Basically, any globally available capability is handled by the framework, but anything fringe case or dealing with a particular platform uses a Native Extension. Basically native code compiled with your application. A whole ecosystem has built up specifically around ANEs. Maybe this would be a nice feature to get into Fuse? This would allow developers to monotize extensions and build up the Fuse ecosystem.

3

Using iOS’s UIImagePickerController to display the Camera UI, you can get access to EXIF data as well as picture orientation information, although I didn’t see orientation showing up in the EXIF dictionary directly.

You get this information by querying the info dictionary from IUIImagePickerControllerDelegate's imagePickerControllerDidFinishPickingMediaWithInfo. There’s a magic "{Exif}" key with unsurprising information, as well as a magic "Orientation" key with what looks like a magic number – I couldn’t find documentation for it, at least. That said, none of this seems to be available when using the “picker” to access the user’s Photo Library.

Another alternative, that seems to work is to query the resulting UIImage (obtained from the info dictionary) for its ImageOrientation, which also works for pictures from the user’s Library.

Save for actually rotating the textures, I put together a sample Fuse app that takes pictures and queries EXIF data and orientation information for it: http://git.io/vv15A

At the moment, the sample is iOS only. It uses native bindings to access platform-specific features and then also falls down to native code for some functions not currently exposed through bindings.

I hope this helps. Cheers!

4

Thanks for providing a solution Joao.

Proper wrapper APIs for camera is in the pipeline :slight_smile:

5

Great, we’ll look at making this work with the information provided, and also looking forward to a full wrapper which honors the EXIF stuff. :slight_smile:

6

On my iOS7 iPhone I get this crash:

2015-04-20 19:26:29.004 CameraApp[4524:60b] Received memory warning.
Picture taken, of size 960 x 1280, with orientation 3

(lldb) bt
 thread #1: tid = 0x1ab2c9, 0x000000019662d980 libobjc.A.dylib_object_set_associative_reference + 404, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=257, address=0xb000000000005002)
    frame #0: 0x000000019662d980 libobjc.A.dylib_object_set_associative_reference + 404
   frame #1: 0x00000001001894d0 CameraApp(anonymous namespace)::AssociationKey<id<Uno_Reference> >::associateValue(this=0x000000010037d850, nativeObject=0xb000000000005002, value=0x0000000178228060) const + 56 at uObjC.Lifetime.mm:20
    frame #2: 0x0000000100188258 CameraApp(anonymous namespace)::_SetAttachedObject(object=0x000000017802fa00, nativeObject=0xb000000000005002) + 148 at uObjC.Lifetime.mm:140
    frame #3: 0x00000001001881b8 CameraApp-[NSObject(self=0xb000000000005002, _cmd=0x0000000100344028, object=0x000000017802fa00) uObjC_setObject:] + 32 at uObjC.Lifetime.mm:155
    frame #4: 0x0000000100188e68 CameraAppuObjC::Lifetime::Attach(object=0x000000017802fa00, nativeObject=0xb000000000005002) + 240 at uObjC.Lifetime.mm:280
    frame #5: 0x000000010011c8f0 CameraAppObjC::Object____initHandle(__this=0x000000017802fa00, __handle=0xb000000000005002) + 24 at ObjC.Object.mm:54
    frame #6: 0x000000010011ca48 CameraAppObjC::Object::initHandle(this=0x000000017802fa00, handle=0xb000000000005002) + 24 at ObjC.Object.h:37
    frame #7: 0x000000010011c960 CameraAppObjC::Object___ObjInit_1(__this=0x000000017802fa00, handle=0xb000000000005002) + 24 at ObjC.Object.mm:90
    frame #8: 0x000000010011ae00 CameraAppiOS::Foundation::NSObject_ObjInit_3(this=0x000000017802fa00, id=0xb000000000005002) + 24 at iOS.Foundation.NSObject.mm:52
    frame #9: 0x000000010011b32c CameraApp`iOS::Foundation::NSValue_ObjInit_5(this=0x000000017802fa00, id=0xb000000000005002) + 24 at iOS.Foundation.NSValue.mm:40
    frame #10: 0x000000010011ab0c CameraAppiOS::Foundation::NSNumber___ObjInit_7(__this=0x000000017802fa00, __id=0xb000000000005002) + 24 at iOS.Foundation.NSNumber.mm:51
    frame #11: 0x000000010011abe8 CameraAppiOS::Foundation::NSNumber::_ObjInit_7(this=0x000000017802fa00, id=0xb000000000005002) + 24 at iOS.Foundation.NSNumber.h:30
    frame #12: 0x000000010011ab50 CameraApp`iOS::Foundation::NSNumberNew_8(this=0x0000000000000000, id=0xb000000000005002) + 56 at iOS.Foundation.NSNumber.mm:57
    frame #13: 0x0000000100073730 CameraAppCameraApp::iOS::Camera__ProcessPictureFromCamera(__this=0x0000000170059f80, image=0x00000001782280e0, info=0x0000000178228080) + 296 at CameraApp.iOS.Camera.cpp:119
    frame #14: 0x00000001000741c4 CameraAppCameraApp::iOS::Camera::ProcessPictureFromCamera(this=0x0000000170059f80, image=0x00000001782280e0, info=0x0000000178228080) + 32 at CameraApp.iOS.Camera.h:59
    frame #15: 0x0000000100073380 CameraAppCameraApp::iOS::Camera__imagePickerControllerDidFinishPickingMediaWithInfo(__this=0x0000000170059f80, picker=0x0000000178225320, info=0x0000000178228080) + 556 at CameraApp.iOS.Camera.cpp:147
    frame #16: 0x000000010017ac1c CameraAppiOS::UIKit::IUIImagePickerControllerDelegate::imagePickerControllerDidFinishPickingMediaWithInfo(this=0x0000000170059f80, picker=0x0000000178225320, info=0x0000000178228080) + 128 at iOS.UIKit.IUIImagePickerControllerDelegate.h:26
    frame #17: 0x000000010017a9cc CameraApp-[uno_IUIImagePickerControllerDelegate_Proxy imagePickerController:didFinishPickingMediaWithInfo:](self=0x0000000170006330, _cmd=0x000000018d7ff575, picker=0x00000001275153a0, info=0x000000017824c4e0) + 192 at IUIImagePickerControllerDelegate.mm:25
    frame #18: 0x0000000191a64938 PhotoLibraryPLNotifyImagePickerOfImageAvailability + 88
    frame #19: 0x0000000191a94e54 PhotoLibrary-[PLCameraView cropOverlay:didFinishSaving:] + 244
    frame #20: 0x0000000191a94cb4 PhotoLibrary-[PLCameraView cropOverlayWasOKed:] + 568
    frame #21: 0x000000018d1150b0 UIKit-[UIApplication sendAction:to:from:forEvent:] + 100
    frame #22: 0x000000018d115044 UIKit-[UIApplication sendAction:toTarget:fromSender:forEvent:] + 24
    frame #23: 0x000000018d0fe520 UIKit-[UIControl _sendActionsForEvents:withEvent:] + 376
    frame #24: 0x000000018d114a44 UIKit-[UIControl touchesEnded:withEvent:] + 584
    frame #25: 0x000000018d1146d8 UIKit-[UIWindow _sendTouchesForEvent:] + 692
    frame #26: 0x000000018d10f370 UIKit-[UIWindow sendEvent:] + 1172
    frame #27: 0x000000018d0e0b50 UIKit-[UIApplication sendEvent:] + 256
    frame #28: 0x000000018d0dec40 UIKit_UIApplicationHandleEventQueue + 8500
    frame #29: 0x000000018a0d77f4 CoreFoundation`CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 24
    frame #30: 0x000000018a0d6b50 CoreFoundation`CFRunLoopDoSources0 + 256
    frame #31: 0x000000018a0d4de8 CoreFoundation__CFRunLoopRun + 632
    frame #32: 0x000000018a015dd0 CoreFoundationCFRunLoopRunSpecific + 452
    frame #33: 0x00000001002683b4 CameraAppUIKit_PumpEvents(_this=0x00000001276067d0) + 88 at SDL_uikitevents.m:57
    frame #34: 0x000000010019cddc CameraAppSDL_PumpEvents + 56 at SDL_events.c:388
    frame #35: 0x00000001002710a8 CameraAppXli::Window::ProcessMessages() + 36 at SDL2Window.cpp:700
    frame #36: 0x000000010026ded4 CameraAppXli::Application::Run(app=0x000000016fd91900, flags=10) + 428 at Application.cpp:43
    frame #37: 0x00000001001822ac CameraAppMain(args=0x000000016fd91998) + 104 at Main.cpp:298
    frame #38: 0x00000001002762c8 CameraAppSDL_main(argc=1, argv=0x0000000178006800) + 228 at Main.cpp:52
    frame #39: 0x0000000100267c60 CameraApp-[SDLUIKitDelegate postFinishLaunch](self=0x0000000170005c20, _cmd=0x00000001003443f0) + 60 at SDL_uikitappdelegate.m:189
    frame #40: 0x000000018ac8c2d0 FoundationNSFireDelayedPerform + 392
    frame #41: 0x000000018a0d7704 CoreFoundation`CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION + 28
    frame #42: 0x000000018a0d7374 CoreFoundation`CFRunLoopDoTimer + 804
    frame #43: 0x000000018a0d509c CoreFoundation__CFRunLoopRun + 1324
    frame #44: 0x000000018a015dd0 CoreFoundationCFRunLoopRunSpecific + 452
    frame #45: 0x000000018fcfdc0c GraphicsServicesGSEventRunModal + 168
    frame #46: 0x000000018d146fc4 UIKitUIApplicationMain + 1156
    frame #47: 0x0000000100267400 CameraAppmain(argc=1, argv=0x000000016fd93c08) + 464 at SDL_uikitappdelegate.m:59
    frame #48: 0x0000000196c13aa0 libdyld.dylibstart + 4
(lldb)
7

Heh, I didn’t try this on an iOS 7 device and the simulator uses the Photo Library and a different code path. From the looks of it, I’d say there’s a mismatch between what my example expects and the information the camera provides on iOS 7.

My bad, I could have been more defensive in writing this and will update the example accordingly.

Still, I think it’s not entirely lost. From the debug output above the backtrace, it seems the app managed to get size and orientation information from the UIImage directly. It fails inside ProcessImageFromCamera where it’s trying to read into EXIF data.

It could be that "{Exif}" is not provided on iOS 7 or that it doesn’t contain a "PixelXDimension" or "PixelYDimension" entry. As I don’t currently have access to an iOS 7 device to play with this, I suggest you take a look at frame #17, -[uno_IUIImagePickerControllerDelegate_Proxy imagePickerController:didFinishPickingMediaWithInfo:] and see what’s in the info dictionary (from lldb’s prompt that’d be: po info).

8 
Picture taken, of size 960 x 1280, with orientation 3

(lldb) bt
 thread #1: tid = 0x20d8f4, 0x000000019662d980 libobjc.A.dylib_object_set_associative_reference + 404, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=257, address=0xb000000000005002)
    frame #0: 0x000000019662d980 libobjc.A.dylib_object_set_associative_reference + 404
   frame #1: 0x000000010016157c CameraApp(anonymous namespace)::AssociationKey<id<Uno_Reference> >::associateValue(this=0x0000000100355850, nativeObject=0xb000000000005002, value=0x0000000170226920) const + 56 at uObjC.Lifetime.mm:20
    frame #2: 0x0000000100160304 CameraApp(anonymous namespace)::_SetAttachedObject(object=0x00000001702268a0, nativeObject=0xb000000000005002) + 148 at uObjC.Lifetime.mm:140
    frame #3: 0x0000000100160264 CameraApp-[NSObject(self=0xb000000000005002, _cmd=0x000000010031c078, object=0x00000001702268a0) uObjC_setObject:] + 32 at uObjC.Lifetime.mm:155
    frame #4: 0x0000000100160f14 CameraAppuObjC::Lifetime::Attach(object=0x00000001702268a0, nativeObject=0xb000000000005002) + 240 at uObjC.Lifetime.mm:280
    frame #5: 0x00000001000f499c CameraAppObjC::Object____initHandle(__this=0x00000001702268a0, __handle=0xb000000000005002) + 24 at ObjC.Object.mm:54
    frame #6: 0x00000001000f4af4 CameraAppObjC::Object::initHandle(this=0x00000001702268a0, handle=0xb000000000005002) + 24 at ObjC.Object.h:37
    frame #7: 0x00000001000f4a0c CameraAppObjC::Object___ObjInit_1(__this=0x00000001702268a0, handle=0xb000000000005002) + 24 at ObjC.Object.mm:90
    frame #8: 0x00000001000f2eac CameraAppiOS::Foundation::NSObject_ObjInit_3(this=0x00000001702268a0, id=0xb000000000005002) + 24 at iOS.Foundation.NSObject.mm:52
    frame #9: 0x00000001000f33d8 CameraApp`iOS::Foundation::NSValue_ObjInit_5(this=0x00000001702268a0, id=0xb000000000005002) + 24 at iOS.Foundation.NSValue.mm:40
    frame #10: 0x00000001000f2bb8 CameraAppiOS::Foundation::NSNumber___ObjInit_7(__this=0x00000001702268a0, __id=0xb000000000005002) + 24 at iOS.Foundation.NSNumber.mm:51
    frame #11: 0x00000001000f2c94 CameraAppiOS::Foundation::NSNumber::_ObjInit_7(this=0x00000001702268a0, id=0xb000000000005002) + 24 at iOS.Foundation.NSNumber.h:30
    frame #12: 0x00000001000f2bfc CameraApp`iOS::Foundation::NSNumberNew_8(this=0x0000000000000000, id=0xb000000000005002) + 56 at iOS.Foundation.NSNumber.mm:57
    frame #13: 0x000000010004b690 CameraAppCameraApp::iOS::Camera__NumberForKey(__this=0x000000017805c710, dict=0x00000001702266e0, key=0x00000001700354a0) + 140 at CameraApp.iOS.Camera.cpp:145
    frame #14: 0x000000010004c144 CameraAppCameraApp::iOS::Camera::NumberForKey(this=0x000000017805c710, dict=0x00000001702266e0, key=0x00000001700354a0) + 32 at CameraApp.iOS.Camera.h:65
    frame #15: 0x000000010004b814 CameraAppCameraApp::iOS::Camera__ProcessPictureFromCamera(__this=0x000000017805c710, image=0x000000017822d1c0, info=0x000000017822d060) + 344 at CameraApp.iOS.Camera.cpp:165
    frame #16: 0x000000010004c344 CameraAppCameraApp::iOS::Camera::ProcessPictureFromCamera(this=0x000000017805c710, image=0x000000017822d1c0, info=0x000000017822d060) + 32 at CameraApp.iOS.Camera.h:66
    frame #17: 0x000000010004b20c CameraAppCameraApp::iOS::Camera__imagePickerControllerDidFinishPickingMediaWithInfo(__this=0x000000017805c710, picker=0x000000017822d0e0, info=0x000000017822d060) + 516 at CameraApp.iOS.Camera.cpp:205
    frame #18: 0x0000000100152cc8 CameraAppiOS::UIKit::IUIImagePickerControllerDelegate::imagePickerControllerDidFinishPickingMediaWithInfo(this=0x000000017805c710, picker=0x000000017822d0e0, info=0x000000017822d060) + 128 at iOS.UIKit.IUIImagePickerControllerDelegate.h:26
    frame #19: 0x0000000100152a78 CameraApp-[uno_IUIImagePickerControllerDelegate_Proxy imagePickerController:didFinishPickingMediaWithInfo:](self=0x0000000170019d30, _cmd=0x000000018d7ff575, picker=0x000000012e5209e0, info=0x000000017805bff0) + 192 at IUIImagePickerControllerDelegate.mm:25
    frame #20: 0x0000000191a64938 PhotoLibraryPLNotifyImagePickerOfImageAvailability + 88
    frame #21: 0x0000000191a94e54 PhotoLibrary-[PLCameraView cropOverlay:didFinishSaving:] + 244
    frame #22: 0x0000000191a94cb4 PhotoLibrary-[PLCameraView cropOverlayWasOKed:] + 568
    frame #23: 0x000000018d1150b0 UIKit-[UIApplication sendAction:to:from:forEvent:] + 100
    frame #24: 0x000000018d115044 UIKit-[UIApplication sendAction:toTarget:fromSender:forEvent:] + 24
    frame #25: 0x000000018d0fe520 UIKit-[UIControl _sendActionsForEvents:withEvent:] + 376
    frame #26: 0x000000018d114a44 UIKit-[UIControl touchesEnded:withEvent:] + 584
    frame #27: 0x000000018d1146d8 UIKit-[UIWindow _sendTouchesForEvent:] + 692
    frame #28: 0x000000018d10f370 UIKit-[UIWindow sendEvent:] + 1172
    frame #29: 0x000000018d0e0b50 UIKit-[UIApplication sendEvent:] + 256
    frame #30: 0x000000018d0dec40 UIKit_UIApplicationHandleEventQueue + 8500
    frame #31: 0x000000018a0d77f4 CoreFoundation`CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 24
    frame #32: 0x000000018a0d6b50 CoreFoundation`CFRunLoopDoSources0 + 256
    frame #33: 0x000000018a0d4de8 CoreFoundation__CFRunLoopRun + 632
    frame #34: 0x000000018a015dd0 CoreFoundationCFRunLoopRunSpecific + 452
    frame #35: 0x0000000100240460 CameraAppUIKit_PumpEvents(_this=0x000000012e60b960) + 88 at SDL_uikitevents.m:57
    frame #36: 0x0000000100174e88 CameraAppSDL_PumpEvents + 56 at SDL_events.c:388
    frame #37: 0x0000000100249154 CameraAppXli::Window::ProcessMessages() + 36 at SDL2Window.cpp:700
    frame #38: 0x0000000100245f80 CameraAppXli::Application::Run(app=0x000000016fdb9900, flags=10) + 428 at Application.cpp:43
    frame #39: 0x000000010015a358 CameraAppMain(args=0x000000016fdb9998) + 104 at Main.cpp:298
    frame #40: 0x000000010024e374 CameraAppSDL_main(argc=1, argv=0x0000000178017da0) + 228 at Main.cpp:52
    frame #41: 0x000000010023fd0c CameraApp-[SDLUIKitDelegate postFinishLaunch](self=0x0000000170017230, _cmd=0x000000010031c440) + 60 at SDL_uikitappdelegate.m:189
    frame #42: 0x000000018ac8c2d0 FoundationNSFireDelayedPerform + 392
    frame #43: 0x000000018a0d7704 CoreFoundation`CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION + 28
    frame #44: 0x000000018a0d7374 CoreFoundation`CFRunLoopDoTimer + 804
    frame #45: 0x000000018a0d509c CoreFoundation__CFRunLoopRun + 1324
    frame #46: 0x000000018a015dd0 CoreFoundationCFRunLoopRunSpecific + 452
    frame #47: 0x000000018fcfdc0c GraphicsServicesGSEventRunModal + 168
    frame #48: 0x000000018d146fc4 UIKitUIApplicationMain + 1156
    frame #49: 0x000000010023f4ac CameraAppmain(argc=1, argv=0x000000016fdbbc08) + 464 at SDL_uikitappdelegate.m:59
    frame #50: 0x0000000196c13aa0 libdyld.dylibstart + 4

I’m guessing you want frame #19 now:

(lldb) po info
{
    UIImagePickerControllerMediaMetadata =     {
        DPIHeight = 72;
        DPIWidth = 72;
        FaceRegions =         {
            Regions =             {
                HeightAppliedTo = 960;
                RegionList =                 (
                                        {
                        AngleInfoRoll = 270;
                        AngleInfoYaw = 0;
                        ConfidenceLevel = 284;
                        FaceID = 1;
                        Height = "0.4458333333333333";
                        Timestamp = 10074616131692;
                        Type = Face;
                        Width = "0.334375";
                        X = "0.6765625";
                        Y = "0.3395833333333333";
                    }
                );
                WidthAppliedTo = 1280;
            };
        };
        Orientation = 6;
        "{Exif}" =         {
            ApertureValue = "2.526068811667587";
            BrightnessValue = "0.1699400072484522";
            ColorSpace = 1;
            DateTimeDigitized = "2015:04:23 19:21:52";
            DateTimeOriginal = "2015:04:23 19:21:52";
            ExposureMode = 0;
            ExposureProgram = 2;
            ExposureTime = "0.06666666666666667";
            FNumber = "2.4";
            Flash = 32;
            FocalLenIn35mmFilm = 33;
            FocalLength = "2.15";
            ISOSpeedRatings =             (
                400
            );
            LensMake = Apple;
            LensModel = "iPhone 5s front camera 2.15mm f/2.4";
            LensSpecification =             (
                "2.15",
                "2.15",
                "2.4",
                "2.4"
            );
            MeteringMode = 5;
            PixelXDimension = 1280;
            PixelYDimension = 960;
            SceneType = 1;
            SensingMethod = 2;
            ShutterSpeedValue = "3.906905022631062";
            SubsecTimeDigitized = 650;
            SubsecTimeOriginal = 650;
            WhiteBalance = 0;
        };
        "{MakerApple}" =         {
            1 = 0;
            3 =             {
                epoch = 0;
                flags = 1;
                timescale = 1000000000;
                value = 419775805300000;
            };
            4 = 1;
            5 = 219;
            6 = 202;
            7 = 1;
        };
        "{TIFF}" =         {
            DateTime = "2015:04:23 19:21:52";
            Make = Apple;
            Model = "iPhone 5s";
            Software = "7.1.2";
            XResolution = 72;
            YResolution = 72;
        };
    };
    UIImagePickerControllerMediaType = "public.image";
    UIImagePickerControllerOriginalImage = "<UIImage: 0x178282e90>";
}

(lldb)
9

This problem lies outside my iOS skills. This crashes:

    private NSNumber NumberForKey(NSDictionary dict, NSString key)
    {
        if (dict == null)
            return null;
        ObjC.ID id = dict.objectForKey(key);
        return id != null ? new NSNumber(id) : null;
    }

on new NSNumber(id), everything looks correct both before and after. id looks like this deeper in the trace:

id    NSCFNumber *    (int)1280    0xb000000000005002
10

Hi, sorry for the delayed reply.

I’ve looked into the NSNumber crash and got to the bottom of the issue. iOS uses specially crafted pointer values for NSNumber and a few other classes – the set of classes itself is not fixed. This is done to avoid allocations and improve performance (Tagged Pointers in Objective C). As it turns out, on iOS 7 this feature doesn’t play nicely with the objc_setAssociatedObject API we use together with native bindings. This is not an issue on iOS 8.

I have added a workaround specifically for iOS 7 which will make it to a future release.