Crash in dmz_YCbCr_to_RGB(_IplImage*, _IplImage*, _IplImage*, _IplImage**)

[nudge]

Hi,

David from Uber here. I was emailing back and forth with Jake (PayPal Technical Account Manager) and he recommended I open a GitHub issue so we can better track this issue.

We are still seeing this elusive crash that was addressed in the following pull request card-io/card.io-iOS-source#57. We see this across iOS versions 7, 8, 9 and iPhone devices 6s, 6, 6 Plus, 5s with around 1k crashes occurring in the past week ending 4 Apr 2016.

CardIO version 5.3.1 (https://github.com/card-io/card.io-iOS-SDK/tree/5.3.1)

Here is the stack trace:

Hardware Model:      iPhone6,1
Process:             UberClient [8093]
Path:                /var/mobile/Containers/Bundle/Application/3FCCECC0-3042-4168-8B0D-695136B95522/UberClient.app/UberClient
Identifier:          com.ubercab.UberClient
Version:             2.128.3 (2.128.3)
Code Type:           ARM-64
Parent Process:      unknown [1]
Date/Time:           2016-04-05 01:43:01.000 -0400
OS Version:          iPhone OS 9.2.1 (13D15)
Report Version:      104
Exception Type:      EXC_BAD_ACCESS (SIGBUS)
Exception Codes:     KERN_INVALID_TASK at 0x0000000000000010
Crashed Thread:      0

Thread 0 Crashed:
0   UberClient                     0x000000010085ba94 dmz_YCbCr_to_RGB() + 272
1   UberClient                     0x000000010085b9e4 dmz_YCbCr_to_RGB() + 96
2   UberClient                     0x0000000100840670 -[CardIOIplImage rgbImageWithY:cb:cr:] + 152
3   UberClient                     0x000000010083c04c -[CardIOVideoFrame imageWithGrayscale:] + 204
4   UberClient                     0x000000010087bd5c -[CardIOView didScanCard:] + 460
5   UberClient                     0x000000010087b924 -[CardIOView videoStream:didProcessFrame:] + 128
6   UberClient                     0x000000010087a2b4 -[CardIOCameraView videoStream:didProcessFrame:] + 484
7   UberClient                     0x0000000100847ba0 -[CardIOVideoStream sendFrameToDelegate:] + 100
8   Foundation                     0x00000001814ebe20 __NSThreadPerformPerform + 340
9   CoreFoundation                 0x0000000180ae0efc __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
10  CoreFoundation                 0x0000000180ae0990 __CFRunLoopDoSources0 + 540
11  CoreFoundation                 0x0000000180ade690 __CFRunLoopRun + 724
12  CoreFoundation                 0x0000000180a0d680 CFRunLoopRunSpecific + 384
13  GraphicsServices               0x0000000181f1c088 GSEventRunModal + 180
14  UIKit                          0x0000000185884d90 UIApplicationMain + 204
15  UberClient                     0x00000001007fd9ec main + 260
16  libdyld.dylib                  0x00000001805ae8b8 start + 4

Thread 0 crashed with ARM-64 Thread State:
  cpsr: 0x0000000060000000     fp: 0x000000016fd569c0     lr: 0x000000010085b9e4     pc: 0x000000010085ba94 
    sp: 0x000000016fd56960     x0: 0x0000010e000001ac     x1: 0x000000018078a8f4    x10: 0x00000000000001ac 
   x11: 0x0000000000000000    x12: 0x00000000000000ac    x13: 0x0000000101ff5020    x14: 0x0000000000000000 
   x15: 0x0000000000000001    x16: 0xfffffffffffffff1    x17: 0x0000000100840910    x18: 0x0000000000000000 
   x19: 0x0000000145886480     x2: 0x0000000000058000    x20: 0x0000000000000000    x21: 0x0000000000000000 
   x22: 0x00000001459188b0    x23: 0x0000000000000003    x24: 0x000000018620c367    x25: 0x0000000000000000 
   x26: 0x0000000000000000    x27: 0x000000019d24e000    x28: 0x000000010209c060    x29: 0x000000016fd569c0 
    x3: 0x0000000000000000     x4: 0x0000000003000001     x5: 0x0000000000000000     x6: 0x0000000000000000 
    x7: 0x0000000000000000     x8: 0x000000010cda8010     x9: 0x0000000000000001

We still cannot reproduce this reliably.

I have a hunch it may be to do with (CMSampleBufferRef)sampleBuffer disappearing before it hits [CardIOIplImage rgbImageWithY:cb:cr] (stack frame 2). According to the Apple documentation (https://developer.apple.com/library/mac/documentation/AVFoundation/Reference/AVCaptureVideoDataOutputSampleBufferDelegate_Protocol/#//apple_ref/occ/intfm/AVCaptureVideoDataOutputSampleBufferDelegate/captureOutput:didOutputSampleBuffer:fromConnection:), sampleBuffer is not guaranteed to stick around after method return, however its image data is being referenced well afterwards.

A direct reference to the pixel buffer of the sampleBuffer is stored in colocatedImage->imageData of CardIOIplImage instances. This buffer is accessed in a callback that is dispatched by performSelectorOnMainThread: in CardIOVideoStream, well after captureOutput:didOutputSampleBuffer:fromConnection: has returned. This memory might have been recycled by the system. Not sure how robust OpenCV is to corrupt image data, but could be worth a look.

if you need more information, feel free to reach out. Thanks for your help!

Cheers,
David

[bluk]

Thanks for the investigation David. We are taking a look.

[bluk]

Thanks for your detailed investigation. I wasn't able to reproduce the crash but based on your description and the Apple documentation, I made a few changes which should help in release 5.3.2. If you still see issues, please leave a comment.

[nudge]

Thanks @bluk , we'll integrate 5.3.2 and let you know our findings.

[designatednerd]

FWIW, we at SpotHero also saw this crash. Will also let you know if we see recurrence after upgrading to 5.3.2.

[designatednerd]

Just had a recurrence again today, after a release of the upgrade to v5.3.2 of the SDK. iPhone 6 running 9.3.1. Looks like the stack trace is largely the same as @nudge's pre-5.3.2 trace:

Crashed: com.apple.main-thread
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000010

0  SpotHero                       0x10027c7bc dmz_YCbCr_to_RGB(_IplImage*, _IplImage*, _IplImage*, _IplImage**) + 4297623484
1  SpotHero                       0x1002613ec +[CardIOIplImage rgbImageWithY:cb:cr:] + 4297511916
2  SpotHero                       0x10025cd50 -[CardIOVideoFrame imageWithGrayscale:] + 4297493840
3  SpotHero                       0x1002a50a8 -[CardIOView didScanCard:] + 4297789608
4  SpotHero                       0x1002a4c70 -[CardIOView videoStream:didProcessFrame:] + 4297788528
5  SpotHero                       0x1002a3610 -[CardIOCameraView videoStream:didProcessFrame:] + 4297782800
6  SpotHero                       0x100268964 -[CardIOVideoStream sendFrameToDelegate:] + 4297541988
7  Foundation                     0x183f9bffc __NSThreadPerformPerform + 340
8  CoreFoundation                 0x18357d124 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
9  CoreFoundation                 0x18357cbb8 __CFRunLoopDoSources0 + 540
10 CoreFoundation                 0x18357a8b8 __CFRunLoopRun + 724
11 CoreFoundation                 0x1834a4d10 CFRunLoopRunSpecific + 384
12 GraphicsServices               0x184d8c088 GSEventRunModal + 180
13 UIKit                          0x188779f70 UIApplicationMain + 204
14 SpotHero                       0x100169de4 main (main.m:14)
15 libdispatch.dylib              0x1830428b8 (Missing)

Crashlytics shows no memory or disk space pressure for this crash, so that's probably not it. FWIW, it's the first crash we've seen for this in about a month.

[bluk]

@designatednerd Thanks for the update. From what you're saying, I'm going to assume that the v5.3.2 fix helped since there wasn't a crash for a month, but there may still be another memory issue. Will go over the code again and see if there's any other noticeable memory issues.

[designatednerd]

@bluk Very possible, but I think @nudge was seeing this crash significantly more often in the first place, so it may be worth seeing how they're doing before you go too far down the rabbit hole.

[nudge]

We're still seeing this crash after integrating 5.3.2 into our builds.

The crash rate looks roughly the same as with pre-CardIO 5.3.2. Seeing around 1k crashes for the week ending 16-Apr 2016. We're still seeing it spread across many device types (iPhone 5-6s Plus) and OS versions (8.4-9.3.1). Will continue to do some more investigation and monitoring.

Incident Identifier: ED252731-4EB1-48C3-9A30-99763F8343C1
CrashReporter Key:   b7a6443459f54b8289e898865e93cdd28fd6138c
Hardware Model:      iPhone6,2
Process:             UberClient [710]
Path:                /var/containers/Bundle/Application/AC23CF6D-268B-418E-98C3-8A1EDCC4B942/UberClient.app/UberClient
Identifier:          com.ubercab.UberClient
Version:             2.133.2 (2.133.2)
Code Type:           ARM-64
Parent Process:      ? [1]
Date/Time:           2016-05-17 18:36:54.000 +1000
OS Version:          iPhone OS 9.3.1 (13E238)
Report Version:      104
Exception Type:      EXC_BAD_ACCESS (SIGBUS)
Exception Codes:     KERN_INVALID_TASK at 0x0000000000000010
Crashed Thread:      0

Thread 0 Crashed:
0   UberClient                     0x0000000100948a3c dmz_YCbCr_to_RGB() + 272
1   UberClient                     0x0000000100948988 dmz_YCbCr_to_RGB() + 92
2   UberClient                     0x000000010092d66c -[CardIOIplImage rgbImageWithY:cb:cr:] + 152
3   UberClient                     0x0000000100928fd0 -[CardIOVideoFrame imageWithGrayscale:] + 204
4   UberClient                     0x0000000100971328 -[CardIOView didScanCard:] + 460
5   UberClient                     0x0000000100970ef0 -[CardIOView videoStream:didProcessFrame:] + 128
6   UberClient                     0x000000010096f890 -[CardIOCameraView videoStream:didProcessFrame:] + 456
7   UberClient                     0x0000000100934be4 -[CardIOVideoStream sendFrameToDelegate:] + 100
8   Foundation                     0x0000000181417ffc __NSThreadPerformPerform + 340
9   CoreFoundation                 0x00000001809f9124 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
10  CoreFoundation                 0x00000001809f8bb8 __CFRunLoopDoSources0 + 540
11  CoreFoundation                 0x00000001809f68b8 __CFRunLoopRun + 724
12  CoreFoundation                 0x0000000180920d10 CFRunLoopRunSpecific + 384
13  GraphicsServices               0x0000000182208088 GSEventRunModal + 180
14  UIKit                          0x0000000185bf5f70 UIApplicationMain + 204
15  UberClient                     0x00000001008e034c main + 260
16  libdyld.dylib                  0x00000001804be8b8 start + 4

Thread 0 crashed with ARM-64 Thread State:
  cpsr: 0x0000000060000000     fp: 0x000000016fd0a9d0     lr: 0x0000000100948988     pc: 0x0000000100948a3c 
    sp: 0x000000016fd0a980     x0: 0x0000010e000001ac     x1: 0x000000018069e8f4    x10: 0x00000000000001ac 
   x11: 0x0000000000000000    x12: 0x000000000000009e    x13: 0x0000000102174ed0    x14: 0x0000000000000000 
   x15: 0x0000000000000001    x16: 0xfffffffffffffff1    x17: 0x000000010092d928    x18: 0x0000000000000000 
   x19: 0x0000000149c8a970     x2: 0x0000000000058000    x20: 0x0000000000000000    x21: 0x0000000000000000 
   x22: 0x000000014889e410    x23: 0x0000000000000003    x24: 0x00000001865a5b1d    x25: 0x0000000000000000 
   x26: 0x0000000000000000    x27: 0x000000019da49000    x28: 0x0000000102198058    x29: 0x000000016fd0a9d0 
    x3: 0x0000000000000000     x4: 0x0000000003000001     x5: 0x0000000000000000     x6: 0x0000000000000000 
    x7: 0x0000000000000000     x8: 0x000000010a19c010     x9: 0x0000000000000001

[designatednerd]

@bluk: Any update on this?

[bluk]

@designatednerd Unfortunately, no update right now. Haven't been able to reproduce this on different devices but will do more audits of the memory management pieces.

[sketchydroide]

From Fabric analysis I found that we only have this crash on the iPad, and I mean 100% iPad.
The iPad only represents about 5% of our iOS devices, so this seems relevant.
I hope it helps.

[AviSergey]

Hi guys.

From Fabric analysis we also constantly have the same issue and even for Card.io version 5.3.2.

Crashed: com.apple.main-thread
0  Rider                          0x1019514b8 dmz_YCbCr_to_RGB(_IplImage*, _IplImage*, _IplImage*, _IplImage**) + 4321137848
1  Rider                          0x1019360e8 +[CardIOIplImage rgbImageWithY:cb:cr:] + 4321026280
2  Rider                          0x101931a4c -[CardIOVideoFrame imageWithGrayscale:] + 4321008204
3  Rider                          0x10197a024 -[CardIOView didScanCard:] + 4321304612
4  Rider                          0x101979bec -[CardIOView videoStream:didProcessFrame:] + 4321303532
5  Rider                          0x10197858c -[CardIOCameraView videoStream:didProcessFrame:] + 4321297804
6  Rider                          0x10193d660 -[CardIOVideoStream sendFrameToDelegate:] + 4321056352
7  Foundation                     0x1826c802c __NSThreadPerformPerform + 340
8  CoreFoundation                 0x181ca909c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
9  CoreFoundation                 0x181ca8b30 __CFRunLoopDoSources0 + 540
10 CoreFoundation                 0x181ca6830 __CFRunLoopRun + 724
11 CoreFoundation                 0x181bd0c50 CFRunLoopRunSpecific + 384
12 GraphicsServices               0x1834b8088 GSEventRunModal + 180
13 UIKit                          0x186eba088 UIApplicationMain + 204
14 Rider                          0x100c89c68 main (AppDelegate.swift:20)
15 libdispatch.dylib              0x18176e8b8 (Missing)

Are there any news regarding this issue?

[sbarow]

Any update on this? Still seeing multiple crashes.

[ghulammustafa]

We're getting multiple reports of crashes because of this. Any update?

[jcayzac]

It looks like -[CardIOIplImage rgbImageWithY:cb:cr:] could implement some sanity checks for NULL pointers, given that the dmz/opencv code it calls doesn't check anything.

[filblue]

Hello, do you guys have any chance to look closer into this? This is a consistent crash producer for our users.

[jcayzac]

The project seems dead (no commit in 5 months)