1. ffmpeg
  2. Commit cf29f49d8ae00bb153c24b5c8a8f6cb150a91de8
Browse code

rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C. Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead. Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>

Xi Wang authored on 2013/01/23 10:58:07
Showing 1 changed files
libavformat/rtpenc_h264.c
History View file @ cf29f49
... ... 
@@ -31,14 +31,14 @@
31 31
32 32 
 static const uint8_t *avc_mp4_find_startcode(const uint8_t *start, const uint8_t *end, int nal_length_size)
33 33 
 {
34 
-    int res = 0;
34 
+    unsigned int res = 0;
35 35
36 36 
     if (end - start < nal_length_size)
37 37 
         return NULL;
38 38 
     while (nal_length_size--)
39 39 
         res = (res << 8) | *start++;
40 40
41 
-    if (start + res > end || res < 0 || start + res < start)
41 
+    if (res > end - start)
42 42 
         return NULL;
43 43
44 44 
     return start + res;