Todo
 
 -- For other people

 - Multithread vc1.

 - Multithread an intra codec like mjpeg (trivial).

 - Fix mpeg1 (see below).
 - Try the first three items under Optimization.
 - Fix h264 (see below).
 - Try mpeg4 (see below).
 
 -- Bug fixes
 
 General critical:
 - Error resilience has to run before ff_report_frame_progress()
 is called. Otherwise there will be race conditions. (This might already
 work.) In general testing error paths should be done more.

 - 'make fate THREADS=2' doesn't pass. Most failures are due to
 bugs in vsync in ffmpeg.c, which are currently obscuring real failures.

 
 h264:

 - Files that aren't parsed (e.g. mp4) and contain PAFF with two
 field pictures in the same packet are not optimal. Modify the
 nals_needed check so that the second field's first slice is
 considered as needed, then uncomment the FIXME code in decode_postinit.
 Ex: http://astrange.ithinksw.net/ffmpeg/mt-samples/PAFF-Chalet-Tire.mp4

 
 mpeg4:
 - Packed B-frames need to be explicitly split up
 when frame threading is on. It's not very fast
 without this.
 - The buffer age optimization is disabled due to
 the way buffers are allocated across threads. The
 branch 'fix_buffer_age' has an attempt to fix it
 which breaks ffplay.
 - Support interlaced.
 
 mpeg1/2:
 - Seeking always prints "first frame not a keyframe"
 with threads on. Currently disabled for this reason.
 
 -- Prove correct
 
 - decode_update_progress() in h264.c

 race_checking branch has some work on h264,

 but not that function. It might be worth putting
 the branch under #ifdef DEBUG in mainline, but
 the code would have to be cleaner.
 - MPV_lowest_referenced_row() and co in mpegvideo.c
 - Same in vp3.
 
 -- Optimization
 

 - Merge h264 decode_update_progress() with loop_filter().
 Add CODEC_CAP_DRAW_HORIZ_BAND as a side effect.

 - EMU_EDGE is always set for h264 PAFF+MT
 because draw_edges() writes into the other field's

 thread's pixels. Needs an option to skip T/B fields.

 - Check update_thread_context() functions and make
 sure they only copy what they need to.
 - Try some more optimization of the "ref < 48; ref++"
 loop in h264.c await_references(), try turning the list0/list1 check
 above into a loop without being slower.
 - Support frame+slice threading at the same time
 by assigning slice_count threads for frame threads
 to use with execute(). This is simpler but unbalanced
 if only one frame thread uses any.
 
 -- Features
 
 - Support streams with width/height changing. This
 requires flushing all current frames (and buffering
 the input in the meantime), closing the codec and
 reopening it. Or don't support it.
 - Support encoding. Might need more threading primitives
 for good ratecontrol; would be nice for audio and libavfilter too.
 - Async decoding part 1: instead of trying to
 start every thread at the beginning, return a picture
 if the earliest thread is already done, but don't wait
 for it. Not sure what effect this would have.
 - Part 2: have an API that doesn't wait for the decoding
 thread, only returns EAGAIN if it's not ready. What will
 it do with the next input packet if it returns that?
 - Have an API that returns finished pictures but doesn't
 require sending new ones. Maybe allow NULL avpkt when
 not at the end of the stream.
 
 -- Samples
 
 http://astrange.ithinksw.net/ffmpeg/mt-samples/
 
 See yuvcmp.c in this directory to compare decoded samples.
 
 For debugging, try commenting out ff_thread_finish_setup calls so
 that only one thread runs at once, and then binary search+
 scatter printfs to look for differences in codec contexts.