ffmpegapi使⽤scale_npp的问题总结
背景
使⽤ffmpeg cuda解码输出的像素格式是119，通过av_hwframe_transfer_data()函数可以设置传输到内存的格式为NV12。
⽽最终需要的像素格式是BGR24。ffmpeg的sws_scale()函数⽀持NV12 YUV420 到BGR24的转换，不⽀持119的转换。
⽬前测试数据显⽰，NV12和YUVJ420P转换bgr24的cpu占⽤分别是13.2% 3.5%，即NV12转换BGR24更慢。这也和NV12的数据组织⽅式有关。
查看sws_scale源码，处理NV12和YUVJ420P的区别如下：
1、NV12初始化时设置chrToYV12，⽽YUVJ420P不设置这个函数指针。
2、nv12ToUV_c函数遍历⼀⾏数据，将交错的UV放⼊两个数组；
3、chr_convert函数中，每⾏数据都会调⽤⼀次nv12ToUV_c；
再上⼀层的swscale函数还有⼀层循环for (; dstY < dstH; dstY++)中调⽤chr_convert；
nv12转RGB⽐yuv420转RGB消耗的CPU多，应该和nv12ToUV_c有关。
为尝试解决NV12转换BGR24的效率问题，尝试在GPU中将NV12转换为YUV420P，使⽤scale_npp的接⼝实现。对应的命令⾏如下，npp 像素格式转换：
ffmpeg -vsync 0 -hwaccel_device 2 -hwaccel cuda -hwaccel_output_format cuda -i ~/vedio/drone1.flv -
vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p" ff22cuda2.yuv
同时查看ffmpeg源码，确认scale_npp⽀持NV12到YUV420P的转换：
vf_scale_npp.c
static const enum [] = {
,
,
,
};
遇到的问题
问题⼀：Impossible to convert between the formats supported by the filter 'in' and the filter 'auto_scaler_0'
原因是 pix_fmt设置错误，需要设置为AV_PIX_FMT_CUDA
avfilter_graph_create_filter()函数作⽤是Create and add a filter instance into an existing graph. 参照doc/examples/filtering_video.c中的函数，调⽤avfilter_graph_create_filter时需设置args。
当前程序设置的是"video_size=1920x1080:pix_fmt=119:time_ba=1/1000:pixel_aspect=1/1"
从AVCodecContext *dec_ctx获取的pix_fmt在解码前是0，需要修改为119才⾏。因为scale_npp的输⼊是显卡上的数据，cuda解码的输出格式就是119。
命令⾏⽅式使⽤scale_npp也必须设置-hwaccel_output_format cuda才⾏。
问题⼆：No hw context provided on input
原因是 input filter需要设置hw_frames_ctx；
1、经查看报错代码在libavfilter/vf_scale_npp.c中的init_processing_chain()函数.
2、查看ffmpeg命令⾏⽅式在调⽤scale_npp的区别，发现fftools/ffmpeg_filter.c中的configure_input_video_filter()函数，在创建filter之后设置了hw_frames_ctx;
if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name, args.str, NULL, fg->graph)) < 0)
goto fail;
par->hw_frames_ctx = ifilter->hw_frames_ctx;
ret = av_buffersrc_parameters_t(ifilter->filter, par);
if (ret < 0)
goto fail;
av_freep(&par);
因此，在demo中通过AVBufferSrcParameters设置hw_frames_ctx即可解决此问题。
ffmeg filter源码相关
filter相关的实现代码都在libavfilter中，scale_npp相关的在libavfilter/vf_scale_npp.c中。
成长需要挫折
⽽ffmpeg命令⾏相关的功能代码均在fftools命令下，包括参数解析，编解码，缩放等。也是调⽤的libavcodec libavfilter等库。
其中filter相关都在fftools/fffmpeg_filter.c⽂件中处理。
重要的函数调⽤堆栈：
(gdb) bt
#0 filter_query_formats (ctx=0x3f65f40) at libavfilter/avfiltergraph.c:333
#1 0x00000000004dc154 in query_formats (graph=graph@entry=0x3f53c80, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:456 #2 0x00000000004dcfaf in graph_config_formats (log_ctx=<optimized out>, graph=<optimized out>) at libavfilter/avfiltergraph.c:1166
#3 avfilter_graph_config (graphctx=0x3f53c80, log_ctx=log_ctx@entry=0x0) at libavfilter/avfiltergraph.c:1277
#4 0x00000000004a3b0d in configure_filtergraph (fg=fg@entry=0x23f1940) at fftools/ffmpeg_filter.c:1107 //初始化filter_graph
#5 0x00000000004b432d in ifilter_nd_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2166
artifact#6 nd_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247
#7 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00,
got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#8 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
#9 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#10 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
童话的歌词#11 transcode () at fftools/ffmpeg.c:4665
(gdb) bt
#0 nppscale_deinterleave (ctx=ctx@entry=0x3f51a80, stage=stage@entry=0x3e836c8, out=0x3f64cc0, in=0x3f67c00) at
libavfilter/vf_scale_npp.c:389
#1 0x00000000005c7840 in nppscale_scale (in=0x3f67c00, out=0x3f67e80, ctx=0x3f51a80) at libavfilter/vf_scale_npp.c:477 //执⾏nppscale转换
#2 nppscale_filter_frame (link=link@entry=0x3f66b40, in=0x3f67c00) at libavfilter/vf_scale_npp.c:526
#3 0x00000000004db273 in ff_filter_frame_framed (frame=0x3f67c00, link=0x3f66b40) at libavfilter/avfilter.c:1066
#4 ff_filter_frame_to_filter (link=0x3f66b40) at libavfilter/avfilter.c:1214
#5 ff_filter_activate_default (filter=<optimized out>) at libavfilter/avfilter.c:1263
#6 ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1424
#7 0x00000000004de97c in ff_filter_graph_run_once (graph=graph@entry=0x3f53c80) at libavfilter/avfiltergraph.c:1456
#8 0x00000000004df918 in push_frame (graph=0x3f53c80) at libavfilter/buffersrc.c:184
#9 av_buffersrc_add_frame_internal (ctx=ctx@entry=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at
addicts
libavfilter/buffersrc.c:247
#10 0x00000000004dff5d in av_buffersrc_add_frame_flags (ctx=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:167
#11 0x00000000004b4349 in ifilter_nd_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2173
#12 nd_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247
third
#13 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00,
got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#14 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
通讯录英文#15 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#16 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
#17 transcode () at fftools/ffmpeg.c:4665
(gdb) bt
#0 hwdownload_filter_frame (link=link@entry=0x3f65700, input=0x3f67e80) at libavfilter/vf_hwdownload.c:152
#1 0x00000000004db273 in ff_filter_frame_framed (frame=0x3f67e80, link=0x3f65700) at libavfilter/avfilter.c:1066
#2 ff_filter_frame_to_filter (link=0x3f65700) at libavfilter/avfilter.c:1214
#3 ff_filter_activate_default (filter=<optimized out>) at libavfilter/avfilter.c:1263
#4 ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1424
#5 0x00000000004de97c in ff_filter_graph_run_once (graph=graph@entry=0x3f53c80) at libavfilter/avfiltergraph.c:1456
#6 0x00000000004df918 in push_frame (graph=0x3f53c80) at libavfilter/buffersrc.c:184
#7 av_buffersrc_add_frame_internal (ctx=ctx@entry=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:247
#8 0x00000000004dff5d in av_buffersrc_add_frame_flags (ctx=0x3f65f40, frame=frame@entry=0x336d880, flags=flags@entry=4) at libavfilter/buffersrc.c:167
#9 0x00000000004b4349 in ifilter_nd_frame (frame=0x336d880, ifilter=0x284aec0) at fftools/ffmpeg.c:2173
#10 nd_frame_to_filters (ist=ist@entry=0x23f4e40, decoded_frame=decoded_frame@entry=0x336d880) at fftools/ffmpeg.c:2247 #11 0x00000000004b4b81 in decode_video (ist=ist@entry=0x23f4e40, pkt=pkt@entry=0x7fffffffda00,
got_output=got_output@entry=0x7fffffffd710, duration_pts=duration_pts@entry=0x7fffffffd718,
eof=eof@entry=0, decode_failed=decode_failed@entry=0x7fffffffd714) at fftools/ffmpeg.c:2446
#12 0x00000000004b6cc2 in process_input_packet (no_eof=0, pkt=0x7fffffffd9a0, ist=0x23f4e40) at fftools/ffmpeg.c:2600
#13 process_input (file_index=<optimized out>) at fftools/ffmpeg.c:4491
#14 0x00000000004b9c53 in transcode_step () at fftools/ffmpeg.c:4611
#15 transcode () at fftools/ffmpeg.c:4665
需要关注的函数：
av_buffersrc_add_frame_flags
av_buffersink_get_frame
avfilter_graph_create_filter
avfilter_graph_par_ptr
avfilter_graph_config
sws_scale
fftools中的函数：
int configure_filtergraph(FilterGraph *fg)
static int configure_input_video_filter(FilterGraph *fg, InputFilter *ifilter, AVFilterInOut *in)
hw_device_tup_for_filter
static int hwaccel_retrieve_data(AVCodecContext *avctx, AVFrame *input)
static int transcode_step(void)
static int ifilter_nd_frame(InputFilter *ifilter, AVFrame *frame)spawned
需要关注的结构体定义：
AVFilterGraph
AVFilterContext
AVBufferSrcParameters
AVFilterInOut
AVFilter
AVCodecContext
新思维教育
AVFrame
参考信息
: This is a scaling filter implemented in . It's primary dependency is the , and it must be explicitly enabled by passing --enable-libnpp, --enable-cuda-nvcc and --enable-nonfree flags to ./configure at compile time when building FFmpeg from source. U this filter in place
of scale_cuda wherever possible.
像素格式说明：
= 0, ///< planar YUV 4:2:0, 12bpp, (1 Cr & Cb sample per 2x2 Y samples)
小车开车步骤= 2, ///< packed RGB 8:8:8, 24bpp,
= 3, ///< packed RGB 8:8:8, 24bpp,
= 12, ///< planar YUV 4:2:0, 12bpp, full scale (JPEG), deprecated in favor of AV_PIX_FMT_YUV420P and tting color_range
= 23, ///< planar YUV 4:2:0, 12bpp, 1 plane for Y and 1 plane for the UV components, which are interleaved (first byte U and the following byte V)
= 119, ///< HW acceleration through CUDA. data[i] contain CUdeviceptr pointers exactly as for system memory frames.
trinaffmpeg编译时configure配置命令（前提需要准备cuda库，nv-codec-headers，libx264，fdk_aac等库）：
PKG_CONFIG_PATH="$HOME/local/lib/pkgconfig" ./configure --prefix="$HOME/local" --pkg-config-flags="--static" --extra-cflags="-
I$HOME/local/include" --extra-ldflags="-L$HOME/local/lib" --extra-libs=-lpthread --extra-libs=-lm --bindir="$HOME/local/bin" --enable-gpl --enable-libfdk_aac --enable-libmp3lame --enable-libx264 --enable-nonfree --enable-gpl --enable-cuda --enable-cuvid --enable-nvdec --enable-nvenc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
编译ffmpeg因nv-codec-headers版本不对导致的报错：
