USB Stream 组件说明

[English]

usb_stream 是基于 ESP32-S2/ESP32-S3 的 USB UVC + UAC 主机驱动程序，支持从 USB 设备读取/写入/控制多媒体流。例如最多同时支持 1 路摄像头 + 1 路麦克风 + 1 路播放器数据流。

特性：

  1. 支持通过 UVC Stream 接口获取视频流，支持批量和同步两种传输模式

  2. 支持通过 UAC Stream 接口获取麦克风数据流，发送播放器数据流

  3. 支持通过 UAC Control 接口控制麦克风音量、静音等特性

  4. 支持自动解析设备配置描述符

  5. 支持对数据流暂停和恢复

USB Stream 用户指南

  • 开发板

    1. 可以使用任何带有 USB 接口的 ESP32-S2/ESP32-S3 开发板，注意该 USB 接口需要能够向外供电

  • USB UVC 功能

    1. 摄像头必须兼容 USB1.1 全速（Fullspeed）模式

    2. 摄像头需要自带 MJPEG 压缩

    3. 用户可通过 uvc_streaming_config() 函数手动指定相机接口、传输模式和图像帧参数

    4. 如使用同步传输摄像头，需要支持设置接口 MPS（Max Packet Size） 为 512

    5. 同步传输模式，图像数据流 USB 传输总带宽应小于 4 Mbps （500 KB/s）

    6. 批量传输模式，图像数据流 USB 传输总带宽应小于 8.8 Mbps （1100 KB/s）

    7. 其它特殊相机要求，请参考示例程序 README.md

  • USB UAC 功能

    1. 音频功能必须兼容 UAC 1.0 协议

    2. 用户需要通过 uac_streaming_config() 函数手动指定 spk/mic 采样率，位宽参数

  • USB UVC + UAC 功能

    1. UVC 和 UAC 功能可以单独启用，例如仅配置 UAC 来驱动一个 USB 耳机，或者仅配置 UVC 来驱动一个 USB 摄像头

    2. 如需同时启用 UVC + UAC，该驱动程序目前仅支持具有摄像头和音频接口的复合设备（Composite Device），不支持同时连接两个单独的设备

USB Stream API 参考

  1. 用户可通过 uvc_config_t 配置摄像头分辨率、帧率参数。通过 uac_config_t 配置音频采样率、位宽等参数。参数说明如下:

uvc_config_t uvc_config = {
    .frame_width = 320, // mjpeg width pixel, for example 320
    .frame_height = 240, // mjpeg height pixel, for example 240
    .frame_interval = FPS2INTERVAL(15), // frame interval (100µs units), such as 15fps
    .xfer_buffer_size = 32 * 1024, // single frame image size, need to be determined according to actual testing, 320 * 240 generally less than 35KB
    .xfer_buffer_a = pointer_buffer_a, // the internal transfer buffer
    .xfer_buffer_b = pointer_buffer_b, // the internal transfer buffer
    .frame_buffer_size = 32 * 1024, // single frame image size, need to determine according to actual test
    .frame_buffer = pointer_frame_buffer, // the image frame buffer
    .frame_cb = &camera_frame_cb, //camera callback, can block in here
    .frame_cb_arg = NULL, // camera callback args
}

uac_config_t uac_config = {
    .mic_bit_resolution = 16, //microphone resolution, bits
    .mic_samples_frequence = 16000, //microphone frequence, Hz
    .spk_bit_resolution = 16, //speaker resolution, bits
    .spk_samples_frequence = 16000, //speaker frequence, Hz
    .spk_buf_size = 16000, //size of speaker send buffer, should be a multiple of spk_ep_mps
    .mic_buf_size = 0, //mic receive buffer size, 0 if not use, else should be a multiple of mic_min_bytes
    .mic_cb = &mic_frame_cb, //mic callback, can not block in here
    .mic_cb_arg = NULL, //mic callback args
};

  1. 使用 uvc_streaming_config() 配置 UVC 驱动，如果设备同时支持音频，可使用 uac_streaming_config() 配置 UAC 驱动

  2. 使用 usb_streaming_start() 打开数据流，之后驱动将响应设备连接和协议协商

  3. 之后，主机将根据用户参数配置，匹配已连接设备的描述符，如果设备无法满足配置要求，驱动将以警告提示

  4. 如果设备满足用户配置要求，主机将持续接收 IN 数据流（UVC 和 UAC 麦克风），并在新帧准备就绪后调用用户的回调

    1. 接收到新的 MJPEG 图像后，将触发 UVC 回调函数。用户可在回调函数中阻塞，因为它在独立的任务上下文中工作

    2. 接收到 mic_min_bytes 字节数据后，将触发 mic 回调。但是这里的回调一定不能以任何方式阻塞，否则会影响下一帧的接收。如果需要对 mic 进行阻塞操作，可以通过 uac_mic_streaming_read 轮询方式代替回调方式

  5. 发送扬声器数据时，用户可通过 uac_spk_streaming_write() 将数据写入内部 ringbuffer，主机将在 USB 空闲时从中获取数据并发送 OUT 数据

  6. 使用 usb_streaming_control() 控制流挂起/恢复。如果 UAC 支持特性单元，可通过其分别控制麦克风和播放器的音量和静音

  7. 使用 usb_streaming_stop() 停止 USB 流，USB 资源将被完全释放。

Bug 报告

ESP32-S2 ECO0 芯片 SPI 屏幕和 USB 同时启用，可能导致屏幕抖动

  1. 在最早版本的 ESP32-S2 ECO0 芯片上, USB 可能污染 SPI 数据 (ESP32-S2 新版本>=ECO1 和 ESP32-S3 均不存在该 Bug)

  2. 软件解决方案

  • spi_ll.h 添加以下接口

//components/hal/esp32s2/include/hal/spi_ll.h
static inline uint32_t spi_ll_tx_get_fifo_cnt(spi_dev_t *hw)
{
    return hw->dma_out_status.out_fifo_cnt;
}

  • 如下在 spi_new_trans 中添加额外检查

// The function is called to send a new transaction, in ISR or in the task.
// Setup the transaction-specified registers and linked-list used by the DMA (or FIFO if DMA is not used)
static void SPI_MASTER_ISR_ATTR spi_new_trans(spi_device_t *dev, spi_trans_priv_t *trans_buf)
{
    //...................
    spi_hal_setup_trans(hal, hal_dev, &hal_trans);
    spi_hal_prepare_data(hal, hal_dev, &hal_trans);

    //Call pre-transmission callback, if any
    if (dev->cfg.pre_cb) dev->cfg.pre_cb(trans);
#if 1
    //USB Bug workaround
    //while (!((spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 12) || (spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == trans->length / 8))) {
    while (trans->length && spi_ll_tx_get_fifo_cnt(SPI_LL_GET_HW(host->id)) == 0) {
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
        __asm__ __volatile__("nop");
    }
#endif
    //Kick off transfer
    spi_hal_user_start(hal);
}

Examples

usb/host/usb_camera_mic_spk

API Reference

Header File

Functions

esp_err_t uvc_streaming_config(const uvc_config_t *config)

Config UVC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Return

esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

Parameters

  • config: parameters defined in uvc_config_t

esp_err_t uac_streaming_config(const uac_config_t *config)

Config UAC streaming with user defined parameters.For normal use, user only need to specify no-optional parameters, and set optional parameters to 0 (the driver will find the correct value from the device descriptors). For quick start mode, user should specify all parameters manually to skip get and process descriptors steps.

Return

esp_err_t ESP_ERR_INVALID_STATE USB streaming is running, user need to stop streaming first ESP_ERR_INVALID_ARG Invalid argument ESP_OK Success

Parameters

  • config: parameters defined in uvc_config_t

esp_err_t usb_streaming_start(void)

Start usb streaming with pre-configs, usb driver will create internal tasks to handle usb data from stream pipe, and run user’s callback after new frame ready.

Return

ESP_ERR_INVALID_STATE streaming not configured, or streaming running already ESP_FAIL start failed ESP_OK start succeed

esp_err_t usb_streaming_stop(void)

Stop current usb streaming, internal tasks will be delete, related resourse will be free.

Return

ESP_ERR_INVALID_STATE streaming not started ESP_ERR_TIMEOUT stop wait timeout ESP_OK stop succeed

esp_err_t usb_streaming_connect_wait(size_t timeout_ms)

Wait for USB device connection.

Return

esp_err_t ESP_ERR_INVALID_STATE: usb streaming not started ESP_ERR_TIMEOUT: timeout ESP_OK: device connected

Parameters

  • timeout_ms: timeout in ms

esp_err_t usb_streaming_state_register(state_callback_t *cb, void *user_ptr)

This function registers a callback for USB streaming, please note that only one callback can be registered, the later registered callback will overwrite the previous one.

Return

esp_err_t

  • ESP_OK Success

  • ESP_ERR_INVALID_STATE USB streaming is running, callback need register before start

Parameters

  • cb: A pointer to a function that will be called when the USB streaming state changes.

  • user_ptr: user_ptr is a void pointer.

esp_err_t usb_streaming_control(usb_stream_t stream, stream_ctrl_t ctrl_type, void *ctrl_value)

Control USB streaming with specific stream and control type.

Return

ESP_ERR_INVALID_ARG invalid arg ESP_ERR_INVALID_STATE driver not configured or not started ESP_ERR_NOT_SUPPORTED current device not support this control type ESP_FAIL control failed ESP_OK succeed

Parameters

  • stream: stream type defined in usb_stream_t

  • ctrl_type: control type defined in stream_ctrl_t

  • ctrl_value: control value

esp_err_t uac_spk_streaming_write(void *data, size_t data_bytes, size_t timeout_ms)

Write data to the speaker buffer, will be send out when USB device is ready.

Return

ESP_ERR_INVALID_STATE spk stream not config ESP_ERR_NOT_FOUND spk interface not found ESP_ERR_TIMEOUT spk ringbuf full ESP_OK succeed

Parameters

  • data: The data to be written.

  • data_bytes: The size of the data to be written.

  • timeout_ms: The timeout value for writing data to the buffer.

esp_err_t uac_mic_streaming_read(void *buf, size_t buf_size, size_t *data_bytes, size_t timeout_ms)

Read data from internal mic buffer, the actual size will be returned.

Return

ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE mic stream not config ESP_ERR_NOT_FOUND mic interface not found ESP_TIMEOUT timeout ESP_OK succeed

Parameters

  • buf: pointer to the buffer to store the received data

  • buf_size: The size of the data buffer.

  • data_bytes: The actual size read from buffer

  • timeout_ms: The timeout value for the read operation.

esp_err_t uac_frame_size_list_get(usb_stream_t stream, uac_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the audio frame size list of current stream, the list contains audio channel number, bit resolution and samples frequence. IF list_size equals 1 and the samples_frequence equals 0, which means the frequency can be set to any value between samples_frequence_min and samples_frequence_max.

Return

esp_err_t

  • ESP_ERR_INVALID_ARG Parameter error

  • ESP_ERR_INVALID_STATE USB device not active

  • ESP_OK Success

Parameters

  • stream: the stream type

  • frame_list: the output frame list, NULL to only get the list size

  • list_size: frame list size

  • cur_index: current frame index

esp_err_t uac_frame_size_reset(usb_stream_t stream, uint8_t ch_num, uint16_t bit_resolution, uint32_t samples_frequence)

Reset audio channel number, bit resolution and samples frequence, please reset when the streaming in suspend state. The new configs will be effective after streaming resume.

Return

esp_err_t

  • ESP_ERR_INVALID_ARG Parameter error

  • ESP_ERR_INVALID_STATE USB device not active

  • ESP_ERR_NOT_FOUND frequency not found

  • ESP_OK Success

  • ESP_FAIL Reset failed

Parameters

  • stream: stream type

  • ch_num: audio channel numbers

  • bit_resolution: audio bit resolution

  • samples_frequence: audio samples frequence

esp_err_t uvc_frame_size_list_get(uvc_frame_size_t *frame_list, size_t *list_size, size_t *cur_index)

Get the frame size list of current connected camera.

Return

esp_err_t ESP_ERR_INVALID_ARG parameter error ESP_ERR_INVALID_STATE uvc stream not config or not active ESP_OK succeed

Parameters

  • frame_list: the frame size list, can be NULL if only want to get list size

  • list_size: the list size

  • cur_index: current frame index

esp_err_t uvc_frame_size_reset(uint16_t frame_width, uint16_t frame_height, uint32_t frame_interval)

Reset the expected frame size and frame interval, please reset when uvc streaming in suspend state.The new configs will be effective after streaming resume.

Note: frame_width and frame_height can be set to 0 at the same time, which means no change on frame size.

Return

esp_err_t

Parameters

  • frame_width: frame width, FRAME_RESOLUTION_ANY means any width

  • frame_height: frame height, FRAME_RESOLUTION_ANY means any height

  • frame_interval: frame interval, 0 means no change

Structures

struct uvc_config

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

Public Members

uint16_t frame_width

Picture width, set FRAME_RESOLUTION_ANY for any resolution

uint16_t frame_height

Picture height, set FRAME_RESOLUTION_ANY for any resolution

uint32_t frame_interval

Frame interval in 100-ns units, 666666 ~ 15 Fps

uint32_t xfer_buffer_size

Transfer buffer size, using double buffer here, must larger than one frame size

uint8_t *xfer_buffer_a

Buffer a for usb payload

uint8_t *xfer_buffer_b

Buffer b for usb payload

uint32_t frame_buffer_size

Frame buffer size, must larger than one frame size

uint8_t *frame_buffer

Buffer for one frame

uvc_frame_callback_t *frame_cb

callback function to handle incoming frame

void *frame_cb_arg

callback function arg Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uvc_xfer_t xfer_type

(optional) UVC stream transfer type, UVC_XFER_ISOC or UVC_XFER_BULK

uint8_t format_index

(optional) Format index of MJPEG

uint8_t frame_index

(optional) Frame index, to choose resolution

uint16_t interface

(optional) UVC stream interface number

uint16_t interface_alt

(optional) UVC stream alternate interface, to choose MPS (Max Packet Size), bulk fix to 0

uint8_t ep_addr

(optional) endpoint address of selected alternate interface

uint32_t ep_mps

(optional) MPS of selected interface_alt

uint32_t flags

(optional) flags to control the driver behavers

struct mic_frame_t

mic frame type

Public Members

void *data

mic data

uint32_t data_bytes

mic data size

uint16_t bit_resolution

bit resolution in buffer

uint32_t samples_frequence

mic sample frequency

struct uvc_frame_size_t

uvc frame type

Public Members

uint16_t width

frame width

uint16_t height

frame height

struct uac_frame_size_t

uac frame type

Public Members

uint8_t ch_num

channel numbers

uint16_t bit_resolution

bit resolution in buffer

uint32_t samples_frequence

sample frequency

uint32_t samples_frequence_min

min sample frequency

uint32_t samples_frequence_max

max sample frequency

struct uac_config_t

UAC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptor parse, or a problem with the device descriptor.

Public Members

uint8_t spk_ch_num

speaker channel numbers, UAC_CH_ANY for any channel number

uint8_t mic_ch_num

microphone channel numbers, UAC_CH_ANY for any channel number

uint16_t mic_bit_resolution

microphone resolution(bits), UAC_BITS_ANY for any bit resolution

uint32_t mic_samples_frequence

microphone frequence(Hz), UAC_FREQUENCY_ANY for any frequency

uint16_t spk_bit_resolution

speaker resolution(bits), UAC_BITS_ANY for any

uint32_t spk_samples_frequence

speaker frequence(Hz), UAC_FREQUENCY_ANY for any frequency

uint32_t spk_buf_size

size of speaker send buffer, should be a multiple of spk_ep_mps

uint32_t mic_buf_size

mic receive buffer size, 0 if not use

mic_callback_t *mic_cb

mic callback, can not block in here!, NULL if not use

void *mic_cb_arg

mic callback args, NULL if not use Optional configs, Users need to specify parameters manually when they want to skip the get and process descriptors steps (used to speed up startup)

uint16_t mic_interface

(optional) microphone stream interface number, set 0 if not use

uint8_t mic_ep_addr

(optional) microphone interface endpoint address

uint32_t mic_ep_mps

(optional) microphone interface endpoint mps

uint16_t spk_interface

(optional) speaker stream interface number, set 0 if not use

uint8_t spk_ep_addr

(optional) speaker interface endpoint address

uint32_t spk_ep_mps

(optional) speaker interface endpoint mps

uint16_t ac_interface

(optional) audio control interface number, set 0 if not use

uint8_t mic_fu_id

(optional) microphone feature unit id, set 0 if not use

uint8_t spk_fu_id

(optional) speaker feature unit id, set 0 if not use

uint32_t flags

(optional) flags to control the driver behavers

Macros

FRAME_RESOLUTION_ANY

any uvc frame resolution

UAC_FREQUENCY_ANY

any uac sample frequency

UAC_BITS_ANY

any uac bit resolution

UAC_CH_ANY

any uac channel number

FPS2INTERVAL(fps)

convert fps to uvc frame interval

FRAME_INTERVAL_FPS_5

5 fps

FRAME_INTERVAL_FPS_10

10 fps

FRAME_INTERVAL_FPS_15

15 fps

FRAME_INTERVAL_FPS_20

20 fps

FRAME_INTERVAL_FPS_30

25 fps

FLAG_UVC_SUSPEND_AFTER_START

suspend uvc after usb_streaming_start

FLAG_UAC_SPK_SUSPEND_AFTER_START

suspend uac speaker after usb_streaming_start

FLAG_UAC_MIC_SUSPEND_AFTER_START

suspend uac microphone after usb_streaming_start

Type Definitions

typedef struct uvc_config uvc_config_t

UVC configurations, for params with (optional) label, users do not need to specify manually, unless there is a problem with descriptors, or users want to skip the get and process descriptors steps.

typedef void() mic_callback_t(mic_frame_t *frame, void *user_ptr)

user callback function to handle incoming mic frames

typedef void() state_callback_t(usb_stream_state_t state, void *user_ptr)

user callback function to handle usb device connection status

Enumerations

enum uvc_xfer_t

UVC stream usb transfer type, most camera using isochronous mode, bulk mode can also be support for higher bandwidth.

Values:

UVC_XFER_ISOC = 0

Isochronous Transfer Mode

UVC_XFER_BULK

Bulk Transfer Mode

UVC_XFER_UNKNOWN

Unknown Mode

enum usb_stream_t

Stream id, used for control.

Values:

STREAM_UVC = 0

usb video stream

STREAM_UAC_SPK

usb audio speaker stream

STREAM_UAC_MIC

usb audio microphone stream

STREAM_MAX

max stream id

enum usb_stream_state_t

USB device connection status.

Values:

STREAM_CONNECTED = 0
STREAM_DISCONNECTED
enum stream_ctrl_t

Stream control type, which also depends on if device support.

Values:

CTRL_NONE = 0

None

CTRL_SUSPEND

streaming suspend control. ctrl_data NULL

CTRL_RESUME

streaming resume control. ctrl_data NULL

CTRL_UAC_MUTE

mute control. ctrl_data (false/true)

CTRL_UAC_VOLUME

volume control. ctrl_data (0~100)

CTRL_MAX

max type value

提供有关此文档的反馈