SpecsMedia
Reference for SpecsMedia, the SPECS NDK low-level media pipeline for demuxing, encoding, decoding, buffers and timed presentation.
SpecsMedia is the low-level media pipeline for SPECS. It provides direct access to media demuxing (SpecsMediaExtractor), codec encode/decode (SpecsMediaCodec), timed presentation (SpecsMediaPresenter), hardware buffers (SpecsMediaBuffer), and GPU texture output (SpecsMediaGLTexture).
Use SpecsMedia when you need raw control over the pipeline—transcoding, frame extraction, custom rendering, or streaming to custom buffers. For playback alone, use SpecsPlayer instead.
Include and link
#include <SpecsMedia.h> // error codes, result types
#include <SpecsMediaFormat.h> // format key-value store
#include <SpecsMediaExtractor.h> // demuxer
#include <SpecsMediaCodec.h> // encoder / decoder
#include <SpecsMediaPresenter.h> // timed display presentation
#include <SpecsMediaBuffer.h> // reference-counted hardware buffers
#include <SpecsMediaGLTexture.h> // SpecsMediaBuffer -> OpenGL ES texture
target_link_libraries(myapp specs_media)
All SpecsMedia headers are in sysroots/armv8a-snap-linux/usr/include/.
Typical pipeline
Media file
│
▼
SpecsMediaExtractor — demux: reads encoded samples track by track
│ encoded samples
▼
SpecsMediaCodec (decoder) — decode: produces raw PCM / raw video frames
│ raw frames (SpecsMediaBuffer when configured with OUTPUT_MEDIA_BUFFER)
▼
SpecsMediaPresenter — present: timed display with optional pose tracking
│ — or —
SpecsMediaGLTexture — update a caller-owned GL texture for your own rendering
│ — or —
Your rendering / processing code
For encoding, feed raw data into SpecsMediaCodec (encoder) and write output to a container.
Result codes (SpecsMedia.h)
// Since: v1.3.0
typedef enum SpecsMedia_Result {
SPECSMEDIA_SUCCESS = 0, // no error
SPECSMEDIA_ERROR_GENERIC = -1, // unspecified error
SPECSMEDIA_ERROR_NOT_INITIALIZED = -2, // module not initialized
SPECSMEDIA_ERROR_BAD_ARGUMENT = -3, // null or invalid argument
SPECSMEDIA_ERROR_NOT_IMPLEMENTED = -4, // functionality not implemented
SPECSMEDIA_ERROR_AGAIN = -5, // resource temporarily unavailable (retry)
SPECSMEDIA_ERROR_NOMEM = -6, // out of memory
SPECSMEDIA_ERROR_TIMED_OUT = -7, // operation timed out
SPECSMEDIA_ERROR_SHUTDOWN = -8, // operation rejected: object is shut down
SPECSMEDIA_ERROR_INVALID_STATE = -9, // operation invalid for current state
SPECSMEDIA_ERROR_UNSUPPORTED = -10, // feature, format, or mode unsupported
SPECSMEDIA_ERROR_RESOURCE_EXHAUSTED = -11 // required resources exhausted; release unused resources before retrying
} SpecsMedia_Result;
// Backward-compatibility alias
// Since: v1.3.0
typedef SpecsMedia_Result SpecsMedia_Result_t;
Logging
The SpecsMedia internals emit logs through a runtime-configurable verbosity. Lower levels include all higher (less-verbose) levels.
// Since: v1.3.0
typedef enum SpecsMedia_LogLevel {
SPECSMEDIA_LOGLEVEL_DEBUG = 0, // debug, info, warning, and error
SPECSMEDIA_LOGLEVEL_INFO = 1, // info, warning, and error
SPECSMEDIA_LOGLEVEL_WARN = 2, // warning and error
SPECSMEDIA_LOGLEVEL_ERROR = 3, // error only
SPECSMEDIA_LOGLEVEL_NONE = 4, // disable SpecsMedia logs
} SpecsMedia_LogLevel_t;
// Set the global SpecsMedia log level.
// Returns SPECSMEDIA_ERROR_BAD_ARGUMENT for an invalid level.
// Since: v1.3.0
SpecsMedia_Result_t SpecsMedia_setLogLevel(SpecsMedia_LogLevel_t level);
Shared encoding types (SpecsMedia.h)
SpecsMedia.h defines a set of shared resolution and encoder types that are reused across the media APIs. Both SpecsRecorder and the SpecsCapture API consume these canonical types rather than defining their own.
// Since: v1.3.0
typedef struct SpecsMedia_Resolution {
uint32_t width; // picture width in pixels
uint32_t height; // picture height in pixels
} SpecsMedia_Resolution_t;
// Since: v1.3.0
typedef enum SpecsMedia_RateControl {
SPECSMEDIA_RATE_CONTROL_DISABLE = 0, // Constant QP (no rate control)
SPECSMEDIA_RATE_CONTROL_CBR = 1, // Constant Bitrate
SPECSMEDIA_RATE_CONTROL_CBR_VFR = 2, // Constant Bitrate, Variable Framerate
SPECSMEDIA_RATE_CONTROL_VBR_CFR = 3, // Variable Bitrate, Constant Framerate
SPECSMEDIA_RATE_CONTROL_VBR_VFR = 4, // Variable Bitrate, Variable Framerate
} SpecsMedia_RateControl_t;
// Since: v1.3.0
typedef enum SpecsMedia_VideoCodec {
SPECSMEDIA_VIDEO_CODEC_DEFAULT = 0, // implementation default (H.265)
SPECSMEDIA_VIDEO_CODEC_H265 = 1, // ITU-T H.265 / HEVC
SPECSMEDIA_VIDEO_CODEC_H264 = 2, // ITU-T H.264 / AVC
} SpecsMedia_VideoCodec_t;
// Since: v1.3.0
typedef enum SpecsMedia_AudioCodec {
SPECSMEDIA_AUDIO_CODEC_DEFAULT = 0, // implementation default (AAC)
SPECSMEDIA_AUDIO_CODEC_AAC = 1, // AAC
} SpecsMedia_AudioCodec_t;
Buffer flags (SpecsMediaCodec.h)
| Constant | Value | Meaning |
|---|---|---|
SPECSMEDIACODEC_BUFFER_FLAG_KEY_FRAME | 0x1 | Buffer contains a keyframe (IDR for H.264) |
SPECSMEDIACODEC_BUFFER_FLAG_CODEC_CONFIG | 0x2 | Buffer contains codec config data (for example SPS/PPS for H.264) |
SPECSMEDIACODEC_BUFFER_FLAG_END_OF_STREAM | 0x4 | Signal end of input; no more buffers after this |
SPECSMEDIACODEC_BUFFER_FLAG_FORMAT_CHANGED | 0x8 | Output format changed before this buffer; call getOutputFormat() |
SpecsMediaFormat
Since: v1.3.0
SpecsMediaFormat is a key-value store used to describe codec parameters. It is used to configure codecs and report output format.
#include <SpecsMediaFormat.h>
Predefined key constants
| Constant | Key string | Typical type |
|---|---|---|
SPECSMEDIAFORMAT_KEY_MIME | "mime" | string, for example, "video/avc", "audio/mp4a-latm" |
SPECSMEDIAFORMAT_KEY_WIDTH | "width" | int32 |
SPECSMEDIAFORMAT_KEY_HEIGHT | "height" | int32 |
SPECSMEDIAFORMAT_KEY_COLOR_FORMAT | "color-format" | int32 |
SPECSMEDIAFORMAT_KEY_CODEC_CONFIG | "codec-config" | buffer (SPS/PPS or AudioSpecificConfig) |
SPECSMEDIAFORMAT_KEY_SAMPLE_RATE | "sample-rate" | int32, Hz |
SPECSMEDIAFORMAT_KEY_CHANNEL_COUNT | "channel-count" | int32 |
SPECSMEDIAFORMAT_KEY_BITS_PER_SAMPLE | "bits-per-sample" | int32 |
SPECSMEDIAFORMAT_KEY_BIT_RATE | "bitrate" | int32, bits/s |
SPECSMEDIAFORMAT_KEY_FRAME_RATE | "frame-rate" | int32, fps |
SpecsMediaFormat functions
Setting a key replaces its value and type. NULL arguments return SPECSMEDIA_ERROR_BAD_ARGUMENT. Getters return SPECSMEDIA_ERROR_NOT_IMPLEMENTED if the key is missing or holds a different type, and leave the outputs unchanged.
A string from getString stays valid until the next getString on the same format or until the format is deleted. A buffer from getBuffer stays valid until the key is set again or the format is deleted. setBuffer copies the data.
// Create an empty format object.
// Since: v1.3.0
SpecsMediaFormat *SpecsMediaFormat_new(void);
// Free a format object. (Do NOT free formats returned by the extractor.)
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_delete(SpecsMediaFormat *format);
// Integer key-value
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_setInt32(SpecsMediaFormat *format,
const char *name, int32_t value);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_getInt32(SpecsMediaFormat *format,
const char *name, int32_t *out_value);
// String key-value. The string from getString stays valid until the next
// getString on the same format or until the format is deleted.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_setString(SpecsMediaFormat *format,
const char *name, const char *value);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_getString(SpecsMediaFormat *format,
const char *name, const char **out_value);
// Binary buffer key-value. setBuffer copies the data. The buffer from getBuffer
// stays valid until the key is set again or the format is deleted.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_setBuffer(SpecsMediaFormat *format,
const char *name,
const uint8_t *data, size_t size);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaFormat_getBuffer(SpecsMediaFormat *format,
const char *name,
const uint8_t **out_data, size_t *out_size);
SpecsMediaExtractor
Since: v1.3.0
SpecsMediaExtractor demuxes a media file: it enumerates tracks and reads raw encoded samples one at a time.
#include <SpecsMediaExtractor.h>
Buffer structure
// Since: v1.3.0
typedef struct {
uint8_t *data; // pointer to buffer (caller-allocated)
size_t size; // bytes written on read
size_t capacity; // total allocated capacity
int64_t presentation_time_us; // PTS in microseconds
uint32_t flags; // SpecsMediaExtractor_BufferFlags
size_t offset; // offset within buffer
} SpecsMediaExtractor_Buffer;
Buffer flags
// Since: v1.3.0
typedef enum {
SPECS_BUFFER_FLAG_KEY_FRAME = 1,
SPECS_BUFFER_FLAG_CODEC_CONFIG = 2,
SPECS_BUFFER_FLAG_END_OF_STREAM = 4,
SPECS_BUFFER_FLAG_PARTIAL_FRAME = 8
} SpecsMediaExtractor_BufferFlags;
Seek modes
// Since: v1.3.0
typedef enum {
SPECS_SEEK_PREVIOUS_SYNC, // seek to sync point at or before time
SPECS_SEEK_NEXT_SYNC, // seek to sync point at or after time
SPECS_SEEK_CLOSEST_SYNC, // seek to nearest sync point
SPECS_SEEK_CLOSEST // seek to nearest frame (may not be sync)
} SpecsMediaExtractor_SeekMode;
SpecsMediaExtractor functions
// Create / destroy
// Since: v1.3.0
SpecsMediaExtractor *SpecsMediaExtractor_create(void);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_delete(SpecsMediaExtractor *extractor);
// Set data source (call one of these). Both return SPECSMEDIA_SUCCESS on
// success and SPECSMEDIA_ERROR_GENERIC if the file cannot be opened or parsed.
// setDataSource returns SPECSMEDIA_ERROR_BAD_ARGUMENT for a NULL extractor or
// an empty path.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_setDataSource(SpecsMediaExtractor *extractor,
const char *path);
// setDataSourceFd returns SPECSMEDIA_ERROR_BAD_ARGUMENT for a NULL extractor or
// a negative fd. The extractor does not take ownership of fd; the caller may
// close it once this returns. offset and length are currently ignored and the
// whole file is read.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_setDataSourceFd(SpecsMediaExtractor *extractor,
int fd, int64_t offset,
int64_t length);
// Enumerate tracks
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_getTrackCount(SpecsMediaExtractor *extractor,
int *count_out);
// Format handle is owned by the extractor — do NOT free it.
// Since: v1.3.0
SpecsMediaFormat *SpecsMediaExtractor_getTrackFormat(SpecsMediaExtractor *extractor,
int track_index);
// Track selection (multiple tracks may be selected simultaneously)
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_selectTrack(SpecsMediaExtractor *extractor,
int track_index);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_unselectTrack(SpecsMediaExtractor *extractor,
int track_index);
// Read current sample (caller must allocate buffer with sufficient capacity)
// Returns SPECSMEDIA_SUCCESS on success, or an error code on failure.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_readSampleData(SpecsMediaExtractor *extractor,
SpecsMediaExtractor_Buffer *buffer);
// Query current sample metadata (call before advance())
// Since: v1.3.0
int SpecsMediaExtractor_getSampleTrackIndex(SpecsMediaExtractor *extractor);
// Since: v1.3.0
int64_t SpecsMediaExtractor_getSampleTime(SpecsMediaExtractor *extractor);
// Since: v1.3.0
uint32_t SpecsMediaExtractor_getSampleFlags(SpecsMediaExtractor *extractor);
// Since: v1.3.0
ssize_t SpecsMediaExtractor_getSampleSize(SpecsMediaExtractor *extractor);
// Advance to next sample. Returns false at end of stream.
// After selectTrack() or seekTo(), call advance() to move to the first sample;
// until then the sample getters report no sample. readSampleData() advances by
// itself when no sample is current.
// Since: v1.3.0
bool SpecsMediaExtractor_advance(SpecsMediaExtractor *extractor);
// Seek to a sample near time_us. mode is a SpecsMediaExtractor_SeekMode value
// selecting which sample is chosen relative to time_us.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaExtractor_seekTo(SpecsMediaExtractor *extractor,
int64_t time_us, int mode);
// True once the extractor has read to the end of the data source (and true for a
// NULL extractor). A successful seekTo() clears the flag. Samples read before the
// end may still be returned after this becomes true — advance() returning false
// is the end-of-samples signal.
// Since: v1.3.0
bool SpecsMediaExtractor_hasCacheReachedEndOfStream(SpecsMediaExtractor *extractor);
SpecsMediaExtractor is declared as a typedef (typedef struct SpecsMediaExtractor SpecsMediaExtractor;), and SpecsMediaExtractor.h includes SpecsMediaFormat.h, so SpecsMediaFormat is available without a separate include.
SpecsMediaCodec
Since: v1.3.0
SpecsMediaCodec encodes or decodes media data. It uses an index-based buffer queue model.
The crypto argument of SpecsMediaCodec_configure() is reserved for future encrypted-content support; pass NULL. The crypto types are opaque handles:
// Opaque handle to media crypto context for encrypted content.
// Since: v1.3.0
typedef struct SpecsMediaCrypto SpecsMediaCrypto;
// Structure containing crypto information for encrypted media.
// Since: v1.3.0
typedef struct SpecsMediaCrypto_info SpecsMediaCrypto_info;
#include <SpecsMediaCodec.h>
Buffer info structure
// Since: v1.3.0
typedef struct {
int32_t size; // data size in bytes
int64_t presentationTimeUs; // PTS in microseconds
uint32_t flags; // SPECSMEDIACODEC_BUFFER_FLAG_*
} SpecsMediaCodec_bufferInfo;
Configure flags
| Constant | Value | Meaning |
|---|---|---|
SPECSMEDIACODEC_CONFIGURE_FLAG_OUTPUT_MEDIA_BUFFER | 0x1 | Configure codec output as SpecsMediaBuffer (use getOutputMediaBuffer() instead of getOutputBuffer()) |
Parameter keys
| Key | String | Description |
|---|---|---|
SPECSMEDIACODEC_PARAMETER_KEY_REQUEST_SYNC_FRAME | "request-sync-frame" | Request an IDR frame from encoder |
SPECSMEDIACODEC_PARAMETER_KEY_VIDEO_BITRATE | "video-bitrate" | Change encoder bitrate at runtime |
Async callback mode
Async callback mode replaces the synchronous dequeueInputBuffer / dequeueOutputBuffer polling loop. When enabled, the codec pushes ready buffer indices to your callbacks.
// Since: v1.3.0
typedef void (*SpecsMediaCodec_OnInputBufferAvailable_t)(SpecsMediaCodec *codec,
size_t idx,
void *userData);
// Since: v1.3.0
typedef void (*SpecsMediaCodec_OnOutputBufferAvailable_t)(SpecsMediaCodec *codec,
size_t idx,
const SpecsMediaCodec_bufferInfo *info,
void *userData);
// Since: v1.3.0
typedef struct {
SpecsMediaCodec_OnInputBufferAvailable_t onInputBufferAvailable;
void *inputUserData;
SpecsMediaCodec_OnOutputBufferAvailable_t onOutputBufferAvailable;
void *outputUserData;
} SpecsMediaCodec_Callbacks;
Async mode is mutually exclusive with dequeueInputBuffer / dequeueOutputBuffer—those APIs return SPECSMEDIA_ERROR_INVALID_STATE while async callbacks are enabled.
Codec lifecycle states
| State | Description |
|---|---|
| NEW | Newly created, not yet configured |
| IDLE | Configured and ready to start, or stopped and normalized |
| RUNNING | Actively accepting input / producing output |
| EOS_DRAINING | Input EOS signaled, output can still be drained |
| ERROR | Codec is in an error state and cannot be used |
Expected flow: create → configure → start → queue/dequeue loop → stop → delete
SpecsMediaCodec functions
// Creation by MIME type (see Supported MIME types) — returns NULL if the type
// is not supported or creation fails.
// Each video input buffer of a decoder holds one access unit: Annex B byte
// stream for H.264/H.265, one compressed frame for VP9. This is the format
// SpecsMediaExtractor_readSampleData() returns, so extracted samples can be
// queued as they are.
// Since: v1.3.0
SpecsMediaCodec *SpecsMediaCodec_createDecoderByType(const char *mime_type);
// Since: v1.3.0
SpecsMediaCodec *SpecsMediaCodec_createEncoderByType(const char *mime_type);
// Cleanup
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_delete(SpecsMediaCodec *codec);
// Configure — valid in NEW or IDLE state.
// crypto: pass NULL for non-encrypted content.
// flags: 0 for standard output, SPECSMEDIACODEC_CONFIGURE_FLAG_OUTPUT_MEDIA_BUFFER
// to get output via SpecsMediaBuffer.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_configure(SpecsMediaCodec *codec,
SpecsMediaFormat *format,
SpecsMediaCrypto *crypto,
uint32_t flags);
// Enable/disable async callback mode (valid in NEW or IDLE state — not while RUNNING).
// Pass NULL to disable async callbacks. Non-NULL callback members enable the
// corresponding async callback; NULL members leave that direction in sync mode.
// User data for each callback is carried in the SpecsMediaCodec_Callbacks struct.
// Returns SPECSMEDIA_ERROR_BAD_ARGUMENT if codec is NULL,
// SPECSMEDIA_ERROR_INVALID_STATE if called outside NEW/IDLE.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_setCallbacks(SpecsMediaCodec *codec,
const SpecsMediaCodec_Callbacks *callbacks);
// Runtime parameter updates (valid while RUNNING or EOS_DRAINING)
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_setParameters(SpecsMediaCodec *codec,
const SpecsMediaFormat *params);
// Lifecycle
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_start(SpecsMediaCodec *codec);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_stop(SpecsMediaCodec *codec);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_flush(SpecsMediaCodec *codec);
// signalEndOfInputStream: valid while RUNNING; moves the codec to EOS_DRAINING.
// Calling it again while draining (EOS_DRAINING) returns SPECSMEDIA_SUCCESS.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_signalEndOfInputStream(SpecsMediaCodec *codec);
// Input buffer management (sync mode — not available while async callbacks are active)
// timeoutUs: 0 = non-blocking, >0 = wait up to N µs, <0 = block indefinitely
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_dequeueInputBuffer(SpecsMediaCodec *codec,
int64_t timeoutUs,
size_t *outIndex);
// Returns pointer to buffer data, sets *out_size. Returns NULL on error.
// Since: v1.3.0
uint8_t *SpecsMediaCodec_getInputBuffer(SpecsMediaCodec *codec, size_t idx,
size_t *out_size);
// ptsUs = presentation timestamp, dtsUs = decode timestamp, both in microseconds
// (dts enables correct B-frame handling). For streams without B-frames (I/P only),
// pass the same value for both. ptsUs is reported back unchanged in
// SpecsMediaCodec_bufferInfo::presentationTimeUs.
// Returns SPECSMEDIA_ERROR_BAD_ARGUMENT if codec is NULL, idx is not a valid
// buffer index, or END_OF_STREAM is set together with data;
// SPECSMEDIA_ERROR_INVALID_STATE if the codec is not running, the buffer is not
// currently dequeued, or offset + size exceeds its capacity.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_queueInputBuffer(SpecsMediaCodec *codec,
size_t idx, size_t offset,
size_t size, uint64_t ptsUs,
uint64_t dtsUs, uint32_t flags);
// Output buffer management (sync mode — not available while async callbacks are active)
// info may be NULL if the metadata is not needed; the buffer must still be released.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_dequeueOutputBuffer(SpecsMediaCodec *codec,
int64_t timeoutUs,
size_t *outIndex,
SpecsMediaCodec_bufferInfo *info);
// Standard output (when configured WITHOUT OUTPUT_MEDIA_BUFFER flag)
// Since: v1.3.0
uint8_t *SpecsMediaCodec_getOutputBuffer(SpecsMediaCodec *codec, size_t idx,
size_t *out_size);
// Hardware buffer output (when configured WITH OUTPUT_MEDIA_BUFFER flag).
// Returned buffer has one retained reference — caller must call SpecsMediaBuffer_release().
// Since: v1.3.0
SpecsMediaBuffer *SpecsMediaCodec_getOutputMediaBuffer(SpecsMediaCodec *codec,
size_t idx);
// releaseOutputBuffer: valid while RUNNING or EOS_DRAINING. Returns
// SPECSMEDIA_ERROR_BAD_ARGUMENT if codec is NULL or idx is not a valid buffer
// index; SPECSMEDIA_ERROR_INVALID_STATE if the codec is not running or draining,
// or the buffer is not currently dequeued (for example, released twice).
// Since: v1.3.0
SpecsMedia_Result SpecsMediaCodec_releaseOutputBuffer(SpecsMediaCodec *codec,
size_t idx);
// Format queries — caller owns the returned format and must call SpecsMediaFormat_delete().
// Since: v1.3.0
SpecsMediaFormat *SpecsMediaCodec_getOutputFormat(SpecsMediaCodec *codec);
// getInputFormat is not yet implemented; it always returns NULL.
// Since: v1.3.0
SpecsMediaFormat *SpecsMediaCodec_getInputFormat(SpecsMediaCodec *codec);
Supported MIME types
These are the mime_type strings SpecsMediaCodec_createDecoderByType() and SpecsMediaCodec_createEncoderByType() accept. Any other type returns NULL.
| MIME type | Codec | Decoder | Encoder |
|---|---|---|---|
video/avc (or video/h264) | H.264 / AVC | Yes | Yes |
video/hevc (or video/h265) | H.265 / HEVC | Yes | Yes |
video/x-vnd.on2.vp9 | VP9 | Yes | No |
audio/aac | AAC | Yes | Yes |
audio/mp4a-latm | AAC | Yes | Yes |
SpecsMediaPresenter
Since: v1.3.0
SpecsMediaPresenter provides timed display presentation of decoded media buffers. Use it when building a custom decode pipeline with SpecsMediaCodec and you need to present video frames to the display with correct timing and optional head-tracked pose.
#include <SpecsMediaPresenter.h>
Sample flags
// Since: v1.3.0
typedef enum SpecsMediaPresenter_SampleFlags {
SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_NONE = 0,
SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_END_OF_STREAM = 1u << 0,
SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_DO_NOT_DISPLAY = 1u << 1,
SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_DISCONTINUITY = 1u << 2,
SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_IMMEDIATE = 1u << 3
} SpecsMediaPresenter_SampleFlags;
DO_NOT_DISPLAY | IMMEDIATE returns an error.
Presenter types
// Since: v1.3.0
typedef struct {
double tx, ty, tz; // translation (meters)
double qx, qy, qz, qw; // rotation quaternion
} SpecsMediaPresenter_Pose;
// Since: v1.3.0
typedef struct {
const char *appName; // application name for display session
uint32_t layerId; // layer selection value; call getLayerId() to read the assigned ID
int32_t width; // target layer width (pixels)
int32_t height; // target layer height (pixels)
} SpecsMediaPresenter_Config;
// Pose provider callback — invoked on render thread per frame.
// Return true with a valid outPose to apply, false to keep the previous pose.
// Since: v1.3.0
typedef bool (*SpecsMediaPresenter_PoseProvider)(
void *userData, uint64_t predictedDisplayTimeUs,
SpecsMediaPresenter_Pose *outPose);
Drop policy
SpecsMediaPresenter uses a bounded internal queue. The drop policy controls what happens when enqueueSample is called with the queue already full.
// Since: v1.3.0
typedef enum SpecsMediaPresenter_DropPolicy {
// Preserve all queued samples and wait for space. enqueueSample follows the
// caller-provided timeoutMs.
SPECS_MEDIA_PRESENTER_DROP_POLICY_BLOCK = 0,
// Drop the oldest queued sample and enqueue the new sample in its place.
// The first sample presented after the drop is automatically tagged with
// SPECS_MEDIA_PRESENTER_SAMPLE_FLAG_DISCONTINUITY.
SPECS_MEDIA_PRESENTER_DROP_POLICY_DROP_OLDEST = 1
} SpecsMediaPresenter_DropPolicy;
The default drop policy is DROP_OLDEST. The policy can only be changed while the presenter is stopped (before SpecsMediaPresenter_start()); once startup has begun, setDropPolicy() returns SPECSMEDIA_ERROR_INVALID_STATE.
SpecsMediaPresenter functions
// Create / destroy
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_create(SpecsMediaPresenter **outDisplay,
const SpecsMediaPresenter_Config *config);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_destroy(SpecsMediaPresenter **display);
// Start / stop presentation (idempotent). start() can return
// SPECSMEDIA_ERROR_RESOURCE_EXHAUSTED — release unused resources before retrying.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_start(SpecsMediaPresenter *display);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_stop(SpecsMediaPresenter *display);
// Enqueue a timed sample. Accepts NV12 and single-plane RGBA8888 DMA-BUF
// buffers. The presenter holds a reference to the buffer until its GPU has
// finished reading it, which is after the frame that shows it is submitted,
// not when this call returns. Unknown sampleFlags bits are rejected.
// - DROP_POLICY_BLOCK: blocks up to timeoutMs if the queue is full.
// - DROP_POLICY_DROP_OLDEST: never blocks; timeoutMs is ignored.
// Returns SPECSMEDIA_ERROR_BAD_ARGUMENT for a null argument or unsupported
// flags, SPECSMEDIA_ERROR_AGAIN or SPECSMEDIA_ERROR_TIMED_OUT when the queue
// stays full under DROP_POLICY_BLOCK, otherwise an error code on failure.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_enqueueSample(
SpecsMediaPresenter *display, SpecsMediaBuffer *buffer,
uint64_t presentationTimeUs, uint32_t sampleFlags, uint32_t timeoutMs);
// Enqueue a sample that the producer's GPU may still be writing. Behaves like
// enqueueSample, except that the presenter orders its draw of the sample after
// acquireFenceFd signals, waiting on the GPU where the driver allows. Producers
// can then hand over a frame right after submitting it rather than blocking in
// glFinish.
// acquireFenceFd: a sync_file fd (for example from eglDupNativeFenceFDANDROID)
// that signals once the buffer's contents are complete, or -1 for none. The
// presenter takes ownership of it, even when the call fails. The presenter's GPU
// waits for the fence where the driver supports it; otherwise its render thread
// waits up to 100 ms and then draws the buffer as it is.
// Returns the same results as enqueueSample.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_enqueueSampleWithFence(
SpecsMediaPresenter *display, SpecsMediaBuffer *buffer,
uint64_t presentationTimeUs, uint32_t sampleFlags, uint32_t timeoutMs,
int acquireFenceFd);
// Configure overflow behaviour. Must be called before start(); returns
// SPECSMEDIA_ERROR_INVALID_STATE after the presenter has started.
// Default policy is SPECS_MEDIA_PRESENTER_DROP_POLICY_DROP_OLDEST.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_setDropPolicy(
SpecsMediaPresenter *display, SpecsMediaPresenter_DropPolicy dropPolicy);
// Drop all queued samples without stopping presentation.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_flush(SpecsMediaPresenter *display);
// Query whether enqueue can accept at least one sample without blocking under
// the current drop policy.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_isReadyForMoreData(
SpecsMediaPresenter *display, bool *outReady);
// Get the layer ID assigned to this presenter. The first successful call
// reserves the display layer; the returned ID stays stable until stop(), and a
// later call reserves a layer again. Can return
// SPECSMEDIA_ERROR_RESOURCE_EXHAUSTED — release unused resources before retrying.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_getLayerId(
SpecsMediaPresenter *display, uint32_t *outLayerId);
// Set or clear a per-frame pose provider callback.
// Pass NULL poseProvider to clear.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_setPoseProvider(
SpecsMediaPresenter *display, SpecsMediaPresenter_PoseProvider poseProvider,
void *userData);
// Set how smoothly the compositor moves the layer toward each frame's pose. The
// compositor eases the quad toward the pose of the latest frame with a
// critically damped spring, stepped for every color field of the display. A
// pose provider that returns a head-locked pose each frame therefore makes the
// layer lazily follow the head without color breakup, which smoothing poses on
// the render thread can't avoid. Has no effect in multi-view projection mode.
// smoothTimeSec: seconds the quad takes to settle on a new pose, or 0 to show
// each frame's pose as is (the default).
// Since: v1.3.0
SpecsMedia_Result
SpecsMediaPresenter_setPoseSmoothing(SpecsMediaPresenter *display,
float smoothTimeSec);
// Set / get the layer z-order. If setZOrder is never called, the presenter
// leaves z-order unset and PlatformGraphics chooses the layer ordering.
// getZOrder returns SPECSMEDIA_ERROR_INVALID_STATE if no custom z-order has
// been set (and leaves *outZOrder unchanged).
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_setZOrder(
SpecsMediaPresenter *display, int32_t zOrder);
// Since: v1.3.0
SpecsMedia_Result SpecsMediaPresenter_getZOrder(
SpecsMediaPresenter *display, int32_t *outZOrder);
Threading
SpecsMediaPresenter control APIs are not thread-safe. Callers must serialize access to a given presenter instance. The pose provider callback fires on an internal render thread.
SpecsMediaBuffer
Since: v1.3.0
SpecsMediaBuffer is a reference-counted hardware buffer handle. Buffers are obtained from SpecsMediaCodec_getOutputMediaBuffer() (when the codec is configured with SPECSMEDIACODEC_CONFIGURE_FLAG_OUTPUT_MEDIA_BUFFER) or created from a DMA-BUF file descriptor with SpecsMediaBuffer_createFromFd(), and can be enqueued to SpecsMediaPresenter_enqueueSample() for display, or written into a caller-owned OpenGL ES texture with SpecsMediaGLTexture_updateFromBuffer().
#include <SpecsMediaBuffer.h>
SpecsMediaBuffer_BackendType_t
Since: v1.3.0
The backend type used to transport the underlying buffer handle.
typedef enum SpecsMediaBuffer_BackendType {
SPECS_MEDIA_BUFFER_BACKEND_UNKNOWN = 0,
SPECS_MEDIA_BUFFER_BACKEND_DMA_BUF_FD = 1,
SPECS_MEDIA_BUFFER_BACKEND_CPU_MEMORY = 2,
SPECS_MEDIA_BUFFER_BACKEND_OPAQUE = 255
} SpecsMediaBuffer_BackendType_t;
SpecsMediaBuffer_Usage_t
Since: v1.3.0
Usage flags describing how the buffer is expected to be used. Multiple values can be combined with bitwise OR.
typedef enum SpecsMediaBuffer_Usage {
SPECS_MEDIA_BUFFER_USAGE_NONE = 0,
SPECS_MEDIA_BUFFER_USAGE_CPU_READ = 1u << 0,
SPECS_MEDIA_BUFFER_USAGE_CPU_WRITE = 1u << 1,
SPECS_MEDIA_BUFFER_USAGE_GPU_SAMPLED = 1u << 2,
SPECS_MEDIA_BUFFER_USAGE_GPU_COLOR_OUTPUT = 1u << 3,
SPECS_MEDIA_BUFFER_USAGE_VIDEO_ENCODE = 1u << 4,
SPECS_MEDIA_BUFFER_USAGE_VIDEO_DECODE = 1u << 5
} SpecsMediaBuffer_Usage_t;
SpecsMediaBuffer_Format_t
Since: v1.3.0
The pixel format of the buffer content.
typedef enum SpecsMediaBuffer_Format {
SPECS_MEDIA_BUFFER_FORMAT_RGB = 0,
SPECS_MEDIA_BUFFER_FORMAT_RGBA = 1,
SPECS_MEDIA_BUFFER_FORMAT_NV12 = 2
} SpecsMediaBuffer_Format_t;
SpecsMediaBuffer_FormatModifier_t
Since: v1.3.0
The layout or compression modifier for the pixel format.
typedef enum SpecsMediaBuffer_FormatModifier {
SPECS_MEDIA_BUFFER_FORMAT_MODIFIER_NONE = 0, /**< Linear/unmodified layout. */
SPECS_MEDIA_BUFFER_FORMAT_MODIFIER_UBWC = 1 /**< Qualcomm UBWC layout. */
} SpecsMediaBuffer_FormatModifier_t;
SpecsMediaBuffer_Desc_t
Since: v1.3.0
Immutable metadata associated with a hardware buffer, passed to SpecsMediaBuffer_createFromFd(). The plane arrays hold up to SPECS_MEDIA_BUFFER_MAX_PLANES entries.
#define SPECS_MEDIA_BUFFER_MAX_PLANES 3
typedef struct SpecsMediaBuffer_Desc {
uint32_t width; /**< Width in pixels. */
uint32_t height; /**< Height in pixels. */
uint64_t timestampNs; /**< Presentation timestamp in nanoseconds. */
SpecsMediaBuffer_Format_t format; /**< Pixel format. */
SpecsMediaBuffer_FormatModifier_t
formatModifier; /**< Pixel layout/compression modifier. */
uint32_t numPlanes; /**< Number of valid entries in plane arrays. */
uint32_t planeStrides[SPECS_MEDIA_BUFFER_MAX_PLANES]; /**< Per-plane row
strides in bytes. */
uint32_t planeOffsets[SPECS_MEDIA_BUFFER_MAX_PLANES]; /**< Per-plane offsets
from buffer start
in bytes. */
uint32_t planeSizes[SPECS_MEDIA_BUFFER_MAX_PLANES]; /**< Per-plane sizes in
bytes. */
SpecsMediaBuffer_Usage_t usage; /**< Intended usage flags (bitmask). */
SpecsMediaBuffer_BackendType_t backendType; /**< Underlying handle type. */
} SpecsMediaBuffer_Desc_t;
| Field | Description |
|---|---|
width | Width in pixels. |
height | Height in pixels. |
timestampNs | Presentation timestamp in nanoseconds. |
format | Pixel format (SpecsMediaBuffer_Format_t). |
formatModifier | Pixel layout or compression modifier (SpecsMediaBuffer_FormatModifier_t). |
numPlanes | Number of valid entries in the plane arrays. |
planeStrides | Per-plane row strides in bytes. |
planeOffsets | Per-plane offsets from the buffer start in bytes. |
planeSizes | Per-plane sizes in bytes. |
usage | Intended usage flags (a bitmask of SpecsMediaBuffer_Usage_t). |
backendType | Underlying handle type (SpecsMediaBuffer_BackendType_t). |
SpecsMediaBuffer_ReleaseCallback_t
Since: v1.3.0
Callback invoked when the final reference to a buffer is released. For a buffer created with SpecsMediaBuffer_createFromFd(), it is called after the implementation has released the adopted DMA-BUF file descriptor.
typedef void (*SpecsMediaBuffer_ReleaseCallback_t)(void *userData);
userData is the releaseUserData passed at create time. Execution context:
- It is invoked synchronously on the thread that performs the final
SpecsMediaBuffer_release(), the one that drops the reference count to zero. - It should return quickly and must not block for long periods.
- It must be reentrant-safe.
- It must not call
SpecsMediaBuffer_*APIs on the buffer instance being released.
SpecsMediaBuffer functions
// Create a refcounted hardware buffer from a DMA-BUF file descriptor. Lets a
// producer that renders its own frames, for example with GLES into a GBM
// buffer, hand them to SpecsMediaPresenter_enqueueSample. The presenter shows
// buffers whose backendType is SPECS_MEDIA_BUFFER_BACKEND_DMA_BUF_FD and whose
// format is NV12 or single-plane RGBA8888.
// desc: immutable buffer metadata; the implementation copies it.
// fd: DMA-BUF file descriptor backing the buffer; must be >= 0. On
// SPECSMEDIA_SUCCESS, SpecsMediaBuffer adopts ownership of fd; on failure,
// ownership stays with the caller.
// onRelease: invoked exactly once when the last reference is released and after
// the adopted fd has been released. May be NULL.
// releaseUserData: user data passed to onRelease.
// outBuffer: receives the created buffer on success.
// Returns SPECSMEDIA_SUCCESS on success, otherwise an error code.
// Since: v1.3.0
SpecsMedia_Result SpecsMediaBuffer_createFromFd(
const SpecsMediaBuffer_Desc_t *desc, int fd,
SpecsMediaBuffer_ReleaseCallback_t onRelease, void *releaseUserData,
SpecsMediaBuffer **outBuffer);
// Increment reference count. Thread-safe.
// Since: v1.3.0
void SpecsMediaBuffer_acquire(SpecsMediaBuffer *buffer);
// Decrement reference count and set *buffer to NULL. Thread-safe.
// Since: v1.3.0
void SpecsMediaBuffer_release(SpecsMediaBuffer **buffer);