audio: tensorflow: keyword detect by singalsu · Pull Request #11092 · thesofproject/sof

singalsu · 2026-08-13T17:45:18Z

No description provided.

…A capture topology

…and topology usage

…library base address

…fest

When CONFIG_LLEXT_TYPE_ELF_RELOCATABLE is active, bypass appending static address flags (-Ttext, --section-start, -Tdata) in the linker helper script. This keeps section base addresses at 0. Also adjust the offset calculator to avoid integer parsing errors when all section addresses are set to 0. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…table modules Implement page-level virtual memory mapping using Zephyr's sys_bitarray utility over the library region. Compile section layout at load-time to allocate virtual addresses and rewrite section sh_addr headers in-place. This enables Zephyr LLEXT to naturally relocate references. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Enable CONFIG_LLEXT_EXPORT_BUILTINS_BY_SLID=y in llext_relocatable.conf to link relocatable LLEXT modules against build-time function signature hashing, providing load-time ABI mismatch protection. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…_mtpm Enable CONFIG_SOF_STAGING=y, CONFIG_CPP=y, CONFIG_STD_CPP17=y, and CONFIG_COMP_TENSORFLOW=m in intel_adsp_ace15_mtpm.conf to allow building the TFLM keyword detection module as an LLEXT module on ARL. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Guard Cadence Tensilica XCC-specific compiler options and tie intrinsic kernels (nn_hifi_lib) under CMAKE_C_COMPILER_ID STREQUAL "Xtensa". When building with open-source Zephyr SDK GCC, use portable TFLM reference kernels and stdc++ to enable building tflm.llext as a relocatable module. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…ation Include <abi.conf> in topology definitions to embed the required IPC4 ABI manifest section (ABI 3.29.1), resolving kernel topology ABI validation. Add sof-arl-cs42l43-l0-cs35l56-l23-tflm.conf to instantiate the real-time TFLM keyword detection pipeline connected to SoundWire DMIC capture on ARL. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Expand SOF_ZEPHYR_VIRTUAL_HEAP_REGION_SIZE to 4MB (0x400000) and increase CONFIG_HEAP_MEM_POOL_SIZE to 32KB (32768) for intel_adsp_ace15_mtpm board. This satisfies dynamic LLEXT module memory requirements for tflm.llext (1.01 MB payload) and prevents -ENOMEM allocation errors in llext_load_elf_data() during section map linking. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…logging Implement stream shutdown summary logging (tflm_log_summary_at_shutdown) in tflm_reset() and tflm_free() to output total inferences, per-category classifications, and KPB trigger events to printk/trace logs at stream end. Add KPB event notification dispatch (tflm_notify_kpb) sending NOTIFIER_ID_KPB_CLIENT_EVT (KPB_EVENT_BEGIN_DRAINING) upon high-confidence keyword detection to initiate pre-roll audio history draining to host PCM stream. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Integrate tflmcly widget into SoundWire feature capture topology and configure multi-pin output routes on drc.1. Add production topology build targets sof-mtl-hda-tflm-kpb and sof-ptl-hda-tflm-kpb for HDA Analog input with Key Phrase Buffer (KPB) pre-roll history management. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…action, and usage Update src/audio/tensorflow/README.md with comprehensive documentation covering: - Dual-path KPB + TFLM Wake-on-Voice architecture - Audio input source abstraction across SoundWire, DMIC, SSP, and HDA - Data Processing (DP) domain vs Low Latency (LL) domain scheduling - Stream shutdown summary event logging ([TFLM STREAM SHUTDOWN SUMMARY]) - Topology v2 build, deployment, and testing usage instructions Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

…pier mfcc.1 in host-gateway-src-mfcc-tflm-capture.conf had no bytes control at all, so mfcc_prepare() hit comp_get_data_blob() returning cd->config == NULL and failed with "configuration is missing" -- the detection path (mfcc -> tflmcly) has never actually started on hardware before this. Attach a real default blob via HDA_MIC_MFCC_PARAMS (mel40.conf/ mel40_compress.conf, generated by setup_mfcc.m's new 40-bin/20ms-hop profile matching TFLM micro_speech's TFLM_FEATURE_SIZE/STRIDE/DURATION), and thread HDA_MIC_MFCC_PARAMS=default through every sof-hda-tflm build target (mtl/ptl/tgl) in tplg-targets-hda-generic.cmake. Add a real "HDA Mic TFLM Detect" host-copier on pipe 1 (was a virtual out_drv sink with nothing behind it) and route tflmcly's output there instead of the terminal virtual.tflm_sink, plus a matching PCM entry in sof-hda-tflm.conf, so the detection pipeline has a real host-facing device to arm/observe (see README: KPB pin 1 -> detection chain; this pipe's host-copier is for arming/instrumentation, not the pin 2 drain target which remains host-copier.0.capture). Fill in in_channels/out_channels (1, mono) on every audio_format block that was missing it and bump src.1's num_input_audio_formats to include a 16kHz passthrough entry, needed once a real 16kHz-native input format is exercised via the new host-copier. MFCC_FRAME_BYTES corrected 344 -> 184 to match mel40's actual per-hop size: 24-byte mfcc_data_header + 40 x int32 Q9.23 mel values (mel80 was 344; mel40 is what this topology's TFLM_FEATURE_SIZE=40 model actually expects).

singalsu · 2026-08-14T07:48:52Z

 CONFIG_COMP_GOOGLE_RTC_AUDIO_PROCESSING=m
 CONFIG_GOOGLE_RTC_AUDIO_PROCESSING_MOCK=y
 CONFIG_COMP_STFT_PROCESS=y
+CONFIG_SOF_STAGING=y


Note: Untested by me.

singalsu · 2026-08-14T07:49:12Z

 CONFIG_COMP_MULTIBAND_DRC=y
 CONFIG_COMP_VOLUME_WINDOWS_FADE=y
 CONFIG_FORMAT_CONVERT_HIFI3=n
+CONFIG_SOF_STAGING=y


Note: Build works only with gcc.

singalsu · 2026-08-14T07:55:37Z

 CONFIG_COMP_ARIA=y
 CONFIG_COMP_CROSSOVER=y
 CONFIG_COMP_DRC=y
-CONFIG_COMP_TESTER=m


This patch is here to enable testing, but can't be merged due to modifying too much normal build. Also this version failed to boot on ARL-S.

Copilot

Pull request overview

This PR adds a TensorFlow Lite Micro (TFLM) keyword-detection (“Wake-on-Voice”) path integrated with MFCC feature extraction and KPB draining, including new topology2 overlays for HDA and SoundWire capture sources plus supporting build, scheduling, and tooling updates.

Changes:

Add topology2 capture overlays (HDA analog, SDW jack, SDW DMIC) that tap an existing capture endpoint into SRC -> KPB -> (WoV drain + MFCC/micsel/TFLM detect) and register new topology build targets.
Add a new tflmcly topology widget class and pipeline templates (host-gateway-micsel-mfcc-tflm-capture, src-kpb-be) and MFCC mel40 blob variants used by the WoV path and benchmarks.
Update firmware/runtime to support DP scheduler init, DP-to-DP bindings via dual ring-buffers, and replace the TFLM model/label plumbing with retraining scripts and new generated headers.

Reviewed changes

Copilot reviewed 49 out of 50 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tools/topology/topology2/sof-hda-generic.conf	Adds optional HDA mic WoV overlay include and required include files
tools/topology/topology2/platform/intel/sdw-jack-tflm-kpb.conf	New SDW jack WoV branch overlay using SRC+KPB and MFCC/TFLM detect
tools/topology/topology2/platform/intel/sdw-dmic-tflm-kpb.conf	New SDW DMIC WoV branch overlay using SRC+KPB and MFCC/TFLM detect
tools/topology/topology2/platform/intel/sdw-dmic-audio-feature.conf	Routes MFCC output through new `tflmcly` widget for audio-feature capture
tools/topology/topology2/platform/intel/hda-mic-tflm-kpb.conf	New HDA analog WoV overlay (adds WoV drain + detect PCMs)
tools/topology/topology2/platform/intel/dmic1-mfcc.conf	Removes redundant MFCC class include (now assumed included by parent)
tools/topology/topology2/include/pipelines/cavs/src-kpb-be.conf	New pipeline class providing SRC->KPB backend capture branch
tools/topology/topology2/include/pipelines/cavs/host-gateway-tdfb-drc-capture.conf	Adjusts DRC widget pin count for capture pipeline
tools/topology/topology2/include/pipelines/cavs/host-gateway-micsel-mfcc-tflm-capture.conf	New detect pipeline class (micsel->mfcc->tflmcly->host)
tools/topology/topology2/include/components/tflm.conf	Defines topology2 widget class for TFLM classifier (`tflmcly`)
tools/topology/topology2/include/components/mfcc/mel40.conf	Adds 40-bin MFCC blob variant
tools/topology/topology2/include/components/mfcc/mel40_compress.conf	Adds compressed 40-bin MFCC blob variant for WoV path
tools/topology/topology2/include/components/kpb.conf	Updates KPB UUID used by topology2 widget definition
tools/topology/topology2/include/common/common_definitions.conf	Adds new feature flags for WoV overlays (SDW jack, SDW DMIC, HDA mic)
tools/topology/topology2/include/common/abi.conf	Adds SOF ABI manifest blob include
tools/topology/topology2/include/bench/mfcc_controls_playback.conf	Adds mel40 option to MFCC bench controls
tools/topology/topology2/include/bench/mfcc_controls_capture.conf	Adds mel40 option to MFCC bench controls
tools/topology/topology2/development/tplg-targets.cmake	Registers new HDA and SDW WoV topology2 build targets
tools/topology/topology2/development/tplg-targets-bench.cmake	Adds mel40 MFCC bench target/config
tools/topology/topology2/cavs-sdw.conf	Includes new pipeline classes and adds WoV overlay include keys
tools/topology/topology2/cavs-benchmark-hda.conf	Adds mfccmel40 bench configs via existing mfccmel includes
tools/rimage/config/tgl.toml.h	Includes TFLM component TOML when enabled
src/platform/intel/cavs/platform.c	Initializes DP scheduler when configured
src/ipc/ipc4/helper.c	Enables DP-to-DP binding with dual ring-buffers and vregion refcounting
src/audio/tensorflow/tune/sof_tflm_verify.py	New off-device verifier for quantized model accuracy metrics
src/audio/tensorflow/tune/sof_tflm_train.py	New training script to emit drop-in model/labels C artifacts
src/audio/tensorflow/tune/sof_tflm_train_pipeline.sh	New end-to-end dataset→features→train pipeline wrapper
src/audio/tensorflow/tune/sof_tflm_prepare_silence_unknown.sh	New Speech Commands v2 silence/unknown dataset prep script
src/audio/tensorflow/tune/sof_tflm_generate_keyword_dataset.sh	New Piper sample-generator keyword dataset synthesis script
src/audio/tensorflow/tune/sof_tflm_generate_keyword_dataset_piper_tts.sh	New single-speaker piper-tts keyword dataset synthesis script
src/audio/tensorflow/tune/sof_tflm_dataset.py	New loader for SOF MFCC hop wire-format into model windows
src/audio/tensorflow/tune/sof_mfcc_extract_features.sh	New testbench-based feature extraction runner producing `.raw` hops
src/audio/tensorflow/tflm-classify.c	Major rework: MFCC-hop parsing, requantization, inference loop, KPB trigger plumbing
src/audio/tensorflow/speech.h	Switches labels/categories to generated header; adds quantization params and arena query APIs
src/audio/tensorflow/speech.cc	Switches to new model data header; expands op resolver; exposes input quant params and arena usage
src/audio/tensorflow/sof_tflm_quantized_model_data.h	Adds generated model header for new model symbol
src/audio/tensorflow/sof_tflm_labels.h	Adds generated labels header used by runtime
src/audio/tensorflow/README.md	Large documentation update describing WoV/TFLM architecture and training flow
src/audio/tensorflow/micro_speech_quantized_model_data.h	Removes old model header include
src/audio/tensorflow/CMakeLists.txt	Updates build gating for nnlib-hifi4, adds required TFLM kernels, and toolchain libc shim for static builds
src/audio/stft_process/stft_process-hifi3.c	Replaces `assert.h` include with panic header
src/audio/stft_process/stft_process-generic.c	Replaces `assert.h` include with panic header
src/audio/mfcc/tune/setup_mfcc.m	Adds mel40 and mel40_compress MFCC blob generation
src/audio/mfcc/mfcc.c	Adjusts DP prepare sizing to consume full MFCC hop per DP wake
src/audio/buffers/ring_buffer.c	Adds vregion_put on ring_buffer free to match new refcounting
src/audio/buffers/audio_buffer.c	Allows secondary buffers on both sides and adds DP-to-DP sync logic
scripts/tensorflow-clone.sh	Makes BASE_DIR configurable via environment override
app/boards/intel_adsp_cavs25.conf	Enables C++/TFLM static build and increases heap/malloc arena sizes

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/audio/tensorflow/README.md:293

The README states the shipped model was retrained “currently against the hey_linux keyword”, but the generated label header checked in by this PR (sof_tflm_labels.h) contains the example labels {silence, unknown, banana, mango, orange}. Please align the documentation with the actual model/labels committed in this PR (or vice versa) so readers don’t follow the wrong training/validation expectations.

The stock model only recognizes `yes`/`no` (plus `silence`/`unknown`). The
shipped `sof_tflm_quantized_model_data.{cc,h}` was retrained end-to-end
against real SOF mel40 features (currently against the `hey_linux`
keyword) using the scripts under [./tune/](./tune/). This section
documents that exact recipe so the model can be reproduced, a different

singalsu · 2026-08-20T15:55:55Z

+# mfcc widget class is already included at the top of the parent
+# sof-hda-generic.conf, so no need to pull include/components/mfcc.conf
+# here again.


This can't be done, the topologies build would fail. I think the topology work for dmic1-mfcc.conf is obsolete. I should replace it with similar pipelines as for Whisper and for keyword recognize and kpb buffer.

Remove the is_relocatable branch that skipped -Wl,-Ttext/--section-start placement and the custom llext_merge.ld section-merge script for CONFIG_LLEXT_TYPE_ELF_RELOCATABLE builds, and remove the clang --target=/--ld-path= hoisting logic. All LLEXT builds now go through the same fixed-section-address layout path unconditionally, and the strip-debug/remove-section objcopy flags applied afterward are dropped as well.

…ted-symbol segment llext_manager_layout_sections() rebases the addresses of recognized sections in place, directly in the raw ELF buffer, before llext_load() ever parses the file. Three problems with the pre-existing rebase and layout logic, all fixed together here: 1. Relocation/symbol staleness after rebase: two classes of data in the file still referenced the OLD (pre-rebase) addresses after this mutation. .rela.dyn/.rela.plt r_offset fields are byte-for-byte copies from the build-time ELF, so llext_link_plt()'s llext_file_offset() lookup failed (Offset not found) for any relocation whose target section moved. R_XTENSA_RELATIVE relocations are treated as a complete no-op by Zephyr's llext core whenever pre_located is set, under the assumption that the stored pointer value is already correct -- SOF's rebase step violates that assumption, so the delta has to be applied to the pointer value at each R_XTENSA_RELATIVE target directly. .symtab/.dynsym st_value fields are taken as final absolute addresses, unmodified, under the same pre_located assumption, feeding both llext_find_sym() lookups and exported symbol addresses. Track (old_addr, size, delta, shdr index) for every section actually rebased, then walk the ELF again applying all three fixups via the new llext_manager_fixup_rela() and llext_manager_fixup_symtab(). 2. Layout packing: .bss must remain immediately contiguous with writable DATA (they share a single VMA mapping, since .bss has no file backing and just extends DATA's mapped-and-zeroed tail), so it is now aliased into LLEXT_MEM_DATA for layout purposes instead of being treated as its own region. A region whose sections are split apart by an interleaving section of a different region (e.g. .exported_sym appearing between two .data-region members) now reuses that region's already-established address delta when it reappears, instead of being re-packed into the forward layout cursor as if it were a fresh region -- which previously inserted a spurious page-aligned gap and could desync .bss from the data region it must stay contiguous with. 3. .exported_sym is now tracked as its own real segment (LIB_MANAGER_EXPORT, new enum value), with its own copy-from-storage pass on load/unload -- some toolchains (GNU ld, unlike the Clang LLEXT overlay which merges it into .rodata) keep it as a distinct allocatable section. It also needs eager, synchronous population from the raw ELF buffer before llext_load() runs (llext_manager_load_sections_early()), because llext_load()'s llext_export_symbols() reads .exported_sym content itself during load, well before SOF's own on-demand per-module copy path (llext_manager_load_module()) would otherwise populate it. Also fixes llext_manager_add_library() to index module manifests by module_id + ctx->mod[i].start_idx instead of module_id + i, for correct indexing with multi-module libraries.

Export ams_send(), ams_helper_register_producer(), ams_helper_unregister_producer(), and ams_helper_prepare_payload() so an LLEXT module can act as an AMS message producer (e.g. a keyword-spotting component signaling KPB directly) without needing these calls statically linked into the base image.

…build src/audio/tensorflow/CMakeLists.txt already lists math_stubs.c as a build source, added in commit c6170c0 'Add KPB trigger notifications and stream shutdown event logging', but the file itself was never added. Provide fmaxf/fminf/roundf/round/floor/expf/logf/frexp, which TFLM's op kernels reference but which are undefined when linking a non-relocatable LLEXT module with the GNU/Zephyr-SDK GCC toolchain, as opposed to the Clang/LLVM toolchain which resolves them from libc.

Add a mel40_10ms.conf block to setup_mfcc.m (frame_length=30.0, frame_shift=10.0, num_mel_bins=40), matching microWakeWord's native MixConv 40-feature/10ms-stride frontend shape, and regenerate all MFCC config blobs via octave-cli. Per-hop byte size is unchanged (184B = 24B header + 40 x int32) since it depends only on bin count. Regenerating the other 6 pre-existing blobs only updated their 'Exported MFCC configuration' timestamp comment -- no functional change.

Consume the lib_uuids dict already populated earlier in the script: when a library's final .bin file did not yet exist at UUID-collection time, its UUIDs were deferred into lib_uuids instead of being symlinked immediately. install_lib() now walks lib_uuids[key] and creates the deferred <uuid>.bin symlink/copy once the library is actually installed.

Add the microWakeWord keyword-spotting component's build scaffolding: Kconfig (CONFIG_COMP_MWW, tristate), CMakeLists.txt (static and LLEXT build paths, mirroring src/audio/tensorflow/'s dual-path shape), mww.toml (rimage module manifest entry), and the llext/ subdirectory (CMakeLists.txt, llext.toml.h, llext-wrap.c reentrant-stub shims) needed to build it as a real loadable LLEXT module. Wire it in: src/audio/Kconfig rsources microwakeword/Kconfig, src/audio/CMakeLists.txt add_subdirectory()s it under CONFIG_COMP_MWW, uuid-registry.txt gets a new mww UUID line, and tools/rimage/config/ptl.toml.h includes mww.toml under the same CONFIG_COMP_MWW guard used by other modular components. No component implementation yet -- mww.c/mww_model.cc follow in the next commit.

mww.c: SOF module adapter (init/prepare/process/reset), single CONFIG_COMP_MWW_MODULE-gated #if/#else shape like src/audio/template/template.c. Consumes MFCC feature hops from the upstream mfcc component, requantizes them per the model's real input_scale/input_zero_point, runs TFLM inference, and acts as an AMS producer of AMS_KPD_MSG_UUID -- mirroring src/samples/audio/detect_test.c's AMS-producer pattern -- sending a KPB wake notification when probability crosses the detection threshold. src/audio/kpb.c's existing AMS-consumer branch needs no changes to receive this. mww_model.cc/.h: TFLM bridge exposing Init_Interpreter()/ TF_ProcessClassify(), structurally like speech.cc/speech.h but for a single-sigmoid probability output rather than tflmcly's 4-way softmax category output, with its own MicroMutableOpResolver sized for microWakeWord's MixConv streaming architecture. mww_model_data.cc/.h: placeholder model data, swappable for a real trained/converted checkpoint.

Enable CONFIG_COMP_MWW=m (real loadable LLEXT), CONFIG_CPP/CONFIG_STD_CPP17 for its TFLM C++ dependency, and CONFIG_AMS=y so aphid's KPB takes the same AMS-consumer path as spider instead of falling through to notifier. Disable CONFIG_COMP_TENSORFLOW on this board: tflmcly (the stock 4-class keyword model) is not yet GNU-toolchain-clean on aphid (undefined sof_ut_log/__assert_no_args/abs) and is not required by mww, which builds its own independent TFLM lib copy. Bump CONFIG_HEAP_MEM_POOL_SIZE and CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE to 32768 and CONFIG_STACK_SIZE_EDF to 32768 for TFLM's memory footprint, and raise CONFIG_LLEXT_HEAP_SIZE from the 8 KB default to 256 KB: mww.llext links in TFLM, whose template-instantiated vtables/typeinfo are emitted STB_WEAK and, now that they are included in ext->sym_tab, blow past the default LLEXT metadata heap size. Also enable CONFIG_XTENSA_ADSP_FATAL_BREADCRUMB_DATA_VADDR and CONFIG_SYS_HEAP_RUNTIME_STATS (useful for diagnosing openmodules-load faults) and set CONFIG_SOF_ZEPHYR_HEAP_SIZE=0x80000.

Add the mww widget class (include/components/mww.conf, mirroring tflm.conf's tflmcly class), the DP-scheduled capture pipeline (host-gateway-src-mfcc-mww-capture.conf: src -> mfcc -> mww -> host-copier, mfcc's bytes-control defaulting to the new mel40_10ms.conf profile), and the top-level topology (sof-hda-mww.conf, mirroring sof-hda-tflm.conf's dual-path KPB routing: dai-copier -> gain -> kpb, pin 1 to the detection path, pin 2 to a host WoV drain PCM). Register the sof-ptl-hda-mww-kpb build target in tplg-targets-hda-generic.cmake for PLATFORM=ptl (aphid).

Document the mww component's architecture and data flow (modeled on src/audio/tensorflow/README.md), the LLEXT fixes required to build and load it as a real module on aphid, and build/deploy/test instructions. Document two open items: (1) MFCC hop cadence stayed ~76ms on hardware after switching the frontend from a 20ms to 10ms hop, not halving as expected, with the leading hypothesis being that pipeline DP scheduling period rather than the MFCC blob's frame_shift governs real invocation rate; (2) an intermittent arecord I/O error against the debug-drain PCM device, confirmed isolated to that debug path and not affecting core detection pipeline correctness.

Copilot

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/audio/tensorflow/tflm-classify.c:412

tflm_process() logs AGC state with comp_info() for every MFCC hop. This runs ~50 times/second and will spam logs and add measurable overhead even when CONFIG_COMP_TENSORFLOW_DEBUG_TRACE is disabled. Gate this log behind the existing debug option (or downgrade to a debug-only trace).

			int32_t agc_gain_q23 = cd->agc_gain_q23;

			comp_info(mod->dev, "tflm agc: peak_q23=%d gain_q23=%d",
				  hop_peak_q23, agc_gain_q23);

src/audio/tensorflow/tflm-classify.c:297

tflm_free() resets g_tflm_initialized when the last instance is freed, but the TFLM backend in speech.cc allocates op_resolver/interpreter with new and never frees them on shutdown. If a stream is opened again later, tflm_prepare() will re-run TF_InitOps() and leak those objects each cycle. Either keep the shared backend initialized for the lifetime of the firmware, or add explicit teardown in the TF backend before clearing this flag.

	tflm_log_summary_at_shutdown(mod);
	if (--g_tflm_instance_count <= 0) {
		g_tflm_instance_count = 0;
		g_tflm_initialized = false;
	}

 DECLARE_TR_CTX(tflm_tr, SOF_UUID(tflmcly_uuid), LOG_LEVEL_INFO);
 DECLARE_MODULE_ADAPTER(tflmcly_interface, tflmcly_uuid, tflm_tr);
-SOF_MODULE_INIT(tflmcly, sys_comp_module_tflmcly_interface_init);
+SOF_MODULE_INIT(tflmcly_interface, sys_comp_module_tflmcly_interface_init);
+


The on-device TFLM wake-word path (KPB -> SRC -> MFCC -> tflmcly) needs a 40-bin/20 ms-hop mel spectrogram to match TFLM micro_speech's front-end shape (FEATURE_SIZE=40, FEATURE_STRIDE_MS=20, FEATURE_DURATION_MS=30). Two variants are required: mel40.conf for the plain audio-buffer path used for off-target sanity checking and mel40_compress.conf for the compress PCM output actually consumed by the wake-word inference component. Add both export blocks to setup_mfcc.m so the blobs stay in lock-step under any future front-end tuning change, and ship the initial generated mel40.conf and mel40_compress.conf so the Wake-on-Voice topology graphs that consume them have matching blobs to reference. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

When the MFCC component runs in the DP (Data Processing) scheduling domain, the DP thread is woken on ibs, which is roughly one FFT hop of input. Sizing the processing capacity to only dev->frames means the DP thread has to re-enter many times per LL tick to nibble through one hop, adding unnecessary scheduling overhead. Raise the max_frames argument passed to mfcc_setup() to at least cd->config->frame_shift when the component is scheduled in the DP domain, so a single DP invocation drains one whole hop. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Add topology graphs and IPC4 ABI manifest bits for the Wake-on-Voice pipeline on HDA and SoundWire jack inputs, plus a bench variant for off-target training and validation. Register the KPB IPC4 UUID, attach the mel40 MFCC binary config blob as a real config, and give the ARL target the HDA_MIC_MFCC_PARAMS it was missing. Wire the dual-path KPB routing so the always-on inference branch and the host capture branch share the same buffer. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Enable KPB, TFLM, MFCC, Gain and Volume on the cAVS 2.5 (TGL/ADL/RPL) board defconfig and size its DP scheduler heap for the TFLM working set. Register the DP scheduler in the cAVS platform init so the DP domain pipelines that host MFCC and TFLM come up cleanly, and add tflmcly.toml to the TGL rimage manifest so the signed image lists the new modules. Also make the tensorflow static CMake wiring portable across xtensa targets: hardcoded /home/lrg zephyr-sdk paths, mcpu names and include directories are replaced with SOC_TOOLCHAIN_NAME and ZEPHYR_SDK_INSTALL_DIR expansions, the -mllvm text-section-literals flag is feature-detected, and HiFi4-only kernels are gated on TENSORFLOW_HAVE_NNLIB_HIFI4. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Document the TFLM Wake-on-Voice architecture: pipeline graph, module UUIDs, topology entry points, static vs LLEXT build variants, MFCC config blob layout, KPB trigger flow, and the on-device vs off-device verification loop. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Add an end-to-end training pipeline for TFLM wake-word models under src/audio/mfcc/tune and src/audio/tensorflow/tune: - Piper-TTS driven keyword dataset generator with per-file Gaussian gain jitter capped 3 dB below full scale so the augmented WAVs directly reflect the level distribution the model will train on without clipping. - Batch feature extractor that runs the SOF testbench MFCC on the WAVs and writes raw mel40 features. - Silence and unknown-class preparation from Speech Commands v2 with the same level-jitter policy (silence exempt). - Python trainer that fits an int8-quantized DS-CNN on the mel features, emits the tflite/C-array pair, and auto-writes a sof_tflm_labels.h header so the runtime labels match the training set without hand-editing. - Off-device verifier that runs the tflite model against a directory of WAVs and reports per-class scores. Refreshes the tensorflow install and training recipe and fixes int8 calibration to use the correct scale and zero-point. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

The existing sof_tflm_generate_keyword_dataset.sh is hard-wired to the English multi-speaker piper-sample-generator (LibriTTS-R). It cannot train a positive class in any other language because the phonemizer inside that checkpoint only knows English graphemes. Add a companion script that drives the regular piper-tts package against any single-speaker voice from rhasspy/piper-voices (Finnish fi_FI-harri, Swedish sv_SE-nst, etc.). To compensate for the missing speaker-embedding axis it (a) randomizes Piper noise-scale / noise-w per utterance while cycling several length-scale values and (b) fans each synthesized clip out into sox pitch-cents plus pitch-preserving tempo copies. Output layout, IR augmentation hook, and gain jitter match the English generator so sof_tflm_train_pipeline.sh consumes it unchanged. Update the tune README with a language-agnostic pipeline table entry, a non-English setup section. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Add a soft AGC in the Q9.23 mel-log domain that automatically clamps peak energy to prevent clipping on loud utterances, while allowing recovery toward a 0 dB target at 0.5 dB/sec during active speech. Requantization is mapped symmetrically from [-1.0, +1.0] in Q9.23 to [-128, 127] in int8 to match model calibration. In addition, track a 49-hop VAD history bitmask from the MFCC header. When the entire sliding context window contains only silence (VAD=0), int8 requantization and neural network inference execution are gated off, substantially reducing DSP MCPS load during idle periods. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Add playback_compatible_d0i3 and capture_compatible_d0i3 attribute definitions to the host copier widget class. This allows topologies to specify D0i3 stream compatibility flags directly on host copier widgets, enabling the DSP to enter and remain in D0i3 during S0ix while the stream is active. Enable capture_compatible_d0i3 on the WoV capture and TFLM detect host copier widgets in the HDA microphone TFLM Wake-on-Voice topology. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

…r Clang build

…cies, and performance metrics

lrgirdwo added 19 commits July 22, 2026 21:07

audio: tflm: add keyword detection logging, host notification, and HD…

6583589

…A capture topology

audio: tflm: fix tensor byte sizing for micro speech int8 input

411ed5e

audio: tflm: integrate keyword detection with KPB for WoV trigger

3776df6

tools: topology2: update WoV pipeline graph with dual-path KPB routing

a6c639c

docs: tensorflow: add comprehensive README for TFLM WoV architecture …

df47967

…and topology usage

ptl: app: enable KPB, TFLM, MFCC, Gain, Volume components and update …

930de94

…library base address

ptl: rimage: include KPB, TFLM, Volume and MFCC in base firmware mani…

ea7449b

…fest

tools: topology2: update KPB IPC4 UUID and HDA TFLM 16kHz WoV topology

cf89a85

singalsu commented Aug 14, 2026

View reviewed changes

Comment thread src/audio/tensorflow/micro_speech_quantized_model_data.cc

singalsu force-pushed the tflm_mfcc_development branch 2 times, most recently from 15aff1e to 80b518b Compare August 14, 2026 12:14

singalsu requested a lite review from Copilot August 14, 2026 13:16

Copilot started reviewing on behalf of singalsu August 14, 2026 13:17 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

singalsu force-pushed the tflm_mfcc_development branch from 80b518b to 2f16184 Compare August 14, 2026 15:04

singalsu requested a lite review from Copilot August 14, 2026 15:05

Copilot started reviewing on behalf of singalsu August 14, 2026 15:09 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

lrgirdwo added 11 commits August 16, 2026 18:15

singalsu force-pushed the tflm_mfcc_development branch 2 times, most recently from 69e9d05 to abe90f1 Compare August 20, 2026 15:57

singalsu requested a lite review from Copilot August 20, 2026 15:57

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of singalsu August 20, 2026 18:57 View session

lrgirdwo and others added 10 commits August 23, 2026 20:27

audio: llext: fix Clang LLEXT build and OBJECT library linking

07e188d

singalsu mentioned this pull request Aug 24, 2026

Audio: Buffers: Add support for DP-to-DP component binding #10562

Open

audio: microwakeword: placement new allocations and arena fallback fo…

984f1f8

…r Clang build

lgirdwood force-pushed the tflm_mfcc_development branch from abe90f1 to 984f1f8 Compare August 24, 2026 16:48

audio: microwakeword: update README with build instructions, dependen…

ac305bd

…cies, and performance metrics

Navigation Menu

Uh oh!

audio: tensorflow: keyword detect - #11092

audio: tensorflow: keyword detect#11092
singalsu wants to merge 44 commits into
thesofproject:mainfrom
singalsu:tflm_mfcc_development

singalsu commented Aug 13, 2026

Uh oh!

Uh oh!

Uh oh!

singalsu Aug 14, 2026

Uh oh!

singalsu Aug 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singalsu Aug 14, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

singalsu Aug 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

singalsu commented Aug 13, 2026

Uh oh!

Uh oh!

Uh oh!

singalsu Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

singalsu Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singalsu Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

singalsu Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants