| New file |
| | |
|---|
| | | # Xiaozhi AI Architecture Audit |
|---|
| | | |
|---|
| | | ä½ ç°å¨è´è´£å¯¹ Xiaozhi AI 项ç®è¿è¡ä¸æ¬¡ **Architecture Audit**ã |
|---|
| | | |
|---|
| | | æ¬æ¬¡ä»»å¡æ¯ï¼ |
|---|
| | | |
|---|
| | | > **åªåæï¼ä¸ä¿®æ¹ä»£ç ã** |
|---|
| | | |
|---|
| | | 项ç®å
å«ä¸¤ä¸ª Repositoryï¼ |
|---|
| | | |
|---|
| | | 1. Xiaozhi Client ââ 硬件端 / Device Runtime |
|---|
| | | 2. Xiaozhi Server ââ Server / AI / Backend |
|---|
| | | |
|---|
| | | åæ¶æä¾äºä¸ä»½ãXiaozhi AI å¼æ¾åºç¨å¹³å° Master Planãã |
|---|
| | | |
|---|
| | | Master Plan æ¯æªæ¥ç®æ æ¶æï¼ä½å½å代ç å¾å¯è½ä¸ç®æ æ¶æåå¨è¾å¤§å·®å¼ã |
|---|
| | | |
|---|
| | | ä½ çä»»å¡ä¸æ¯ç«å³å®ç° Master Planï¼èæ¯ï¼ |
|---|
| | | |
|---|
| | | > æ¾åºå½åç³»ç»æ¯ä»ä¹ãå·²ç»å
·å¤ä»ä¹ã缺å°ä»ä¹ï¼ä»¥ååºè¯¥ä»¥ä»ä¹æå°ä»£ä»·éæ¥æ¼è¿å°ç®æ æ¶æã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## ä¸ãç¦æ¢äºé¡¹ |
|---|
| | | |
|---|
| | | æ¬é¶æ®µç¦æ¢ï¼ |
|---|
| | | |
|---|
| | | - ä¿®æ¹ä»£ç |
|---|
| | | - å é¤ä»£ç |
|---|
| | | - éæ代ç |
|---|
| | | - å建æ°çä¸å¡æ¨¡å |
|---|
| | | - ä¿®æ¹æ°æ®åº |
|---|
| | | - ä¿®æ¹ API |
|---|
| | | - ä¿®æ¹åè®® |
|---|
| | | - ä¿®æ¹é
ç½® |
|---|
| | | - ä¿®æ¹ç¡¬ä»¶ç«¯é»è¾ |
|---|
| | | |
|---|
| | | åªå
è®¸ï¼ |
|---|
| | | |
|---|
| | | - é
读代ç |
|---|
| | | - åæ代ç |
|---|
| | | - 建ç«æ¶æè®¤ç¥ |
|---|
| | | - è¾åºåææ¥å |
|---|
| | | - æåºå»ºè®® |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # äºãAudit èå´ |
|---|
| | | |
|---|
| | | å¿
é¡»åæ¶åæ Client å Serverã |
|---|
| | | |
|---|
| | | ä¸è¦åªçç®å½ç»æã |
|---|
| | | |
|---|
| | | éè¦è¿½è¸ªçå®ä»£ç è°ç¨é¾ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # ä¸ãClient Audit |
|---|
| | | |
|---|
| | | éç¹åæï¼ |
|---|
| | | |
|---|
| | | ## 1. å½å Runtime |
|---|
| | | |
|---|
| | | åçï¼ |
|---|
| | | |
|---|
| | | - Client å½åå¯å¨æµç¨æ¯ä»ä¹ï¼ |
|---|
| | | - 主循ç¯å¨åªéï¼ |
|---|
| | | - å½å Conversation / Chat Runtime å¨åªéï¼ |
|---|
| | | - ç¨æ·è¯´è¯ â ASR â LLM â TTS â ææ¾çå®æ´è°ç¨é¾æ¯ä»ä¹ï¼ |
|---|
| | | - Audio Input / Output å¦ä½å®ç°ï¼ |
|---|
| | | - æ¯å¦å·²ç»åå¨ Event / Message / Command æºå¶ï¼ |
|---|
| | | - Server ä¸ Client 使ç¨ä»ä¹éä¿¡åè®®ï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. 硬件è½å |
|---|
| | | |
|---|
| | | æ´çå½åå·²ç»åå¨çï¼ |
|---|
| | | |
|---|
| | | - 麦å
é£ |
|---|
| | | - æ¬å£°å¨ |
|---|
| | | - é³é¢ææ¾ |
|---|
| | | - é³é¢å½é³ |
|---|
| | | - VAD |
|---|
| | | - æé® |
|---|
| | | - LED |
|---|
| | | - å±å¹ï¼å¦ææï¼ |
|---|
| | | - SD Card |
|---|
| | | - ç½ç» |
|---|
| | | - MQTT / WebSocket / HTTP çéä¿¡è½å |
|---|
| | | |
|---|
| | | 并æåºåªäºè½åå·²ç»å¯ä»¥ç´æ¥ä½ä¸ºæªæ¥ Skillã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. æ¾åºå½åå¯ä»¥æ½è±¡æ Skill çè½å |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | play_audio |
|---|
| | | record_audio |
|---|
| | | listen |
|---|
| | | speak |
|---|
| | | wait |
|---|
| | | display |
|---|
| | | led |
|---|
| | | button |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä½ä¸è¦å¼ºè¡åé ä¸åå¨çè½åã |
|---|
| | | |
|---|
| | | å¿
é¡»åºäºçå®ä»£ç åæã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 4. å½å Chat Architecture |
|---|
| | | |
|---|
| | | éç¹åçï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Audio Input |
|---|
| | | â |
|---|
| | | VAD |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Conversation |
|---|
| | | â |
|---|
| | | LLM |
|---|
| | | â |
|---|
| | | TTS |
|---|
| | | â |
|---|
| | | Audio Output |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®é
代ç ä¸åå«å¯¹åºä»ä¹æ¨¡åãç±»ãå½æ°åæä»¶ï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 5. Task Runtime æå
¥ç¹ |
|---|
| | | |
|---|
| | | éç¹åæï¼ |
|---|
| | | |
|---|
| | | æªæ¥å¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åºè¯¥æå
¥ç°æ Client çä»ä¹ä½ç½®ï¼ |
|---|
| | | |
|---|
| | | éç¹åçï¼ |
|---|
| | | |
|---|
| | | > å¦ä½åå°âä¸ç ´åç°æ Chat åè½âçæ
åµä¸å¢å Task Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # åãServer Audit |
|---|
| | | |
|---|
| | | éç¹åæï¼ |
|---|
| | | |
|---|
| | | ## 1. Server å½åæ¶æ |
|---|
| | | |
|---|
| | | åæï¼ |
|---|
| | | |
|---|
| | | - Server å¯å¨å
¥å£ |
|---|
| | | - API |
|---|
| | | - WebSocket |
|---|
| | | - MQTT |
|---|
| | | - Session |
|---|
| | | - User |
|---|
| | | - Device |
|---|
| | | - Conversation |
|---|
| | | - Audio |
|---|
| | | - ASR |
|---|
| | | - LLM |
|---|
| | | - TTS |
|---|
| | | |
|---|
| | | ä¹é´çå
³ç³»ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. å½å对è¯é¾è·¯ |
|---|
| | | |
|---|
| | | å®æ´è¿½è¸ªï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device |
|---|
| | | â |
|---|
| | | Server |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Conversation |
|---|
| | | â |
|---|
| | | LLM |
|---|
| | | â |
|---|
| | | TTS |
|---|
| | | â |
|---|
| | | Device |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æåºçå®ä»£ç å®ç°ä½ç½®ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. æ°æ®åº |
|---|
| | | |
|---|
| | | ååºå½åä¸ä»¥ä¸å¯¹è±¡ç¸å
³çæ°æ®è¡¨ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User |
|---|
| | | Device |
|---|
| | | Conversation |
|---|
| | | Message |
|---|
| | | Audio |
|---|
| | | Session |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 以åå
¶ä»ä¸æªæ¥ Task ç³»ç»å¯è½ç¸å
³ç表ã |
|---|
| | | |
|---|
| | | 说æï¼ |
|---|
| | | |
|---|
| | | - 表ç»æ |
|---|
| | | - ä¸»é® |
|---|
| | | - å¤é® |
|---|
| | | - å
³ç³» |
|---|
| | | - å½åæ°æ®çå½å¨æ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # äºãéç¹å¯»æ¾æªæ¥ Domain Model çè½ç¹ |
|---|
| | | |
|---|
| | | 对以ä¸æ¦å¿µéä¸åæï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | Step |
|---|
| | | Skill |
|---|
| | | InteractionMode |
|---|
| | | TaskInstance |
|---|
| | | StepInstance |
|---|
| | | Event |
|---|
| | | Audio |
|---|
| | | Assessment |
|---|
| | | Metric |
|---|
| | | Schedule |
|---|
| | | Plan |
|---|
| | | Subscription |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 对äºæ¯ä¸ä¸ªæ¦å¿µåçï¼ |
|---|
| | | |
|---|
| | | 1. å½å代ç ä¸æ¯å¦å·²ç»åå¨ç±»ä¼¼æ¦å¿µï¼ |
|---|
| | | 2. å¦æåå¨ï¼å¨åªéï¼ |
|---|
| | | 3. æ¯å¦å¯ä»¥å¤ç¨ï¼ |
|---|
| | | 4. æ¯å¦éè¦æ©å±ï¼ |
|---|
| | | 5. æ¯å¦éè¦æ°å»ºï¼ |
|---|
| | | 6. æéåæ¾å¨åªä¸ª Repository / Moduleï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # å
ãéç¹åæ InteractionMode |
|---|
| | | |
|---|
| | | Master Plan ä¸ï¼ |
|---|
| | | |
|---|
| | | > InteractionMode ä¸æ¯ä¸å¡åºæ¯ï¼èæ¯ Xiaozhi ç硬件交äºæ¹å¼ã |
|---|
| | | |
|---|
| | | éç¹ç¡®è®¤å½å代ç æ¯å¦åå¨ç±»ä¼¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Chat |
|---|
| | | Audio Playback |
|---|
| | | Recording |
|---|
| | | Listening |
|---|
| | | Translation |
|---|
| | | Game |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æè
å
¶ä»æ¨¡å¼ã |
|---|
| | | |
|---|
| | | åæï¼ |
|---|
| | | |
|---|
| | | > InteractionMode æéåæ¾å¨ ClientãServerï¼è¿æ¯ä¸¤è¾¹é½æï¼ |
|---|
| | | |
|---|
| | | 并ç»åºæ¨èã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # ä¸ãéç¹åæ Task Runtime |
|---|
| | | |
|---|
| | | è¿æ¯æ¬æ¬¡ Audit æéè¦çé¨åã |
|---|
| | | |
|---|
| | | åçï¼ |
|---|
| | | |
|---|
| | | å½åç³»ç»åºè¯¥å¦ä½ä»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Conversation Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | éæ¥æ¼è¿ä¸ºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device Runtime |
|---|
| | | âââ Chat Runtime |
|---|
| | | âââ Task Runtime |
|---|
| | | âââ Step |
|---|
| | | âââ Skill |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸è¦ç´æ¥è®¾è®¡æ°ä»£ç ã |
|---|
| | | |
|---|
| | | å
åæï¼ |
|---|
| | | |
|---|
| | | > å½å代ç å¨åªéæéåæå
¥è¿ä¸ª Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # å
«ãéç¹åæ Audio / Recording |
|---|
| | | |
|---|
| | | æªæ¥ Server éè¦ä¿åå¿ç«¥ Task æ§è¡è¿ç¨ä¸çè¯é³ã |
|---|
| | | |
|---|
| | | å æ¤åæï¼ |
|---|
| | | |
|---|
| | | - å½åå½é³å¨åªé产çï¼ |
|---|
| | | - é³é¢æ¯å¦å·²ç»ä¸ä¼ Serverï¼ |
|---|
| | | - é³é¢æ ¼å¼æ¯ä»ä¹ï¼ |
|---|
| | | - æ¯å¦å¯ä»¥ä¿ååå§é³é¢ï¼ |
|---|
| | | - å½åé³é¢çå½å¨ææ¯ä»ä¹ï¼ |
|---|
| | | - æ¯å¦åå¨ Audio IDï¼ |
|---|
| | | - æ¯å¦åå¨ Session IDï¼ |
|---|
| | | - æ¯å¦å¯ä»¥å
³èå° TaskInstance / StepInstanceï¼ |
|---|
| | | |
|---|
| | | æåºæªæ¥æå°æ¹é æ¹æ¡ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # ä¹ãéç¹åæ Event |
|---|
| | | |
|---|
| | | æ£æ¥å½åæ¯å¦å·²ç»åå¨ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Event |
|---|
| | | Message |
|---|
| | | Command |
|---|
| | | State |
|---|
| | | Callback |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | çæºå¶ã |
|---|
| | | |
|---|
| | | å¦æåå¨ï¼ |
|---|
| | | |
|---|
| | | > å¤æè½å¦æ¼è¿æç»ä¸ Event Architectureã |
|---|
| | | |
|---|
| | | å¦æä¸åå¨ï¼ |
|---|
| | | |
|---|
| | | > æåºæåéçæå
¥ä½ç½®ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # åãæç»è¾åº Audit Report |
|---|
| | | |
|---|
| | | æç»ä¸è¦åªè¾åºæ»ç»ã |
|---|
| | | |
|---|
| | | å¿
é¡»è¾åºä»¥ä¸å
å®¹ï¼ |
|---|
| | | |
|---|
| | | ## 1. Current Architecture |
|---|
| | | |
|---|
| | | åå«ç»åºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Client Architecture |
|---|
| | | Server Architecture |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æå¥½ä½¿ç¨ Mermaidã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. Current Runtime Call Graph |
|---|
| | | |
|---|
| | | ç»åºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¨æ·è¯´è¯ |
|---|
| | | â |
|---|
| | | ... |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 对åºçå®ä»£ç 模åã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. Existing Capability Map |
|---|
| | | |
|---|
| | | æ´çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Capability |
|---|
| | | Current Implementation |
|---|
| | | Repository |
|---|
| | | Module |
|---|
| | | Reusable? |
|---|
| | | Required Change |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 4. Domain Mapping |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Master Plan Domain |
|---|
| | | â |
|---|
| | | Current Code |
|---|
| | | â |
|---|
| | | Gap |
|---|
| | | â |
|---|
| | | Recommended Location |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 5. Architecture Gap |
|---|
| | | |
|---|
| | | æç¡®ååºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | P0 å¿
é¡»è§£å³ |
|---|
| | | P1 åç»è§£å³ |
|---|
| | | P2 æªæ¥å¹³å°åè§£å³ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 6. Recommended Evolution Architecture |
|---|
| | | |
|---|
| | | ä¸æ¯è¦æ±ç«å³å®ç°ã |
|---|
| | | |
|---|
| | | èæ¯è¯´æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Current |
|---|
| | | â |
|---|
| | | Phase 1 |
|---|
| | | â |
|---|
| | | Phase 2 |
|---|
| | | â |
|---|
| | | Phase 3 |
|---|
| | | â |
|---|
| | | Target Architecture |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 7. æå
³é®çä¿®æ¹ç¹ |
|---|
| | | |
|---|
| | | æç»æç¡®ååºï¼ |
|---|
| | | |
|---|
| | | ### Client |
|---|
| | | |
|---|
| | | éè¦æ¹åªäºæ¨¡åã |
|---|
| | | |
|---|
| | | ### Server |
|---|
| | | |
|---|
| | | éè¦æ¹åªäºæ¨¡åã |
|---|
| | | |
|---|
| | | ### Database |
|---|
| | | |
|---|
| | | éè¦æ°å¢/ä¿®æ¹åªäºå®ä½ã |
|---|
| | | |
|---|
| | | ### Protocol |
|---|
| | | |
|---|
| | | éè¦å¢å åªäºæ¶æ¯ã |
|---|
| | | |
|---|
| | | ### API |
|---|
| | | |
|---|
| | | éè¦å¢å åªäºæ¥å£ã |
|---|
| | | |
|---|
| | | ä½æ¤é¶æ®µåªæåºï¼ä¸å®ç°ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # åä¸ãæç»å¿
é¡»åççæ ¸å¿é®é¢ |
|---|
| | | |
|---|
| | | 请ç¹å«åç以ä¸é®é¢ï¼ |
|---|
| | | |
|---|
| | | ### Q1 |
|---|
| | | |
|---|
| | | å½å Chat Runtime è½å¦ç´æ¥ä½ä¸ºæªæ¥ Task Runtime çä¸ä¸ªå Runtimeï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q2 |
|---|
| | | |
|---|
| | | Task Runtime æéåæ¾å¨ Client è¿æ¯ Serverï¼ |
|---|
| | | |
|---|
| | | 请ç»åå½å代ç åæï¼èä¸æ¯åªæ ¹æ®ç论åçã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q3 |
|---|
| | | |
|---|
| | | Step ç¶æåºè¯¥ç± Client ä¿åãServer ä¿åï¼è¿æ¯ä¸¤è¾¹é½æï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q4 |
|---|
| | | |
|---|
| | | Task æ§è¡è¿ç¨ä¸ Server ä¸ Client çç¶æåæ¥æºå¶åºè¯¥å¦ä½è®¾è®¡ï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q5 |
|---|
| | | |
|---|
| | | App / Task / Resource æ¯å¦å¯ä»¥å®å
¨éè¿ JSON + Audio Resource ä¸åï¼èä¸éè¦ OTAï¼ |
|---|
| | | |
|---|
| | | å¦æä¸è½ï¼è¯·æåºåªäºè½åå¿
é¡»åºåå° Firmwareã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q6 |
|---|
| | | |
|---|
| | | å½å代ç è·ç¦»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | â |
|---|
| | | Device Capability |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ大çæ¶æéç¢æ¯ä»ä¹ï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q7 |
|---|
| | | |
|---|
| | | å¦ä½ä»¥æå°ä¿®æ¹å®ç°ç¬¬ä¸ä¸ªï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯å¤è¯» Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸ç ´åç°æ Chatï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### Q8 |
|---|
| | | |
|---|
| | | å¦ä½ä»¥æå°ä¿®æ¹å®ç°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯è·è¯» |
|---|
| | | â |
|---|
| | | å½é³ |
|---|
| | | â |
|---|
| | | ä¸ä¼ |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # åäºãæç»ç»è®º |
|---|
| | | |
|---|
| | | æåç»åºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Architecture Audit Conclusion |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 并æç¡®ï¼ |
|---|
| | | |
|---|
| | | 1. å½åæ¶æå¯ä»¥ä¿çä»ä¹ |
|---|
| | | 2. å½åæ¶æå¿
é¡»æ¹ä»ä¹ |
|---|
| | | 3. 第ä¸é¶æ®µåºè¯¥æ¹ä»ä¹ |
|---|
| | | 4. åªäºäºæ
ç°å¨ç»å¯¹ä¸è¦å |
|---|
| | | 5. ä¸ä¸é¶æ®µåºè¯¥å¦ä½çæ Phase 1 Coding Plan |
|---|
| | | |
|---|
| | | å次强è°ï¼ |
|---|
| | | |
|---|
| | | > æ¬æ¬¡ä»»å¡çç®æ ä¸æ¯å代ç ï¼èæ¯å»ºç«å¯¹å½åç³»ç»çå®æ´æ¶æ认ç¥ï¼å¹¶æ¾å°ä» Current Architecture å° Master Plan çæå°æ¼è¿è·¯å¾ã |
|---|
| | | |
|---|
| | | å®æ Audit ååæ¢ã |
|---|
| | | |
|---|
| | | çå¾
ä¸ä¸æ¥æ令ã |
|---|
| New file |
| | |
|---|
| | | # Phase 1 Coding Plan |
|---|
| | | |
|---|
| | | > File: `phase-1.md` |
|---|
| | | > Date: 2026-08-30 |
|---|
| | | > Updated: é»è®¤ chat.conversation pendingï¼é
ç½/å¼æºä¸»å¨åèµ·ï¼ä¸ééæåæ¢ |
|---|
| | | > Based on: Master Plan · Architecture Audit · Client Plan V1.1 · Server Plan V1.1 · Cross-System Review |
|---|
| | | > **æ¬é¶æ®µäº§åºè®¾è®¡ä¸ç¼ç ä»»å¡æ¸
åãææ¬æ件å®æ½ï¼ä¸ V1.1 Plan å²çªæ¶ï¼ä»¥æ¬æ为åã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 0. Cross-System Review ç»è®º |
|---|
| | | |
|---|
| | | Client Plan ä¸ Server Plan **æ¹åå¯åå**ï¼Server ç¼æãClient æ§è¡ãChat çè·¯å¾ä¸å¨ãMCP ä¼å
ï¼ï¼ä½ä¸è½æ两份ææ¡£çãåæ Phase 1ãåèªå¼å·¥ãå¿
é¡»å
对é½ä¸é¢ä¸æ¡ï¼å¦åå¤è¯»åç¿»è¯é½ä¼åä¸å®æç ´å Chatã |
|---|
| | | |
|---|
| | | ### 0.1 å·²ç»å¯¹é½ï¼ä¿æï¼ |
|---|
| | | |
|---|
| | | | å¥çº¦ | å«ä¹ | |
|---|
| | | |------|------| |
|---|
| | | | æå¨ç¶æå¨ Server | 设å¤ä¸æä¹
å App/Task åå² | |
|---|
| | | | åºä»¶æ ä¸å¡åæ¯ | ç¦æ¢ `if (app == english)` | |
|---|
| | | | DeviceState â InteractionMode | ä¸æå¤è¯»/ç¿»è¯å¡è¿ Listening/Speaking è¿ç§»è¡¨ | |
|---|
| | | | åè®®è¯ä¹å»ç» | `hello/listen/abort/tts/stt/mcp` ä¸æ¹å段å«ä¹ | |
|---|
| | | | å¯å¤ç¨ç¡¬ä»¶ | Sening `self.mp3.play` / `self.mp3.stop` / `self.power_off`ï¼`PauseSdCardMp3` | |
|---|
| | | |
|---|
| | | ### 0.2 åæå²çªä¸ Phase 1 å³è®® |
|---|
| | | |
|---|
| | | | ç¼ºå£ | åæ | å³è®® | |
|---|
| | | |------|------|------| |
|---|
| | | | App å± | Client 说 Phase 1 ä¸å Appï¼Server 说 App 表 Phase 3 | **åè App Runtimeï¼ä»
Server JSON + `active_app_id`ï¼**ã设å¤æ App ç®å½ãæ App Store | |
|---|
| | | | å¤è¯» N 次 | åæ¹é½å设 Server 循ç¯ç `play.completed` | **å½å `self.mp3.play` ç«å³è¿å `accepted`**ãN 次循ç¯æ¾è¿**è®¾å¤ Skill åæ°** `repeat`ï¼Server åªä¸å `name/repeat/power_off_after` | |
|---|
| | | | ç¿»è¯ | Client æ TRANSLATION Mode æ¾å° Phase 3 | **App å¤ç¨ ASR+TTS 管线ï¼è·³è¿é²è LLMï¼åªæ´é²ç¿»è¯ MCP** | |
|---|
| | | | 软å
³æº | 两份 Plan é½æªåå
¥å¤è¯»éªæ¶ | å¤ç¨ç°æ `self.power_off` â `RequestPowerOffStandby` | |
|---|
| | | | å MQTT | å
许并å | **è²å® snlp MQTT æ¯å°ç¨åºå³æ¶å¯å¨ / 计åä»»å¡çæ§å¶é¢ï¼ä¼å
级æé«ï¼**ã`START_APP_TASK` åªè¡¨è¾¾ãå¯å¨åªä¸ª App/Taskãï¼**å
·ä½è°ç¨åªäº Skill ç±è¯¥ Task çé
ç½®å³å®**ãç¦æ¢ç¨ `PLAY_MEDIA` åå
ãå¯å¨ Appãã | |
|---|
| | | |
|---|
| | | ### 0.3 ååæ»å¾ï¼Phase 1ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è¿å
¥ééï¼ä¼å
级ä»é«å°ä½ï¼ |
|---|
| | | â å°ç¨åºå³æ¶å¼å§ ââè²å® MQTTââ⺠æé«ï¼ææä¸å |
|---|
| | | â¡ å°ç¨åº Schedule ââå°ç¹/é¹éå¤éââ⺠å¯ææ chat |
|---|
| | | ⢠离线è¯é³åæ¢ ââMultiNet å£ä»¤ââ⺠交äºé App/Task |
|---|
| | | â |
|---|
| | | â¼ |
|---|
| | | ç»ä¸ start_app_task(app, task, params, source) |
|---|
| | | â |
|---|
| | | â¼ |
|---|
| | | App Runtime (Server + 设å¤æ§è¡å¨) |
|---|
| | | active_app_id é»è®¤ = chat |
|---|
| | | ââââââââââââââââââââ¼âââââââââââââââââââ |
|---|
| | | â â â |
|---|
| | | chat english_learning english_translation |
|---|
| | | mode=CHAT mode=PLAY_AUDIO mode=TRANSLATION |
|---|
| | | â â â |
|---|
| | | âââââââââ Device Skillï¼MCP / æ¬å°ææ¾å¨ï¼âââââââ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åä¸æ¶å»åªè¿è¡ä¸ä¸ª App / ä¸ä¸ª Taskã |
|---|
| | | |
|---|
| | | **é»è®¤ï¼æ²¡ææ´é«ä¼å
级å
¥å£æ¶ï¼ï¼æ°¸è¿æ¯ç¬¬ä¸ä¸ª App ç default Task = `chat.conversation`ï¼åæå°æºå¯¹è¯è½åï¼ä½æ´ä¸»å¨å¼å£ï¼ã** |
|---|
| | | |
|---|
| | | | äºä»¶ | è¿å
¥ | |
|---|
| | | |------|------| |
|---|
| | | | é¦æ¬¡é
ç½æå | `chat` / default task `conversation`ï¼å°æºå
åèµ·ï¼å¦ãæ们æ¥è天å§ãï¼ | |
|---|
| | | | ä¹åæ®éå¼æº / 软硬å
³æºåå¯å¨ | **继ç»** è¿ä»½ chat pending taskï¼ä¸è¦åå¨çº¯çå¾
é®è¯ç Idle | |
|---|
| | | | ç¨æ·è¯´ãæ们æ¥èå¤©å§ / èä¼å¤©å§ãæå¤éåè天 | ååï¼æä¿æï¼`chat.conversation` | |
|---|
| | | | å°ç¨åºå³æ¶ / 计åå°ç¹ / 离线å App | æç¦»å¼ chat | |
|---|
| | | | RTC é¹éä¸ pending **ä¸æ¯** chatï¼ä¾å¦å¤è¯»ï¼ | æ§è¡è¯¥è®¡å Taskï¼ä¸è¦å
é²è | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 1. Phase 1 ç®æ |
|---|
| | | |
|---|
| | | å **å
¨é¢ä½èç App æ¡æ¶**ï¼å¹¶å¨æ¡æ¶ä¸è·ä¸ä¸ª Appï¼ |
|---|
| | | |
|---|
| | | 1. **`chat`ï¼default App + default Taskï¼** â åæå°æºå¯¹è¯è½åï¼é
ç½åä¸æ¯æ¬¡é»è®¤å¼æºé½è¿å
¥å¹¶ **主å¨åèµ·è天**ï¼ä¸æ¯åªçç¨æ·æé® |
|---|
| | | 2. **`english_learning`** â å¬Â·å¤è¯»æºï¼æ¬å° SD å¡ MP3 ææ¾ **N 次**ï¼ç»æå **å¯é软å
³æº** |
|---|
| | | 3. **`english_translation`** â è±è¯ç¿»è¯ Appï¼Server æå
¥ **åªåç¿»è¯ç MCP**ï¼è¯¥ App ä¸ä¸é²èãä¸è°å
¶ä»æ件 |
|---|
| | | |
|---|
| | | ### 1.1 éç®æ ï¼æç¡®ä¸åï¼ |
|---|
| | | |
|---|
| | | - è·è¯»ãå½é³å½æ¡£ãAssessmentãTaskInstance æ£å¼è¡¨ |
|---|
| | | - Xiaozhi åè®®æ° `type=task`ï¼å°ç¨åºèµ°è²å® MQTTï¼è§ §2.7ï¼ |
|---|
| | | - 家é¿å°ç¨åº UI / å®æ´ Java Scheduler å¾®æå¡ / App Store ä¸æ¶ |
|---|
| | | - æè¯æ¥é¾ãæ°åè诵ãå£ç®çå°æªåå¨ç Task æ¬ä½ï¼è¯é³èååªé¢çæå¾æ§½ï¼ |
|---|
| | | - å并 Xiaozhi MQTT ä¸ snlp/v1 为ä¸æ¡ç©çè¿æ¥ |
|---|
| | | - æ¹ Opus çè·¯å¾ãDeviceState åæ³è¿ç§»è¡¨ |
|---|
| | | - é Sening æ¿ç SD å¤è¯»ï¼ç¿»è¯ App ä¸ä¾èµ SDï¼å¯å
¨æ¿ï¼ |
|---|
| | | - ç¨ `play_music`ï¼Server TTS ééï¼åå
å¤è¯» |
|---|
| | | |
|---|
| | | **å¿
é¡»åå
¥æ¬é¶æ®µæ¡æ¶ï¼æé©å¯æµï¼å°ç¨åºå¯ååï¼ï¼** ä¸ç§è¿å
¥ééãä¼å
级æ¢å ãå
³æºå¤éåç»è·è®¡åä»»å¡ã离线å£ä»¤ä¸åæ¯ç©ºåè°ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. æ¡æ¶è®¾è®¡ï¼æ¬é¶æ®µéç¹ï¼ |
|---|
| | | |
|---|
| | | ### 2.1 æ¦å¿µ |
|---|
| | | |
|---|
| | | | æ¦å¿µ | Phase 1 è½å° | ä¸è½å° | |
|---|
| | | |------|----------------|--------| |
|---|
| | | | App | Server 注å表 + è¿æ¥ä¸ `active_app_id` | åºä»¶ App ManagerãSD `/apps` å®è£
| |
|---|
| | | | Task | `english_learning` å
ä¸ä¸ªãå¤è¯»ãå¨ä½ | Workflow / StepInstance | |
|---|
| | | | InteractionMode | è¿æ¥å段 `CHAT \| PLAY_AUDIO \| TRANSLATION` | æ° DeviceState | |
|---|
| | | | Skill | ç°æ MCP tools + æ©å±åæ° | 设å¤ä¾§ ASR/è¯å | |
|---|
| | | | Default App / Task | `chat` + `conversation` | æ App ç空 Idle | |
|---|
| | | |
|---|
| | | Chat **å°±æ¯** 设å¤ä¸ç第ä¸ä¸ª Appï¼å
¶ **default Task å为 `conversation`**ï¼çäºä»å¤©å°æºç¡¬ä»¶çå¬/说/LLM 对è¯ï¼å ä¸ãå°æºæ´å¤å°ä¸»å¨å¼å£ãã |
|---|
| | | |
|---|
| | | 没æå°ç¨åºå³æ¶ã没æå°æ计åã没æ离线å App æ¶ï¼**ä»»ä½å¯å¨é½è¿å
¥ææ¢å¤è¿ä¸ Task**ï¼ä¸è¦åå¨ãé
好ç½åå¹²çç¨æ·å
说è¯ãã |
|---|
| | | |
|---|
| | | ### 2.1.1 å¯å¨è§£æï¼é»è®¤æ°¸è¿æ¯ chat.conversation |
|---|
| | | |
|---|
| | | 设å¤æ¬å°æä¹
åä¸ä»½ **current/pending**ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | pending_task = { |
|---|
| | | app_id, task_id, params, source, updated_at |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åºå / é¦æ¬¡é
ç½æååçåå¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | app_id = chat |
|---|
| | | task_id = conversation |
|---|
| | | source = default_chat |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ¯æ¬¡å¼æºï¼å«è½¯å
³æºåå¯å¨ãæçµåä¸çµï¼æä¸é¢è§£æï¼**ä»ä¸å°ä¸ç¬¬ä¸æ¡å½ä¸**ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 1. ééâ ç¼åçå³æ¶ MQTT æ令å°æªæ§è¡ â æ§è¡å® |
|---|
| | | 2. ééâ¡ é¹éå°æä¸ pending ä¸æ¯ chat.conversation â æ§è¡è®¡å Task |
|---|
| | | 3. å¦å â æ¢å¤ pendingï¼è¥ç©ºæéæ³ â chat.conversation |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ï¼ |
|---|
| | | |
|---|
| | | - **第ä¸æ¬¡é
ç½æå** â è¿å
¥ç¬¬ä¸ä¸ª App ç default Taskï¼`chat.conversation` |
|---|
| | | - **ä¹åå
³æºåå¼** â é»è®¤ä»æ¯è¿ä»½ chat pendingï¼ç»§ç»è天 Taskï¼ï¼ä¸æ¯ç©ºç½ Idle |
|---|
| | | - **åªæ** å°ç¨åºå³æ¶ãå°ç¨åºè®¡åã离线è¯é³åæ¢ æä¼æ¹å pending å¹¶ç¦»å¼ chat |
|---|
| | | - 计å Task è·å®è¥é
ç½®ãåå° defaultãâ pending åå `chat.conversation` |
|---|
| | | |
|---|
| | | `conversation` ä¹æ¯ä¸ä¸ªæ£å¼ Taskï¼ä¸æ¯ã没æ Taskããä¸ééåæ¢æ¶ï¼chat ä¸å¤è¯»ãç¿»è¯åä¸å¥ `start_app_task`ã |
|---|
| | | |
|---|
| | | ### 2.2 App 注å表ï¼Serverï¼é
置驱å¨ï¼ |
|---|
| | | |
|---|
| | | 建议æ件ï¼`xiaozhi-esp32-server/main/xiaozhi-server/data/apps.json` |
|---|
| | | ï¼æ `config.yaml` ä¸ `apps:`ãä¼å
ç¬ç« JSONï¼é¿å
å模åé
ç½®ç¼ å¨ä¸èµ·ãï¼ |
|---|
| | | |
|---|
| | | ```json |
|---|
| | | { |
|---|
| | | "default_app_id": "chat", |
|---|
| | | "apps": { |
|---|
| | | "chat": { |
|---|
| | | "name": "è天", |
|---|
| | | "mode": "CHAT", |
|---|
| | | "default_task": "conversation", |
|---|
| | | "tool_policy": "inherit_agent", |
|---|
| | | "system_prompt_mode": "agent_proactive", |
|---|
| | | "tasks": { |
|---|
| | | "conversation": { |
|---|
| | | "interaction_mode": "CHAT", |
|---|
| | | "skills": [], |
|---|
| | | "note": "åæ Xiaozhi Chat Runtimeï¼ç± Runtime 主å¨åèµ·ï¼ä¸å¦æ MCP" |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | "english_learning": { |
|---|
| | | "name": "è±è¯å¦ä¹ ", |
|---|
| | | "mode": "PLAY_AUDIO", |
|---|
| | | "tool_policy": "allowlist", |
|---|
| | | "tools": ["switch_app", "start_repeater", "self.mp3.play", "self.mp3.stop", "self.mp3.get_list", "self.power_off"], |
|---|
| | | "system_prompt_mode": "none", |
|---|
| | | "tasks": { |
|---|
| | | "repeater": { |
|---|
| | | "interaction_mode": "PLAY_AUDIO", |
|---|
| | | "skills": [ |
|---|
| | | { "name": "self.mp3.play", "args_from": ["name", "repeat", "power_off_after"] } |
|---|
| | | ] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | "english_translation": { |
|---|
| | | "name": "è±è¯ç¿»è¯", |
|---|
| | | "mode": "TRANSLATION", |
|---|
| | | "tool_policy": "allowlist", |
|---|
| | | "tools": ["switch_app", "translate"], |
|---|
| | | "system_prompt_mode": "translator", |
|---|
| | | "tasks": { |
|---|
| | | "live_translate": { |
|---|
| | | "interaction_mode": "TRANSLATION", |
|---|
| | | "skills": [ |
|---|
| | | { "name": "translate", "side": "server_mcp" } |
|---|
| | | ] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | - `inherit_agent`ï¼æ²¿ç¨å½å Agent å·²ç»å®çæ件 + è®¾å¤ MCPï¼**ç°ç½è¡ä¸º**ï¼ |
|---|
| | | - `allowlist`ï¼åªæååéçå·¥å
·äº¤ç» LLM / ç¼æå¨ |
|---|
| | | - åºä»¶ **ä¸è¯»å** æ¤æ件 |
|---|
| | | |
|---|
| | | ### 2.3 è¿æ¥æï¼Server å
åï¼ä¸å»ºè¡¨ï¼ |
|---|
| | | |
|---|
| | | å¨ `ConnectionHandler` å¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | active_app_id: str = "chat" |
|---|
| | | interaction_mode: str = "CHAT" # ä¸å½å App.mode åæ¥ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åæ¢ Appï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | switch_app(app_id) |
|---|
| | | â æ ¡éªæ³¨å表 |
|---|
| | | â åå½åå¨ä½ï¼å¤è¯»å MCP self.mp3.stopï¼ |
|---|
| | | â active_app_id / interaction_mode æ´æ° |
|---|
| | | â å·æ°å·¥å
·å表ï¼invalidate ToolManager cacheï¼ |
|---|
| | | â æ system_prompt_mode åæ¢æç¤ºè¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | `switch_app` å¨ **chat** ä¸å App ç½ååä¸é½å¯ç¨ï¼ä»¥ä¾¿ãéåºç¿»è¯ / éåºå¤è¯»ï¼åå°è天ãã |
|---|
| | | |
|---|
| | | ### 2.4 Mode 对å®æ¶é¾è·¯çåæµ |
|---|
| | | |
|---|
| | | æå
¥ç¹ï¼`startToChat` / `ConnectionHandler.chat` **ä¹å**ï¼`receiveAudioHandle.py`ã`connection.py`ï¼ã |
|---|
| | | |
|---|
| | | | mode | ASR | LLM chat() | TTS | 设å¤ææ¾ | |
|---|
| | | |------|-----|------------|-----|----------| |
|---|
| | | | CHAT | æ¯ | æ¯ï¼ç°ç½ï¼ | æ¯ | å¦ï¼é¤éæä»¶ï¼ | |
|---|
| | | | PLAY_AUDIO | å¦ï¼å¿½ç¥ä¸è¡å½å¯¹è¯ï¼ | å¦ | ä»
å¯å¨æ¶ä¸å¥æ示ï¼å¯éï¼ | SD MP3 | |
|---|
| | | | TRANSLATION | æ¯ | **å¦** | æ¯ï¼ç¿»è¯ç»æï¼ | å¦ | |
|---|
| | | |
|---|
| | | PLAY_AUDIO æé´è¥æ¶å°**ç¨æ·å¤éè¯** / ç¨æ·ä¸»å¨ `listen`ï¼å¤ä½ `chat`ï¼èµ°ç°ç½ãä¸è¦å¨å¤è¯»ä¸æç¨æ·è¯´è¯éè¿ LLMã |
|---|
| | | **RTC é¹éå¤é**èµ°ééâ¡ï¼ä¸èµ°è¿æ¡å¤ä½ã |
|---|
| | | |
|---|
| | | ### 2.5 å·¥å
·è¿æ»¤ |
|---|
| | | |
|---|
| | | æå
¥ç¹ï¼`UnifiedToolHandler.get_functions()` / `ToolManager.get_all_tools()`ã |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | if tool_policy == inherit_agent: |
|---|
| | | ç°ç½å
¨é |
|---|
| | | else: |
|---|
| | | åªä¿ç allowlist ä¸åå¨çå·¥å
·ï¼å«è®¾å¤ MCP å sanitize åï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç¿»è¯ App ä¸ **ç¦æ¢** åºç° weather / play_music / 设å¤æç
§çã |
|---|
| | | è¿æ¯ãåªåç¿»è¯ç MCPãè½æç«çåæï¼ä¸æ¯ MCP èªå·±è°¦èï¼èæ¯ **App ç½ååæå«çå·¥å
·ææ**ã |
|---|
| | | |
|---|
| | | ### 2.6 Client æ¡æ¶èè´£ï¼èï¼ |
|---|
| | | |
|---|
| | | è®¾å¤ **ä¸å®ç°ä¸å¡ App**ï¼æ `EnglishLearningApp` ç±»ï¼ï¼ä½å¿
é¡»æ **è¿å
¥ééç½å
³**ï¼ |
|---|
| | | |
|---|
| | | 1. Chat Runtime åè®®ä¸çè·¯å¾ä¿æï¼hello/listen/ttsï¼ |
|---|
| | | 2. é
ç½æåä¸é»è®¤å¼æºè¿å
¥ `chat.conversation`ï¼å¹¶è®©å°æº **主å¨å¼å£**ï¼è§ §3.1ï¼ |
|---|
| | | 3. Skill å¯è¢« MCP è°ç¨ |
|---|
| | | 4. `self.mp3.play` æ¯æ `repeat` + `power_off_after`ï¼ä»
å½ Task é
ç½®äºè¯¥ Skillï¼ |
|---|
| | | 5. é³é¢äºæ¥ç»§ç»ç¨ `PauseSdCardMp3` |
|---|
| | | 6. 软å
³æºç»§ç»ç¨ `RequestPowerOffStandby` |
|---|
| | | 7. è²å® MQTT è½ä¸å `START_APP_TASK` 并æ¢å å½åä»»å¡ï¼Â§2.7 ééâ ï¼ |
|---|
| | | 8. å¼æºè§£æ pendingï¼æ æ´é«ä¼å
级åæ¢å¤ chat.conversationï¼é¹éé chat 计ååæ§è¡è®¡åï¼Â§2.1.1ï¼ |
|---|
| | | 9. 离线è¯é³å½ä»¤åè°è¿å
¥åæ¢å
¥å£ï¼ãæ们æ¥èå¤©å§ / èä¼å¤©å§ãæ å°å `chat.conversation` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### 2.7 ä¸ç§è¿å
¥ App / Task çæ¹å¼ï¼æ¡æ¶æ ¸å¿ï¼ |
|---|
| | | |
|---|
| | | å°æ¥è¿å
¥ãæ App çæ Taskãåªæä¸æ¡ééãPhase 1 å°±æ **ä¼å
级ãæ¢å ãè½½è·ãå
³æºå¤é** å®æ»ï¼é¿å
以åæ¯ä¸ª App èªå·±æ¢è®¾å¤ã |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç»ä¸å
¥å£ï¼ |
|---|
| | | start_app_task(app_id, task_id, params, source) |
|---|
| | | |
|---|
| | | source: |
|---|
| | | mqtt_immediate ä¼å
级 3 æé« |
|---|
| | | schedule ä¼å
级 2 |
|---|
| | | voice ä¼å
级 1 |
|---|
| | | default_chat ä¼å
级 0 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ¢å è§åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ°è¯·æ±.priority > å½å.priority â ä¸æå½å App/Taskï¼ç«å»åæ¢ |
|---|
| | | æ°è¯·æ±.priority = å½å.priority â åééåå°è¦çï¼å°ç¨åºè¿ç¹ä¸¤æ¬¡ï¼åè
çæï¼ |
|---|
| | | æ°è¯·æ±.priority < å½å.priority â æç»æ忽ç¥ï¼ä¾å¦å¤è¯»ç± MQTT æèµ·æ¶ï¼è¯é³åç¿»è¯æ æï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ãä¸æãç»ä¸å¨ä½ï¼è®¾å¤ä¾§ï¼ä¸ä¸å¡æ å
³ï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åæ¢å½åæ£å¨è·ç Skillï¼æãå¬ãTTSãæ¾ç¤ºä»»å¡çï¼ |
|---|
| | | â åæãæ° Task é
ç½®ãæ§è¡å
¶ skills[] |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åºä»¶ä»ç¶ **ä¸å** `if english`ï¼MQTT/è¯é³/é¹éåªå¸¦ `app_id` + `task_id` + JSON paramsã |
|---|
| | | **æ§è¡åªäº Skill ä¸ç±ééå³å®ï¼åªç±è¯¥ App/Task çé
ç½®å³å®ã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | #### æè½è§£æï¼ææééå
±ç¨ï¼å¿
é¡»æç¡®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | START_APP_TASK / start_app_task |
|---|
| | | â |
|---|
| | | â¼ |
|---|
| | | æ¥ App 注å表[app_id].tasks[task_id] |
|---|
| | | â |
|---|
| | | â¼ |
|---|
| | | 读å该 Task çï¼ |
|---|
| | | interaction_mode |
|---|
| | | skills[] â çæ£è¦è°ç¨çè½å |
|---|
| | | params schema |
|---|
| | | â |
|---|
| | | â¼ |
|---|
| | | æ skills[] ä¾æ¬¡ï¼ææ Stepï¼è°ç¨ |
|---|
| | | è®¾å¤ MCP Skillï¼å¦ self.mp3.* / self.power_off / displayï¼ |
|---|
| | | å/æ Server Skillï¼å¦ translate MCP / ASR / TTSï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¦ç¹ï¼ |
|---|
| | | |
|---|
| | | - **ééåªè´è´£ãä½æ¶ãç¨ä»ä¹ä¼å
级å¯å¨åªä¸ª Taskã**ï¼ä¸è´è´£ãè¿ä¸ª Task ææ¾ MP3ãã |
|---|
| | | - **`self.mp3.play` åªæ¯ `english_learning.repeater` è¿ä»½é
ç½®éç Skill**ï¼ä¸æ¯ææ App/Task çé»è®¤æè½ã |
|---|
| | | - ç¿»è¯ Task é
çæ¯ Server MCP `translate`ï¼å°æ¥è·è¯»ä¼é
`play` + `listen`ï¼å£ç®ä¼é
`speak` + `listen` + Server å¤é¢ãåä¸æ¡ MQTT `START_APP_TASK` é½è½å¯å¨ã |
|---|
| | | - `PLAY_MEDIA` æ¯è²å®åå²ä¸çãææ¬å°æ件ãæ令ï¼**ä¸æ¯ App åè®®**ãå³ä½¿æ Task ç Skill è¯ä¹ä¸ä¹æ¯ææ¾ï¼ä¹åºèµ°è¯¥ Task é
ç½®ç Skill åï¼Phase 1 å¤è¯»ç¨ `self.mp3.play`ï¼ï¼ä¸è¦è®©å°ç¨åºç´æ¥å `PLAY_MEDIA` åå
ãæå¼è±è¯ Appãã |
|---|
| | | |
|---|
| | | Phase 1 å·²æ Task ä¸ Skill 对ç
§ï¼ |
|---|
| | | |
|---|
| | | | App | Task | é
ç½®ä¸ç skillsï¼ç¤ºä¾ï¼ | |
|---|
| | | |-----|------|-------------------------| |
|---|
| | | | `chat` | `conversation`ï¼**default**ï¼ | ç°æ Chat Runtimeï¼å°æºä¸»å¨åèµ·ï¼æ 强å¶é¢å¤ MCP | |
|---|
| | | | `english_learning` | `repeater` | `self.mp3.play`ï¼+ å¯é `self.power_off`ï¼ | |
|---|
| | | | `english_translation` | `live_translate` | Server MCP `translate` + ç°æ TTS | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | #### ééâ å°ç¨åºå³æ¶å¼å§ï¼ä¼å
级æé«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææºå°ç¨åº |
|---|
| | | é App â é Task â ç«å³å¼å§ |
|---|
| | | â è²å® MQTT ä¸å START_APP_TASKï¼åªå¸¦ app_id / task_id / params / sourceï¼ |
|---|
| | | â æ 论å½åæ¯ chatãç¿»è¯è¿æ¯å«ç Taskï¼å
¨é¨ä¸æ |
|---|
| | | â start_app_task(...) |
|---|
| | | â æ该 Task é
ç½®ç skills[] æ§è¡ï¼å¤è¯»æä¼è°å° self.mp3.playï¼ç¿»è¯åè° translateï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **为ä»ä¹èµ°è²å® MQTT èä¸æ¯ Xiaozhi WSï¼** å°ç¨åºå¯è½å¨è®¾å¤ Idleã软å
³æºæ¢å¤åãæ对è¯ééæªæå¼æ¶å°±è¦è½ä¸åï¼è²å® `snlp/v1/{device_id}/cmd/ctl|task` 已常驻ãXiaozhi `type=mcp` / Server MCP ä»æ¯ **Skill è°ç¨é¢**ãMQTT ç `START_APP_TASK` æ¯ **Task å¯å¨é¢**ãäºè
ä¸è¦æ··æä¸æ¡ãææ¾æ令ãã |
|---|
| | | |
|---|
| | | å»ºè®®æ° actionï¼æå¨ç°æ `BoardMqtt_DispatchAction`ï¼ï¼ |
|---|
| | | |
|---|
| | | ```json |
|---|
| | | { |
|---|
| | | "header": { "device_id": "...", "req_id": "...", "timestamp": 0 }, |
|---|
| | | "action": "START_APP_TASK", |
|---|
| | | "payload": { |
|---|
| | | "app_id": "english_learning", |
|---|
| | | "task_id": "repeater", |
|---|
| | | "params": { |
|---|
| | | "name": "lesson001.mp3", |
|---|
| | | "repeat": 10, |
|---|
| | | "power_off_after": false |
|---|
| | | }, |
|---|
| | | "source": "mqtt_immediate" |
|---|
| | | } |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | `payload.params` çåæ®µç± **该 Task ç schema** å³å®ï¼ä¸æ¯å
¨å±åºå®ä¸º MP3 æ件åãå¤è¯» Task æéè¦ `name/repeat/power_off_after`ï¼ç¿»è¯ Task å¯ä»¥å ä¹æ paramsï¼è·è¯»ä»¥åä¼æ `resource_id` çã |
|---|
| | | |
|---|
| | | è®¾å¤ ack èµ°ç°æ `rpt/ack`ãè¥å½åæä½ä¼å
级任å¡ï¼å
ååææ° Task ç `skills[]` å¯å¨ã |
|---|
| | | |
|---|
| | | **Phase 1 ç¼ç ï¼** åè®® + æ¢å + è½å¯å¨æ¬é¶æ®µå·²æ taskï¼`repeater` / `translate` / å `chat`ï¼ãå°ç¨åº UI å¯ååï¼ç¨ MQTT 客æ·ç«¯èè°å³å¯ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | #### ééâ¡ å°ç¨åº Scheduleï¼ä¼å
级第äºï¼ââå
³æºèªå¨æ§è¡ç主ç®æ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å°ç¨åºè®¾ç½®è®¡å |
|---|
| | | ä¾å¦ï¼æ¯å¤© 07:00 è±è¯å¦ä¹ / å¬-å¤è¯» æ è·è¯» |
|---|
| | | â Server ä¿å Scheduleï¼æå¨ï¼ |
|---|
| | | â åæ¥å°è®¾å¤ï¼RTC é¹é SET_ALARM + æ¬å° pending_task |
|---|
| | | â å°ç¹ï¼ |
|---|
| | | è¥è®¾å¤å¼çä¸å½åæ¯ chatï¼ææ´ä½ä¼å
级ï¼â åæ¢å¹¶å¼å§ |
|---|
| | | è¥è®¾å¤è½¯å
³æº / 硬件å
³æºåç± RTC å¤é â èç½åç«å»æ§è¡ pendingï¼ä¸è¿ chat |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ¯ **å°æºç¡¬ä»¶éå¯ / é¹éå¤éç主è¦äº§åç®æ **ï¼ä¸æ¯ä¸ºäºåå°å¾
æºè天ï¼èæ¯å°ç¹èªå¨å¬è¯»ãè·è¯»çã |
|---|
| | | |
|---|
| | | ç°æ代ç å·²ç»å
·å¤åæªè½åï¼**å°æªæ¥å° App/Task**ï¼ |
|---|
| | | |
|---|
| | | | å·²æ | ç¼ºå£ | |
|---|
| | | |------|------| |
|---|
| | | | MQTT `SET_ALARM` / `UNSET_ALARM` + DS3231 | é¹éåªå¤éï¼ä¸å¸¦ app/task | |
|---|
| | | | 软å
³æº `RequestPowerOffStandby`ï¼é¹é `schedule_power_off_restore_from_alarm` | `RestoreFromPowerOffStandby` åª `ResumePlaylistIfUnfinished` | |
|---|
| | | | 软å
³æºæé´å
³éå¤éè¯ä¸ç¦»çº¿å½ä»¤ | 符åãå
³æºååªæé¹éè½æèµ·æ¥ã | |
|---|
| | | |
|---|
| | | **Phase 1 å¿
须补ç设å¤è¯ä¹ï¼** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | NVS / SD ä¿å pending_task.json |
|---|
| | | { app_id, task_id, params, source: "schedule", alarm_id } |
|---|
| | | |
|---|
| | | é¹é触åä¸ä» power-off æ¢å¤ |
|---|
| | | â è¥ pending æ¯é chat ç计å Task â æ§è¡å®ï¼ç¦æ¢å
é²è |
|---|
| | | â è¥ pending æ¯ chat.conversation æ没æé chat 计å â è¿å
¥ chat.conversation 并主å¨å¼å£ |
|---|
| | | â 计å Task æåç»æåæé
ç½®åå chat.conversation |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¼æºä¸å·²å¨ chatï¼å°ç¹åçæ¢å ï¼priority 2 > 0ï¼ã |
|---|
| | | æ£å¨æ§è¡ MQTT å³æ¶ä»»å¡ï¼priority 3ï¼æ¶ï¼æ¬æ¬¡ Schedule 触åè·³è¿æ延åï¼ä¸æ¢ MQTTã |
|---|
| | | |
|---|
| | | Server 侧å®æ´ Plan 表ã家é¿æ¥å UIï¼**æ¬é¶æ®µä¸å»º Java 表**ï¼å¯ç¨ `apps.json` + 设å¤é¹é + pending æ件å
è·éã7 ç¹å¤éåèªå¨å¤è¯»ããè·è¯» Task æ¬ä½ä»éæ¬é¶æ®µï¼ä½ pending ç `task_id` å段å¿
é¡»é¢çã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | #### éé⢠离线è¯é³äº¤äºåæ¢ï¼ä¼å
级第ä¸ï¼ |
|---|
| | | |
|---|
| | | 硬件 **å·²ç»å®ç°ç¦»çº¿è¯é³è¯å«ï¼ESP-SR MultiNetï¼**ï¼ç®çå°±æ¯ä¸ºå°æ¥å App / Task / InteractionMode ååå¤ã**å½åçæ¬æ²¡ææ¥å°åæ¢ã** |
|---|
| | | |
|---|
| | | ç°ç¶ï¼Seningï¼ï¼ |
|---|
| | | |
|---|
| | | | ä½ç½® | è¡ä¸º | |
|---|
| | | |------|------| |
|---|
| | | | `afe_wake_word.cc` / `custom_wake_word.cc` | MultiNet æ£æµç¦»çº¿å½ä»¤ | |
|---|
| | | | å£ä»¤ | ä»
`åæ¢æ¨¡å¼`ï¼æ¼é³ `qie huan mo shi`ï¼ | |
|---|
| | | | `Application` â `Board_Handle_Offline_Command` | å¹é
ååª `ESP_LOGI`ï¼**æ åæ¢** | |
|---|
| | | | `CustomWakeWord` ç `index.json` commands | å·²æ `command / text / action` ç»æï¼å¯æ©å±ï¼å°æªç¨äº App | |
|---|
| | | |
|---|
| | | ç®æ ä½éªï¼å°æ¥ï¼ä¸éäºæ¬é¶æ®µä¸ä¸ª Appï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å°æºï¼ç°å¨æ们æ¥å个游æå§ï¼ |
|---|
| | | æ¯æ³åæ°åè诵ãæè¯æ¥é¾ï¼ |
|---|
| | | è¿æ¯ 10 以å
å åæ³å£ç®ï¼è¿æ¯ xxï¼ |
|---|
| | | ç¨æ·ï¼ï¼ç¦»çº¿å½ä»¤ / çå¥ï¼ |
|---|
| | | ç¨åºè½å¤å®æå¾ â start_app_task(å¯¹åº app, å¯¹åº task, source=voice) |
|---|
| | | æå¾ä¸æ¸
â åé®ä¸æ¬¡ / çå¨å½å App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¤å®ååï¼ |
|---|
| | | |
|---|
| | | - **ä¼å
æ¬å° MultiNet å£ä»¤**ï¼ä½å»¶è¿ãå¯å¨æªå¼ Xiaozhi 对è¯ééæ¶å·¥ä½ï¼ |
|---|
| | | - å£ä»¤è¡¨æãå½å交äºæ¥ãå è½½ï¼èåæ¥åªæå 个åéï¼ä¸è¦ä¸æ¬¡æ³¨åå
¨é¨ä¸å¡å¥ï¼ |
|---|
| | | - æ¬å°æ æ³å¤å®æ¶ï¼æå
许走äºç«¯ ASR+æå¾ï¼æ¤æ¶å·²å¨ chat 管线ï¼priority ä»ç® voiceï¼ |
|---|
| | | - è¯é³ **ä¸è½** ææééâ MQTT å³æ¶ä»»å¡ï¼ä¹ **ä¸è½** æææ£å¨æ§è¡çéé⡠计åä»»å¡ |
|---|
| | | |
|---|
| | | **Phase 1 ç¼ç ï¼æé©ï¼ä¸æ¯åå®æ´æ¸¸æå¹³å°ï¼ï¼** |
|---|
| | | |
|---|
| | | 1. `Board_Handle_Offline_Command` ä¸å空转ï¼è¿å
¥ `voice_entry`ï¼ä¾å¦æå¼ãé Appãèåæï¼ |
|---|
| | | 2. Phase 1 èåå¿
é¡»å
å«åå°è天ï¼ãæ们æ¥è天å§ããèä¼å¤©å§ãâ `chat.conversation`ï¼æ常ç¨ï¼ |
|---|
| | | 3. å
¶å®åéï¼`english_learning.repeater` / `english_translation`ï¼ãæ°åè诵 / æè¯æ¥é¾ / å£ç®ãä½æå¾æ§½é¢ç |
|---|
| | | 4. ä¸å¨åºä»¶åæ»æ¸¸æä¸å¡ï¼å£ä»¤ â `app_id+task_id` æ å°è¡¨ï¼JSON/Assetsï¼ï¼ä¸ MQTT è½½è·åä¸å¥ |
|---|
| | | |
|---|
| | | ä¸å¤éè¯åå·¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | é
ç½æå / æ®éå¼æº â chat.conversationï¼ä¸»å¨å¼å£ï¼ |
|---|
| | | ãæ们æ¥è天å§ãçå£ä»¤ â chat.conversation |
|---|
| | | å¤éè¯ â åå°æä¿æ chat.conversationï¼å¯æ¥ç主å¨ä¸å¥ï¼ |
|---|
| | | 离线ãåæ¢æ¨¡å¼ã â éé⢠éå
¶å® App/Task |
|---|
| | | RTC é¹é + é chat 计å â éé⡠计å Taskï¼ä¸è¦å
é²è |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 软å
³æºæé´ä¸¤è
é½å
³éï¼ç°æé»è¾ï¼ï¼åªæé¹éè½æ¢å¤ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ### 2.8 è¿å
¥éé vs Server App Runtime |
|---|
| | | |
|---|
| | | | éé | 设å¤æ¯å¦å¿
é¡»å¨çº¿ Xiaozhi WS | æå¨ | |
|---|
| | | |------|------------------------------|------| |
|---|
| | | | â MQTT å³æ¶ | ä¸å¿
å
å¼å¯¹è¯éé | å°ç¨åºè¯·æ±ï¼è®¾å¤ç«å³æ¢å æ§è¡ | |
|---|
| | | | â¡ Schedule | å°ç¹æ¶åèç½å³å¯ | Server å计åï¼è®¾å¤åé¹é + pending | |
|---|
| | | | ⢠è¯é³ | 纯å£ä»¤å¯ç¦»çº¿å¤å®ï¼æ¾æ¸
对è¯éè¦ WS | Server 注å表ï¼æ¬å°åªåå£ä»¤âid | |
|---|
| | | |
|---|
| | | ä¸ééæ±åå°åä¸å¥ `start_app_task` â **æ¥ Task é
ç½® â è° `skills[]`**ãMQTT ä¸è¦å为ãææ¾ãåç¬èµ° `PLAY_MEDIA` ä¸å¡åæ¯ï¼å¦ååä¸å¤è¯» Task ä¼åºç°ä¸¤å¥æ§è¡å¨ãå
¶å® Task ä¹ä¸æ ·ï¼ééä¸ç»å®æç§ Skillã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. App è§æ ¼ |
|---|
| | | |
|---|
| | | ### 3.1 chat · default Task `conversation` |
|---|
| | | |
|---|
| | | è¿æ¯è®¾å¤ä¸ç **第ä¸ä¸ª Appãé»è®¤ Task**ï¼å¯¹åºåæå°æº AI 硬件ç对è¯åè½ã |
|---|
| | | |
|---|
| | | **ä½æ¶è¿å
¥ï¼åªåºè¿äºï¼å¤å å¯å¨è§£æï¼ï¼** |
|---|
| | | |
|---|
| | | - é¦æ¬¡é
ç½æå |
|---|
| | | - ä¹åæ¯æ¬¡å¼æº / 软硬å
³æºåå¯å¨ï¼ä¸æ²¡ææ´é«ä¼å
级ç计å Task |
|---|
| | | - å£ä»¤æ对è¯ï¼ãæ们æ¥è天å§ããèä¼å¤©å§ããå¤éåè |
|---|
| | | - `switch_app("chat")` / MQTT `app_id=chat, task_id=conversation` |
|---|
| | | - å
¶å® Task æ£å¸¸ç»æä¸é
ç½®åå° default |
|---|
| | | |
|---|
| | | **ä½æ¶ä¸è¿ chatï¼** å°ç¨åºå³æ¶å¼å§äºå«ç Taskï¼é¹éå¤éç pending æ¯å¤è¯»/ç¿»è¯çé chat 计åã |
|---|
| | | |
|---|
| | | **åè®®ä¸ç¶ææºï¼** Listening â Speakingãhello/listen/ttsãAgent æ件 **ä¿æ**ãæ¡æ¶åªæè¿æ¡é¾è·¯ç»è®°ä¸º `chat.conversation`ã |
|---|
| | | |
|---|
| | | **ä¸åç¨åºçå·®å¼ï¼æ¬é¶æ®µè¦åï¼ï¼å°æºæ´å¤å°åèµ·ï¼èä¸æ¯å®å
¨è¢«å¨çç¨æ·é®ã** |
|---|
| | | |
|---|
| | | åè¡ä¸ºï¼é
ç½å Idle â çå¤é â çç¨æ·è¯´è¯ â åçã |
|---|
| | | ç®æ è¡ä¸ºï¼è¿å
¥ `conversation` åï¼å°æº **å
å¼å£** é请è天ï¼å¹¶å¨å·åºæ¶å¯ä»¥ååèµ·ä¸è½®ï¼èä¸æ¯ä¸ç´æ²é»ã |
|---|
| | | |
|---|
| | | Phase 1 主å¨åèµ·çæå°å®ç°ï¼ä¸å¦é Appï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è¿å
¥ chat.conversation |
|---|
| | | â æå¼é³é¢ééï¼è¥å°æªæå¼ï¼ |
|---|
| | | â TTS 主å¨å¥ï¼é
ç½®æè§è²æ示驱å¨ï¼ |
|---|
| | | ä¾ï¼ãæ们æ¥è天å§ãããæ³èç¹ä»ä¹åï¼ã |
|---|
| | | â è¿å
¥ Listeningï¼çç¨æ· |
|---|
| | | â ä¹åæç°ç½ ASR â LLM â TTS |
|---|
| | | â è¥é¿æ¶é´æ ç¨æ·è¯´è¯ï¼å¯å主å¨ä¸å¥ï¼æ¬¡æ°/é´éå¯é
ï¼é¿å
æ»å¾ªç¯å¬ä¿ï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ç°è½ç¹å»ºè®®ï¼ |
|---|
| | | |
|---|
| | | - Serverï¼`system_prompt_mode = agent_proactive`ï¼é¼å±å
é®ãçå¥åèµ·ï¼ï¼å¯éè¿æ¥å»ºç«åä¸åä¸å¥ `tts` èä¸çç¨æ· STT |
|---|
| | | - Clientï¼é
ç½æå / é»è®¤å¼æºè§£æå° `conversation` åï¼ä¸è¦åªåå¨ Idle çæé®ï¼åºèµ°ç°æãå¼å§å¬ãè·¯å¾å¹¶å
许å
æ TTS |
|---|
| | | - å£ä»¤ãæ们æ¥è天å§ãä¸ä¸»å¨å¥å
±ç¨åä¸ Taskï¼ä¸è¦åæ第äºä¸ªè天 App |
|---|
| | | |
|---|
| | | **pendingï¼** å
³æºåè¥æ£å¨ chatï¼pending ä¿æ `chat.conversation`ãåå¼æºç»§ç»è¯¥ Taskï¼å¯å主å¨æ个æå¼ï¼ä¸å¿
å¤è¿°å
³æºåæåä¸å¥ï¼ä½è§è²è®°å¿ä»èµ°ç°æ Server memoryï¼ã |
|---|
| | | |
|---|
| | | **åå½çº¢çº¿ï¼** ç¨æ·ä¸æ¦å¼å§è¯´è¯ï¼ææãTTSãå·²ææ件ä¸ç°ç½ä¸è´ã主å¨å¼å£ **ä¸å¾** æ¹ Opus çè·¯å¾æ listen è¯ä¹ã |
|---|
| | | |
|---|
| | | ### 3.2 english_learning · å¬-å¤è¯»æº |
|---|
| | | |
|---|
| | | **ç¨æ·ä½éª** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è¿å
¥è±è¯å¦ä¹ / å¼å§å¤è¯» |
|---|
| | | â ææ¾ SD å¡ä¸æå® MP3 |
|---|
| | | â å®æ´ææ¾ N 次ï¼æ件已å¨æ¬å°ï¼ä¸ä¸è½½ï¼ |
|---|
| | | â è¥ power_off_after=trueï¼è½¯å
³æºï¼ç°æ power-off standbyï¼ |
|---|
| | | â å¦ååå¨ Idleï¼å¤éåå° chat |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **åæ°ï¼Server ä¸åï¼è®¾å¤æ§è¡ï¼** |
|---|
| | | |
|---|
| | | | åæ° | 说æ | |
|---|
| | | |------|------| |
|---|
| | | | `name` | ä¸ `self.mp3.get_list` ç `files[].name` å®å
¨ä¸è´ | |
|---|
| | | | `repeat` | æ£æ´æ° Nï¼æ´é¦å¾ªç¯æ¬¡æ°ï¼ä¸æ¯æå¡å¨è¿è° N 次 playï¼ | |
|---|
| | | | `power_off_after` | boolï¼é»è®¤ false | |
|---|
| | | |
|---|
| | | **为ä»ä¹å¾ªç¯å¨è®¾å¤ä¸** |
|---|
| | | |
|---|
| | | `self.mp3.play` ä»æ¥å¨æ件å¼å§ææ¾åå³è¿å `{"accepted":true}`ï¼15s å
ååéå¤è°ç¨è¿ä¼ `duplicate_ignored`ãServer æ æ³å¯é ãæå®åè°ä¸ä¸æ¬¡ããæ `repeat` æ¾è¿ææ¾å¨å¾ªç¯ï¼ä¸ä¾èµæ°åè®®äºä»¶ï¼ä¹ä¸æãè±è¯ãåè¿åºä»¶ã |
|---|
| | | |
|---|
| | | **ç¼æï¼Serverï¼** |
|---|
| | | |
|---|
| | | æ件 `start_repeater(name, repeat, power_off_after)`ï¼ |
|---|
| | | |
|---|
| | | 1. `active_app_id = english_learning`ï¼`mode = PLAY_AUDIO` |
|---|
| | | 2. å¯é TTSï¼ãå¼å§å¤è¯» N 次ãï¼çãå¯å
³ï¼ |
|---|
| | | 3. Device MCPï¼`self.mp3.play` + æ°åæ° |
|---|
| | | 4. `Action.RECORD` æ `NONE`ï¼**ä¸å** `chat()` |
|---|
| | | |
|---|
| | | 缺æ件ï¼å
`self.mp3.get_list`ï¼æ²¡æå TTS æ示并ä¿æ chatã |
|---|
| | | |
|---|
| | | **软å
³æº** |
|---|
| | | |
|---|
| | | 设å¤å¨ç¬¬ N 次èªç¶ç»æï¼é stopãé TTS ææï¼åï¼è¥ `power_off_after`ï¼è°ç¨ç°æ `Application::RequestPowerOffStandby()`ï¼ä¸ `self.power_off` åä¸è·¯å¾ï¼ã |
|---|
| | | 被å¤éè¯/æé®/`self.mp3.stop` ææå **ä¸å
³æº**ï¼å¹¶åºè®© Server å¨ä¸æ¬¡ `listen` æ¶åå° chatã |
|---|
| | | |
|---|
| | | **ç®æ æ¿ï¼** `sening-xiaozhi-s3`ãå
¶ä»æ¿æ SD MP3 æ¶ï¼`start_repeater` è¿åæç¡®é误ï¼ä¸å¾æ¹ Chatã |
|---|
| | | |
|---|
| | | ### 3.3 english_translation |
|---|
| | | |
|---|
| | | **ç¨æ·ä½éª** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è¿å
¥è±è¯ç¿»è¯ |
|---|
| | | ç¨æ·è¯´ä¸æ â 设å¤æè±æ |
|---|
| | | ç¨æ·è¯´è±æ â 设å¤æä¸æ |
|---|
| | | ä¸é²èãä¸è®²ç¬è¯ãä¸è°å¤©æ° |
|---|
| | | ãéåºç¿»è¯ãâ chat |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **Server MCPï¼æ¬é¶æ®µæ ¸å¿æ°å¢ï¼** |
|---|
| | | |
|---|
| | | å¨ `data/.mcp_server_settings.json` 注åç¬ç« MCP Serverï¼ä¾å¦å `english-translate`ï¼ |
|---|
| | | |
|---|
| | | ```json |
|---|
| | | { |
|---|
| | | "mcpServers": { |
|---|
| | | "english-translate": { |
|---|
| | | "command": "python", |
|---|
| | | "args": ["mcp_servers/translate/server.py"] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | MCP å·¥å
·ï¼å»ºè®®åªæ´é²ä¸ä¸ªï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | translate(text, source_lang?, target_lang?) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è§åï¼ |
|---|
| | | |
|---|
| | | - æªæå®è¯è¨ï¼ä¸æâenï¼è±æâzhï¼å
¶å®è¯è¨é»è®¤è¯æä¸æ |
|---|
| | | - **åªè¿åè¯æ**ï¼æ 解éãæ æ¼é³è¯¾ãæ follow-up æé® |
|---|
| | | - å®ç°ï¼MCP è¿ç¨å
ç¨ç¬ç«ãç¿»è¯ç³»ç»æ示ãè°ç°æ LLM HTTPï¼è¯» xiaozhi é
ç½®ï¼ï¼**ä¸è¦**èµ° `ConnectionHandler.chat`ï¼é¿å
å¥å¨é²èï¼ |
|---|
| | | - 该 MCP ä¸æ³¨å play_music / 设å¤æ§å¶ |
|---|
| | | |
|---|
| | | **å®æ¶è·¯å¾** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ASR ææ¬ |
|---|
| | | â interaction_mode == TRANSLATION |
|---|
| | | â ç´æ¥ ServerMCPExecutor.execute("translate", {text}) |
|---|
| | | â TTS æ result |
|---|
| | | â ä¸è¿å
¥ chat()ãä¸åé¿ Dialogueï¼å¯éè®°ä¸æ¡ historyï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸è¦ï¼å
让é²è LLMãå³å®æ¯å¦ç¿»è¯ããå·¥å
·ç½åå + æè·¯ `chat()` åä¿é©ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 4. æ¶åº |
|---|
| | | |
|---|
| | | ### 4.1 å¤è¯» |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¨æ·ï¼å¨ chatï¼ï¼ãå¤è¯» lesson001 ä¸éç¶åå
³æºã |
|---|
| | | â LLM function call start_repeater |
|---|
| | | â Server: app=english_learning, mode=PLAY_AUDIO |
|---|
| | | â MCP self.mp3.play(name=lesson001.mp3, repeat=3, power_off_after=true) |
|---|
| | | â 设å¤æ¬å°ææ¾ 3 次 |
|---|
| | | â RequestPowerOffStandby |
|---|
| | | |
|---|
| | | ç¨æ·å¤é |
|---|
| | | â listen/detect |
|---|
| | | â Server å¤ä½ app=chat, mode=CHAT |
|---|
| | | â ç°ç½å¯¹è¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.2 ç¿»è¯ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¨æ·ï¼ãæå¼ç¿»è¯ã |
|---|
| | | â switch_app(english_translation) |
|---|
| | | â å·¥å
·å表ä»
switch_app + translate |
|---|
| | | ç¨æ·è¯´è¯ â ASR |
|---|
| | | â translate MCP â TTS |
|---|
| | | ç¨æ·ï¼ãéåºç¿»è¯ã |
|---|
| | | â switch_app(chat) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.3 Defaultï¼é
ç½ä¸æ¯æ¬¡å¼æº |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | é¦æ¬¡é
ç½æå |
|---|
| | | â pending = chat.conversation |
|---|
| | | â start_app_task(chat, conversation, source=default_chat) |
|---|
| | | â å°æºä¸»å¨ï¼ãæ们æ¥è天å§ã |
|---|
| | | â Listening |
|---|
| | | |
|---|
| | | ä¹åå
³æºåå¯å¨ï¼æ å°æçé chat 计åï¼ |
|---|
| | | â 读å pending = chat.conversation |
|---|
| | | â å次è¿å
¥è¯¥ Taskï¼å°æºå¯å主å¨æå¼ |
|---|
| | | â ä¸æ¯ç©ºç½ Idle å¹²çé®è¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.3b ç¨æ·è¦è天 |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 离线æ对è¯ï¼ãæ们æ¥èå¤©å§ / èä¼å¤©å§ã |
|---|
| | | â start_app_task(chat, conversation, source=voice) |
|---|
| | | â ææå½åæ´ä½ä¼å
级ï¼è¥æ£å¨ MQTT å³æ¶ä»»å¡åä¸æ¢ |
|---|
| | | â å°æºä¸»å¨æ¥è¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.4 å°ç¨åºå³æ¶å¼å§ï¼ééâ ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | MQTT START_APP_TASK app=english_learning task=repeater |
|---|
| | | â è¥æ£å¨ç¿»è¯æ chat æè¯é³èåï¼å
¨é¨ä¸æ |
|---|
| | | â æ SD å¤è¯» |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.5 计åä»»å¡ + å
³æºå¤éï¼ééâ¡ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | SET_ALARM 07:00 + pending_task=repeater |
|---|
| | | â å¤é´è½¯å
³æºï¼power_offï¼ |
|---|
| | | â 07:00 DS3231 é¹é |
|---|
| | | â RestoreFromPowerOffStandby |
|---|
| | | â æ§è¡ pending_taskï¼ä¸è¦å
chatï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 4.6 离线è¯é³åæ¢ï¼ééâ¢ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¨æ·è¯´ãåæ¢æ¨¡å¼ãï¼ç°æ MultiNet å£ä»¤ï¼ |
|---|
| | | â Board_Handle_Offline_Command è¿å
¥ voice_entry |
|---|
| | | â ï¼Phase 1ï¼å¯åå¬ãæå¼ç¿»è¯ / å¼å§å¤è¯» / åå°è天ã |
|---|
| | | â start_app_task(source=voice) |
|---|
| | | â è¥å½åæ¯ MQTT å³æ¶ä»»å¡ï¼å¿½ç¥è¯é³åæ¢ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 5. Client æ¹å¨æ¸
å |
|---|
| | | |
|---|
| | | ä»åºï¼`xiaozhi-esp32-2.2.6` |
|---|
| | | ååï¼æå°æ¹å¨ï¼ä¸å¡åä¸ä¸åºä»¶ã |
|---|
| | | |
|---|
| | | | ID | ä»»å¡ | æ件 | 说æ | |
|---|
| | | |----|------|------|------| |
|---|
| | | | C1 | `self.mp3.play` å¢å `repeat`ï¼intï¼é»è®¤ 1ï¼ã`power_off_after`ï¼boolï¼é»è®¤ falseï¼ | `main/boards/sening-xiaozhi-s3/sening_xiaozhi_s3.cc` | å·¥å
·æè¿°åæ¥æ´æ°ï¼`repeat=1` å
¼å®¹æ§è°ç¨ | |
|---|
| | | | C2 | SD ææ¾å¨æ¯æå¾ªç¯ N 次 | `sening` ç `SDCard_Play_MP3*` å®ç°å¤ï¼`system.c` / ææ¾ä»»å¡ï¼ | èªç¶æå®æ计æ°ï¼`self.mp3.stop`ãTTS Pause ææååæ¶å©ä½æ¬¡æ°ä¸ä¸å
³æº | |
|---|
| | | | C3 | N 次å®æä¸ `power_off_after` â `RequestPowerOffStandby` | åä¸ + `application.cc` å·²ææ¥å£ | ç¦æ¢æ°åä¸å¥å
³æº | |
|---|
| | | | C4 | ä¿æ `PauseSdCardMp3` ä¸ Chat TTS äºæ¥ | `application.cc` | ä¸æ¹çè·¯å¾ï¼å¤è¯»ä¸ TTS ä»å¯æå MP3 | |
|---|
| | | | C5 | 确认 `self.mp3.get_status` è½åºå playing / 循ç¯å©ä½ï¼å¯éï¼ | `sening_xiaozhi_s3.cc` | Phase 1 å¯éå段 `repeat_left`ï¼éå¿
é¡» | |
|---|
| | | | C6 | **ç¦æ¢** æ°å¢ English App ç±»ãDeviceStateãXiaozhi `type=task` | â | Review 红线 | |
|---|
| | | | C7 | MQTT `START_APP_TASK` åå + æ¢å + **æ task é
ç½®è° skills[]** | `mqtt.c` ååï¼æ§è¡èµ°ç»ä¸ start_app_task | ä¸è¦å¨ MQTT éåæ» play MP3ï¼ack èµ° `rpt/ack` | |
|---|
| | | | C8 | pending æä¹
åï¼å¼æºè§£æ §2.1.1 | NVS/SD + `RestoreFromPowerOffStandby` | é»è®¤æ¢å¤ `chat.conversation`ï¼ä»
é chat 计åé¹éæä¸è¿é²è | |
|---|
| | | | C9 | 离线å½ä»¤åè°æ¹ä¸ºè¿å
¥ééâ¢ï¼ä¸ååªææ¥å¿ | `system.c` `Board_Handle_Offline_Command`ï¼å£ä»¤è¡¨å¯æ© | å
æ¯æåå°å·²æ App/Taskï¼æ¸¸æç±» task_id é¢ç | |
|---|
| | | |
|---|
| | | é Sening æ¿ï¼ä¸æ¹ SD/MQTT/RTCãMCP æ `self.mp3.play` æ¶ç± Server æ¥éã离线 MultiNet å¨å
¶å®æ¿è¥åå¨ï¼åæ ·åªåæé©ãä¸åè±è¯ä¸å¡ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 6. Server æ¹å¨æ¸
å |
|---|
| | | |
|---|
| | | ä»åºï¼`xiaozhi-esp32-server` |
|---|
| | | ç©çè¾¹çï¼åªå¨ **xiaozhi-server (Python)**ãmanager-api **ä¸æ¹è¡¨**ã |
|---|
| | | |
|---|
| | | ### 6.1 æ¡æ¶ |
|---|
| | | |
|---|
| | | | ID | ä»»å¡ | å»ºè®®è·¯å¾ | |
|---|
| | | |----|------|----------| |
|---|
| | | | S1 | App 注å表å è½½ | `core/app_runtime/registry.py` + `data/apps.json` | |
|---|
| | | | S2 | è¿æ¥æé»è®¤ `chat` / `conversation` / CHAT | `core/connection.py` | ä¸è®¾å¤ pending å¯¹é½ | |
|---|
| | | | S2b | è¿å
¥ conversation åä¸»å¨ TTS åèµ·ï¼æ们æ¥è天å§ï¼ | è¿æ¥æ¬¢è¿ / è§è² prompt | ä¸è¦çç¨æ·ç¬¬ä¸å¥æ说è¯ï¼å¯é
ç½®å¼å
³ | |
|---|
| | | | S3 | `switch_app` æ件 | `plugins_func/functions/switch_app.py` | |
|---|
| | | | S4 | å·¥å
·ç½ååè¿æ»¤ | `unified_tool_handler.py` æ `unified_tool_manager.py`ï¼åæ¢å `refresh_tools()` | |
|---|
| | | | S5 | `startToChat` / `chat` åæ mode åæµ | `receiveAudioHandle.py`ã`connection.py` | |
|---|
| | | | S6 | **ç¨æ·**å¤é / `listen.state=detect` å¤ä½ default app | `listenMessageHandler.py` | **ä¸è¦**å¨é¹é pending æ§è¡æé´å½æç¨æ·å¤éæ¸
ä»»å¡ | |
|---|
| | | | S14 | ç»ä¸ `start_app_task(app, task, params, source)` + ä¼å
级 | `core/app_runtime/` | MQTT/è¯é³/æ件é½èµ°è¿é | |
|---|
| | | | S15 | ééâ¢ï¼äºç«¯ä»
ä½å£ä»¤ä¸å¤æ¶çæ¾æ¸
ï¼é»è®¤ä¸è¦ç MQTT/Schedule | åä¸ | ä¸ç¦»çº¿å½ä»¤æ å°è¡¨å¯¹é½ | |
|---|
| | | |
|---|
| | | chat ä¸ S4 å¿
é¡»çä»·äºç°ç½ï¼`inherit_agent`ï¼ï¼å¦å default App éªæ¶å¤±è´¥ã |
|---|
| | | |
|---|
| | | ### 6.2 å¤è¯» |
|---|
| | | |
|---|
| | | | ID | ä»»å¡ | å»ºè®®è·¯å¾ | |
|---|
| | | |----|------|----------| |
|---|
| | | | S7 | `start_repeater` æ件 | `plugins_func/functions/start_repeater.py` | |
|---|
| | | | S8 | éè¿ DeviceMCPExecutor è° `self.mp3.play` | å¤ç¨ç°æ device MCPï¼ä¸è¦èµ° `play_music` | |
|---|
| | | | S9 | Agent æ件ç»å® | 管ç端æé
ç½®æ `switch_app`ã`start_repeater` å å°ç®æ Agentï¼chat éè¦è½å App | |
|---|
| | | |
|---|
| | | ### 6.3 ç¿»è¯ MCP |
|---|
| | | |
|---|
| | | | ID | ä»»å¡ | å»ºè®®è·¯å¾ | |
|---|
| | | |----|------|----------| |
|---|
| | | | S10 | ç¬ç« MCP Server è¿ç¨ï¼åªæ³¨å `translate` | `main/xiaozhi-server/mcp_servers/translate/server.py` | |
|---|
| | | | S11 | åå
¥ `data/.mcp_server_settings.json` | ç°æ `ServerMCPManager` ä¼å è½½ | |
|---|
| | | | S12 | TRANSLATION 模å¼ï¼ASR åç´æ¥ `execute_tool(translate)` å TTS | ä¸è¦å
`chat()` | |
|---|
| | | | S13 | translator ç³»ç»æ示ï¼ä»
switch_app æ¶ç» LLM ç¨ï¼ | çï¼è¯´æå½åæ¯ç¿»è¯æºãéåºéè°ç¨ switch_app | |
|---|
| | | |
|---|
| | | `translate` ç LLM è°ç¨åºä½¿ç¨ç¬ç«ç promptï¼æ¸©åº¦ä½ï¼ç¦æ¢ function callã |
|---|
| | | |
|---|
| | | ### 6.4 é
置示ä¾ï¼S11ï¼ |
|---|
| | | |
|---|
| | | ```json |
|---|
| | | "english-translate": { |
|---|
| | | "command": "python", |
|---|
| | | "args": ["mcp_servers/translate/server.py"] |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å·¥ä½ç®å½ä¸º `xiaozhi-server`ãä¾èµï¼ç°æ `mcp` Python å
ï¼Server å·²ç¨ï¼ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 7. åè®®ä¸æ¶æ¯ |
|---|
| | | |
|---|
| | | Phase 1 **ä¸æ°å¢** Xiaozhi WS `type=task`ãå°ç¨åº/计åèµ° **è²å® MQTT**ã |
|---|
| | | |
|---|
| | | | æ¹å | æ¶æ¯ | ç¨é | |
|---|
| | | |------|------|------| |
|---|
| | | | ä¸è¡ WS | ç°æ `type=mcp` `tools/call` | play / stop / power_off / 设å¤å表 | |
|---|
| | | | ä¸è¡ WS | ç°æ `type=tts` | ç¿»è¯ç»æãå¤è¯»æ示ãè¯é³èåæé® | |
|---|
| | | | ä¸è¡ WS | ç°æ `listen` / `mcp` result | ç¨æ·å¤éå¤ä½ãå·¥å
·è¿å | |
|---|
| | | | ä¸è¡ MQTT | **æ°** `START_APP_TASK` | ééâ å³æ¶å¼å§ï¼å¯å¸¦ `source=schedule` ä½è¡¥å | |
|---|
| | | | ä¸è¡ MQTT | ç°æ `SET_ALARM` | ééâ¡å°ç¹å¤é | |
|---|
| | | | ä¸è¡ MQTT | ç°æ `rpt/ack` | å¯å¨/失败åæ§ | |
|---|
| | | | â | ä¸æ°å¢ play.completed | ç±è®¾å¤å
é¨ repeat æ¶å | |
|---|
| | | |
|---|
| | | MCP play åæ°ååå
¼å®¹ï¼æ§å®¢æ·ç«¯æ `repeat` æ¶æ 1 次ãä¸å
³æºã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 8. æ°æ®åº / API |
|---|
| | | |
|---|
| | | | 项 | Phase 1 | |
|---|
| | | |----|---------| |
|---|
| | | | MySQL æ°è¡¨ | æ | |
|---|
| | | | `/task/instance` | æ | |
|---|
| | | | chat_history | ç¿»è¯å¯ç»§ç»æç°ç½ä¸æ¥ææ¬ï¼ä¸å¼ºå¶æ° audio å段 | |
|---|
| | | | Redis | ä¸å App ç¶æï¼è¿æ¥å
åå³å¯ï¼æ线å³å defaultï¼ | |
|---|
| | | |
|---|
| | | Xiaozhi WS æ线éè¿ = æ° `session_id` = é»è®¤ chatã |
|---|
| | | **ä¾å¤ï¼** 设å¤æ¬å° `pending_task`ï¼ééâ¡ï¼å¨é¹éå¤éåä»ç¶æ§è¡ï¼ä¸ä¾èµæ§ `session_id`ãä¸æ¢å¤ãåæªè¢«ç¨æ·ææçå¤è¯»ãã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 9. å®æ½é¡ºåºï¼å»ºè®® PR åçï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | P1-A æ¡æ¶éª¨æ¶ |
|---|
| | | S1âS5ãS14ãswitch_app |
|---|
| | | + è¿å
¥ééï¼ä¼å
级ãC7 æ¢å ãC9 离线å£ä»¤æé©ãC8 pending å½¢ç¶ |
|---|
| | | éªæ¶ï¼é»è®¤ä»æ¯å®æ´ Chatï¼switch_app å¾è¿ä¸ç ´åå¯¹è¯ |
|---|
| | | éªæ¶ï¼MQTT START_APP_TASK è½ææ chatï¼ç¦»çº¿ãåæ¢æ¨¡å¼ãä¸åæ¯ç©º log |
|---|
| | | |
|---|
| | | P1-B ç¿»è¯ App |
|---|
| | | S10âS13 |
|---|
| | | éªæ¶ï¼ç¿»è¯ App å
åªåºè¯æï¼æ 天æ°/é³ä¹å·¥å
·ï¼éåºå chat |
|---|
| | | |
|---|
| | | P1-C å¤è¯»æº |
|---|
| | | C1âC4 + S7âS9 |
|---|
| | | éªæ¶ï¼SD æ件æ N 次ï¼å¯é软å
³æºï¼ææä¸å
³æºï¼Chat TTS ä»è½æå MP3 |
|---|
| | | |
|---|
| | | P1-D åå½ |
|---|
| | | ä¸å App çé¿å¯¹è¯ãææãå·²ææ件ãOTAãé
ç½ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å
A å B å Cï¼ç¿»è¯ä¸ä¾èµ SDï¼å¯å¨é Sening ä¸å
æµæ¡æ¶ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 10. æµè¯è®¡å |
|---|
| | | |
|---|
| | | ### 10.1 chat defaultï¼`conversation`ï¼ |
|---|
| | | |
|---|
| | | - [ ] é¦æ¬¡é
ç½æååèªå¨è¿å
¥ chat.conversationï¼ä¸å°æº **å
说è¯** |
|---|
| | | - [ ] å
³æºåå¼ï¼æ å
¶å®è®¡åï¼ä»è¿å
¥ chat.conversationï¼ä¸æ¯çº¯ Idle |
|---|
| | | - [ ] ç¨æ·è¯´è¯åçææãTTSãåæ件ä¸ç°ç½ä¸è´ |
|---|
| | | - [ ] ãæ们æ¥èå¤©å§ / èä¼å¤©å§ãåå conversation |
|---|
| | | - [ ] ä»ä¸è°ç¨å
¶å®ééæ¶ï¼ä¸ä¼èªå·±è·³å°å¤è¯»/ç¿»è¯ |
|---|
| | | |
|---|
| | | ### 10.2 æ¡æ¶ |
|---|
| | | |
|---|
| | | - [ ] `switch_app("english_translation")` åå·¥å
·å表ä¸å« weather |
|---|
| | | - [ ] `switch_app("chat")` åå·¥å
·å表æ¢å¤ |
|---|
| | | - [ ] éæ³ `app_id` 被æç»å¹¶çå¨å½å App |
|---|
| | | - [ ] MQTT `START_APP_TASK` æ£å¨ç¿»è¯æ¶è½ä¸æ并æ¹å»å¤è¯»ï¼ééâ > â¢/chatï¼ |
|---|
| | | - [ ] å¤è¯»ç± MQTT æèµ·æ¶ï¼ç¦»çº¿ãåæ¢æ¨¡å¼ãä¸è½æå®åèµ° |
|---|
| | | - [ ] 软å
³æº + `SET_ALARM` + pending å¤è¯»ï¼é¹éå¤éåç´æ¥ææ¾ï¼ä¸å
è¿å
¥é²è |
|---|
| | | - [ ] ç¨æ·å¤éè¯ä»åå° chatï¼ä¸é¹éå¤éåºåï¼ |
|---|
| | | - [ ] ãåæ¢æ¨¡å¼ã触å `voice_entry`ï¼ä¸ååªææ¥å¿ï¼ |
|---|
| | | |
|---|
| | | ### 10.3 ç¿»è¯ |
|---|
| | | |
|---|
| | | - [ ] ä¸æå¥ â è±æ TTS |
|---|
| | | - [ ] è±æå¥ â ä¸æ TTS |
|---|
| | | - [ ] ä¸åçã讲个ç¬è¯ãç±»é²èï¼åºç¿»è¯è¯¥å¥ææ示说è¦è¯çå
容ï¼ç± translate prompt å®ä¸æ¡å¹¶åè¿æµè¯ï¼ |
|---|
| | | - [ ] ãéåºç¿»è¯ãåå° chatï¼ç¬è¯ç±»æ¢å¤å¯ç¨ |
|---|
| | | |
|---|
| | | ### 10.4 å¤è¯»ï¼Sening + å¡ä¸å·²æ mp3ï¼ |
|---|
| | | |
|---|
| | | - [ ] `get_list` ææ件 |
|---|
| | | - [ ] N=2 å®æ´å¬å®ä¸¤é |
|---|
| | | - [ ] `power_off_after=false` ç»æåå¯å¤éè¿ chat |
|---|
| | | - [ ] `power_off_after=true` ç»æåè¿ç°æ软å
³æºï¼å¤éè¯å
³éï¼ä¸ä»æ¥ `self.power_off` ä¸è´ï¼ |
|---|
| | | - [ ] ææ¾ä¸ `stop` æå¤éï¼å©ä½éæ°åæ¶ã**ä¸**å
³æº |
|---|
| | | - [ ] å¤è¯»ä¸è¥æ Chat TTSï¼MP3 pause/resume è¡ä¸ºä¸åå½ |
|---|
| | | |
|---|
| | | ### 10.5 è´ä¾ |
|---|
| | | |
|---|
| | | - [ ] æ `self.mp3.play` çæ¿è°ç¨ `start_repeater` â è¯é³é误ï¼Chat ä»å¯ç¨ |
|---|
| | | - [ ] ä¸åºç° `play_music` æè¯¾ç¨ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 11. éªæ¶æ åï¼Phase 1 Doneï¼ |
|---|
| | | |
|---|
| | | 1. **æ¡æ¶ï¼** 第ä¸ä¸ª App ç default Task æ¯ `chat.conversation`ï¼é
ç½åä¸é»è®¤å¼æºé½è¿å
¥å®ï¼åæ¶åªæä¸ä¸ª activeã |
|---|
| | | 2. **主å¨è天ï¼** è¿å
¥ conversation åå°æºå
åèµ·ï¼ä¸æ¯å®å
¨è¢«å¨çé®ãç¨æ·åºçåç对è¯é¾è·¯ä¸ç°ç½ä¸è´ã |
|---|
| | | 3. **ä¸ééï¼** åªæå°ç¨åºå³æ¶ã计åã离线åæ¢æä¼ç¦»å¼ chatï¼é¹éé chat pending æè·³è¿é²èã |
|---|
| | | 4. **å¤è¯» / ç¿»è¯ï¼** è§åéªæ¶ã |
|---|
| | | 5. **è¾¹çï¼** æ Java æ°è¡¨ãæ Xiaozhi `type=task`ãæ åºä»¶è±è¯ä¸å¡ç±»ãæ é³é¢çè·¯å¾éåã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 12. é£é©ä¸ææ¯åºï¼è®°å
¥ Phase 2ï¼æ¬é¶æ®µæ¥åï¼ |
|---|
| | | |
|---|
| | | | åº | 说æ | |
|---|
| | | |----|------| |
|---|
| | | | å¤è¯»ä»
Sening | å
¶å®æ¿éåç»æ play_audio æä¸ºæ ¸å¿ Skill | |
|---|
| | | | æ play.completed åè®® | Server ä¸ç¥éæå°ç¬¬å éï¼ç¶æ以设å¤ä¸ºå | |
|---|
| | | | App ç¶æä¸è½åº | WS æ线å³å chatï¼**pending_task å¿
é¡»å¨è®¾å¤æ¬å°**ï¼å¦åé¹éå¤é丢失 | |
|---|
| | | | ç¿»è¯ MCP åè°ä¸æ¬¡ LLM | æ¯ãæ¹ç³»ç»æ示硬è¯ãå¤ä¸è·³ï¼æ¢çæ¯å·¥å
·é离ä¸ãåªåç¿»è¯ã | |
|---|
| | | | åæ§å¶é¢ | å°ç¨åº/é¹éèµ° snlpï¼å¯¹è¯/æè½èµ° Xiaozhi WS+MCPãä¸å¨ Phase 1 å并 | |
|---|
| | | | 离线å½ä»¤è¯å¾å° | ç°ä»
ãåæ¢æ¨¡å¼ãï¼èåå£ä»¤ä¸æ¸¸æ Task åç»å Assets/å½ä»¤è¡¨ | |
|---|
| | | | å°ç¨åºæªå | ééâ â¡ç¨ MQTT/é¹éèè°ï¼UI åç½® | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 13. ç¼ç 约æï¼ç»å®æ½ Agentï¼ |
|---|
| | | |
|---|
| | | 1. å
读åæ¹ï¼`application.cc` JSON/MCPã`sening_xiaozhi_s3.cc` mp3/power_offã`connection.py`ã`receiveAudioHandle.py`ã`unified_tool_handler.py`ã`server_mcp/mcp_manager.py`ã |
|---|
| | | 2. æ¯ä¸ª PR å
è· Chat åå½ååä¸å¡ã |
|---|
| | | 3. ç¦æ¢ä¸º Demo æ°å¢ `EnglishRepeatManager` / `TranslationApp` åºä»¶ç±»ã |
|---|
| | | 4. ç¦æ¢ä¿®æ¹ `hello/listen/tts/stt` è¯ä¹ã |
|---|
| | | 5. `repeat` é»è®¤ 1ï¼ä¿è¯æ§ MCP è°ç¨è
å
¼å®¹ã |
|---|
| | | 6. è¿å
¥ééå¿
须带 `source` ä¸ä¼å
级ï¼ç¦æ¢ä¸ä¸ªå
¥å£ååä¸å¥ stop/playã |
|---|
| | | 7. æ®éå¼æº / é
ç½æåç¦æ¢åæãåª Idle çç¨æ·ãï¼é»è®¤ pending æ¯ `chat.conversation` 并å
许å
TTSã |
|---|
| | | 8. ä»
å½é¹é pending 为 **é chat 计å Task** æ¶ï¼ç¦æ¢å
èµ°é²èã |
|---|
| | | |
|---|
| | | å®æ½ä» **P1-A** å¼å§ãæªå®æ Aï¼å«è¿å
¥ééæé©ä¸é»è®¤ conversationï¼ä¹åä¸è¦æ¹ææ¾å¨å¾ªç¯ã |
|---|
| | |
|---|
| | | Schedule([this]() { |
|---|
| | | // Play the success sound to indicate the device is ready |
|---|
| | | audio_service_.PlaySound(Lang::Sounds::OGG_SUCCESS); |
|---|
| | | Board::GetInstance().OnDeviceReady(); |
|---|
| | | }); |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | |
|---|
| | | void Application::StopListening() { |
|---|
| | | xEventGroupSetBits(event_group_, MAIN_EVENT_STOP_LISTENING); |
|---|
| | | } |
|---|
| | | |
|---|
| | | void Application::SendListenDetect(const std::string& text, bool open_if_idle) { |
|---|
| | | Schedule([this, text, open_if_idle]() { |
|---|
| | | if (!protocol_ || text.empty()) { |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | if (protocol_->IsAudioChannelOpened()) { |
|---|
| | | protocol_->SendWakeWordDetected(text); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | if (open_if_idle) { |
|---|
| | | WakeWordInvoke(text); |
|---|
| | | } |
|---|
| | | }); |
|---|
| | | } |
|---|
| | | |
|---|
| | | void Application::HandleToggleChatEvent() { |
|---|
| | |
|---|
| | | SetDeviceState(kDeviceStateIdle); |
|---|
| | | } |
|---|
| | | audio_service_.EnableWakeWordDetection(true); |
|---|
| | | BoardMqtt_ResumePlaylistIfUnfinished(); |
|---|
| | | if (!Board::GetInstance().OnRestoredFromPowerOff(network_ready)) { |
|---|
| | | BoardMqtt_ResumePlaylistIfUnfinished(); |
|---|
| | | } |
|---|
| | | }); |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | * Sends MAIN_EVENT_STOP_LISTENING to be handled in Run() |
|---|
| | | */ |
|---|
| | | void StopListening(); |
|---|
| | | void SendListenDetect(const std::string& text, bool open_if_idle = true); |
|---|
| | | |
|---|
| | | void Reboot(); |
|---|
| | | void WakeWordInvoke(const std::string& wake_word); |
|---|
| | |
|---|
| | | virtual void SetPowerSaveLevel(PowerSaveLevel level) = 0; |
|---|
| | | virtual std::string GetBoardJson() = 0; |
|---|
| | | virtual std::string GetDeviceStatusJson() = 0; |
|---|
| | | virtual void OnDeviceReady() {} |
|---|
| | | virtual bool OnRestoredFromPowerOff(bool network_ready) { (void)network_ready; return false; } |
|---|
| | | }; |
|---|
| | | |
|---|
| | | #define DECLARE_BOARD(BOARD_CLASS_NAME) \ |
|---|
| New file |
| | |
|---|
| | | #include "app_task.h" |
|---|
| | | |
|---|
| | | #include "application.h" |
|---|
| | | #include "mqtt.h" |
|---|
| | | #include "system.h" |
|---|
| | | |
|---|
| | | #include <cJSON.h> |
|---|
| | | #include <esp_log.h> |
|---|
| | | #include <nvs.h> |
|---|
| | | #include <strings.h> |
|---|
| | | #include <string.h> |
|---|
| | | #include <stdio.h> |
|---|
| | | #include <stdlib.h> |
|---|
| | | |
|---|
| | | #define TAG "app_task" |
|---|
| | | #define APP_TASK_NVS_NS "app_task" |
|---|
| | | #define APP_TASK_NVS_KEY "pending" |
|---|
| | | #define APP_TASK_JSON_MAX 768 |
|---|
| | | #define APP_TASK_ID_MAX 32 |
|---|
| | | #define APP_TASK_SOURCE_MAX 24 |
|---|
| | | |
|---|
| | | static char s_current_app[APP_TASK_ID_MAX] = "chat"; |
|---|
| | | static char s_current_task[APP_TASK_ID_MAX] = "conversation"; |
|---|
| | | static char s_current_source[APP_TASK_SOURCE_MAX] = "default_chat"; |
|---|
| | | static int s_current_priority = 0; |
|---|
| | | |
|---|
| | | typedef struct { |
|---|
| | | const char *app_id; |
|---|
| | | const char *task_id; |
|---|
| | | const char *skills[4]; |
|---|
| | | } app_task_skill_map_t; |
|---|
| | | |
|---|
| | | /* Configuration table: channels only pass app_id/task_id; skills are looked up here. */ |
|---|
| | | static const app_task_skill_map_t k_task_skills[] = { |
|---|
| | | {"chat", "conversation", {NULL, NULL, NULL, NULL}}, |
|---|
| | | {"english_learning", "repeater", {"self.mp3.play", NULL, NULL, NULL}}, |
|---|
| | | {"english_translation", "live_translate", {"translate", NULL, NULL, NULL}}, |
|---|
| | | }; |
|---|
| | | |
|---|
| | | int Board_AppTask_SourcePriority(const char *source) |
|---|
| | | { |
|---|
| | | if (!source) { |
|---|
| | | return 1; |
|---|
| | | } |
|---|
| | | if (strcmp(source, "mqtt_immediate") == 0) { |
|---|
| | | return 3; |
|---|
| | | } |
|---|
| | | if (strcmp(source, "schedule") == 0) { |
|---|
| | | return 2; |
|---|
| | | } |
|---|
| | | if (strcmp(source, "voice") == 0) { |
|---|
| | | return 1; |
|---|
| | | } |
|---|
| | | return 0; |
|---|
| | | } |
|---|
| | | |
|---|
| | | int Board_AppTask_CurrentPriority(void) |
|---|
| | | { |
|---|
| | | return s_current_priority; |
|---|
| | | } |
|---|
| | | |
|---|
| | | static const app_task_skill_map_t *find_task_spec(const char *app_id, const char *task_id) |
|---|
| | | { |
|---|
| | | if (!app_id) { |
|---|
| | | return NULL; |
|---|
| | | } |
|---|
| | | const char *tid = (task_id && task_id[0]) ? task_id : NULL; |
|---|
| | | for (size_t i = 0; i < sizeof(k_task_skills) / sizeof(k_task_skills[0]); ++i) { |
|---|
| | | if (strcmp(k_task_skills[i].app_id, app_id) != 0) { |
|---|
| | | continue; |
|---|
| | | } |
|---|
| | | if (!tid || strcmp(k_task_skills[i].task_id, tid) == 0) { |
|---|
| | | return &k_task_skills[i]; |
|---|
| | | } |
|---|
| | | } |
|---|
| | | return NULL; |
|---|
| | | } |
|---|
| | | |
|---|
| | | static bool is_chat_task(const char *app_id, const char *task_id) |
|---|
| | | { |
|---|
| | | return app_id && strcmp(app_id, "chat") == 0 && |
|---|
| | | (!task_id || task_id[0] == '\0' || strcmp(task_id, "conversation") == 0); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static void copy_id(char *dest, size_t dest_size, const char *src, const char *fallback) |
|---|
| | | { |
|---|
| | | const char *value = (src && src[0]) ? src : fallback; |
|---|
| | | strlcpy(dest, value, dest_size); |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t Board_SavePendingTask(const char *app_id, const char *task_id, const char *params_json, |
|---|
| | | const char *source, const char *alarm_id) |
|---|
| | | { |
|---|
| | | cJSON *root = cJSON_CreateObject(); |
|---|
| | | if (!root) { |
|---|
| | | return ESP_ERR_NO_MEM; |
|---|
| | | } |
|---|
| | | cJSON_AddStringToObject(root, "app_id", app_id && app_id[0] ? app_id : "chat"); |
|---|
| | | cJSON_AddStringToObject(root, "task_id", task_id && task_id[0] ? task_id : "conversation"); |
|---|
| | | cJSON_AddStringToObject(root, "source", source && source[0] ? source : "default_chat"); |
|---|
| | | cJSON_AddStringToObject(root, "alarm_id", alarm_id ? alarm_id : ""); |
|---|
| | | if (params_json && params_json[0]) { |
|---|
| | | cJSON *params = cJSON_Parse(params_json); |
|---|
| | | if (params) { |
|---|
| | | cJSON_AddItemToObject(root, "params", params); |
|---|
| | | } else { |
|---|
| | | cJSON_AddRawToObject(root, "params", params_json); |
|---|
| | | } |
|---|
| | | } else { |
|---|
| | | cJSON_AddItemToObject(root, "params", cJSON_CreateObject()); |
|---|
| | | } |
|---|
| | | |
|---|
| | | char *printed = cJSON_PrintUnformatted(root); |
|---|
| | | cJSON_Delete(root); |
|---|
| | | if (!printed) { |
|---|
| | | return ESP_ERR_NO_MEM; |
|---|
| | | } |
|---|
| | | |
|---|
| | | nvs_handle_t handle; |
|---|
| | | esp_err_t err = nvs_open(APP_TASK_NVS_NS, NVS_READWRITE, &handle); |
|---|
| | | if (err == ESP_OK) { |
|---|
| | | err = nvs_set_str(handle, APP_TASK_NVS_KEY, printed); |
|---|
| | | if (err == ESP_OK) { |
|---|
| | | err = nvs_commit(handle); |
|---|
| | | } |
|---|
| | | nvs_close(handle); |
|---|
| | | } |
|---|
| | | free(printed); |
|---|
| | | return err; |
|---|
| | | } |
|---|
| | | |
|---|
| | | static bool load_pending_fields(char *app_id, size_t app_size, char *task_id, size_t task_size, |
|---|
| | | char *source, size_t source_size, char *params_json, size_t params_size) |
|---|
| | | { |
|---|
| | | nvs_handle_t handle; |
|---|
| | | if (nvs_open(APP_TASK_NVS_NS, NVS_READONLY, &handle) != ESP_OK) { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | char buf[APP_TASK_JSON_MAX]; |
|---|
| | | size_t length = sizeof(buf); |
|---|
| | | esp_err_t err = nvs_get_str(handle, APP_TASK_NVS_KEY, buf, &length); |
|---|
| | | nvs_close(handle); |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | |
|---|
| | | cJSON *root = cJSON_Parse(buf); |
|---|
| | | if (!root) { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | const cJSON *app = cJSON_GetObjectItem(root, "app_id"); |
|---|
| | | const cJSON *task = cJSON_GetObjectItem(root, "task_id"); |
|---|
| | | const cJSON *src = cJSON_GetObjectItem(root, "source"); |
|---|
| | | const cJSON *params = cJSON_GetObjectItem(root, "params"); |
|---|
| | | copy_id(app_id, app_size, cJSON_IsString(app) ? app->valuestring : NULL, "chat"); |
|---|
| | | copy_id(task_id, task_size, cJSON_IsString(task) ? task->valuestring : NULL, "conversation"); |
|---|
| | | copy_id(source, source_size, cJSON_IsString(src) ? src->valuestring : NULL, "default_chat"); |
|---|
| | | if (params_json && params_size > 0) { |
|---|
| | | params_json[0] = '\0'; |
|---|
| | | if (params) { |
|---|
| | | char *printed = cJSON_PrintUnformatted(params); |
|---|
| | | if (printed) { |
|---|
| | | strlcpy(params_json, printed, params_size); |
|---|
| | | free(printed); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | cJSON_Delete(root); |
|---|
| | | return app_id[0] != '\0'; |
|---|
| | | } |
|---|
| | | |
|---|
| | | static void preempt_current_skills(void) |
|---|
| | | { |
|---|
| | | SDCard_Stop_Play(); |
|---|
| | | BoardMqtt_StopPlaylistPlayback(); |
|---|
| | | Application::GetInstance().Schedule([]() { |
|---|
| | | auto state = Application::GetInstance().GetDeviceState(); |
|---|
| | | if (state == kDeviceStateSpeaking) { |
|---|
| | | Application::GetInstance().AbortSpeaking(kAbortReasonNone); |
|---|
| | | } |
|---|
| | | }); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static void notify_server(const char *app_id, const char *task_id, const char *source, bool open_if_idle) |
|---|
| | | { |
|---|
| | | char detect[160]; |
|---|
| | | snprintf(detect, sizeof(detect), "[app_task]%s|%s|%s", |
|---|
| | | app_id && app_id[0] ? app_id : "chat", |
|---|
| | | task_id && task_id[0] ? task_id : "conversation", |
|---|
| | | source && source[0] ? source : "default_chat"); |
|---|
| | | Application::GetInstance().SendListenDetect(detect, open_if_idle); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static void dispatch_skill(const char *skill, const char *app_id, const char *task_id, |
|---|
| | | const char *params_json, const char *source) |
|---|
| | | { |
|---|
| | | if (!skill || skill[0] == '\0') { |
|---|
| | | notify_server(app_id, task_id, source, true); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | if (strcmp(skill, "self.mp3.play") == 0) { |
|---|
| | | char name[SDCARD_PATH_MAX_LEN] = {0}; |
|---|
| | | float speed = 1.0f; |
|---|
| | | int repeat = 1; |
|---|
| | | bool power_off_after = false; |
|---|
| | | if (params_json && params_json[0]) { |
|---|
| | | cJSON *params = cJSON_Parse(params_json); |
|---|
| | | if (params) { |
|---|
| | | const cJSON *name_item = cJSON_GetObjectItem(params, "name"); |
|---|
| | | const cJSON *speed_item = cJSON_GetObjectItem(params, "speed"); |
|---|
| | | const cJSON *repeat_item = cJSON_GetObjectItem(params, "repeat"); |
|---|
| | | const cJSON *power_item = cJSON_GetObjectItem(params, "power_off_after"); |
|---|
| | | if (cJSON_IsString(name_item) && name_item->valuestring) { |
|---|
| | | strlcpy(name, name_item->valuestring, sizeof(name)); |
|---|
| | | } |
|---|
| | | if (cJSON_IsNumber(speed_item)) { |
|---|
| | | speed = static_cast<float>(speed_item->valuedouble); |
|---|
| | | } |
|---|
| | | if (cJSON_IsNumber(repeat_item)) { |
|---|
| | | repeat = repeat_item->valueint; |
|---|
| | | } |
|---|
| | | if (cJSON_IsBool(power_item)) { |
|---|
| | | power_off_after = cJSON_IsTrue(power_item); |
|---|
| | | } else if (cJSON_IsNumber(power_item)) { |
|---|
| | | power_off_after = power_item->valueint != 0; |
|---|
| | | } |
|---|
| | | cJSON_Delete(params); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | if (name[0] == '\0') { |
|---|
| | | ESP_LOGW(TAG, "self.mp3.play missing name"); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | if (repeat < 1) { |
|---|
| | | repeat = 1; |
|---|
| | | } |
|---|
| | | esp_err_t err = SDCard_Play_MP3_With_Speed_Repeat(name, speed, repeat, power_off_after); |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | ESP_LOGW(TAG, "self.mp3.play failed: %s", esp_err_to_name(err)); |
|---|
| | | Board_OnSkillPlaybackFinished(true, false); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | notify_server(app_id, task_id, source, false); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | if (strcmp(skill, "translate") == 0) { |
|---|
| | | notify_server(app_id, task_id, source, true); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | ESP_LOGW(TAG, "unsupported skill: %s", skill); |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t Board_StartAppTask(const char *app_id, const char *task_id, const char *params_json, const char *source) |
|---|
| | | { |
|---|
| | | if (!app_id || app_id[0] == '\0') { |
|---|
| | | return ESP_ERR_INVALID_ARG; |
|---|
| | | } |
|---|
| | | const char *src = (source && source[0]) ? source : "mqtt_immediate"; |
|---|
| | | const char *tid = (task_id && task_id[0]) ? task_id : "conversation"; |
|---|
| | | const app_task_skill_map_t *spec = find_task_spec(app_id, tid); |
|---|
| | | if (!spec) { |
|---|
| | | ESP_LOGW(TAG, "unknown app/task: %s/%s", app_id, tid); |
|---|
| | | return ESP_ERR_NOT_FOUND; |
|---|
| | | } |
|---|
| | | |
|---|
| | | int new_priority = Board_AppTask_SourcePriority(src); |
|---|
| | | if (new_priority < s_current_priority) { |
|---|
| | | ESP_LOGW(TAG, "reject %s/%s source=%s prio %d < current %d", |
|---|
| | | app_id, tid, src, new_priority, s_current_priority); |
|---|
| | | return ESP_ERR_INVALID_STATE; |
|---|
| | | } |
|---|
| | | |
|---|
| | | preempt_current_skills(); |
|---|
| | | copy_id(s_current_app, sizeof(s_current_app), app_id, "chat"); |
|---|
| | | copy_id(s_current_task, sizeof(s_current_task), tid, "conversation"); |
|---|
| | | copy_id(s_current_source, sizeof(s_current_source), src, "default_chat"); |
|---|
| | | s_current_priority = new_priority; |
|---|
| | | Board_SavePendingTask(app_id, tid, params_json, src, NULL); |
|---|
| | | |
|---|
| | | ESP_LOGI(TAG, "start app_task %s/%s source=%s", app_id, tid, src); |
|---|
| | | if (!spec->skills[0]) { |
|---|
| | | dispatch_skill(NULL, app_id, tid, params_json, src); |
|---|
| | | return ESP_OK; |
|---|
| | | } |
|---|
| | | for (int i = 0; i < 4 && spec->skills[i]; ++i) { |
|---|
| | | dispatch_skill(spec->skills[i], app_id, tid, params_json, src); |
|---|
| | | } |
|---|
| | | return ESP_OK; |
|---|
| | | } |
|---|
| | | |
|---|
| | | void Board_OnSkillPlaybackFinished(bool stopped, bool power_off_requested) |
|---|
| | | { |
|---|
| | | (void)power_off_requested; |
|---|
| | | ESP_LOGI(TAG, "skill playback %s", stopped ? "stopped" : "finished"); |
|---|
| | | copy_id(s_current_app, sizeof(s_current_app), "chat", "chat"); |
|---|
| | | copy_id(s_current_task, sizeof(s_current_task), "conversation", "conversation"); |
|---|
| | | copy_id(s_current_source, sizeof(s_current_source), "default_chat", "default_chat"); |
|---|
| | | s_current_priority = 0; |
|---|
| | | Board_SavePendingTask("chat", "conversation", "{}", "default_chat", NULL); |
|---|
| | | } |
|---|
| | | |
|---|
| | | void Board_ResolveBootOrRestore(bool from_alarm_wake) |
|---|
| | | { |
|---|
| | | char app_id[APP_TASK_ID_MAX] = "chat"; |
|---|
| | | char task_id[APP_TASK_ID_MAX] = "conversation"; |
|---|
| | | char source[APP_TASK_SOURCE_MAX] = "default_chat"; |
|---|
| | | char params[APP_TASK_JSON_MAX] = "{}"; |
|---|
| | | bool has_pending = load_pending_fields(app_id, sizeof(app_id), task_id, sizeof(task_id), |
|---|
| | | source, sizeof(source), params, sizeof(params)); |
|---|
| | | if (!has_pending) { |
|---|
| | | Board_SavePendingTask("chat", "conversation", "{}", "default_chat", NULL); |
|---|
| | | Board_StartAppTask("chat", "conversation", "{}", "default_chat"); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | |
|---|
| | | if (from_alarm_wake && strcmp(source, "schedule") == 0 && !is_chat_task(app_id, task_id)) { |
|---|
| | | ESP_LOGI(TAG, "alarm restore runs pending %s/%s", app_id, task_id); |
|---|
| | | Board_StartAppTask(app_id, task_id, params, "schedule"); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | |
|---|
| | | Board_StartAppTask("chat", "conversation", "{}", "default_chat"); |
|---|
| | | } |
|---|
| | | |
|---|
| | | void Board_EnterVoiceEntry(void) |
|---|
| | | { |
|---|
| | | if (s_current_priority >= 2) { |
|---|
| | | ESP_LOGI(TAG, "ignore voice_entry, current priority=%d", s_current_priority); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | | Application::GetInstance().SendListenDetect("[voice_entry]", true); |
|---|
| | | } |
|---|
| | | |
|---|
| | | bool Board_Handle_Voice_Command(const char *command) |
|---|
| | | { |
|---|
| | | if (!command || command[0] == '\0') { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | if (strcmp(command, "åæ¢æ¨¡å¼") == 0 || strcmp(command, "qie huan mo shi") == 0) { |
|---|
| | | Board_EnterVoiceEntry(); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | if (strstr(command, "æ们æ¥è天å§") || strstr(command, "èä¼å¤©å§")) { |
|---|
| | | if (s_current_priority >= 2) { |
|---|
| | | ESP_LOGI(TAG, "ignore voice chat switch, priority=%d", s_current_priority); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | Board_StartAppTask("chat", "conversation", "{}", "voice"); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | if (strstr(command, "æå¼ç¿»è¯") || strstr(command, "å¼å§ç¿»è¯")) { |
|---|
| | | if (s_current_priority >= 2) { |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | Board_StartAppTask("english_translation", "live_translate", "{}", "voice"); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | if (strstr(command, "å¼å§å¤è¯»") || strstr(command, "æå¼å¤è¯»")) { |
|---|
| | | if (s_current_priority >= 2) { |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | Board_EnterVoiceEntry(); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | if (strstr(command, "æ°åè诵") || strstr(command, "æè¯æ¥é¾") || strstr(command, "å£ç®")) { |
|---|
| | | ESP_LOGI(TAG, "reserved voice slot: %s", command); |
|---|
| | | Board_EnterVoiceEntry(); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | return false; |
|---|
| | | } |
|---|
| New file |
| | |
|---|
| | | #ifndef APP_TASK_H |
|---|
| | | #define APP_TASK_H |
|---|
| | | |
|---|
| | | #include "esp_err.h" |
|---|
| | | #include <stdbool.h> |
|---|
| | | |
|---|
| | | #ifdef __cplusplus |
|---|
| | | extern "C" { |
|---|
| | | #endif |
|---|
| | | |
|---|
| | | int Board_AppTask_SourcePriority(const char *source); |
|---|
| | | esp_err_t Board_StartAppTask(const char *app_id, const char *task_id, const char *params_json, const char *source); |
|---|
| | | esp_err_t Board_SavePendingTask(const char *app_id, const char *task_id, const char *params_json, const char *source, const char *alarm_id); |
|---|
| | | void Board_ResolveBootOrRestore(bool from_alarm_wake); |
|---|
| | | void Board_EnterVoiceEntry(void); |
|---|
| | | bool Board_Handle_Voice_Command(const char *command); |
|---|
| | | void Board_OnSkillPlaybackFinished(bool stopped, bool power_off_requested); |
|---|
| | | int Board_AppTask_CurrentPriority(void); |
|---|
| | | |
|---|
| | | #ifdef __cplusplus |
|---|
| | | } |
|---|
| | | #endif |
|---|
| | | |
|---|
| | | #endif |
|---|
| | |
|---|
| | | #include "system.h" |
|---|
| | | #include "app_task.h" |
|---|
| | | |
|---|
| | | #include "audio_codec.h" |
|---|
| | | #include "board.h" |
|---|
| | | #include "application.h" |
|---|
| | | |
|---|
| | | #include <esp_log.h> |
|---|
| | | #include <freertos/FreeRTOS.h> |
|---|
| | |
|---|
| | | typedef struct { |
|---|
| | | char path[SDCARD_PATH_MAX_LEN]; |
|---|
| | | float speed; |
|---|
| | | int repeat; |
|---|
| | | bool power_off_after; |
|---|
| | | } mp3_player_task_arg_t; |
|---|
| | | |
|---|
| | | static TaskHandle_t mp3_player_task_handle; |
|---|
| | | static volatile bool mp3_stop_requested; |
|---|
| | | static volatile bool mp3_pause_requested; |
|---|
| | | static volatile int mp3_repeat_left; |
|---|
| | | |
|---|
| | | static std::vector<int16_t> convert_channels(const int16_t *input, size_t frame_count, uint8_t src_channels, uint8_t dst_channels) |
|---|
| | | { |
|---|
| | |
|---|
| | | char path[SDCARD_PATH_MAX_LEN]; |
|---|
| | | strlcpy(path, task_arg->path, sizeof(path)); |
|---|
| | | float speed = task_arg->speed; |
|---|
| | | int repeat = task_arg->repeat < 1 ? 1 : task_arg->repeat; |
|---|
| | | bool power_off_after = task_arg->power_off_after; |
|---|
| | | free(task_arg); |
|---|
| | | |
|---|
| | | FILE *fp = fopen(path, "rb"); |
|---|
| | |
|---|
| | | ESP_LOGE(TAG, "open MP3 failed: %s", path); |
|---|
| | | mp3_player_task_handle = nullptr; |
|---|
| | | mp3_stop_requested = false; |
|---|
| | | mp3_repeat_left = 0; |
|---|
| | | vTaskDelete(nullptr); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | |
|---|
| | | fclose(fp); |
|---|
| | | mp3_player_task_handle = nullptr; |
|---|
| | | mp3_stop_requested = false; |
|---|
| | | mp3_repeat_left = 0; |
|---|
| | | vTaskDelete(nullptr); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | |
|---|
| | | fclose(fp); |
|---|
| | | mp3_player_task_handle = nullptr; |
|---|
| | | mp3_stop_requested = false; |
|---|
| | | mp3_repeat_left = 0; |
|---|
| | | vTaskDelete(nullptr); |
|---|
| | | return; |
|---|
| | | } |
|---|
| | |
|---|
| | | ESP_LOGE(TAG, "no memory for MP3 playback"); |
|---|
| | | ret = ESP_AUDIO_ERR_MEM_LACK; |
|---|
| | | } else { |
|---|
| | | ESP_LOGI(TAG, "play MP3: %s, speed: %.2fx", path, speed); |
|---|
| | | ESP_LOGI(TAG, "play MP3: %s, speed: %.2fx, repeat: %d, power_off_after: %d", path, speed, repeat, power_off_after ? 1 : 0); |
|---|
| | | |
|---|
| | | if (wait_if_paused()) { |
|---|
| | | codec->EnableOutput(true); |
|---|
| | | } |
|---|
| | | |
|---|
| | | for (int pass = 0; pass < repeat && ret == ESP_AUDIO_ERR_OK && !mp3_stop_requested; ++pass) { |
|---|
| | | mp3_repeat_left = repeat - pass; |
|---|
| | | if (pass > 0) { |
|---|
| | | if (fseek(fp, 0, SEEK_SET) != 0) { |
|---|
| | | ESP_LOGE(TAG, "rewind MP3 failed: %s", path); |
|---|
| | | break; |
|---|
| | | } |
|---|
| | | esp_audio_simple_dec_close(decoder); |
|---|
| | | decoder = nullptr; |
|---|
| | | ret = esp_audio_simple_dec_open(&dec_cfg, &decoder); |
|---|
| | | if (ret != ESP_AUDIO_ERR_OK || !decoder) { |
|---|
| | | ESP_LOGE(TAG, "reopen MP3 decoder failed: %d", ret); |
|---|
| | | break; |
|---|
| | | } |
|---|
| | | } |
|---|
| | | |
|---|
| | | while (ret == ESP_AUDIO_ERR_OK && !mp3_stop_requested) { |
|---|
| | | if (!wait_if_paused()) { |
|---|
| | |
|---|
| | | break; |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | |
|---|
| | | if (rate_cvt_handle) { |
|---|
| | |
|---|
| | | } |
|---|
| | | free(in_buf); |
|---|
| | | free(out_buf); |
|---|
| | | esp_audio_simple_dec_close(decoder); |
|---|
| | | if (decoder) { |
|---|
| | | esp_audio_simple_dec_close(decoder); |
|---|
| | | } |
|---|
| | | esp_audio_simple_dec_unregister_default(); |
|---|
| | | esp_audio_dec_unregister_default(); |
|---|
| | | fclose(fp); |
|---|
| | | |
|---|
| | | ESP_LOGI(TAG, "MP3 playback %s: %s", mp3_stop_requested ? "stopped" : "finished", path); |
|---|
| | | bool stopped = mp3_stop_requested; |
|---|
| | | bool should_power_off = power_off_after && !stopped; |
|---|
| | | mp3_player_task_handle = nullptr; |
|---|
| | | mp3_stop_requested = false; |
|---|
| | | mp3_pause_requested = false; |
|---|
| | | mp3_repeat_left = 0; |
|---|
| | | if (should_power_off) { |
|---|
| | | Application::GetInstance().RequestPowerOffStandby(); |
|---|
| | | } |
|---|
| | | Board_OnSkillPlaybackFinished(stopped, power_off_after); |
|---|
| | | vTaskDelete(nullptr); |
|---|
| | | } |
|---|
| | | |
|---|
| | | extern "C" esp_err_t SDCard_Play_MP3_File(const char *path) |
|---|
| | | { |
|---|
| | | return SDCard_Play_MP3_File_With_Speed(path, 1.0f); |
|---|
| | | return SDCard_Play_MP3_File_With_Speed_Repeat(path, 1.0f, 1, false); |
|---|
| | | } |
|---|
| | | |
|---|
| | | extern "C" esp_err_t SDCard_Play_MP3_File_With_Speed(const char *path, float speed) |
|---|
| | | { |
|---|
| | | return SDCard_Play_MP3_File_With_Speed_Repeat(path, speed, 1, false); |
|---|
| | | } |
|---|
| | | |
|---|
| | | extern "C" esp_err_t SDCard_Play_MP3_File_With_Speed_Repeat(const char *path, float speed, int repeat, bool power_off_after) |
|---|
| | | { |
|---|
| | | if (!path || path[0] == '\0' || !is_supported_speed(speed)) { |
|---|
| | | return ESP_ERR_INVALID_ARG; |
|---|
| | | } |
|---|
| | | if (repeat < 1) { |
|---|
| | | repeat = 1; |
|---|
| | | } |
|---|
| | | |
|---|
| | | bool start_paused = mp3_pause_requested; |
|---|
| | |
|---|
| | | } |
|---|
| | | strlcpy(arg->path, path, sizeof(arg->path)); |
|---|
| | | arg->speed = speed; |
|---|
| | | arg->repeat = repeat; |
|---|
| | | arg->power_off_after = power_off_after; |
|---|
| | | mp3_stop_requested = false; |
|---|
| | | mp3_pause_requested = start_paused; |
|---|
| | | mp3_repeat_left = repeat; |
|---|
| | | |
|---|
| | | BaseType_t ret = xTaskCreate(mp3_player_task, "mp3_player", MP3_PLAYER_STACK_SIZE, arg, MP3_PLAYER_PRIORITY, &mp3_player_task_handle); |
|---|
| | | if (ret != pdPASS) { |
|---|
| | | free(arg); |
|---|
| | | mp3_player_task_handle = nullptr; |
|---|
| | | mp3_repeat_left = 0; |
|---|
| | | return ESP_FAIL; |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | { |
|---|
| | | return mp3_player_task_handle != nullptr && mp3_pause_requested; |
|---|
| | | } |
|---|
| | | |
|---|
| | | extern "C" int SDCard_Get_Repeat_Left(void) |
|---|
| | | { |
|---|
| | | return mp3_repeat_left; |
|---|
| | | } |
|---|
| | |
|---|
| | | #include "freertos/task.h" |
|---|
| | | #include "mqtt_client.h" |
|---|
| | | #include "system.h" |
|---|
| | | #include "app_task.h" |
|---|
| | | |
|---|
| | | #define BOARD_MQTT_KEEPALIVE_SEC 120 |
|---|
| | | #define BOARD_MQTT_STATE_REPORT_INTERVAL_MS 5000 |
|---|
| | |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t err = DS2312_Set_Alarm_Id_Saved(id->valuestring, hour, minute, second, repeat_mask); |
|---|
| | | return err == ESP_OK ? BoardMqtt_ResultOk("alarm updated") : |
|---|
| | | BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, esp_err_to_name(err)); |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, esp_err_to_name(err)); |
|---|
| | | } |
|---|
| | | |
|---|
| | | cJSON *app_id = cJSON_GetObjectItem(payload, "app_id"); |
|---|
| | | cJSON *task_id = cJSON_GetObjectItem(payload, "task_id"); |
|---|
| | | if (cJSON_IsString(app_id) && app_id->valuestring && app_id->valuestring[0]) { |
|---|
| | | const char *tid = (cJSON_IsString(task_id) && task_id->valuestring) ? task_id->valuestring : ""; |
|---|
| | | cJSON *params = cJSON_GetObjectItem(payload, "params"); |
|---|
| | | char *params_json = params ? cJSON_PrintUnformatted(params) : NULL; |
|---|
| | | Board_SavePendingTask(app_id->valuestring, tid, params_json, "schedule", id->valuestring); |
|---|
| | | free(params_json); |
|---|
| | | } |
|---|
| | | return BoardMqtt_ResultOk("alarm updated"); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static board_mqtt_action_result_t BoardMqtt_HandleUnsetAlarm(cJSON *payload) |
|---|
| | |
|---|
| | | return BoardMqtt_ResultOk("manifest accepted"); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static board_mqtt_action_result_t BoardMqtt_HandleStartAppTask(cJSON *payload) |
|---|
| | | { |
|---|
| | | if (!payload) { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "missing payload"); |
|---|
| | | } |
|---|
| | | cJSON *app_id = cJSON_GetObjectItem(payload, "app_id"); |
|---|
| | | cJSON *task_id = cJSON_GetObjectItem(payload, "task_id"); |
|---|
| | | cJSON *source = cJSON_GetObjectItem(payload, "source"); |
|---|
| | | cJSON *params = cJSON_GetObjectItem(payload, "params"); |
|---|
| | | if (!cJSON_IsString(app_id) || !app_id->valuestring || app_id->valuestring[0] == '\0') { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "missing app_id"); |
|---|
| | | } |
|---|
| | | const char *tid = (cJSON_IsString(task_id) && task_id->valuestring) ? task_id->valuestring : ""; |
|---|
| | | const char *src = (cJSON_IsString(source) && source->valuestring) ? source->valuestring : "mqtt_immediate"; |
|---|
| | | char *params_json = params ? cJSON_PrintUnformatted(params) : NULL; |
|---|
| | | esp_err_t err = Board_StartAppTask(app_id->valuestring, tid, params_json, src); |
|---|
| | | free(params_json); |
|---|
| | | if (err == ESP_ERR_NOT_FOUND) { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_NOT_FOUND, "unknown app or task"); |
|---|
| | | } |
|---|
| | | if (err == ESP_ERR_INVALID_STATE) { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_BUSY, "preempted by higher priority task"); |
|---|
| | | } |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, esp_err_to_name(err)); |
|---|
| | | } |
|---|
| | | return BoardMqtt_ResultOk("app task started"); |
|---|
| | | } |
|---|
| | | |
|---|
| | | static board_mqtt_action_result_t BoardMqtt_DispatchAction(const char *req_id, const char *action, cJSON *payload) |
|---|
| | | { |
|---|
| | | if (strcmp(action, "SET_VOLUME") == 0) return BoardMqtt_HandleSetVolume(payload); |
|---|
| | |
|---|
| | | if (strcmp(action, "UPDATE_MANIFEST") == 0 || strcmp(action, "SYNC_COURSE_TASK") == 0) return BoardMqtt_HandleUpdateManifest(req_id, payload); |
|---|
| | | if (strcmp(action, "DOWNLOAD") == 0) return BoardMqtt_HandleDownload(req_id, payload); |
|---|
| | | if (strcmp(action, "OTA_UPDATE") == 0) return BoardMqtt_HandleOtaUpdate(req_id, payload); |
|---|
| | | if (strcmp(action, "START_APP_TASK") == 0) return BoardMqtt_HandleStartAppTask(payload); |
|---|
| | | return BoardMqtt_ResultFail(BOARD_MQTT_ERR_UNSUPPORTED, "unsupported action"); |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | #include "power_save_timer.h" |
|---|
| | | #include "adc_battery_monitor.h" |
|---|
| | | #include "wifi_manager.h" |
|---|
| | | #include "app_task.h" |
|---|
| | | |
|---|
| | | #include <esp_log.h> |
|---|
| | | #include <esp_efuse_table.h> |
|---|
| | |
|---|
| | | static float last_mp3_play_speed = 1.0f; |
|---|
| | | static int64_t last_mp3_play_time = 0; |
|---|
| | | mcp_server.AddTool("self.mp3.play", |
|---|
| | | "Play one MP3 file from the SD card. The name argument must exactly match a files[].name from self.mp3.get_list. Optional speed supports only 0.5, 0.75, 1.0, 1.25, 1.5, or 2.0; non-1.0 speeds change pitch and affect only SD card MP3 playback. For play, random play, switch song, or named-song requests, call this tool before telling the user playback has started. Do not call this tool repeatedly for the same user request after it has returned accepted=true.", |
|---|
| | | "Play one MP3 file from the SD card. The name argument must exactly match a files[].name from self.mp3.get_list. Optional speed supports only 0.5, 0.75, 1.0, 1.25, 1.5, or 2.0; non-1.0 speeds change pitch and affect only SD card MP3 playback. Optional repeat is the number of full-file loops (default 1). Optional power_off_after requests existing power-off standby after a natural finish, not after stop. For play, random play, switch song, or named-song requests, call this tool before telling the user playback has started. Do not call this tool repeatedly for the same user request after it has returned accepted=true.", |
|---|
| | | PropertyList({ |
|---|
| | | Property("name", kPropertyTypeString), |
|---|
| | | Property("speed", kPropertyTypeNumber, 1.0) |
|---|
| | | Property("speed", kPropertyTypeNumber, 1.0), |
|---|
| | | Property("repeat", kPropertyTypeInteger, 1, 1, 99), |
|---|
| | | Property("power_off_after", kPropertyTypeBoolean, false) |
|---|
| | | }), |
|---|
| | | [](const PropertyList& properties) -> ReturnValue { |
|---|
| | | std::string name = properties["name"].value<std::string>(); |
|---|
| | | float speed = static_cast<float>(properties["speed"].value<double>()); |
|---|
| | | int repeat = properties["repeat"].value<int>(); |
|---|
| | | bool power_off_after = properties["power_off_after"].value<bool>(); |
|---|
| | | if (repeat < 1) { |
|---|
| | | repeat = 1; |
|---|
| | | } |
|---|
| | | int64_t now = esp_timer_get_time(); |
|---|
| | | if (name == last_mp3_play_name && std::fabs(speed - last_mp3_play_speed) < 0.001f && now - last_mp3_play_time < 15000000) { |
|---|
| | | return std::string("{\"accepted\":true,\"duplicate_ignored\":true}"); |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t err = SDCard_Play_MP3_With_Speed(name.data(), speed); |
|---|
| | | esp_err_t err = SDCard_Play_MP3_With_Speed_Repeat(name.data(), speed, repeat, power_off_after); |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | return std::string("{\"accepted\":false,\"error\":\"") + esp_err_to_name(err) + |
|---|
| | | "\",\"retry\":\"use exact files[].name from self.mp3.get_list\"}"; |
|---|
| | |
|---|
| | | last_mp3_play_name = name; |
|---|
| | | last_mp3_play_speed = speed; |
|---|
| | | last_mp3_play_time = now; |
|---|
| | | return std::string("{\"accepted\":true}"); |
|---|
| | | return std::string("{\"accepted\":true,\"repeat\":") + std::to_string(repeat) + |
|---|
| | | ",\"power_off_after\":" + (power_off_after ? "true" : "false") + "}"; |
|---|
| | | }); |
|---|
| | | |
|---|
| | | mcp_server.AddTool("self.mp3.get_status", |
|---|
| | |
|---|
| | | PropertyList(), |
|---|
| | | [](const PropertyList& properties) -> ReturnValue { |
|---|
| | | return std::string("{\"playing\":") + (SDCard_Is_Playing() ? "true" : "false") + |
|---|
| | | ",\"tts_paused\":" + (SDCard_Is_Paused() ? "true" : "false") + "}"; |
|---|
| | | ",\"tts_paused\":" + (SDCard_Is_Paused() ? "true" : "false") + |
|---|
| | | ",\"repeat_left\":" + std::to_string(SDCard_Get_Repeat_Left()) + "}"; |
|---|
| | | }); |
|---|
| | | |
|---|
| | | mcp_server.AddTool("self.mp3.stop", |
|---|
| | |
|---|
| | | InitializeTools(); |
|---|
| | | } |
|---|
| | | |
|---|
| | | void OnDeviceReady() override { |
|---|
| | | xTaskCreate([](void *arg) { |
|---|
| | | (void)arg; |
|---|
| | | vTaskDelay(pdMS_TO_TICKS(1500)); |
|---|
| | | Board_ResolveBootOrRestore(false); |
|---|
| | | vTaskDelete(nullptr); |
|---|
| | | }, "boot_app_task", 4096, nullptr, 3, nullptr); |
|---|
| | | } |
|---|
| | | |
|---|
| | | bool OnRestoredFromPowerOff(bool network_ready) override { |
|---|
| | | (void)network_ready; |
|---|
| | | Board_ResolveBootOrRestore(true); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | |
|---|
| | | virtual void SetNetworkEventCallback(NetworkEventCallback callback) override { |
|---|
| | | WifiBoard::SetNetworkEventCallback([this, callback](NetworkEvent event, const std::string& data) { |
|---|
| | | if (event == NetworkEvent::Connected) { |
|---|
| | |
|---|
| | | #include "system.h" |
|---|
| | | #include "config.h" |
|---|
| | | #include "app_task.h" |
|---|
| | | #include "display/lvgl_display/jpg/jpeg_to_image.h" |
|---|
| | | |
|---|
| | | #include <esp_heap_caps.h> |
|---|
| | |
|---|
| | | if (!command || Board_Is_Power_Off()) { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | if (Board_Handle_Voice_Command(command)) { |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | if (strcmp(command, OFFLINE_COMMAND_WORD) != 0 && strcmp(command, OFFLINE_COMMAND_WORD_PINYIN) != 0) { |
|---|
| | | return false; |
|---|
| | | } |
|---|
| | | |
|---|
| | | Board_Offline_Command_Callback(); |
|---|
| | | Board_EnterVoiceEntry(); |
|---|
| | | return true; |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | |
|---|
| | | esp_err_t SDCard_Play_MP3_With_Speed(char *mp3_name, float speed) |
|---|
| | | { |
|---|
| | | return SDCard_Play_MP3_With_Speed_Repeat(mp3_name, speed, 1, false); |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t SDCard_Play_MP3_With_Speed_Repeat(char *mp3_name, float speed, int repeat, bool power_off_after) |
|---|
| | | { |
|---|
| | | if (!is_supported_mp3_speed(speed)) { |
|---|
| | | ESP_LOGW(TAG, "invalid MP3 playback speed: %.2f", speed); |
|---|
| | | SDCard_Resume_Play(); |
|---|
| | |
|---|
| | | return ESP_ERR_NOT_FOUND; |
|---|
| | | } |
|---|
| | | |
|---|
| | | esp_err_t err = SDCard_Play_MP3_File_With_Speed(path, speed); |
|---|
| | | esp_err_t err = SDCard_Play_MP3_File_With_Speed_Repeat(path, speed, repeat, power_off_after); |
|---|
| | | if (err != ESP_OK) { |
|---|
| | | SDCard_Resume_Play(); |
|---|
| | | ESP_LOGE(TAG, "start MP3 playback failed: %s", esp_err_to_name(err)); |
|---|
| | |
|---|
| | | * @return ESP_OK 表示ææ¾ä»»å¡å·²æåå¯å¨ï¼ESP_ERR_INVALID_ARG 表示é度ææ件åéæ³ã |
|---|
| | | */ |
|---|
| | | esp_err_t SDCard_Play_MP3_With_Speed(char *mp3_name, float speed); |
|---|
| | | esp_err_t SDCard_Play_MP3_With_Speed_Repeat(char *mp3_name, float speed, int repeat, bool power_off_after); |
|---|
| | | void SDCard_Show_Matching_JPG_For_MP3_Path(const char *mp3_path); |
|---|
| | | |
|---|
| | | /** |
|---|
| | |
|---|
| | | * @return ESP_OK 表示ææ¾ä»»å¡å·²æåå¯å¨ï¼ESP_ERR_INVALID_ARG 表示路å¾æé度éæ³ã |
|---|
| | | */ |
|---|
| | | esp_err_t SDCard_Play_MP3_File_With_Speed(const char *path, float speed); |
|---|
| | | esp_err_t SDCard_Play_MP3_File_With_Speed_Repeat(const char *path, float speed, int repeat, bool power_off_after); |
|---|
| | | int SDCard_Get_Repeat_Left(void); |
|---|
| | | esp_err_t SDCard_Map_Business_Path(const char *business_path, char *sd_path, size_t sd_path_size); |
|---|
| | | esp_err_t SDCard_Prepare_Business_File(const char *business_path, |
|---|
| | | char *final_path, |
|---|
| | |
|---|
| | | .secrets.yaml |
|---|
| | | .private_config.yaml |
|---|
| | | .env.development |
|---|
| | | !main/manager-web/.env.development |
|---|
| | | |
|---|
| | | # model files |
|---|
| | | main/xiaozhi-server/models/SenseVoiceSmall/model.pt |
|---|
| | |
|---|
| | | /** |
|---|
| | | * æ æåºä»¶URL |
|---|
| | | */ |
|---|
| | | String INVALID_FIRMWARE_URL = "http://xiaozhi.server.com:8002/xiaozhi/otaMag/download/NOT_ACTIVATED_FIRMWARE_THIS_IS_A_INVALID_URL"; |
|---|
| | | String INVALID_FIRMWARE_URL = "http://192.168.0.107:8002/xiaozhi/otaMag/download/NOT_ACTIVATED_FIRMWARE_THIS_IS_A_INVALID_URL"; |
|---|
| | | |
|---|
| | | /** |
|---|
| | | * åå
¸ç±»å |
|---|
| | |
|---|
| | | |
|---|
| | | if (StringUtils.isBlank(wsUrl) || wsUrl.equals("null")) { |
|---|
| | | log.error("WebSocketå°åæªé
ç½®ï¼è¯·ç»å½æºæ§å°ï¼å¨åæ°ç®¡çæ¾å°ãserver.websocketãé
ç½®"); |
|---|
| | | wsUrl = "ws://xiaozhi.server.com:8000/xiaozhi/v1/"; |
|---|
| | | wsUrl = "ws://192.168.0.107:8000/xiaozhi/v1/"; |
|---|
| | | websocket.setUrl(wsUrl); |
|---|
| | | } else { |
|---|
| | | String[] wsUrls = wsUrl.split("\\;"); |
|---|
| | |
|---|
| | | websocket.setUrl(wsUrls[RandomUtil.randomInt(0, wsUrls.length)]); |
|---|
| | | } else { |
|---|
| | | log.error("WebSocketå°åæªé
ç½®ï¼è¯·ç»å½æºæ§å°ï¼å¨åæ°ç®¡çæ¾å°ãserver.websocketãé
ç½®"); |
|---|
| | | websocket.setUrl("ws://xiaozhi.server.com:8000/xiaozhi/v1/"); |
|---|
| | | websocket.setUrl("ws://192.168.0.107:8000/xiaozhi/v1/"); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | |
|---|
| | |
|---|
| | | -- å¢å æ¯å¦å
许ç¨æ·æ³¨ååæ° |
|---|
| | | delete from `sys_params` where id in (103,104); |
|---|
| | | INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (103, 'server.allow_user_register', 'false', 'boolean', 1, 'æ¯å¦è¿è¡ç®¡çå以å¤ç人注å'); |
|---|
| | | INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (104, 'server.fronted_url', 'http://xiaozhi.server.com', 'string', 1, 'ä¸åå
ä½éªè¯ç æ¶æ¾ç¤ºçæ§å¶é¢æ¿å°å'); |
|---|
| | | INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (104, 'server.fronted_url', 'http://192.168.0.107', 'string', 1, 'ä¸åå
ä½éªè¯ç æ¶æ¾ç¤ºçæ§å¶é¢æ¿å°å'); |
|---|
| | | |
|---|
| | | -- ä¿®æ£CosyVoiceSiliconflowé³è² |
|---|
| | | delete from `ai_tts_voice` where tts_model_id = 'TTS_CosyVoiceSiliconflow'; |
|---|
| New file |
| | |
|---|
| | | VUE_APP_API_BASE_URL=/xiaozhi |
|---|
| | | VUE_APP_PUBLIC_PATH=/ |
|---|
| | | # æ¯å¦å¼å¯CDN |
|---|
| | | VUE_APP_USE_CDN=false |
|---|
| | |
|---|
| | | * å¼åæ¶èªå¨è¯»å使ç¨.env.developmentæ件 |
|---|
| | | * ç¼è¯æ¶èªå¨è¯»å使ç¨.env.productionæ件 |
|---|
| | | */ |
|---|
| | | const DEV_API_SERVICE = process.env.VUE_APP_API_BASE_URL |
|---|
| | | const DEV_API_SERVICE = process.env.VUE_APP_API_BASE_URL || '/xiaozhi' |
|---|
| | | |
|---|
| | | /** |
|---|
| | | * æ ¹æ®å¼åç¯å¢è¿åæ¥å£url |
|---|
| | |
|---|
| | | </el-form-item> |
|---|
| | | </el-form> |
|---|
| | | |
|---|
| | | <teleport v-if="chunkedCallInfoFields.length"> |
|---|
| | | <template v-if="chunkedCallInfoFields.length"> |
|---|
| | | <div class="section-title">{{ $t('modelConfigDialog.callInfo') }}</div> |
|---|
| | | <div class="divider"></div> |
|---|
| | | </teleport> |
|---|
| | | </template> |
|---|
| | | |
|---|
| | | <el-form :model="formData.configJson" label-width="auto" label-position="left" class="custom-form"> |
|---|
| | | <div v-for="(row, rowIndex) in chunkedCallInfoFields" :key="rowIndex" class="form-row"> |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <el-dialog |
|---|
| | | :title="$t('cache.dialogTitle')" |
|---|
| | | :visible.sync="visible" |
|---|
| | | :visible.sync="dialogVisible" |
|---|
| | | width="70%" |
|---|
| | | :before-close="handleClose" |
|---|
| | | > |
|---|
| | |
|---|
| | | }, |
|---|
| | | data() { |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible, |
|---|
| | | isLoading: true, |
|---|
| | | cacheAvailable: false, |
|---|
| | | cacheData: { |
|---|
| | |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(newVal) { |
|---|
| | | this.dialogVisible = newVal; |
|---|
| | | if (newVal) { |
|---|
| | | this.loadCacheData(); |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val); |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | methods: { |
|---|
| | |
|---|
| | | window.location.reload(); |
|---|
| | | }, |
|---|
| | | |
|---|
| | | handleClose() { |
|---|
| | | this.$emit('update:visible', false); |
|---|
| | | handleClose(done) { |
|---|
| | | this.dialogVisible = false; |
|---|
| | | if (typeof done === 'function') { |
|---|
| | | done(); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }; |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <CustomDialog |
|---|
| | | :title="title" |
|---|
| | | :visible.sync="visible" |
|---|
| | | :visible.sync="dialogVisible" |
|---|
| | | width="800px" |
|---|
| | | @confirm="submit" |
|---|
| | | @close="cancel" |
|---|
| | |
|---|
| | | |
|---|
| | | data() { |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible, |
|---|
| | | uploadProgress: 0, |
|---|
| | | uploadStatus: '', |
|---|
| | | isUploading: false, |
|---|
| | |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(val) { |
|---|
| | | this.dialogVisible = val; |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val); |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | computed: { |
|---|
| | | isTypeDisabled() { |
|---|
| | | // å¦ææidï¼è¯´ææ¯ç¼è¾æ¨¡å¼ï¼ç¦ç¨ç±»åéæ© |
|---|
| | |
|---|
| | | </el-form-item> |
|---|
| | | </el-form> |
|---|
| | | |
|---|
| | | <teleport v-if="chunkedCallInfoFields.length"> |
|---|
| | | <template v-if="chunkedCallInfoFields.length"> |
|---|
| | | <div class="section-title">{{ $t("modelConfigDialog.callInfo") }}</div> |
|---|
| | | <div class="section-divider"></div> |
|---|
| | | </teleport> |
|---|
| | | </template> |
|---|
| | | |
|---|
| | | <el-form :model="form.configJson" ref="callInfoForm" label-width="auto" label-position="left"> |
|---|
| | | <template> |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <CustomDialog |
|---|
| | | :title="title" |
|---|
| | | :visible.sync="visible" |
|---|
| | | :visible.sync="dialogVisible" |
|---|
| | | width="600px" |
|---|
| | | class="param-dialog-wrapper" |
|---|
| | | @confirm="submit" |
|---|
| | |
|---|
| | | }, |
|---|
| | | data() { |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible, |
|---|
| | | dialogKey: Date.now(), |
|---|
| | | saving: false, |
|---|
| | | valueTypeOptions: [ |
|---|
| | |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(newVal) { |
|---|
| | | this.dialogVisible = newVal; |
|---|
| | | if (newVal) { |
|---|
| | | if (this.form.paramValue) { |
|---|
| | | // å¦ææ¯ json ç±»åï¼æ ¼å¼åæ¾ç¤º |
|---|
| | |
|---|
| | | // å½å¯¹è¯æ¡å
³éæ¶ï¼éç½®savingç¶æ |
|---|
| | | this.saving = false; |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }; |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <CustomDialog |
|---|
| | | :title="title" |
|---|
| | | :visible.sync="visible" |
|---|
| | | :visible.sync="dialogVisible" |
|---|
| | | width="57%" |
|---|
| | | class="provider-dialog-wrapper" |
|---|
| | | @confirm="submit" |
|---|
| | |
|---|
| | | }, |
|---|
| | | data() { |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible, |
|---|
| | | saving: false, |
|---|
| | | isAllFieldsSelected: false, |
|---|
| | | tableKey: 0 |
|---|
| | |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(val) { |
|---|
| | | this.dialogVisible = val; |
|---|
| | | if (!val) { |
|---|
| | | this.resetForm(); |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }; |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <CustomDialog |
|---|
| | | :title="title" |
|---|
| | | :visible.sync="visible" |
|---|
| | | :visible.sync="dialogVisible" |
|---|
| | | width="700px" |
|---|
| | | class="param-dialog-wrapper" |
|---|
| | | @confirm="submit" |
|---|
| | |
|---|
| | | }; |
|---|
| | | |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible, |
|---|
| | | dialogKey: Date.now(), |
|---|
| | | saving: false, |
|---|
| | | localForm: { |
|---|
| | |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(newVal) { |
|---|
| | | this.dialogVisible = newVal; |
|---|
| | | if (newVal) { |
|---|
| | | this.localForm.id = this.form.id; |
|---|
| | | this.localForm.fileName = this.form.fileName || ''; |
|---|
| | |
|---|
| | | } else { |
|---|
| | | this.saving = false; |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val); |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }; |
|---|
| | |
|---|
| | | <template> |
|---|
| | | <el-dialog :visible.sync="visible" width="400px" center> |
|---|
| | | <el-dialog :visible.sync="dialogVisible" width="400px" center> |
|---|
| | | <div |
|---|
| | | style="margin: 0 10px 10px;display: flex;align-items: center;gap: 10px;font-weight: 700;font-size: 20px;text-align: left;color: #3d4566;"> |
|---|
| | | <div |
|---|
| | |
|---|
| | | visible: { type: Boolean, required: true }, |
|---|
| | | password: { type: String, default: '' } |
|---|
| | | }, |
|---|
| | | data() { |
|---|
| | | return { |
|---|
| | | dialogVisible: this.visible |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | watch: { |
|---|
| | | visible(val) { |
|---|
| | | this.dialogVisible = val |
|---|
| | | }, |
|---|
| | | dialogVisible(val) { |
|---|
| | | this.$emit('update:visible', val) |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | methods: { |
|---|
| | | closeDialog() { |
|---|
| | | this.$emit('update:visible', false) |
|---|
| | | this.dialogVisible = false |
|---|
| | | }, |
|---|
| | | copyPassword() { |
|---|
| | | navigator.clipboard.writeText(this.password) |
|---|
| | |
|---|
| | | return path.join(__dirname, dir) |
|---|
| | | } |
|---|
| | | |
|---|
| | | // ç¡®ä¿å è½½ .env æ件 |
|---|
| | | // Vue CLI ä¼æ mode å è½½ .env.*ï¼è¿éå读ä¸éï¼ç» vue.config.js èªèº«ç¨ |
|---|
| | | dotenv.config(); |
|---|
| | | dotenv.config({ path: path.resolve(__dirname, `.env.${process.env.NODE_ENV || 'development'}`) }); |
|---|
| | | |
|---|
| | | // å®ä¹CDNèµæºå表ï¼ç¡®ä¿Service Workerä¹è½è®¿é® |
|---|
| | | const cdnResources = { |
|---|
| | |
|---|
| | | } |
|---|
| | | }); |
|---|
| | | |
|---|
| | | // å¯ç¨ä¼å设置 |
|---|
| | | config.optimization.usedExports(true); |
|---|
| | | config.optimization.concatenateModules(true); |
|---|
| | | config.optimization.minimize(true); |
|---|
| | | // webpack 5 å¼å模å¼é»è®¤ä¹ä¼å¼ usedExportsï¼ä¼æ Vue 2 SFC ç style |
|---|
| | | // default export treeshake æï¼å¤§é style0 warningï¼ãminimize è¿ä¼ç ´å |
|---|
| | | // Vue 2 渲æå½æ°ï¼å¯¼è´é¡µé¢ç©ºç½å setAttribute æ¥éãè¿äºåªç»ç产æ建ã |
|---|
| | | if (process.env.NODE_ENV === 'production') { |
|---|
| | | config.optimization.usedExports(true); |
|---|
| | | config.optimization.concatenateModules(true); |
|---|
| | | config.optimization.minimize(true); |
|---|
| | | } else { |
|---|
| | | config.optimization.usedExports(false); |
|---|
| | | config.optimization.concatenateModules(false); |
|---|
| | | config.optimization.minimize(false); |
|---|
| | | } |
|---|
| | | |
|---|
| | | // Vue 2 + vue-style-loader éè¦ css-loader ç CommonJS å¯¼åº |
|---|
| | | const ruleTypes = ['css', 'postcss', 'scss', 'sass', 'less', 'stylus'] |
|---|
| | | ruleTypes.forEach(ruleType => { |
|---|
| | | const rule = config.module.rule(ruleType) |
|---|
| | | if (!rule) return |
|---|
| | | const oneOfs = rule.oneOfs && rule.oneOfs.store |
|---|
| | | if (!oneOfs) return |
|---|
| | | oneOfs.forEach(oneOf => { |
|---|
| | | if (oneOf.uses.has('css-loader')) { |
|---|
| | | oneOf.use('css-loader').tap(options => { |
|---|
| | | options = options || {} |
|---|
| | | options.esModule = false |
|---|
| | | return options |
|---|
| | | }) |
|---|
| | | } |
|---|
| | | }) |
|---|
| | | }) |
|---|
| | | }, |
|---|
| | | configureWebpack: config => { |
|---|
| | | if (process.env.NODE_ENV === 'production') { |
|---|
| | |
|---|
| | | enable_wakeup_words_response_cache: true |
|---|
| | | # å¼åºæ¯å¦åå¤å¤éè¯ |
|---|
| | | enable_greeting: true |
|---|
| | | # Phase 1ï¼è¿å
¥ chat.conversation å主å¨å¼å£ |
|---|
| | | enable_proactive_greeting: true |
|---|
| | | proactive_chat_greeting: æ们æ¥è天å§ã |
|---|
| | | # 说å®è¯æ¯å¦å¼å¯æç¤ºé³ |
|---|
| | | enable_stop_tts_notify: false |
|---|
| | | # 说å®è¯æ¯å¦å¼å¯æ示é³ï¼é³æå°å |
|---|
| | |
|---|
| | | - get_weather |
|---|
| | | - get_news_from_newsnow |
|---|
| | | - play_music |
|---|
| | | - switch_app |
|---|
| | | - start_repeater |
|---|
| | | function_call: |
|---|
| | | # ä¸éè¦å¨type |
|---|
| | | type: function_call |
|---|
| | |
|---|
| | | # play_musicæ¯æå¡å¨èªå¸¦çé³ä¹ææ¾ï¼hass_play_musicæ¯éè¿home assistantæ§å¶çç¬ç«å¤é¨ç¨åºé³ä¹ææ¾ |
|---|
| | | # å¦æç¨äºhass_play_musicï¼å°±ä¸è¦å¼å¯play_musicï¼ä¸¤è
åªçä¸ä¸ª |
|---|
| | | - play_music |
|---|
| | | - switch_app |
|---|
| | | - start_repeater |
|---|
| | | #- hass_state |
|---|
| | | #- hass_play_music |
|---|
| | | |
|---|
| New file |
| | |
|---|
| | | """Phase 1 App Runtime: chat / english_learning / english_translation.""" |
|---|
| | | |
|---|
| | | from .registry import get_app, get_task, get_registry, default_app_id |
|---|
| | | from .runtime import start_app_task, switch_app, filter_tools, on_user_wakeup |
|---|
| | | |
|---|
| | | __all__ = [ |
|---|
| | | "get_app", |
|---|
| | | "get_task", |
|---|
| | | "get_registry", |
|---|
| | | "default_app_id", |
|---|
| | | "start_app_task", |
|---|
| | | "switch_app", |
|---|
| | | "filter_tools", |
|---|
| | | "on_user_wakeup", |
|---|
| | | ] |
|---|
| New file |
| | |
|---|
| | | { |
|---|
| | | "default_app_id": "chat", |
|---|
| | | "apps": { |
|---|
| | | "chat": { |
|---|
| | | "name": "è天", |
|---|
| | | "mode": "CHAT", |
|---|
| | | "default_task": "conversation", |
|---|
| | | "tool_policy": "inherit_agent", |
|---|
| | | "system_prompt_mode": "agent_proactive", |
|---|
| | | "tasks": { |
|---|
| | | "conversation": { |
|---|
| | | "interaction_mode": "CHAT", |
|---|
| | | "skills": [] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | "english_learning": { |
|---|
| | | "name": "è±è¯å¦ä¹ ", |
|---|
| | | "mode": "PLAY_AUDIO", |
|---|
| | | "default_task": "repeater", |
|---|
| | | "tool_policy": "allowlist", |
|---|
| | | "tools": [ |
|---|
| | | "switch_app", |
|---|
| | | "start_repeater", |
|---|
| | | "self_mp3_play", |
|---|
| | | "self_mp3_stop", |
|---|
| | | "self_mp3_get_list", |
|---|
| | | "self_power_off" |
|---|
| | | ], |
|---|
| | | "system_prompt_mode": "none", |
|---|
| | | "tasks": { |
|---|
| | | "repeater": { |
|---|
| | | "interaction_mode": "PLAY_AUDIO", |
|---|
| | | "skills": [ |
|---|
| | | { |
|---|
| | | "name": "self.mp3.play", |
|---|
| | | "args_from": ["name", "repeat", "power_off_after"] |
|---|
| | | } |
|---|
| | | ] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | "english_translation": { |
|---|
| | | "name": "è±è¯ç¿»è¯", |
|---|
| | | "mode": "TRANSLATION", |
|---|
| | | "default_task": "live_translate", |
|---|
| | | "tool_policy": "allowlist", |
|---|
| | | "tools": ["switch_app", "translate"], |
|---|
| | | "system_prompt_mode": "translator", |
|---|
| | | "tasks": { |
|---|
| | | "live_translate": { |
|---|
| | | "interaction_mode": "TRANSLATION", |
|---|
| | | "skills": [ |
|---|
| | | { |
|---|
| | | "name": "translate", |
|---|
| | | "side": "server_mcp" |
|---|
| | | } |
|---|
| | | ] |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| | | } |
|---|
| New file |
| | |
|---|
| | | """Load the Phase 1 App / Task registry from JSON.""" |
|---|
| | | |
|---|
| | | import json |
|---|
| | | import os |
|---|
| | | from typing import Any, Dict, Optional, Tuple |
|---|
| | | |
|---|
| | | from config.config_loader import get_project_dir |
|---|
| | | from config.logger import setup_logging |
|---|
| | | |
|---|
| | | TAG = __name__ |
|---|
| | | logger = setup_logging() |
|---|
| | | |
|---|
| | | _REGISTRY: Optional[Dict[str, Any]] = None |
|---|
| | | |
|---|
| | | _BUNDLED_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "apps.json") |
|---|
| | | |
|---|
| | | |
|---|
| | | def _load_registry() -> Dict[str, Any]: |
|---|
| | | candidates = [ |
|---|
| | | os.path.join(get_project_dir(), "data", "apps.json"), |
|---|
| | | _BUNDLED_PATH, |
|---|
| | | ] |
|---|
| | | for path in candidates: |
|---|
| | | if not os.path.isfile(path): |
|---|
| | | continue |
|---|
| | | try: |
|---|
| | | with open(path, "r", encoding="utf-8") as f: |
|---|
| | | data = json.load(f) |
|---|
| | | if isinstance(data, dict) and data.get("apps"): |
|---|
| | | logger.bind(tag=TAG).info(f"å·²å è½½ App 注å表: {path}") |
|---|
| | | return data |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).error(f"å è½½ App 注å表失败 {path}: {e}") |
|---|
| | | logger.bind(tag=TAG).warning("æªæ¾å° apps.jsonï¼ä½¿ç¨å
ç½®æå°æ³¨å表") |
|---|
| | | return { |
|---|
| | | "default_app_id": "chat", |
|---|
| | | "apps": { |
|---|
| | | "chat": { |
|---|
| | | "name": "è天", |
|---|
| | | "mode": "CHAT", |
|---|
| | | "default_task": "conversation", |
|---|
| | | "tool_policy": "inherit_agent", |
|---|
| | | "system_prompt_mode": "agent_proactive", |
|---|
| | | "tasks": {"conversation": {"interaction_mode": "CHAT", "skills": []}}, |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | } |
|---|
| | | |
|---|
| | | |
|---|
| | | def get_registry() -> Dict[str, Any]: |
|---|
| | | global _REGISTRY |
|---|
| | | if _REGISTRY is None: |
|---|
| | | _REGISTRY = _load_registry() |
|---|
| | | return _REGISTRY |
|---|
| | | |
|---|
| | | |
|---|
| | | def default_app_id() -> str: |
|---|
| | | return get_registry().get("default_app_id") or "chat" |
|---|
| | | |
|---|
| | | |
|---|
| | | def get_app(app_id: str) -> Optional[Dict[str, Any]]: |
|---|
| | | if not app_id: |
|---|
| | | return None |
|---|
| | | return get_registry().get("apps", {}).get(app_id) |
|---|
| | | |
|---|
| | | |
|---|
| | | def get_task(app_id: str, task_id: str) -> Tuple[Optional[Dict[str, Any]], Optional[Dict[str, Any]]]: |
|---|
| | | app = get_app(app_id) |
|---|
| | | if not app: |
|---|
| | | return None, None |
|---|
| | | tasks = app.get("tasks") or {} |
|---|
| | | if not task_id: |
|---|
| | | task_id = app.get("default_task") or "" |
|---|
| | | task = tasks.get(task_id) |
|---|
| | | if not task: |
|---|
| | | return app, None |
|---|
| | | return app, task |
|---|
| New file |
| | |
|---|
| | | """Unified start_app_task / switch_app / mode helpers.""" |
|---|
| | | |
|---|
| | | from __future__ import annotations |
|---|
| | | |
|---|
| | | import uuid |
|---|
| | | from typing import Any, Dict, Optional, TYPE_CHECKING |
|---|
| | | |
|---|
| | | from config.logger import setup_logging |
|---|
| | | from plugins_func.register import Action, ActionResponse |
|---|
| | | |
|---|
| | | from .registry import default_app_id, get_app, get_task |
|---|
| | | |
|---|
| | | if TYPE_CHECKING: |
|---|
| | | from core.connection import ConnectionHandler |
|---|
| | | |
|---|
| | | TAG = __name__ |
|---|
| | | logger = setup_logging() |
|---|
| | | |
|---|
| | | SOURCE_PRIORITY = { |
|---|
| | | "mqtt_immediate": 3, |
|---|
| | | "schedule": 2, |
|---|
| | | "voice": 1, |
|---|
| | | "default_chat": 0, |
|---|
| | | } |
|---|
| | | |
|---|
| | | TRANSLATOR_PROMPT = ( |
|---|
| | | "ä½ æ¯ä¸å°ç¿»è¯æºãåªè¾åºè¯æï¼ä¸è¦è§£éãä¸è¦æ¼é³è¯¾ãä¸è¦è¿½é®ãä¸è¦é²èã" |
|---|
| | | "æªæå®è¯è¨æ¶ï¼ä¸æè¯æè±æï¼è±æè¯æä¸æï¼å
¶å®è¯è¨é»è®¤è¯æä¸æã" |
|---|
| | | ) |
|---|
| | | |
|---|
| | | CHAT_PROACTIVE_HINT = ( |
|---|
| | | "ä½ æ¯ç¨æ·çè¯é³ä¼ä¼´ãè¿å
¥å¯¹è¯åå¯ä»¥å
主å¨çå¥é请è天ï¼ä¾å¦ãæ们æ¥è天å§ãã" |
|---|
| | | "ç¨æ·å¼å£åææ£å¸¸å¯¹è¯åçï¼ä¸è¦åå¤å¬ä¿ã" |
|---|
| | | ) |
|---|
| | | |
|---|
| | | VOICE_MENU = "æ³è天ãå¬å¤è¯»ï¼è¿æ¯æå¼ç¿»è¯ï¼ä¹å¯ä»¥è¯´æ们æ¥è天å§ã" |
|---|
| | | |
|---|
| | | CHAT_PHRASES = ("æ们æ¥è天å§", "èä¼å¤©å§", "åå°è天", "éåºç¿»è¯", "ç»æç¿»è¯", "éåºå¤è¯»") |
|---|
| | | TRANSLATE_PHRASES = ("æå¼ç¿»è¯", "å¼å§ç¿»è¯", "è±è¯ç¿»è¯", "ç¿»è¯æ¨¡å¼") |
|---|
| | | REPEATER_PHRASES = ("å¼å§å¤è¯»", "æå¼å¤è¯»", "è±è¯å¦ä¹ ", "å¬å¤è¯»") |
|---|
| | | |
|---|
| | | DEVICE_PLAY_TOOL = "self_mp3_play" |
|---|
| | | DEVICE_STOP_TOOL = "self_mp3_stop" |
|---|
| | | |
|---|
| | | |
|---|
| | | def source_priority(source: str) -> int: |
|---|
| | | return SOURCE_PRIORITY.get(source or "voice", 1) |
|---|
| | | |
|---|
| | | |
|---|
| | | def _ensure_runtime_fields(conn: "ConnectionHandler") -> None: |
|---|
| | | if not hasattr(conn, "active_app_id"): |
|---|
| | | conn.active_app_id = default_app_id() |
|---|
| | | if not hasattr(conn, "current_task_id"): |
|---|
| | | conn.current_task_id = "conversation" |
|---|
| | | if not hasattr(conn, "interaction_mode"): |
|---|
| | | conn.interaction_mode = "CHAT" |
|---|
| | | if not hasattr(conn, "current_source"): |
|---|
| | | conn.current_source = "default_chat" |
|---|
| | | if not hasattr(conn, "tool_policy"): |
|---|
| | | conn.tool_policy = "inherit_agent" |
|---|
| | | if not hasattr(conn, "tool_allowlist"): |
|---|
| | | conn.tool_allowlist = [] |
|---|
| | | if not hasattr(conn, "original_system_prompt"): |
|---|
| | | conn.original_system_prompt = conn.prompt |
|---|
| | | |
|---|
| | | |
|---|
| | | def filter_tools(conn: "ConnectionHandler", tools: Dict[str, Any]) -> Dict[str, Any]: |
|---|
| | | """inherit_agent keeps the live agent toolset; allowlist keeps named tools only.""" |
|---|
| | | _ensure_runtime_fields(conn) |
|---|
| | | if getattr(conn, "tool_policy", "inherit_agent") != "allowlist": |
|---|
| | | return tools |
|---|
| | | allow = set(getattr(conn, "tool_allowlist", None) or []) |
|---|
| | | if not allow: |
|---|
| | | return {} |
|---|
| | | filtered = {} |
|---|
| | | for name, definition in tools.items(): |
|---|
| | | if name in allow: |
|---|
| | | filtered[name] = definition |
|---|
| | | return filtered |
|---|
| | | |
|---|
| | | |
|---|
| | | def _refresh_tools(conn: "ConnectionHandler") -> None: |
|---|
| | | handler = getattr(conn, "func_handler", None) |
|---|
| | | if handler and hasattr(handler, "upload_functions_desc"): |
|---|
| | | handler.upload_functions_desc() |
|---|
| | | |
|---|
| | | |
|---|
| | | def _apply_prompt(conn: "ConnectionHandler", app: Dict[str, Any]) -> None: |
|---|
| | | mode = app.get("system_prompt_mode") or "none" |
|---|
| | | if conn.original_system_prompt is None: |
|---|
| | | conn.original_system_prompt = conn.prompt |
|---|
| | | if mode == "translator": |
|---|
| | | conn.change_system_prompt( |
|---|
| | | "ä½ ç°å¨æ¯è±è¯ç¿»è¯æºãç¨æ·è¯´è¯ååºè°ç¨ translateï¼ç¨æ·è¯´éåºç¿»è¯ãåå°è天æ¶è°ç¨ switch_appï¼app_id=chatã" |
|---|
| | | "ä¸è¦è®²ç¬è¯ãä¸è¦æ¥å¤©æ°ãä¸è¦é²èã" |
|---|
| | | ) |
|---|
| | | elif mode == "agent_proactive": |
|---|
| | | base = conn.original_system_prompt or conn.prompt or "" |
|---|
| | | if CHAT_PROACTIVE_HINT not in (base or ""): |
|---|
| | | conn.change_system_prompt((base + "\n" + CHAT_PROACTIVE_HINT).strip()) |
|---|
| | | else: |
|---|
| | | conn.change_system_prompt(base) |
|---|
| | | else: |
|---|
| | | if conn.original_system_prompt: |
|---|
| | | conn.change_system_prompt(conn.original_system_prompt) |
|---|
| | | |
|---|
| | | |
|---|
| | | async def _execute_named_tool(conn: "ConnectionHandler", tool_name: str, arguments: Dict[str, Any]): |
|---|
| | | handler = getattr(conn, "func_handler", None) |
|---|
| | | if not handler or not hasattr(handler, "tool_manager"): |
|---|
| | | return ActionResponse(action=Action.ERROR, response="å·¥å
·å¤çå¨æªå°±ç»ª") |
|---|
| | | try: |
|---|
| | | return await handler.tool_manager.execute_tool(tool_name, arguments or {}) |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).error(f"æ§è¡å·¥å
· {tool_name} 失败: {e}") |
|---|
| | | return ActionResponse(action=Action.ERROR, response=str(e)) |
|---|
| | | |
|---|
| | | |
|---|
| | | async def _stop_current_skills(conn: "ConnectionHandler") -> None: |
|---|
| | | mode = getattr(conn, "interaction_mode", "CHAT") |
|---|
| | | if mode == "PLAY_AUDIO": |
|---|
| | | await _execute_named_tool(conn, DEVICE_STOP_TOOL, {}) |
|---|
| | | conn.awaiting_voice_app_choice = False |
|---|
| | | |
|---|
| | | |
|---|
| | | def _speak(conn: "ConnectionHandler", text: str) -> None: |
|---|
| | | if not text: |
|---|
| | | return |
|---|
| | | from core.handle.intentHandler import speak_txt |
|---|
| | | |
|---|
| | | if not getattr(conn, "sentence_id", None): |
|---|
| | | conn.sentence_id = uuid.uuid4().hex |
|---|
| | | try: |
|---|
| | | speak_txt(conn, text) |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).warning(f"主å¨ææ¥å¤±è´¥: {e}") |
|---|
| | | |
|---|
| | | |
|---|
| | | def _greeting_text(conn: "ConnectionHandler") -> str: |
|---|
| | | return (conn.config or {}).get("proactive_chat_greeting") or "æ们æ¥è天å§ã" |
|---|
| | | |
|---|
| | | |
|---|
| | | async def start_app_task( |
|---|
| | | conn: "ConnectionHandler", |
|---|
| | | app_id: str, |
|---|
| | | task_id: str = "", |
|---|
| | | params: Optional[Dict[str, Any]] = None, |
|---|
| | | source: str = "voice", |
|---|
| | | execute_skills: bool = True, |
|---|
| | | ) -> ActionResponse: |
|---|
| | | """Single entry for MQTT / schedule / voice / default chat.""" |
|---|
| | | _ensure_runtime_fields(conn) |
|---|
| | | params = params or {} |
|---|
| | | app, task = get_task(app_id, task_id) |
|---|
| | | if not app: |
|---|
| | | return ActionResponse(action=Action.ERROR, response=f"æªç¥ç App: {app_id}") |
|---|
| | | if not task: |
|---|
| | | default_task = app.get("default_task") or "" |
|---|
| | | app, task = get_task(app_id, default_task) |
|---|
| | | if not task: |
|---|
| | | return ActionResponse(action=Action.ERROR, response=f"æªç¥ç Task: {app_id}/{task_id}") |
|---|
| | | |
|---|
| | | resolved_task_id = task_id or app.get("default_task") or "conversation" |
|---|
| | | new_p = source_priority(source) |
|---|
| | | cur_p = source_priority(getattr(conn, "current_source", "default_chat")) |
|---|
| | | same_task = ( |
|---|
| | | getattr(conn, "active_app_id", "") == app_id |
|---|
| | | and getattr(conn, "current_task_id", "") == resolved_task_id |
|---|
| | | ) |
|---|
| | | if new_p < cur_p and not same_task: |
|---|
| | | logger.bind(tag=TAG).info( |
|---|
| | | f"æç»ä½ä¼å
çº§ä»»å¡ {source}({new_p}) < {conn.current_source}({cur_p})" |
|---|
| | | ) |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.NONE, |
|---|
| | | response="å½åææ´é«ä¼å
级çä»»å¡ï¼å
å®æåååæ¢", |
|---|
| | | ) |
|---|
| | | |
|---|
| | | if not same_task or new_p >= cur_p: |
|---|
| | | await _stop_current_skills(conn) |
|---|
| | | |
|---|
| | | conn.active_app_id = app_id |
|---|
| | | conn.current_task_id = resolved_task_id |
|---|
| | | conn.interaction_mode = task.get("interaction_mode") or app.get("mode") or "CHAT" |
|---|
| | | conn.current_source = source |
|---|
| | | conn.tool_policy = app.get("tool_policy") or "inherit_agent" |
|---|
| | | conn.tool_allowlist = list(app.get("tools") or []) |
|---|
| | | _apply_prompt(conn, app) |
|---|
| | | _refresh_tools(conn) |
|---|
| | | |
|---|
| | | logger.bind(tag=TAG).info( |
|---|
| | | f"start_app_task app={app_id} task={resolved_task_id} mode={conn.interaction_mode} source={source}" |
|---|
| | | ) |
|---|
| | | |
|---|
| | | if execute_skills: |
|---|
| | | for skill in task.get("skills") or []: |
|---|
| | | skill_name = (skill or {}).get("name") or "" |
|---|
| | | if skill_name in ("self.mp3.play", DEVICE_PLAY_TOOL): |
|---|
| | | play_args = { |
|---|
| | | "name": params.get("name") or "", |
|---|
| | | "repeat": int(params.get("repeat") or 1), |
|---|
| | | "power_off_after": bool(params.get("power_off_after") or False), |
|---|
| | | } |
|---|
| | | if params.get("speed") is not None: |
|---|
| | | play_args["speed"] = params.get("speed") |
|---|
| | | if not play_args["name"]: |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.RESPONSE, |
|---|
| | | response="没ææå®è¦å¤è¯»çæ件åï¼è¯·å
ç¨ self.mp3.get_list 确认 files[].name", |
|---|
| | | ) |
|---|
| | | result = await _execute_named_tool(conn, DEVICE_PLAY_TOOL, play_args) |
|---|
| | | if result and result.action in (Action.ERROR, Action.NOTFOUND): |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.RESPONSE, |
|---|
| | | response=result.response or "è¿å°è®¾å¤æ²¡ææ¬å°å¤è¯»ææ¾è½å", |
|---|
| | | ) |
|---|
| | | elif skill_name == "self.power_off": |
|---|
| | | await _execute_named_tool(conn, "self_power_off", {}) |
|---|
| | | |
|---|
| | | if conn.interaction_mode == "CHAT": |
|---|
| | | enable = (conn.config or {}).get("enable_proactive_greeting", True) |
|---|
| | | woken = bool(getattr(conn, "just_woken_up", False)) |
|---|
| | | if enable and not woken: |
|---|
| | | _speak(conn, _greeting_text(conn)) |
|---|
| | | |
|---|
| | | app_name = app.get("name") or app_id |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.NONE, |
|---|
| | | result=f"{app_id}/{resolved_task_id}", |
|---|
| | | response=f"å·²æå¼{app_name}", |
|---|
| | | ) |
|---|
| | | |
|---|
| | | |
|---|
| | | async def switch_app(conn: "ConnectionHandler", app_id: str, source: str = "voice") -> ActionResponse: |
|---|
| | | app = get_app(app_id) |
|---|
| | | if not app: |
|---|
| | | return ActionResponse(action=Action.ERROR, response=f"ä¸æ¯æç App: {app_id}") |
|---|
| | | task_id = app.get("default_task") or "" |
|---|
| | | return await start_app_task(conn, app_id, task_id, {}, source) |
|---|
| | | |
|---|
| | | |
|---|
| | | async def on_user_wakeup(conn: "ConnectionHandler") -> None: |
|---|
| | | """User wake word returns to chat; do not treat [app_task] as wakeup.""" |
|---|
| | | _ensure_runtime_fields(conn) |
|---|
| | | if getattr(conn, "interaction_mode", "CHAT") == "CHAT" and getattr(conn, "active_app_id", "chat") == "chat": |
|---|
| | | return |
|---|
| | | await start_app_task(conn, "chat", "conversation", {}, source="voice") |
|---|
| | | |
|---|
| | | |
|---|
| | | def match_voice_switch(text: str) -> Optional[tuple]: |
|---|
| | | """Map a short phrase to (app_id, task_id) or ('__menu__', '').""" |
|---|
| | | if not text: |
|---|
| | | return None |
|---|
| | | _, filtered = _strip(text) |
|---|
| | | if filtered in ("åæ¢æ¨¡å¼", "qie huan mo shi"): |
|---|
| | | return ("__menu__", "") |
|---|
| | | for p in CHAT_PHRASES: |
|---|
| | | if p in filtered: |
|---|
| | | return ("chat", "conversation") |
|---|
| | | for p in TRANSLATE_PHRASES: |
|---|
| | | if p in filtered: |
|---|
| | | return ("english_translation", "live_translate") |
|---|
| | | for p in REPEATER_PHRASES: |
|---|
| | | if p in filtered: |
|---|
| | | return ("english_learning", "repeater") |
|---|
| | | if any(k in filtered for k in ("æ°åè诵", "æè¯æ¥é¾", "å£ç®", "å åæ³")): |
|---|
| | | return ("__reserved__", "") |
|---|
| | | return None |
|---|
| | | |
|---|
| | | |
|---|
| | | def _strip(text: str) -> tuple: |
|---|
| | | from core.utils.util import remove_punctuation_and_length |
|---|
| | | |
|---|
| | | return remove_punctuation_and_length(text) |
|---|
| | | |
|---|
| | | |
|---|
| | | def is_exit_translation(text: str) -> bool: |
|---|
| | | _, filtered = _strip(text) |
|---|
| | | return any(p in filtered for p in ("éåºç¿»è¯", "ç»æç¿»è¯", "åå°è天", "æ们æ¥è天å§", "èä¼å¤©å§")) |
|---|
| | | |
|---|
| | | |
|---|
| | | def _plain_user_text(text: str) -> str: |
|---|
| | | """ASR å¯è½æåæå
å¨ JSON éï¼ç¿»è¯åªè¦ contentã""" |
|---|
| | | if not text: |
|---|
| | | return "" |
|---|
| | | raw = str(text).strip() |
|---|
| | | if raw.startswith("{") and raw.endswith("}"): |
|---|
| | | try: |
|---|
| | | import json |
|---|
| | | |
|---|
| | | data = json.loads(raw) |
|---|
| | | if isinstance(data, dict) and data.get("content"): |
|---|
| | | return str(data["content"]).strip() |
|---|
| | | except Exception: |
|---|
| | | pass |
|---|
| | | return raw |
|---|
| | | |
|---|
| | | |
|---|
| | | def _looks_like_tool_error(text: str) -> bool: |
|---|
| | | if not text: |
|---|
| | | return True |
|---|
| | | s = str(text) |
|---|
| | | if "Error executing tool" in s or "isError=True" in s or "isError=true" in s: |
|---|
| | | return True |
|---|
| | | if "æªé
ç½®" in s and "api_key" in s: |
|---|
| | | return True |
|---|
| | | return False |
|---|
| | | |
|---|
| | | |
|---|
| | | async def run_translation(conn: "ConnectionHandler", text: str) -> str: |
|---|
| | | """Translate with the connection LLM first (æºæ§å°å·²é
ç模å). MCP is fallback only.""" |
|---|
| | | text = _plain_user_text(text) |
|---|
| | | if not text: |
|---|
| | | return "" |
|---|
| | | |
|---|
| | | from plugins_func.register import all_function_registry |
|---|
| | | |
|---|
| | | item = all_function_registry.get("translate") |
|---|
| | | if item: |
|---|
| | | try: |
|---|
| | | maybe = item.func(conn, text=text) |
|---|
| | | if hasattr(maybe, "__await__"): |
|---|
| | | maybe = await maybe |
|---|
| | | if isinstance(maybe, ActionResponse) and maybe.action not in ( |
|---|
| | | Action.ERROR, |
|---|
| | | Action.NOTFOUND, |
|---|
| | | ): |
|---|
| | | out = (maybe.response or maybe.result or "").strip() |
|---|
| | | if out and not _looks_like_tool_error(out): |
|---|
| | | return out |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).warning(f"ç¿»è¯æ件失败: {e}") |
|---|
| | | |
|---|
| | | handler = getattr(conn, "func_handler", None) |
|---|
| | | if handler and handler.has_tool("translate"): |
|---|
| | | result = await _execute_named_tool(conn, "translate", {"text": text}) |
|---|
| | | if result and result.action not in (Action.ERROR, Action.NOTFOUND): |
|---|
| | | out = _extract_translate_text(result.response or result.result or "") |
|---|
| | | if out and not _looks_like_tool_error(out): |
|---|
| | | return out |
|---|
| | | |
|---|
| | | llm = getattr(conn, "llm", None) |
|---|
| | | if llm and hasattr(llm, "response_no_stream"): |
|---|
| | | try: |
|---|
| | | return ( |
|---|
| | | llm.response_no_stream(TRANSLATOR_PROMPT, text, temperature=0.1) or "" |
|---|
| | | ).strip() |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).error(f"ç¿»è¯ LLM 失败: {e}") |
|---|
| | | return "" |
|---|
| | | |
|---|
| | | |
|---|
| | | def _extract_translate_text(raw: Any) -> str: |
|---|
| | | if raw is None: |
|---|
| | | return "" |
|---|
| | | if isinstance(raw, str): |
|---|
| | | text = raw.strip() |
|---|
| | | if text.startswith("{") and "text" in text: |
|---|
| | | try: |
|---|
| | | import json |
|---|
| | | |
|---|
| | | data = json.loads(text) |
|---|
| | | if isinstance(data, dict): |
|---|
| | | return str(data.get("text") or data.get("content") or text) |
|---|
| | | except Exception: |
|---|
| | | pass |
|---|
| | | return text |
|---|
| | | if hasattr(raw, "content"): |
|---|
| | | content = raw.content |
|---|
| | | if isinstance(content, list) and content: |
|---|
| | | first = content[0] |
|---|
| | | return getattr(first, "text", None) or str(first) |
|---|
| | | return str(content) |
|---|
| | | return str(raw).strip() |
|---|
| | |
|---|
| | | # æ è®°å½åæ¯å¦ä¸ºæ¥çµæ¥å¬æ¨¡å¼ |
|---|
| | | self.incoming_call = None |
|---|
| | | |
|---|
| | | # Phase 1 App Runtimeï¼è¿æ¥å
åï¼ä¸å»ºè¡¨ï¼ |
|---|
| | | self.active_app_id = "chat" |
|---|
| | | self.current_task_id = "conversation" |
|---|
| | | self.interaction_mode = "CHAT" |
|---|
| | | self.current_source = "default_chat" |
|---|
| | | self.tool_policy = "inherit_agent" |
|---|
| | | self.tool_allowlist = [] |
|---|
| | | self.original_system_prompt = None |
|---|
| | | self.awaiting_voice_app_choice = False |
|---|
| | | |
|---|
| | | async def handle_connection(self, ws: websockets.ServerConnection): |
|---|
| | | try: |
|---|
| | | # è·åè¿è¡ä¸çäºä»¶å¾ªç¯ï¼å¿
é¡»å¨å¼æ¥ä¸ä¸æä¸ï¼ |
|---|
| | |
|---|
| | | self._init_prompt_enhancement() |
|---|
| | | """注å
¥å·¥å
·è°ç¨few-shot示ä¾ï¼ä»
function_call模å¼ï¼""" |
|---|
| | | self._inject_tool_call_fewshot() |
|---|
| | | if self.original_system_prompt is None: |
|---|
| | | self.original_system_prompt = self.prompt |
|---|
| | | |
|---|
| | | except Exception as e: |
|---|
| | | self.logger.bind(tag=TAG).error(f"å®ä¾åç»ä»¶å¤±è´¥: {e}") |
|---|
| | |
|---|
| | | actual_text = text |
|---|
| | | |
|---|
| | | try: |
|---|
| | | # å°è¯è§£æJSONæ ¼å¼çè¾å
¥ |
|---|
| | | # FunASR çä¼æç»æå
æ {"content","language","emotion"}ï¼å£°çº¹åºæ¯è¿æ speaker |
|---|
| | | if text.strip().startswith("{") and text.strip().endswith("}"): |
|---|
| | | data = json.loads(text) |
|---|
| | | if "speaker" in data and "content" in data: |
|---|
| | | speaker_name = data["speaker"] |
|---|
| | | actual_content = data["content"] |
|---|
| | | conn.logger.bind(tag=TAG).info(f"解æå°è¯´è¯äººä¿¡æ¯: {speaker_name}") |
|---|
| | | |
|---|
| | | # ä»
å¨è¯¥è¯´è¯äººé¦æ¬¡åºç°æ¶ä¿ç {"speaker":...} JSONï¼è®©æ¨¡åèªç¶ç§°å¼ä¸æ¬¡ï¼ |
|---|
| | | # åç»è½®é为纯ææ¬ï¼é¿å
æ¯è½®éå¤åºç°åå诱导模ååå¤ç§°å¼ |
|---|
| | | if speaker_name not in conn.introduced_speakers: |
|---|
| | | conn.introduced_speakers.add(speaker_name) |
|---|
| | | actual_text = text |
|---|
| | | else: |
|---|
| | | if isinstance(data, dict) and "content" in data: |
|---|
| | | speaker_name = data.get("speaker") |
|---|
| | | actual_content = data.get("content") or "" |
|---|
| | | if speaker_name: |
|---|
| | | conn.logger.bind(tag=TAG).info(f"解æå°è¯´è¯äººä¿¡æ¯: {speaker_name}") |
|---|
| | | # ä»
å¨è¯¥è¯´è¯äººé¦æ¬¡åºç°æ¶ä¿ç {"speaker":...} JSONï¼è®©æ¨¡åèªç¶ç§°å¼ä¸æ¬¡ï¼ |
|---|
| | | # åç»è½®é为纯ææ¬ï¼é¿å
æ¯è½®éå¤åºç°åå诱导模ååå¤ç§°å¼ |
|---|
| | | if speaker_name not in conn.introduced_speakers: |
|---|
| | | conn.introduced_speakers.add(speaker_name) |
|---|
| | | actual_text = text |
|---|
| | | else: |
|---|
| | | actual_text = actual_content |
|---|
| | | elif actual_content: |
|---|
| | | actual_text = actual_content |
|---|
| | | except (json.JSONDecodeError, KeyError): |
|---|
| | | except (json.JSONDecodeError, KeyError, TypeError): |
|---|
| | | # å¦æ解æ失败ï¼ç»§ç»ä½¿ç¨åå§ææ¬ |
|---|
| | | pass |
|---|
| | | |
|---|
| | |
|---|
| | | await max_out_size(conn) |
|---|
| | | return |
|---|
| | | |
|---|
| | | from core.app_runtime.runtime import ( |
|---|
| | | is_exit_translation, |
|---|
| | | match_voice_switch, |
|---|
| | | run_translation, |
|---|
| | | start_app_task, |
|---|
| | | VOICE_MENU, |
|---|
| | | ) |
|---|
| | | from core.handle.intentHandler import speak_txt |
|---|
| | | |
|---|
| | | mode = getattr(conn, "interaction_mode", "CHAT") |
|---|
| | | |
|---|
| | | if getattr(conn, "awaiting_voice_app_choice", False): |
|---|
| | | mapped = match_voice_switch(actual_text) |
|---|
| | | conn.awaiting_voice_app_choice = False |
|---|
| | | if mapped and mapped[0] == "__reserved__": |
|---|
| | | speak_txt(conn, "è¿ä¸ªæ¸¸æè¿æ²¡å¼æ¾ï¼æ们å
è天å§ã") |
|---|
| | | await start_app_task(conn, "chat", "conversation", {}, source="voice") |
|---|
| | | return |
|---|
| | | if mapped and mapped[0] == "__menu__": |
|---|
| | | conn.awaiting_voice_app_choice = True |
|---|
| | | speak_txt(conn, VOICE_MENU) |
|---|
| | | return |
|---|
| | | if mapped and mapped[0] != "__menu__": |
|---|
| | | params = {} |
|---|
| | | await start_app_task(conn, mapped[0], mapped[1], params, source="voice") |
|---|
| | | return |
|---|
| | | speak_txt(conn, VOICE_MENU) |
|---|
| | | conn.awaiting_voice_app_choice = True |
|---|
| | | return |
|---|
| | | |
|---|
| | | mapped = match_voice_switch(actual_text) |
|---|
| | | if mapped and mapped[0] == "__menu__": |
|---|
| | | conn.awaiting_voice_app_choice = True |
|---|
| | | speak_txt(conn, VOICE_MENU) |
|---|
| | | return |
|---|
| | | if mapped and mapped[0] == "__reserved__": |
|---|
| | | speak_txt(conn, "è¿ä¸ªè¿æ²¡å¼æ¾ï¼å
继ç»å½åå
容å§ã") |
|---|
| | | return |
|---|
| | | if mapped and mapped[0] in ("english_translation", "english_learning", "chat"): |
|---|
| | | if mapped[0] == "english_learning" and mapped[1] == "repeater": |
|---|
| | | speak_txt(conn, "å¤è¯»éè¦æå® SD å¡ä¸çæ件åï¼æç¨å°ç¨åºå¼å§ã") |
|---|
| | | return |
|---|
| | | await start_app_task(conn, mapped[0], mapped[1], {}, source="voice") |
|---|
| | | return |
|---|
| | | |
|---|
| | | if mode == "PLAY_AUDIO": |
|---|
| | | conn.logger.bind(tag=TAG).info("PLAY_AUDIO 模å¼å¿½ç¥ä¸è¡å¯¹è¯") |
|---|
| | | return |
|---|
| | | |
|---|
| | | if mode == "TRANSLATION": |
|---|
| | | if is_exit_translation(actual_text): |
|---|
| | | await start_app_task(conn, "chat", "conversation", {}, source="voice") |
|---|
| | | return |
|---|
| | | await send_stt_message(conn, actual_text) |
|---|
| | | conn.client_abort = False |
|---|
| | | translated = await run_translation(conn, actual_text) |
|---|
| | | if not translated: |
|---|
| | | translated = "没æå¬æ¸
è¦ç¿»è¯çå
容ï¼è¯·å说ä¸æ¬¡ã" |
|---|
| | | if not conn.sentence_id: |
|---|
| | | import uuid as _uuid |
|---|
| | | conn.sentence_id = _uuid.uuid4().hex |
|---|
| | | speak_txt(conn, translated) |
|---|
| | | return |
|---|
| | | |
|---|
| | | # manual 模å¼ä¸ä¸æææ£å¨ææ¾çå
容 |
|---|
| | | if conn.client_is_speaking and conn.client_listen_mode != "manual": |
|---|
| | | await handleAbortMessage(conn) |
|---|
| | |
|---|
| | | original_text |
|---|
| | | ) |
|---|
| | | |
|---|
| | | # App Runtime: 设å¤ä¸æ¥ç Task å¯å¨ï¼å¤ç¨ listen detect.textï¼ä¸æ°å¢ WS typeï¼ |
|---|
| | | if original_text.startswith("[app_task]"): |
|---|
| | | payload = original_text[len("[app_task]"):].strip() |
|---|
| | | parts = [p.strip() for p in payload.split("|") if p.strip()] |
|---|
| | | app_id = parts[0] if parts else "chat" |
|---|
| | | task_id = parts[1] if len(parts) > 1 else "" |
|---|
| | | source = parts[2] if len(parts) > 2 else "mqtt_immediate" |
|---|
| | | conn.logger.bind(tag=TAG).info( |
|---|
| | | f"æ¶å° app_task: {app_id}/{task_id} source={source}" |
|---|
| | | ) |
|---|
| | | from core.app_runtime.runtime import start_app_task |
|---|
| | | |
|---|
| | | start_time = time.time() |
|---|
| | | while time.time() - start_time < 3: |
|---|
| | | if conn.tts: |
|---|
| | | break |
|---|
| | | await asyncio.sleep(0.1) |
|---|
| | | await start_app_task(conn, app_id, task_id, {}, source=source, execute_skills=False) |
|---|
| | | return |
|---|
| | | |
|---|
| | | if original_text.startswith("[voice_entry]"): |
|---|
| | | from core.app_runtime.runtime import VOICE_MENU |
|---|
| | | from core.handle.intentHandler import speak_txt |
|---|
| | | |
|---|
| | | conn.awaiting_voice_app_choice = True |
|---|
| | | conn.sentence_id = uuid.uuid4().hex |
|---|
| | | speak_txt(conn, VOICE_MENU) |
|---|
| | | return |
|---|
| | | |
|---|
| | | # æ£æ¥æ¯å¦æ¯è®¾å¤å¼å«æ令 [device_call] |
|---|
| | | if original_text.startswith("[device_call]"): |
|---|
| | | # æå tag åçææ¬ |
|---|
| | |
|---|
| | | conn.client_is_speaking = False |
|---|
| | | elif is_wakeup_words: |
|---|
| | | conn.just_woken_up = True |
|---|
| | | from core.app_runtime.runtime import on_user_wakeup |
|---|
| | | |
|---|
| | | await on_user_wakeup(conn) |
|---|
| | | # ä¸æ¥çº¯æåæ°æ®ï¼å¤ç¨ASRä¸æ¥åè½ï¼ä½ä¸æä¾é³é¢æ°æ®ï¼ |
|---|
| | | enqueue_asr_report(conn, "å¿ï¼ä½ 好å", []) |
|---|
| | | await startToChat(conn, "å¿ï¼ä½ 好å") |
|---|
| | |
|---|
| | | |
|---|
| | | import asyncio |
|---|
| | | import os |
|---|
| | | import sys |
|---|
| | | import json |
|---|
| | | from typing import Dict, Any, List |
|---|
| | | |
|---|
| | |
|---|
| | | |
|---|
| | | def load_config(self) -> Dict[str, Any]: |
|---|
| | | """å è½½MCPæå¡é
ç½®""" |
|---|
| | | if len(self.config_path) == 0: |
|---|
| | | return {} |
|---|
| | | servers: Dict[str, Any] = {} |
|---|
| | | if self.config_path: |
|---|
| | | try: |
|---|
| | | with open(self.config_path, "r", encoding="utf-8") as f: |
|---|
| | | config = json.load(f) |
|---|
| | | servers = config.get("mcpServers", {}) or {} |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).error( |
|---|
| | | f"Error loading MCP config from {self.config_path}: {e}" |
|---|
| | | ) |
|---|
| | | servers = {} |
|---|
| | | return self._ensure_builtin_servers(servers) |
|---|
| | | |
|---|
| | | try: |
|---|
| | | with open(self.config_path, "r", encoding="utf-8") as f: |
|---|
| | | config = json.load(f) |
|---|
| | | return config.get("mcpServers", {}) |
|---|
| | | except Exception as e: |
|---|
| | | logger.bind(tag=TAG).error( |
|---|
| | | f"Error loading MCP config from {self.config_path}: {e}" |
|---|
| | | ) |
|---|
| | | return {} |
|---|
| | | def _ensure_builtin_servers(self, servers: Dict[str, Any]) -> Dict[str, Any]: |
|---|
| | | """Register the Phase 1 translate MCP even if data/.mcp_server_settings.json is missing.""" |
|---|
| | | if "english-translate" in servers: |
|---|
| | | return servers |
|---|
| | | script = os.path.join( |
|---|
| | | get_project_dir(), "mcp_servers", "translate", "server.py" |
|---|
| | | ) |
|---|
| | | if os.path.isfile(script): |
|---|
| | | servers["english-translate"] = { |
|---|
| | | "command": sys.executable or "python", |
|---|
| | | "args": [script], |
|---|
| | | } |
|---|
| | | logger.bind(tag=TAG).info("已注åå
ç½® english-translate MCP") |
|---|
| | | return servers |
|---|
| | | |
|---|
| | | async def _init_server(self, name: str, srv_config: Dict[str, Any]): |
|---|
| | | """åå§åå个MCPæå¡""" |
|---|
| | |
|---|
| | | except Exception as e: |
|---|
| | | self.logger.error(f"è·å{tool_type.value}å·¥å
·æ¶åºé: {e}") |
|---|
| | | |
|---|
| | | try: |
|---|
| | | from core.app_runtime.runtime import filter_tools |
|---|
| | | |
|---|
| | | all_tools = filter_tools(self.conn, all_tools) |
|---|
| | | except Exception as e: |
|---|
| | | self.logger.error(f"æ App è¿æ»¤å·¥å
·å¤±è´¥: {e}") |
|---|
| | | |
|---|
| | | self._cached_tools = all_tools |
|---|
| | | return all_tools |
|---|
| | | |
|---|
| | |
|---|
| | | "å½åæ¯æä¸ç§ä¼ è¾æ¨¡å¼ï¼stdio(æ åè¾å
¥è¾åº), sse(Server-Sent Events), streamable-http(æµå¼HTTP)ã" |
|---|
| | | ], |
|---|
| | | "mcpServers": { |
|---|
| | | "english-translate": { |
|---|
| | | "command": "python", |
|---|
| | | "args": ["mcp_servers/translate/server.py"], |
|---|
| | | "des": "Phase 1 è±è¯ç¿»è¯ MCPï¼åªæ´é² translate" |
|---|
| | | }, |
|---|
| | | "Home Assistant": { |
|---|
| | | "command": "mcp-proxy", |
|---|
| | | "args": [ |
|---|
| New file |
| | |
|---|
| | | #!/usr/bin/env python3 |
|---|
| | | """Standalone translation MCP. Only exposes translate. Does not call ConnectionHandler.chat.""" |
|---|
| | | |
|---|
| | | from __future__ import annotations |
|---|
| | | |
|---|
| | | import os |
|---|
| | | from pathlib import Path |
|---|
| | | |
|---|
| | | import httpx |
|---|
| | | import yaml |
|---|
| | | from mcp.server.fastmcp import FastMCP |
|---|
| | | |
|---|
| | | ROOT = Path(__file__).resolve().parents[2] |
|---|
| | | TRANSLATOR_PROMPT = ( |
|---|
| | | "ä½ æ¯ä¸å°ç¿»è¯æºãåªè¾åºè¯æï¼ä¸è¦è§£éãä¸è¦æ¼é³è¯¾ãä¸è¦è¿½é®ãä¸è¦é²èã" |
|---|
| | | "æªæå®è¯è¨æ¶ï¼ä¸æè¯æè±æï¼è±æè¯æä¸æï¼å
¶å®è¯è¨é»è®¤è¯æä¸æã" |
|---|
| | | ) |
|---|
| | | |
|---|
| | | mcp = FastMCP("english-translate") |
|---|
| | | |
|---|
| | | |
|---|
| | | def _read_yaml(path: Path) -> dict: |
|---|
| | | if not path.is_file(): |
|---|
| | | return {} |
|---|
| | | with path.open("r", encoding="utf-8") as f: |
|---|
| | | data = yaml.safe_load(f) or {} |
|---|
| | | return data if isinstance(data, dict) else {} |
|---|
| | | |
|---|
| | | |
|---|
| | | def _load_llm_config() -> dict: |
|---|
| | | default_cfg = _read_yaml(ROOT / "config.yaml") |
|---|
| | | custom_cfg = _read_yaml(ROOT / "data" / ".config.yaml") |
|---|
| | | |
|---|
| | | def merge(base, overlay): |
|---|
| | | out = dict(base) |
|---|
| | | for k, v in overlay.items(): |
|---|
| | | if isinstance(v, dict) and isinstance(out.get(k), dict): |
|---|
| | | out[k] = merge(out[k], v) |
|---|
| | | else: |
|---|
| | | out[k] = v |
|---|
| | | return out |
|---|
| | | |
|---|
| | | cfg = merge(default_cfg, custom_cfg) |
|---|
| | | selected = (cfg.get("selected_module") or {}).get("LLM") |
|---|
| | | llm_map = cfg.get("LLM") or {} |
|---|
| | | if selected and selected in llm_map: |
|---|
| | | item = dict(llm_map[selected]) |
|---|
| | | item["_name"] = selected |
|---|
| | | return item |
|---|
| | | if llm_map: |
|---|
| | | name, item = next(iter(llm_map.items())) |
|---|
| | | data = dict(item) |
|---|
| | | data["_name"] = name |
|---|
| | | return data |
|---|
| | | return {} |
|---|
| | | |
|---|
| | | |
|---|
| | | def _chat_complete(text: str, source_lang: str, target_lang: str) -> str: |
|---|
| | | llm = _load_llm_config() |
|---|
| | | api_key = llm.get("api_key") or os.environ.get("OPENAI_API_KEY") or "" |
|---|
| | | if api_key and ("ä½ ç" in str(api_key) or "key" == str(api_key).lower()): |
|---|
| | | api_key = os.environ.get("OPENAI_API_KEY") or api_key |
|---|
| | | base_url = llm.get("base_url") or llm.get("url") or "https://api.openai.com/v1" |
|---|
| | | model = llm.get("model_name") or llm.get("model") or "gpt-4o-mini" |
|---|
| | | if not api_key or "ä½ ç" in str(api_key): |
|---|
| | | raise RuntimeError("ç¿»è¯ MCP æªé
ç½®å¯ç¨ç LLM api_key") |
|---|
| | | |
|---|
| | | extra = "" |
|---|
| | | if source_lang or target_lang: |
|---|
| | | extra = f"\næºè¯è¨:{source_lang or 'auto'} ç®æ è¯è¨:{target_lang or 'auto'}" |
|---|
| | | user = f"ç¿»è¯ä¸é¢å
容ï¼åªè¾åºè¯æï¼{extra}\n{text}" |
|---|
| | | |
|---|
| | | url = str(base_url).rstrip("/") |
|---|
| | | if not url.endswith("/chat/completions"): |
|---|
| | | if url.endswith("/v1"): |
|---|
| | | url = url + "/chat/completions" |
|---|
| | | else: |
|---|
| | | url = url + "/v1/chat/completions" |
|---|
| | | |
|---|
| | | headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"} |
|---|
| | | payload = { |
|---|
| | | "model": model, |
|---|
| | | "temperature": 0.1, |
|---|
| | | "messages": [ |
|---|
| | | {"role": "system", "content": TRANSLATOR_PROMPT}, |
|---|
| | | {"role": "user", "content": user}, |
|---|
| | | ], |
|---|
| | | } |
|---|
| | | with httpx.Client(timeout=60.0) as client: |
|---|
| | | resp = client.post(url, headers=headers, json=payload) |
|---|
| | | resp.raise_for_status() |
|---|
| | | data = resp.json() |
|---|
| | | choices = data.get("choices") or [] |
|---|
| | | if not choices: |
|---|
| | | raise RuntimeError("ç¿»è¯æ¥å£æ²¡æè¿åå
容") |
|---|
| | | msg = (choices[0].get("message") or {}).get("content") or "" |
|---|
| | | return str(msg).strip() |
|---|
| | | |
|---|
| | | |
|---|
| | | @mcp.tool() |
|---|
| | | def translate(text: str, source_lang: str = "", target_lang: str = "") -> str: |
|---|
| | | """Translate text. Chinese->English, English->Chinese unless languages are set. Return translation only.""" |
|---|
| | | if not text or not str(text).strip(): |
|---|
| | | return "" |
|---|
| | | return _chat_complete(str(text).strip(), source_lang or "", target_lang or "") |
|---|
| | | |
|---|
| | | |
|---|
| | | if __name__ == "__main__": |
|---|
| | | os.chdir(ROOT) |
|---|
| | | mcp.run(transport="stdio") |
|---|
| New file |
| | |
|---|
| | | from typing import TYPE_CHECKING |
|---|
| | | |
|---|
| | | from plugins_func.register import register_function, ToolType, ActionResponse, Action |
|---|
| | | |
|---|
| | | if TYPE_CHECKING: |
|---|
| | | from core.connection import ConnectionHandler |
|---|
| | | |
|---|
| | | start_repeater_function_desc = { |
|---|
| | | "type": "function", |
|---|
| | | "function": { |
|---|
| | | "name": "start_repeater", |
|---|
| | | "description": "å¨è®¾å¤ SD å¡ä¸å¤è¯»æå® MP3ãname å¿
é¡»ä¸ self.mp3.get_list è¿åç files[].name å®å
¨ä¸è´ãrepeat æ¯æ´é¦å¾ªç¯æ¬¡æ°ãä¸è¦ç¨ play_musicã", |
|---|
| | | "parameters": { |
|---|
| | | "type": "object", |
|---|
| | | "properties": { |
|---|
| | | "name": { |
|---|
| | | "type": "string", |
|---|
| | | "description": "SD å¡ MP3 æ件åï¼å¿
é¡»ä¸ get_list ç files[].name ä¸è´", |
|---|
| | | }, |
|---|
| | | "repeat": { |
|---|
| | | "type": "integer", |
|---|
| | | "description": "æ´é¦å¾ªç¯æ¬¡æ°ï¼é»è®¤ 1", |
|---|
| | | }, |
|---|
| | | "power_off_after": { |
|---|
| | | "type": "boolean", |
|---|
| | | "description": "æå®åæ¯å¦è½¯å
³æºï¼é»è®¤ false", |
|---|
| | | }, |
|---|
| | | }, |
|---|
| | | "required": ["name"], |
|---|
| | | }, |
|---|
| | | }, |
|---|
| | | } |
|---|
| | | |
|---|
| | | |
|---|
| | | @register_function("start_repeater", start_repeater_function_desc, ToolType.SYSTEM_CTL) |
|---|
| | | async def start_repeater( |
|---|
| | | conn: "ConnectionHandler", |
|---|
| | | name: str, |
|---|
| | | repeat: int = 1, |
|---|
| | | power_off_after: bool = False, |
|---|
| | | ): |
|---|
| | | from core.app_runtime.runtime import start_app_task |
|---|
| | | |
|---|
| | | try: |
|---|
| | | repeat_n = int(repeat or 1) |
|---|
| | | except (TypeError, ValueError): |
|---|
| | | repeat_n = 1 |
|---|
| | | if repeat_n < 1: |
|---|
| | | repeat_n = 1 |
|---|
| | | |
|---|
| | | result = await start_app_task( |
|---|
| | | conn, |
|---|
| | | "english_learning", |
|---|
| | | "repeater", |
|---|
| | | { |
|---|
| | | "name": name, |
|---|
| | | "repeat": repeat_n, |
|---|
| | | "power_off_after": bool(power_off_after), |
|---|
| | | }, |
|---|
| | | source="voice", |
|---|
| | | ) |
|---|
| | | if result.action in (Action.ERROR, Action.RESPONSE): |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.RESPONSE, |
|---|
| | | result=result.result, |
|---|
| | | response=result.response or "æ æ³å¼å§å¤è¯»", |
|---|
| | | ) |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.NONE, |
|---|
| | | result=result.result, |
|---|
| | | response=result.response, |
|---|
| | | ) |
|---|
| New file |
| | |
|---|
| | | from typing import TYPE_CHECKING |
|---|
| | | |
|---|
| | | from plugins_func.register import register_function, ToolType, ActionResponse, Action |
|---|
| | | |
|---|
| | | if TYPE_CHECKING: |
|---|
| | | from core.connection import ConnectionHandler |
|---|
| | | |
|---|
| | | switch_app_function_desc = { |
|---|
| | | "type": "function", |
|---|
| | | "function": { |
|---|
| | | "name": "switch_app", |
|---|
| | | "description": "åæ¢è®¾å¤å½å Appãæå¼ç¿»è¯ãæå¼è±è¯å¦ä¹ /å¤è¯»ãåå°è天ãéåºç¿»è¯æ¶è°ç¨ãapp_id åªè½æ¯ chatãenglish_learningãenglish_translationã", |
|---|
| | | "parameters": { |
|---|
| | | "type": "object", |
|---|
| | | "properties": { |
|---|
| | | "app_id": { |
|---|
| | | "type": "string", |
|---|
| | | "description": "ç®æ Appï¼chatï¼è天ï¼ãenglish_learningï¼å¤è¯»ï¼ãenglish_translationï¼ç¿»è¯ï¼", |
|---|
| | | } |
|---|
| | | }, |
|---|
| | | "required": ["app_id"], |
|---|
| | | }, |
|---|
| | | }, |
|---|
| | | } |
|---|
| | | |
|---|
| | | |
|---|
| | | @register_function("switch_app", switch_app_function_desc, ToolType.SYSTEM_CTL) |
|---|
| | | async def switch_app(conn: "ConnectionHandler", app_id: str): |
|---|
| | | from core.app_runtime.runtime import switch_app as runtime_switch |
|---|
| | | |
|---|
| | | result = await runtime_switch(conn, app_id, source="voice") |
|---|
| | | if result.action == Action.ERROR: |
|---|
| | | return ActionResponse( |
|---|
| | | action=Action.RESPONSE, |
|---|
| | | result=result.response, |
|---|
| | | response=result.response or "æä¸å¼è¿ä¸ªåºç¨", |
|---|
| | | ) |
|---|
| | | if str(result.result or "").startswith("chat/"): |
|---|
| | | return ActionResponse(action=Action.NONE, result=result.result, response=result.response) |
|---|
| | | name = result.response or "å·²åæ¢" |
|---|
| | | return ActionResponse(action=Action.RESPONSE, result=result.result, response=name) |
|---|
| New file |
| | |
|---|
| | | from typing import TYPE_CHECKING |
|---|
| | | |
|---|
| | | from plugins_func.register import register_function, ToolType, ActionResponse, Action |
|---|
| | | from core.app_runtime.runtime import TRANSLATOR_PROMPT |
|---|
| | | |
|---|
| | | if TYPE_CHECKING: |
|---|
| | | from core.connection import ConnectionHandler |
|---|
| | | |
|---|
| | | translate_function_desc = { |
|---|
| | | "type": "function", |
|---|
| | | "function": { |
|---|
| | | "name": "translate", |
|---|
| | | "description": "æç¨æ·è¿å¥è¯ç¿»è¯æå¦ä¸ç§è¯è¨ãåªè¿åè¯æãä¸æè¯è±æï¼è±æè¯ä¸æãä¸è¦è§£éã", |
|---|
| | | "parameters": { |
|---|
| | | "type": "object", |
|---|
| | | "properties": { |
|---|
| | | "text": {"type": "string", "description": "è¦ç¿»è¯çåæ"}, |
|---|
| | | "source_lang": { |
|---|
| | | "type": "string", |
|---|
| | | "description": "æºè¯è¨ä»£ç ï¼å¯ç©ºï¼ç©ºåèªå¨å¤æ", |
|---|
| | | }, |
|---|
| | | "target_lang": { |
|---|
| | | "type": "string", |
|---|
| | | "description": "ç®æ è¯è¨ä»£ç ï¼å¯ç©ºï¼ç©ºåä¸æâè±æ", |
|---|
| | | }, |
|---|
| | | }, |
|---|
| | | "required": ["text"], |
|---|
| | | }, |
|---|
| | | }, |
|---|
| | | } |
|---|
| | | |
|---|
| | | |
|---|
| | | def _build_user_prompt(text: str, source_lang: str, target_lang: str) -> str: |
|---|
| | | extra = "" |
|---|
| | | if source_lang or target_lang: |
|---|
| | | extra = f"\næºè¯è¨:{source_lang or 'auto'} ç®æ è¯è¨:{target_lang or 'auto'}" |
|---|
| | | return f"ç¿»è¯ä¸é¢å
容ï¼åªè¾åºè¯æï¼{extra}\n{text}" |
|---|
| | | |
|---|
| | | |
|---|
| | | @register_function("translate", translate_function_desc, ToolType.SYSTEM_CTL) |
|---|
| | | def translate( |
|---|
| | | conn: "ConnectionHandler", |
|---|
| | | text: str, |
|---|
| | | source_lang: str = "", |
|---|
| | | target_lang: str = "", |
|---|
| | | ): |
|---|
| | | llm = getattr(conn, "llm", None) |
|---|
| | | if not llm or not text: |
|---|
| | | return ActionResponse(action=Action.ERROR, response="ç¿»è¯æå¡ä¸å¯ç¨") |
|---|
| | | try: |
|---|
| | | out = llm.response_no_stream( |
|---|
| | | TRANSLATOR_PROMPT, |
|---|
| | | _build_user_prompt(text, source_lang or "", target_lang or ""), |
|---|
| | | temperature=0.1, |
|---|
| | | ) |
|---|
| | | out = (out or "").strip() |
|---|
| | | if not out: |
|---|
| | | return ActionResponse(action=Action.ERROR, response="没æå¾å°è¯æ") |
|---|
| | | return ActionResponse(action=Action.RESPONSE, result=out, response=out) |
|---|
| | | except Exception as e: |
|---|
| | | return ActionResponse(action=Action.ERROR, response=str(e)) |
|---|
| New file |
| | |
|---|
| | | # å°æº AI Server Architecture Plan |
|---|
| | | |
|---|
| | | > ææ¡£ï¼å°æº AI Server ââ Taskãè¯é³åæä¸ç¨æ·è¯ä¼°å¹³å°æ¶æ Plan |
|---|
| | | > Version: V1.1 |
|---|
| | | > Date: 2026-08-30 |
|---|
| | | > æ¬æå¨ V1.0 æ¦å¿µç¨¿ä¸ï¼ä¾æ®ãXiaozhi AI å¼æ¾åºç¨å¹³å° Master Planãä¸ãArchitecture Auditã修订ã |
|---|
| | | > **åªè®¾è®¡ï¼ä¸ç¼ç ã** |
|---|
| | | > ä¸ãå°æº AI 硬件å¼æ¾åºç¨å¹³å° ââ æ»ä½æ¶æä¸å¼å PlanãV1.1 æ对é
读ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 0. æ¬æå®ä½ |
|---|
| | | |
|---|
| | | æ¬æåªå®ä¹ **Server**ï¼ä»åº `xiaozhi-esp32-server`ï¼å¦ä½ä»ãå®æ¶è¯é³å¯¹è¯å端ãæ¼è¿ä¸ºãTask ç¼æ + è¯é³å¯¹è±¡ + è¯ä¼°ãå¹³å°ã |
|---|
| | | |
|---|
| | | 硬件æ§è¡ç»èè§ Client Planãæ¬æåªè§å® **对设å¤çå¥çº¦** ä¸ **äºç«¯æ¨¡å / æ°æ® / API**ã |
|---|
| | | |
|---|
| | | å·¥ä½åºå¦æ `smartlearning/`ï¼è¯¾ç¨ JSON + MQTT DOWNLOAD/PLAY_MEDIAï¼ãå®æ´æ¥è¿ App Syncï¼ä½**æªæ¥å
¥** Xiaozhi Serverãæ¬ææå®å½ä½å¯å¸æ¶çæè·¯ï¼èä¸æ¯ç¬¬ä¸å¥é¿æä¸å¡äºã |
|---|
| | | |
|---|
| | | **ç¸å¯¹ V1.0 çå
³é®ä¿®æ£ï¼** |
|---|
| | | |
|---|
| | | | V1.0 | V1.1ï¼æ¬æ¬¡ï¼ | |
|---|
| | | |------|----------------| |
|---|
| | | | ä¸ä¸æ¥æ Gateway/ä¸å Assessment å¾®æå¡ | **é»è¾æ¨¡ååå¼ï¼ç©çä»æ¯ Python xiaozhi-server + Java manager-api** | |
|---|
| | | | Phase 1 å°±åè·è¯»+åè+å£ç®ä¸å¥è¯ä¼° | Phase 1 = è±è¯å¤è¯»ç¼æä¸ **ä¸è¯ä¼°**ï¼Phase 2 = è·è¯» + ASR ææ¬æ¯å¯¹ | |
|---|
| | | | 设å¤ä¸ Server åæä¸å¥å®æ´ Task Runtime | **æå¨ Task Runtime åªå¨ Server**ï¼è®¾å¤æ¯ Task Executor | |
|---|
| | | | é³é¢è§å为对象åå¨ï¼ä½æªå¯¹é½ç°ç¶ | æ¿è®¤ç°ç¶æ¯ `ai_agent_chat_audio` LONGBLOBï¼æ¼è¿åè¿å¯¹è±¡åå¨ | |
|---|
| | | | Agent æªåºå | **Agent â App**ï¼Agent 继ç»å人设/模åé
ç½® | |
|---|
| | | | 8ï½10 个平å°é¶æ®µæ··å¨ä¸èµ· | ä¸ Client 对é½ä¸º Phase 1/2/3ï¼å
¶åè¯ä¼°å¹³å°åæåé¶æ®µ | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 1. å½å Server æ¯ä»ä¹ï¼Audit äºå®ï¼ |
|---|
| | | |
|---|
| | | åæ ï¼ä¸æ¯å¾®æå¡ç½ï¼ |
|---|
| | | |
|---|
| | | | 模å | å
¥å£ | èè´£ | |
|---|
| | | |------|------|------| |
|---|
| | | | xiaozhi-server | `main/xiaozhi-server/app.py` | WS `/xiaozhi/v1/`ãOTA HTTPãå®æ¶ ASR/LLM/TTS | |
|---|
| | | | manager-api | `AdminApplication.java` | ç¨æ·/设å¤/Agent/æ件/è天è½åº | |
|---|
| | | | manager-web / mobile | â | 管ç端 | |
|---|
| | | | digital-human | â | æµè¯ç«¯ï¼é主é¾è·¯ | |
|---|
| | | |
|---|
| | | ä¸æ¬¡å¯¹è¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device â ConnectionHandler (session_id = uuid4) |
|---|
| | | â VAD â ASR â intent / chat() |
|---|
| | | â UnifiedToolHandler (Plugin / MCP / IoT) |
|---|
| | | â TTS â type=tts/stt/llm/mcp + Opus |
|---|
| | | â å¯é report â POST /agent/chat-history/report |
|---|
| | | â ai_agent_chat_history + ai_agent_chat_audio (WAV) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å
³é®ä»£ç ï¼ |
|---|
| | | |
|---|
| | | | é¶æ®µ | ä½ç½® | |
|---|
| | | |------|------| |
|---|
| | | | å
¥ç« | `core/connection.py` `_route_message` | |
|---|
| | | | VAD/ASR | `core/handle/receiveAudioHandle.py` | |
|---|
| | | | æå¾ | `core/handle/intentHandler.py` | |
|---|
| | | | LLM | `ConnectionHandler.chat` | |
|---|
| | | | TTS åºç« | `core/handle/sendAudioHandle.py` | |
|---|
| | | | ä¸æ¥ | `core/handle/reportHandle.py` â WAV | |
|---|
| | | | å·¥å
· | `plugins_func/*` + `unified_tool_handler.py` | |
|---|
| | | |
|---|
| | | **å·²æå¯å¤ç¨ï¼** AgentãPluginãMCPãfunction callãsession_idãå¯é WAVã`play_music`ï¼ä½èµ° TTS ééï¼**ä¸éå**å¤è¯»å¾ªç¯ï¼ã |
|---|
| | | |
|---|
| | | **ä¸åå¨ï¼** AppãTaskãStepãTaskInstanceãInteractionModeãAssessmentãPlanãSubscriptionã |
|---|
| | | `session_id` æ¯ **WS è¿æ¥çº§**ï¼ä¸æ¯ TaskInstanceã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. ç®æ ï¼ä»æ¯ V1.0 æ¿æ¯ï¼ä½åå±è½å°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Xiaozhi Server |
|---|
| | | âââ Device Gateway â ç°æ WS / MQTT ç½å
³ï¼ä¿ç |
|---|
| | | âââ Chat Orchestration â ç°æ ConnectionHandler.chat |
|---|
| | | âââ Task Runtime â æ°å¢ï¼æå¨ç¼æ |
|---|
| | | âââ Media / Audio Object â ä»ãè天é件ãæ¼è¿ä¸ºå¯æ Step çé³é¢å¯¹è±¡ |
|---|
| | | âââ Assessment â Phase 2 èµ·ï¼å
ææ¬æ¯å¯¹ï¼åè¯é³ç¹å¾ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»æ°æ®æµï¼ç®æ æï¼ä¸æ¯ Phase 1ï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Schedule â TaskInstance â Step â InteractionMode |
|---|
| | | â Device Skill â å¿ç«¥è¯é³ |
|---|
| | | â ä¿å Audio 对象 â ASR â Assessment â Metric â 家é¿æ¥å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Phase 1 **æªæå¨**ãMode=PLAY_AUDIO 驱å¨è®¾å¤ææ¾ãï¼ä¸è¦æ±è¯ä¼°ä¸å®¶é¿æ¥åã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. è责边çï¼ä¸ Client Plan 对é½ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Server |
|---|
| | | å建 / æ¨è¿ / ç»æ TaskInstance |
|---|
| | | æå® InteractionMode |
|---|
| | | è°ç¨è®¾å¤ Skillï¼MCP æ type=taskï¼ |
|---|
| | | ASR / LLM / TTS / Assessment |
|---|
| | | é³é¢å¯¹è±¡ä¸ step å
³è |
|---|
| | | App JSON ä¸èµæº URL çæå¨ |
|---|
| | | |
|---|
| | | Client |
|---|
| | | æ§è¡ Skillãåæ¢ Modeãåæ¥ Event |
|---|
| | | ä¸æä¹
å Step åå²ãä¸è¯å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **ç¦æ¢**å¨ Python 对è¯çè·¯å¾éåæ» `if english_repeat` ä½ä¸ºé¿æå½¢æï¼Phase 1 å
许ä¸ä¸ª **ç¼ææ件**ï¼Phase 2 å¿
é¡»åå° Task Runtime + JSON å®ä¹ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 4. é»è¾æ¨¡å vs ç©çé¨ç½² |
|---|
| | | |
|---|
| | | V1.0 ååºç Gateway / Task Service / Assessment Service / Emotion ⦠**å
¨é¨ä½ä¸ºé»è¾è¾¹ç**ã |
|---|
| | | |
|---|
| | | ç©çé¨ç½²ä¿æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | xiaozhi-server (Python) å®æ¶ï¼è¿æ¥ãASRãLLMãTTSãMode åæ¢ãTask æ¥è¿ï¼è¿è¡æ¶ï¼ |
|---|
| | | manager-api (Java) é
ç½®ä¸æä¹
åï¼UserãDeviceãAgentãTask å®ä¹/å®ä¾ãé³é¢å
æ°æ®ãè¯ä¼°ç»æ |
|---|
| | | 对象åå¨ï¼Phase 2+ï¼ åå§é³é¢ï¼ç¬¬ä¸æå¯ç»§ç» BLOBï¼ä½ API æ audio_id è®¿é® |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸è¦ä¸ºç¬¬ä¸ä¸ªå¤è¯» Task ææ°è¿ç¨ã |
|---|
| | | |
|---|
| | | 模åæ å°ï¼ |
|---|
| | | |
|---|
| | | | é»è¾æ¨¡å | Phase | è½ç¹ | |
|---|
| | | |----------|-------|------| |
|---|
| | | | Device Gateway | 0 å·²æ | `websocket_server.py` / MQTT gateway | |
|---|
| | | | Chat Orchestration | 0 å·²æ | `ConnectionHandler` | |
|---|
| | | | Interaction Mode å¼å
³ | 1 | `ConnectionHandler` ä¸ `interaction_mode`ï¼é CHAT è·³è¿ `chat()` | |
|---|
| | | | Repeat Orchestrator | 1 | `plugins_func` æç¬ç«è模åï¼MCP `self.mp3.play` | |
|---|
| | | | Task Runtime | 2 | Python è¿è¡æ¶ç¶ææº + Java å®ä¾è¡¨ | |
|---|
| | | | Audio Object | 2 | æ©å±ä¸æ¥ï¼æ `step_instance_id`ï¼éæ¥è¿åº LONGBLOB | |
|---|
| | | | Assessment v0 | 2 | expected_text vs ASRï¼Java æ Python åå¯ï¼ç»æå
¥åº | |
|---|
| | | | App / Resource | 3 | å¸æ¶ smartlearning æ¸
å + ä¸è½½ URLï¼Java 为æå¨ | |
|---|
| | | | Scheduler / Plan | 3 | manager-api + å®æ¶ï¼ä¸è¦ç¨è®¾å¤ RTC é¹éåå
| |
|---|
| | | | Voice Feature / Baseline / Emotion | 3+ | å¼æ¥éåï¼ç¦æ¢è¿å®æ¶ TTS è·¯å¾ | |
|---|
| | | | Subscription / Developer API | æ´å | ä¸å Phase 1â2 | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 5. ç°æ对è¯é¾è·¯ï¼å¿
é¡»ä¿ç |
|---|
| | | |
|---|
| | | æ Task æ¶è¡ä¸ºä¸ç°ç½ä¸è´ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | listen â ASR â intent/LLM/tools â TTS |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åè®® `hello/listen/abort/iot/mcp/ping` ä¸ä¸è¡ `stt/tts/llm/mcp` **ç¦æ¢æ¹è¯ä¹**ã |
|---|
| | | |
|---|
| | | æå
¥æ¹å¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | _route_message / startToChat / chat() |
|---|
| | | ä¹åå¤æ interaction_mode |
|---|
| | | CHAT â åè·¯å¾ |
|---|
| | | PLAY_AUDIO / FOLLOW_READ â ä¸è°ç¨ chat() |
|---|
| | | FOLLOW_READ çè¯é³ä»èµ° ASR.receive_audioï¼ç»æåå
¥ Step èé Dialogue |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | `play_music` 继ç»æå¡ãç¨æ·è¯´ææ¾ææ²ãã**ä¸è¦**ç¨å®å®ç°è±è¯å¤è¯»ï¼å®å ç¨ TTS éåï¼åè¯¾ç¨ SD ææ¾å²çªï¼ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 6. Agent vs App |
|---|
| | | |
|---|
| | | | | Agentï¼å·²æ `ai_agent`ï¼ | Appï¼æ°å»ºï¼Phase 3ï¼ | |
|---|
| | | |--|--------------------------|----------------------| |
|---|
| | | | æ¯ä»ä¹ | 人设 + ASR/LLM/TTS/æ件é
ç½® | ä¸å¡å®¹å¨ï¼è±è¯èå¸ãå£ç®â¦ï¼ | |
|---|
| | | | è°ç¨ | æ¯å°è®¾å¤å½åç»å®ä¸ä¸ª | å¿ç«¥å¯è®¢é
å¤ä¸ªï¼åæ¶åª Active ä¸ä¸ª | |
|---|
| | | | Phase 1 | **继ç»ç¨ Agent è· Chat** | ä¸å»º App 表ä¹å¯ä»¥åå¤è¯» Demo | |
|---|
| | | |
|---|
| | | Phase 1 å¤è¯»å¯ä»¥æå¨ãå½å Agent + ä¸ä¸ªæ件ãï¼ä¸å¿
å
æ App å®ä½ãPhase 3 åæ Task Definition å½å° App ä¸ã |
|---|
| | | |
|---|
| | | ä¸è¦æ `ai_agent` æ¹å为 Appï¼ä¹ä¸è¦å¨ Agent ä¸å Workflow JSON ä½ä¸ºé¿æ模åï¼ä¸´æ¶å¯ä»¥ï¼æ£å¼æ¨¡åè§ç¬¬ 10 èï¼ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 7. Task Runtimeï¼Server æå¨ï¼ |
|---|
| | | |
|---|
| | | ### 7.1 èè´£ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å è½½ Task Definition |
|---|
| | | å建 TaskInstance |
|---|
| | | æ Step æ¨è¿ |
|---|
| | | 设置 Connection.interaction_mode |
|---|
| | | ä¸å skill.execute / MCP |
|---|
| | | çå¾
Device Event æ ASR å®æ |
|---|
| | | å StepInstance |
|---|
| | | å³å®ä¸ä¸æ¥ / ç»æ / 失败 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿è¡æ¶å¯æ´»å¨ `ConnectionHandler` æç `TaskRuntime` 对象ï¼æ¯è¿æ¥ä¸ä¸ªå½å Instanceï¼ãæä¹
åå¨ manager-apiã |
|---|
| | | |
|---|
| | | ### 7.2 ç¶æ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | CREATED â READY â RUNNING |
|---|
| | | â WAITING_DEVICE | WAITING_USER | WAITING_AI |
|---|
| | | â COMPLETED | FAILED | CANCELLED | TIMEOUT |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 设å¤åªç¼å RUNNING çå½å stepï¼éè¿å Server éæ¾æç»æã |
|---|
| | | |
|---|
| | | ### 7.3 Definition ä¸ Instance å离ï¼ä¿ç V1.0ï¼ |
|---|
| | | |
|---|
| | | Definitionï¼æ¨¡æ¿ï¼è±è¯å¤è¯» 10 éï¼ã |
|---|
| | | Instanceï¼ææ¥æ设å¤çä¸æ¬¡æ§è¡ï¼è¿åº¦ãç¶æãç»æï¼ã |
|---|
| | | |
|---|
| | | ### 7.4 Phase 1 ç¹ä¾ |
|---|
| | | |
|---|
| | | 没ææ£å¼ Runtime ä¹å¯ä»¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ件 english_repeat(n, resource) |
|---|
| | | loop: MCP play â wait play.completed æè¶
æ¶ |
|---|
| | | mode=PLAY_AUDIO æé´ç¦æ¢ chat() |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ¯ä¸ºäºéªè¯é¾è·¯ï¼Phase 2 ç¨ JSON Step æ¿æ¢è¯¥å¾ªç¯ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 8. InteractionMode å¨ Server çå«ä¹ |
|---|
| | | |
|---|
| | | Server æ¯ Mode ç **æå®è
**ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Step.mode = PLAY_AUDIO | FOLLOW_READ | CHAT | LISTEN | ⦠|
|---|
| | | â conn.interaction_mode = ⦠|
|---|
| | | â å¯éä¸è¡ mode.setï¼Phase 2ï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | | Mode | Server è¡ä¸º | |
|---|
| | | |------|-------------| |
|---|
| | | | CHAT | ç°æ `startToChat` / `chat()` | |
|---|
| | | | PLAY_AUDIO | ä¸ ASR 对è¯ãä¸ LLMï¼é©±å¨ playï¼çå®æäºä»¶ | |
|---|
| | | | FOLLOW_READ | å
playï¼åæ¶å¬ï¼ASR ææ¬è¿ Stepï¼**ä¸**è¿ Dialogue/LLM | |
|---|
| | | | LISTEN | åª ASRï¼ç»æç»å½å Step | |
|---|
| | | | QUIZ/GAME | Phase 3ï¼åºé¢å¨ Serverï¼è®¾å¤ CHAT æ TTS+LISTEN | |
|---|
| | | |
|---|
| | | V1.0 çãInteraction Runtime ä½ä¸ºç¬ç«æå¡ãé级为 **ConnectionHandler ä¸ç mode å段 + TaskRuntime çç¥è¡¨**ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 9. é³é¢æ¶æ |
|---|
| | | |
|---|
| | | ### 9.1 ç°ç¶ |
|---|
| | | |
|---|
| | | - å
¥ï¼Opus æµï¼å
é¨ ASR PCM 16k |
|---|
| | | - åºï¼Opusï¼hello éå¸¸è§ 24k / 60msï¼ |
|---|
| | | - è½åºï¼`chat_history_conf==2` æ¶ PCM/Opus **转 WAV** è¿ `ai_agent_chat_audio.audio`ï¼æ³¨éå opusï¼å®é
WAVï¼ |
|---|
| | | - `audio_id` ä»
å¨ä¸æ¥ååå¨ï¼history æ `session_id`ï¼**æ ** task/step |
|---|
| | | |
|---|
| | | ### 9.2 ååï¼ä¿ç V1.0ï¼ |
|---|
| | | |
|---|
| | | > è·è¯»/è¯ä¼°ä¸è¦åªå ASR ææ¬ãåå§é³é¢æ¯å¯éåæèµäº§ã |
|---|
| | | |
|---|
| | | ä½ Phase 1 å¤è¯» **å¯ä»¥ä¸åç¨æ·é³é¢**ï¼ç¨æ·å¯è½æ²¡è¯´è¯ï¼ã |
|---|
| | | |
|---|
| | | ### 9.3 æ¼è¿ |
|---|
| | | |
|---|
| | | **Phase 2 æå°ï¼** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ²¿ç¨ reportHandle WAV ä¸æ¥ |
|---|
| | | å段å¢å task_instance_id, step_instance_id, purpose=follow_read |
|---|
| | | audio_id åå
¥ step_instance |
|---|
| | | ä»å¯æå BLOB |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **Phase 2+ ç®æ ï¼V1.0 对象åå¨ï¼ï¼** |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | DB: audio_id, user/device, task/step, storage_key, duration, format, sample_rate |
|---|
| | | Blob: 对象åå¨ raw wav/opus |
|---|
| | | ä¸è¦æé¿æ课ç¨é³é¢ç»§ç»å å¨ MySQL LONGBLOB |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **Segmentï¼** è·è¯»ä¸æ¥ä¸æ®µç¨æ·é³é¢å³å¯ï¼åè¯çº§ååæ¾å°è¯ä¼°æ¨¡åéè¦æ¶ååãä¸è¦ Phase 2 å°±ä¸å®æ´ AudioSession æ ã |
|---|
| | | |
|---|
| | | **设å¤æ件å½é³ / pending ä¸ä¼ ï¼** Client Phase 3ï¼Server é¢ç upload APIï¼ä¸é»å¡è·è¯» v0ã |
|---|
| | | |
|---|
| | | æ¶é´æ³ï¼è¿æ¥çº§è¶³å¤å v0ï¼NTP ä¸ç²¾ç¡® latency æ¾è¯ä¼° Phase 3ãSening å·²ææ¶éåæ¥æ令ï¼å¯å¤ç¨ï¼ä¸ç® Server æ°æå¡ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 10. æ°æ®æ¨¡å |
|---|
| | | |
|---|
| | | ### 10.1 ç°æ表ï¼ä¿çï¼Chat 继ç»ç¨ï¼ |
|---|
| | | |
|---|
| | | | 表 | ä½ç¨ | |
|---|
| | | |----|------| |
|---|
| | | | `sys_user` | è´¦å· | |
|---|
| | | | `ai_device` | 设å¤ï¼`user_id` `agent_id` `mac_address` | |
|---|
| | | | `ai_agent` | 人设ä¸æ¨¡åï¼`chat_history_conf` | |
|---|
| | | | `ai_agent_chat_history` | ä¼è¯ææ¬ï¼`session_id` `audio_id` `chat_type` | |
|---|
| | | | `ai_agent_chat_audio` | é³é¢ BLOB | |
|---|
| | | | `ai_agent_plugin_mapping` | Agent â æ件 | |
|---|
| | | | `ai_ota` / 模å / RAG ç | ä¸ Task æ å
³ï¼ä¸å¨ | |
|---|
| | | |
|---|
| | | Chat åå² **ä¸è¦**æ¹è¯ä¹å»å Taskï¼ç¨æ°è¡¨æå¯ç©ºå¤é®æ©å±ã |
|---|
| | | |
|---|
| | | ### 10.2 Phase 2 æå°æ°è¡¨ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | task_definition id, app_id(å¯ç©º), name, version, json_body, interaction é»è®¤ mode |
|---|
| | | task_instance id, device_id, task_definition_id, session_id, |
|---|
| | | status, current_step, started_at, ended_at |
|---|
| | | step_instance id, task_instance_id, step_index, mode, status, |
|---|
| | | audio_id, asr_text, result_json, started_at, ended_at |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第ä¸æ `json_body` å¯å
åµ stepsï¼ä¸å¿
å
æ `task_step` å®ä¹è¡¨ã |
|---|
| | | |
|---|
| | | ### 10.3 Phase 3+ï¼V1.0 æ¸
åçåéï¼æéï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | app / app_version / app_subscription |
|---|
| | | plan / schedule |
|---|
| | | audio_segment # ä» chat_audio è¿åº |
|---|
| | | assessment / assessment_metric |
|---|
| | | user_metric_baseline |
|---|
| | | daily_summary |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | V1.0 ç emotion_signalãdeveloper 宽表 **ä¸è¦**åå
¥ Phase 2 DDLã |
|---|
| | | |
|---|
| | | Child å®ä½ï¼ç°ç½æ¯ User+DeviceãPhase 3 è¥éè¦ãä¸ä¸ªå®¶é¿å¤ä¸ªå©åãåæ `child`ï¼Phase 1â2 ç¨ `user_id + device_id`ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 11. Assessment |
|---|
| | | |
|---|
| | | ä¿ç V1.0 ååï¼æ¶ç´§è½å°ã |
|---|
| | | |
|---|
| | | ### 11.1 ä¸å±ï¼æ¦å¿µä¿çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | L1 ä»»å¡å¯¹é / ææ¬å¹é
â Phase 2 v0 |
|---|
| | | L2 è¯é³ç¹å¾ï¼è¯éãåé¡¿ï¼ â å¼æ¥ï¼Phase 3 |
|---|
| | | L3 è¶å¿ / åºçº¿ / åä¸åº¦ â è¦æè¶³å¤ Instance |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ
绪åªå **ä¿¡å·** ä¸æ¯è¯æï¼V1.0 §20 ä¿çï¼ï¼ä¸ **æäº** L1ã |
|---|
| | | |
|---|
| | | ### 11.2 è±è¯è·è¯» v0ï¼Phase 2ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | expected_text vs ASR text |
|---|
| | | â åç¡®ç / æ¯å¦è¯´å® |
|---|
| | | â å³æ¶ TTS åé¦ï¼èµ°ç°æ TTSï¼ |
|---|
| | | â ç»æåå
¥ step_instance.result_json |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸å¨ Phase 2 ä¸åé³æ¨¡åãé³ç´ 对é½ãword timingï¼æåå ASR åå§ JSONï¼è¯ä¼°å¨åè¡¥ï¼ã |
|---|
| | | |
|---|
| | | ### 11.3 å£ç® / åè / æè¯ |
|---|
| | | |
|---|
| | | å½ Phase 3+ Task Definitionï¼ç¨ QUIZ/GAME + ASR 解æã**ä¸è¦**为å®ä»¬å
æ¹ Clientã |
|---|
| | | |
|---|
| | | ### 11.4 è¯ä¼°çæ¬åï¼ä¿ç V1.0ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | assessment.model_version |
|---|
| | | åä¸ step å¯è¿½å æ°è¯ä¼°è¡ï¼ç¦æ¢è¦çæ§ç»æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®æ¶è·¯å¾åªå L1ï¼L2+ èµ°éåï¼é¿å
å¡ä½ TTSã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 12. åè®®ä¸ API |
|---|
| | | |
|---|
| | | ### 12.1 设å¤åè®® |
|---|
| | | |
|---|
| | | Phase 1ï¼åªå¢å **MCP è°ç¨**ï¼å·²æ `type=mcp`ï¼ãServer ä¸»å¨ `tools/call` ææ¾ã |
|---|
| | | |
|---|
| | | Phase 2ï¼ä¸ Client Plan ç¸åç `task.*` / `mode.set` / å®æäºä»¶ãPython `TextMessageType` å¢å æ举ï¼Handler ä¸ `hello/listen/mcp` 并åã |
|---|
| | | |
|---|
| | | äºè¿å¶ Opus **ä¸æ¹**ãMQTT ç½å
³ 16 åè头é»è¾ä¿æã |
|---|
| | | |
|---|
| | | ### 12.2 HTTP APIï¼manager-apiï¼ |
|---|
| | | |
|---|
| | | Phase 1ï¼å¯æ æ° APIï¼æ件 + 管ç端ç°æé
置足å¤è§¦å Demoï¼ã |
|---|
| | | |
|---|
| | | Phase 2ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | POST /task/instance å¯å¨ï¼deviceId, taskId, paramsï¼ |
|---|
| | | GET /task/instance/{id} æ¥è¯¢ç¶æä¸ steps |
|---|
| | | POST /agent/chat-history/report æ©å±å¯é task/step å段ï¼å
¼å®¹æ§å®¢æ·ç«¯ï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Phase 3ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | /apps /resources /schedule /assessment /reports |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | V1.0 ä¸æ¬¡éºå¼ `/analytics` `/developer` ââ åç½®ã |
|---|
| | | |
|---|
| | | ä¸æ¥ä»èµ°ç°æ `manage_api_client.report`ï¼é¿å
åé ä¸æ¡è天管éã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 13. Plugin / MCP å¤ç¨ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Server Plugin ç¼æãä¸å¡å¤æãè° ASR ç»æ |
|---|
| | | Device MCP ææ¾ãé³éãå±ãä¸è½½ï¼è®¾å¤ Skillï¼ |
|---|
| | | Server MCP å¤é¨å·¥å
·ï¼ä¸ Task æ 强ç»å® |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è±è¯å¤è¯» Phase 1ï¼Server Plugin è° Device MCPã |
|---|
| | | ä¸è¦æå¤è¯»åè¿åºä»¶ï¼ä¹ä¸è¦æ°å»ºä¸ `UnifiedToolHandler` å¹³è¡ç第äºå¥å·¥å
·æ»çº¿ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 14. smartlearning å¸æ¶çç¥ |
|---|
| | | |
|---|
| | | | å·²æè½å | å½å± | |
|---|
| | | |----------|------| |
|---|
| | | | è¯¾ç¨ JSON / manifest | Phase 3 App Resourceï¼æå¨è¿å
¥ manager-api | |
|---|
| | | | MQTT DOWNLOAD / PLAY_MEDIA | Client æ¿çº§å·²å®ç°ï¼Server æç»ç» Xiaozhi è¿æ¥ä¸ååçè¯ä¹ | |
|---|
| | | | åºæ¯è¯å« HTTP | ä¸è¿å
¥ Xiaozhi 主é¾è·¯ï¼é¤é产åæç¡®è¦ | |
|---|
| | | |
|---|
| | | **ç¦æ¢**让 Xiaozhi Server ä¸ smartlearning é¿æåååä¸è®¾å¤ç¶æãPhase 1 Demo è¥åªè½è· Seningï¼å
许ææ¶æ snlpï¼ä½ Coding Plan å¿
é¡»æ æææ¯åºã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 15. åé¶æ®µï¼Serverï¼ |
|---|
| | | |
|---|
| | | ä¸ Client Plan ç¼å·ä¸è´ã |
|---|
| | | |
|---|
| | | ### Phase 1 â å¤è¯»ç¼æï¼ä¸ç ´å Chat |
|---|
| | | |
|---|
| | | - `ConnectionHandler.interaction_mode`ï¼`CHAT` | `PLAY_AUDIO` |
|---|
| | | - PLAY_AUDIO æ¶ï¼ä¸ `chat()`ãä¸æ课ç¨å½ç¨æ·æå¾ |
|---|
| | | - æ件ï¼å¾ªç¯ N 次 MCP/`PLAY_MEDIA`ï¼æçä»·ï¼ |
|---|
| | | - **ä¸å»º** Task 表ãä¸è¯ä¼°ãä¸æ¹ history è¯ä¹ãä¸æå¾®æå¡ |
|---|
| | | - Chat åå½ï¼mode å CHAT æè¿æ¥ä»æç°ç½ listen |
|---|
| | | |
|---|
| | | éªæ¶ï¼ç®¡ç端æä¸å¥è§¦åå¯è®©è®¾å¤å¤è¯»ï¼åæ¶å¤éè天ä¸ç°ç½ä¸è´ã |
|---|
| | | |
|---|
| | | ### Phase 2 â Task Runtime åç + è·è¯» v0 |
|---|
| | | |
|---|
| | | - `task_definition` / `task_instance` / `step_instance` |
|---|
| | | - åè®® `task.*` + `mode.set`ï¼è¥ MCP ä¸å¤è¡¨è¾¾æ¥è¿ï¼ |
|---|
| | | - FOLLOW_READï¼ASR åå
¥ stepï¼æ©å± report å
³è audio_id |
|---|
| | | - Assessment v0ï¼ææ¬æ¯å¯¹ + ç»æ JSON |
|---|
| | | - ä»ä¸è¦æ±å¯¹è±¡åå¨ãåºçº¿ãæ
绪ã家é¿å°ç¨åº |
|---|
| | | |
|---|
| | | éªæ¶ï¼ä¸æ¬¡è·è¯» Instance å¯æ¥è¯¢å°å step çé³é¢ä¸ææ¬ä¸åæ°ã |
|---|
| | | |
|---|
| | | ### Phase 3 â å¹³å°ä¸è¯ä¼°æ·±å |
|---|
| | | |
|---|
| | | - App / Resource Syncï¼å¸æ¶è¯¾ç¨æ¸
åï¼ |
|---|
| | | - Schedule / Planï¼å®¶é¿ãæ¯å¤© 19:30ãï¼ |
|---|
| | | - é³é¢è¿å¯¹è±¡åå¨ |
|---|
| | | - å¼æ¥ Voice Featureï¼å¯é Baseline |
|---|
| | | - QUIZ çæ° Task åªå Definitionï¼ä¸å åºä»¶ä¸å¡ |
|---|
| | | - åå¾åï¼Emotion SignalãSubscriptionãDeveloper APIï¼V1.0 ååæ®µï¼ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 16. ç¦æ¢äºé¡¹ï¼Serverï¼ |
|---|
| | | |
|---|
| | | 1. 为 Task éå ASR/TTS Provider æ WS å¸§æ ¼å¼ |
|---|
| | | 2. ç¨ `play_music` åå
课ç¨å¤è¯» |
|---|
| | | 3. æ Task è¡å¡è¿ `ai_agent_chat_history` å½æ£å¼æ¨¡å |
|---|
| | | 4. Phase 1 ä¸æ
绪/å¿ç/åä¸åº¦å¤§ç |
|---|
| | | 5. Phase 1 æå¾®æå¡ / ä¸ç¬ç« MQ è¯ä¼°é群 |
|---|
| | | 6. æ¹ hello/listen/tts å段å«ä¹ |
|---|
| | | 7. 让设å¤æ为 Step æå¨ |
|---|
| | | 8. æªè¯» `ConnectionHandler` å°±æ°åä¸å¥å¯¹è¯æå¡å¨ |
|---|
| | | 9. ä¸ smartlearning æ°¸ä¹
åæ è°åº¦ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 17. AI Coding 约æ |
|---|
| | | |
|---|
| | | 1. å
读 `connection.py`ã`receiveAudioHandle.py`ã`reportHandle.py`ãAgent ä¸æ¥ APIãç°æ表ç»æã |
|---|
| | | 2. é»è®¤ Chat å¿
é¡»å¯å
³ Task ç¬ç«è¿è¡ã |
|---|
| | | 3. å®æ¶è·¯å¾åªå Mode åæµä¸ L1 è¯ä¼°ï¼éåæå¼æ¥ã |
|---|
| | | 4. æ°è¡¨æ°åè®®é½è¦ååå
¼å®¹æ§è®¾å¤ï¼å¿½ç¥æªç¥ type / å¯ç©º task å段ï¼ã |
|---|
| | | 5. Phase 1 è½ç¨æ件解å³åä¸æ¹ Java DDLã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 18. ä¸ Client Plan çå¥çº¦ |
|---|
| | | |
|---|
| | | Server ä¿è¯ï¼ |
|---|
| | | |
|---|
| | | - CHAT è¿æ¥è¡ä¸ºååå
¼å®¹ |
|---|
| | | - é CHAT mode ä¸è¯¯è§¦å LLM |
|---|
| | | - Instance æå¨ç¶æå¯æ¥è¯¢ |
|---|
| | | - Phase 2 é³é¢è½æå° step |
|---|
| | | |
|---|
| | | Server è¦æ± Clientï¼ |
|---|
| | | |
|---|
| | | - æä¾å¯è°ç¨ç play_audioï¼è³å°ä¸åç®æ æ¿ï¼ |
|---|
| | | - FOLLOW_READ æ¶ç»§ç»ä¸ä¼ Opus |
|---|
| | | - å®æäºä»¶å¯å
³è session / instance |
|---|
| | | - ä¸å¨åºä»¶åè¯ä¼° |
|---|
| | | |
|---|
| | | ç»ä¸éªæ¶é¾ï¼ç®æ æï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Server å建 Instance â mode + skill â Device æ§è¡ â Event/Audio â ASR â Assessment â æ¥è¯¢ Instance |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Phase 1 åªéªè¯ååï¼å建æå¾ â play å¾ªç¯ â Chat ä»å¯ç¨ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 19. ä¸ä¸é¶æ®µ |
|---|
| | | |
|---|
| | | ä¸ä¸æ¥ä¸æ¯å®ç°è¯ä¼°å¹³å°ï¼èæ¯äº§åºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Phase 1 Coding Plan |
|---|
| | | âââ Serverï¼mode æè·¯ + å¤è¯»æ件 |
|---|
| | | âââ Clientï¼å¤ç¨ MCP/PLAY_MEDIA + é³é¢äºæ¥ï¼å·²æ PauseSdCardMp3ï¼ |
|---|
| | | âââ åå½ï¼ç°æ Chat E2E |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | V1.0 ä¸ç家é¿æ¥åå级ã跨任å¡åæãå¼åè
åæçï¼ä¿ç为 Phase 3 以åç产åéå½ï¼ä¸è¿å
¥è¿ææ¶ææ½å·¥å¾ã |
|---|
| New file |
| | |
|---|
| | | # Xiaozhi AI å¼æ¾åºç¨å¹³å° |
|---|
| | | # 顶å±æ¶æä¸ AI Coding Master Plan |
|---|
| | | |
|---|
| | | > Version: V1.0 |
|---|
| | | > Date: 2026-08-30 |
|---|
| | | > Purpose: ä½ä¸º Xiaozhi AI 硬件端ãServer 端ãå°ç¨åºç«¯ä»¥åæªæ¥å¼åè
å¼æ¾å¹³å°çæé«å±æ¶æè®¾è®¡ä¸ AI Coding æ»ä½æ导ææ¡£ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 1. 项ç®æç»ç®æ |
|---|
| | | |
|---|
| | | å°ç°æ Xiaozhi AI ä»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | AI 对è¯ç¡¬ä»¶ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | éæ¥åçº§ä¸ºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | AI Application Terminal |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»å½¢æç±»ä¼¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææº |
|---|
| | | âââ å¯ä»¥å®è£
å¤ä¸ª App |
|---|
| | | âââ æ¯ä¸ª App å
å«å¤ä¸ª Task |
|---|
| | | âââ æ¯ä¸ª Task å
å«å¤ä¸ª Step |
|---|
| | | âââ Step è°ç¨åºå±è½å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä½æ¯ Xiaozhi ä¸ææºçæ ¸å¿åºå«æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææºï¼ |
|---|
| | | ç¨æ·ä¸»å¨éæ© App â 主å¨æä½ |
|---|
| | | |
|---|
| | | Xiaozhiï¼ |
|---|
| | | 家é¿é
ç½® â ç³»ç»è°åº¦ â Xiaozhi 主å¨æ§è¡ â å¿ç«¥åä¸ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ Xiaozhi æç»ä¸æ¯å纯çè天æºå¨äººï¼èæ¯ï¼ |
|---|
| | | |
|---|
| | | > **ä¸ä¸ªå¯ä»¥æ¿è½½å¤ä¸ª AI Appï¼å¹¶æ ¹æ®å®¶é¿é
置主å¨æ§è¡ Taskï¼åæ¶æç»è®°å½ãè¯ä¼°ååæå¿ç«¥è¡ä¸ºç AI ç»ç«¯å¹³å°ã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 2. æç»äº§åå½¢æ |
|---|
| | | |
|---|
| | | æç»æ´ä¸ªç³»ç»ç±ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Xiaozhi Device |
|---|
| | | â |
|---|
| | | â |
|---|
| | | Xiaozhi Server |
|---|
| | | â |
|---|
| | | âââ App Platform |
|---|
| | | âââ Task Platform |
|---|
| | | âââ Assessment Platform |
|---|
| | | âââ User Data Platform |
|---|
| | | âââ Developer Platform |
|---|
| | | âââ Subscription Platform |
|---|
| | | âââ Parent Platform |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç»æã |
|---|
| | | |
|---|
| | | ç¨æ·å
³ç³»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å¹³å° |
|---|
| | | â |
|---|
| | | âââ App A |
|---|
| | | â âââ Task 1 |
|---|
| | | â âââ Task 2 |
|---|
| | | â âââ Task 3 |
|---|
| | | â |
|---|
| | | âââ App B |
|---|
| | | â âââ Task 1 |
|---|
| | | â âââ Task 2 |
|---|
| | | â |
|---|
| | | âââ App C |
|---|
| | | âââ ... |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸ä¸ªå©åå¯ä»¥è®¢é
å¤ä¸ª Appã |
|---|
| | | |
|---|
| | | ä½æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åä¸æ¶é´ |
|---|
| | | åªè½è¿è¡ä¸ä¸ª App / Task Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ä¸è¦æ± Xiaozhi 硬件åæ¶è¿è¡å¤ä¸ªå¤æåºç¨ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 3. æ ¸å¿æ¶æåå |
|---|
| | | |
|---|
| | | æ´ä¸ªç³»ç»å¿
é¡»åæ以ä¸ååã |
|---|
| | | |
|---|
| | | ## 3.1 App æ¯ä¸å¡è½åçå®¹å¨ |
|---|
| | | |
|---|
| | | App ä¸åºè¯¥çåäºç¡¬ä»¶åè½ã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯å¦ä¹ App |
|---|
| | | åå¿å½ App |
|---|
| | | çæºæ¸¸æ App |
|---|
| | | AI 主æ人 App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | é½æ¯é»è¾åºç¨ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3.2 Task æ¯ç³»ç»çæ£æ§è¡çåºæ¬ä¸å¡åä½ |
|---|
| | | |
|---|
| | | ç³»ç»æç»æ§è¡çæ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Scenario |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯ App |
|---|
| | | âââ è±è¯è·è¯» 10 次 |
|---|
| | | âââ åè¯å¤ä¹ |
|---|
| | | âââ è±è¯å¬å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å
¶ä¸ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯è·è¯» 10 次 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å°±æ¯ä¸ä¸ª Taskã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 4. Scenario / Interaction Mode çæç»å®ä¹ |
|---|
| | | |
|---|
| | | æ¬é¡¹ç®ä¸ä¸åæ Scenario çè§£ä¸ºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä¸å¡åºæ¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èå®ä¹ä¸ºï¼ |
|---|
| | | |
|---|
| | | > **Device Interaction Mode** |
|---|
| | | |
|---|
| | | å³ï¼ |
|---|
| | | |
|---|
| | | > Xiaozhi å¨æ§è¡ Task æ¶ï¼åºè¯¥ä»¥ä»ä¹æ¹å¼ä¸ç¨æ·äº¤äºã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | CHAT |
|---|
| | | PLAY_AUDIO |
|---|
| | | REPEAT |
|---|
| | | FOLLOW_READ |
|---|
| | | QUESTION_ANSWER |
|---|
| | | TRANSLATION |
|---|
| | | CALCULATION |
|---|
| | | MEMORY |
|---|
| | | GAME |
|---|
| | | SILENT |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | æå® Interaction Mode |
|---|
| | | â |
|---|
| | | Device Runtime |
|---|
| | | â |
|---|
| | | 以对åºæ¹å¼æ§è¡ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Taskï¼ |
|---|
| | | å¬è±è¯å¤è¯»æº 10 é |
|---|
| | | |
|---|
| | | interactionModeï¼ |
|---|
| | | PLAY_AUDIO |
|---|
| | | |
|---|
| | | stepsï¼ |
|---|
| | | ææ¾ â çå¾
â ææ¾ â çå¾
... |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸éè¦é¢å¤å»ºç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Scenario |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ ·çä¸å¡å±çº§ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 5. Task æ¯æ ¸å¿æ§è¡æ½è±¡ |
|---|
| | | |
|---|
| | | ç»ä¸æ¨¡åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | âââ Task |
|---|
| | | âââ Step |
|---|
| | | âââ Action / Skill |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | English App |
|---|
| | | âââ English Repeat 10 Times |
|---|
| | | âââ Play Audio |
|---|
| | | âââ Wait |
|---|
| | | âââ Record |
|---|
| | | âââ Evaluate |
|---|
| | | âââ Repeat |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 6. Step æ¯çæ£çæ§è¡åå
|
|---|
| | | |
|---|
| | | Task è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åä»ä¹ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Step è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å
·ä½æä¹ä¸æ¥ä¸æ¥å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Taskï¼ |
|---|
| | | è±è¯è·è¯» |
|---|
| | | |
|---|
| | | Step 1ï¼ |
|---|
| | | ææ¾æ åé³é¢ |
|---|
| | | |
|---|
| | | Step 2ï¼ |
|---|
| | | çå¾
ç¨æ·è¯´è¯ |
|---|
| | | |
|---|
| | | Step 3ï¼ |
|---|
| | | å½é³ |
|---|
| | | |
|---|
| | | Step 4ï¼ |
|---|
| | | ASR |
|---|
| | | |
|---|
| | | Step 5ï¼ |
|---|
| | | æ¯è¾ |
|---|
| | | |
|---|
| | | Step 6ï¼ |
|---|
| | | è¯å |
|---|
| | | |
|---|
| | | Step 7ï¼ |
|---|
| | | åé¦ |
|---|
| | | |
|---|
| | | Step 8ï¼ |
|---|
| | | è¿å
¥ä¸ä¸å¥ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 7. Skill / Atomic Capability |
|---|
| | | |
|---|
| | | Step ä¸ç´æ¥æä½ç¡¬ä»¶ã |
|---|
| | | |
|---|
| | | Step è°ç¨ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Skill |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | play_audio |
|---|
| | | record_audio |
|---|
| | | listen |
|---|
| | | speak |
|---|
| | | wait |
|---|
| | | count |
|---|
| | | asr |
|---|
| | | evaluate |
|---|
| | | display |
|---|
| | | led |
|---|
| | | button |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | â |
|---|
| | | Device Capability |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 8. Device ä¸ Server çè责边ç |
|---|
| | | |
|---|
| | | ## Device |
|---|
| | | |
|---|
| | | è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ§è¡ |
|---|
| | | ææ¾ |
|---|
| | | å½é³ |
|---|
| | | éé |
|---|
| | | ç¼å |
|---|
| | | éä¿¡ |
|---|
| | | åºç¡ç¶æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## Server |
|---|
| | | |
|---|
| | | è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è°åº¦ |
|---|
| | | Task |
|---|
| | | App |
|---|
| | | ç¨æ· |
|---|
| | | æ°æ® |
|---|
| | | ASR |
|---|
| | | AI |
|---|
| | | è¯ä¼° |
|---|
| | | åæ |
|---|
| | | é¿æè¶å¿ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 9. App ä¸åºè¯¥æ为硬件åºä»¶ |
|---|
| | | |
|---|
| | | æªæ¥æ°å¢ Appï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä¸è½½é
ç½® |
|---|
| | | ä¸è½½èµæº |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä¿®æ¹ C++ åºä»¶ |
|---|
| | | éæ° OTA |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | çæ³æ
åµä¸ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ°å¢ App |
|---|
| | | â |
|---|
| | | App Manifest |
|---|
| | | Task Definition |
|---|
| | | Step Definition |
|---|
| | | Resource |
|---|
| | | Assessment Definition |
|---|
| | | â |
|---|
| | | Server |
|---|
| | | â |
|---|
| | | Device Sync |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å³å¯è¿è¡ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 10. App Manifest |
|---|
| | | |
|---|
| | | æ¯ä¸ä¸ª App é½æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | app.json |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ä¹ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | appId |
|---|
| | | name |
|---|
| | | version |
|---|
| | | developerId |
|---|
| | | description |
|---|
| | | icon |
|---|
| | | tasks |
|---|
| | | resources |
|---|
| | | permissions |
|---|
| | | assessment |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 11. Task Definition |
|---|
| | | |
|---|
| | | Task å®ä¹ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | taskId |
|---|
| | | name |
|---|
| | | description |
|---|
| | | version |
|---|
| | | steps |
|---|
| | | parameters |
|---|
| | | interactionMode |
|---|
| | | assessment |
|---|
| | | resources |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```json |
|---|
| | | { |
|---|
| | | "taskId": "english_repeat_10", |
|---|
| | | "interactionMode": "PLAY_AUDIO", |
|---|
| | | "parameters": { |
|---|
| | | "repeatCount": 10 |
|---|
| | | } |
|---|
| | | } |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 12. Resource |
|---|
| | | |
|---|
| | | App å¯ä»¥æºå¸¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | MP3 |
|---|
| | | WAV |
|---|
| | | å¾ç |
|---|
| | | JSON |
|---|
| | | ææ¬ |
|---|
| | | é
ç½® |
|---|
| | | 模ååæ° |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿äºå±äºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Application Resource |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯åºä»¶ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 13. OTA ä¸ App æ´æ°å¿
须彻åºåºå |
|---|
| | | |
|---|
| | | ## OTA |
|---|
| | | |
|---|
| | | ç¨äºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device Firmware |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åºå±é©±å¨ |
|---|
| | | éä¿¡åè®® |
|---|
| | | é³é¢é©±å¨ |
|---|
| | | ç³»ç» Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## App Sync |
|---|
| | | |
|---|
| | | ç¨äºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | Config |
|---|
| | | Resource |
|---|
| | | Audio |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ°å¢è±è¯ App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åªéè¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Sync App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ é OTAã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 14. SD Card çå®ä½ |
|---|
| | | |
|---|
| | | 16GB SD Card 主è¦ç¨äºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App Resources |
|---|
| | | Audio Resources |
|---|
| | | Cache |
|---|
| | | Offline Tasks |
|---|
| | | Pending Upload Audio |
|---|
| | | Temporary Files |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | /sdcard/ |
|---|
| | | apps/ |
|---|
| | | resources/ |
|---|
| | | audio/ |
|---|
| | | cache/ |
|---|
| | | recordings/ |
|---|
| | | pending/ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 15. Server Task Runtime |
|---|
| | | |
|---|
| | | Server å¿
须建ç«ç»ä¸ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å建 TaskInstance |
|---|
| | | å¯å¨ Task |
|---|
| | | æ¨è¿ Step |
|---|
| | | çå¾
Device |
|---|
| | | å¤ç Event |
|---|
| | | å¤æ Step å®æ |
|---|
| | | è¿å
¥ä¸ä¸ Step |
|---|
| | | ç»æ Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 16. Task Instance |
|---|
| | | |
|---|
| | | ä¸è¦ç´æ¥ä¿®æ¹ Task Definitionã |
|---|
| | | |
|---|
| | | æ¯ä¸æ¬¡å®é
æ§è¡ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | TaskInstance |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Taskï¼ |
|---|
| | | è±è¯è·è¯» 10 次 |
|---|
| | | |
|---|
| | | TaskInstanceï¼ |
|---|
| | | 2026-08-30 19:30 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¿åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | startTime |
|---|
| | | endTime |
|---|
| | | status |
|---|
| | | currentStep |
|---|
| | | result |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 17. Scheduler |
|---|
| | | |
|---|
| | | 家é¿éè¿å°ç¨åºé
ç½®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ¯å¤© 19:30 |
|---|
| | | è±è¯å¦ä¹ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Serverï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Schedule |
|---|
| | | â |
|---|
| | | Scheduler |
|---|
| | | â |
|---|
| | | TaskInstance |
|---|
| | | â |
|---|
| | | Device |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ï¼ |
|---|
| | | |
|---|
| | | > 家é¿é
ç½®çæ¯âä»ä¹æ¶ååä»ä¹ Taskâï¼ä¸æ¯ç´æ¥æ§å¶ç¡¬ä»¶ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 18. Plan |
|---|
| | | |
|---|
| | | è¿ä¸æ¥å
è®¸ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Plan |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ¯æ¥å¦ä¹ 计å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å
å«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 19:30 è±è¯è·è¯» |
|---|
| | | 19:45 å£ç® |
|---|
| | | 19:55 è®°å¿æ¸¸æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Plan |
|---|
| | | â |
|---|
| | | Schedule |
|---|
| | | â |
|---|
| | | TaskInstance |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 19. Device Runtime |
|---|
| | | |
|---|
| | | Device åªéè¦ç解æ ååè®®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | APP_SYNC |
|---|
| | | TASK_START |
|---|
| | | STEP_START |
|---|
| | | SKILL_EXECUTE |
|---|
| | | TASK_PAUSE |
|---|
| | | TASK_RESUME |
|---|
| | | TASK_STOP |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 以åå Server ä¸æ¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | TASK_STARTED |
|---|
| | | STEP_STARTED |
|---|
| | | STEP_COMPLETED |
|---|
| | | AUDIO_PLAYED |
|---|
| | | SPEECH_STARTED |
|---|
| | | SPEECH_STOPPED |
|---|
| | | TASK_COMPLETED |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 20. Interaction Runtime |
|---|
| | | |
|---|
| | | Device Runtime å
é¨å¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Interaction Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ ¹æ®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | interactionMode |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | éæ©äº¤äºæ¹å¼ã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | PLAY_AUDIO |
|---|
| | | â Audio Player |
|---|
| | | |
|---|
| | | FOLLOW_READ |
|---|
| | | â Audio Player |
|---|
| | | â Recorder |
|---|
| | | â VAD |
|---|
| | | |
|---|
| | | CHAT |
|---|
| | | â ASR |
|---|
| | | â LLM |
|---|
| | | â TTS |
|---|
| | | |
|---|
| | | TRANSLATION |
|---|
| | | â ASR |
|---|
| | | â Translation |
|---|
| | | â TTS |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 21. Chat ä¸åæ¯å¯ä¸ Runtime |
|---|
| | | |
|---|
| | | ç°æ Xiaozhi å¾å¯è½ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Audio |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | LLM |
|---|
| | | â |
|---|
| | | TTS |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿ç§æ¶æã |
|---|
| | | |
|---|
| | | å¿
é¡»å级æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Runtime |
|---|
| | | â |
|---|
| | | Interaction Runtime |
|---|
| | | â |
|---|
| | | ââââââââââ¬âââââââââ¬âââââââââ¬âââââââââ |
|---|
| | | Chat Audio Repeat Game ... |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Chat åªæ¯å
¶ä¸ä¸ç§è½åã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 22. Server Audio Architecture |
|---|
| | | |
|---|
| | | Server ä¿åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Original Audio |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯åªä¿åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ASR Text |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç»æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | AudioSession |
|---|
| | | âââ AudioSegment |
|---|
| | | âââ Original Audio |
|---|
| | | âââ ASR |
|---|
| | | âââ Voice Feature |
|---|
| | | âââ Assessment |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 23. ASR |
|---|
| | | |
|---|
| | | ç»ä¸ ASR Serviceï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Audio |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Transcript |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¿åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | text |
|---|
| | | confidence |
|---|
| | | word timing |
|---|
| | | word confidence |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 24. Voice Analysis |
|---|
| | | |
|---|
| | | ç»ä¸æåï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | speechDuration |
|---|
| | | pauseCount |
|---|
| | | pauseDuration |
|---|
| | | speechRate |
|---|
| | | responseLatency |
|---|
| | | volume |
|---|
| | | pitch |
|---|
| | | energy |
|---|
| | | prosody |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 25. Task Assessment |
|---|
| | | |
|---|
| | | ä¸å Task 使ç¨ä¸åè¯ä¼°çç¥ã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯è·è¯» |
|---|
| | | â pronunciation |
|---|
| | | â fluency |
|---|
| | | â accuracy |
|---|
| | | |
|---|
| | | æ°ååè |
|---|
| | | â sequenceAccuracy |
|---|
| | | â memoryAccuracy |
|---|
| | | â responseTime |
|---|
| | | |
|---|
| | | å£ç® |
|---|
| | | â accuracy |
|---|
| | | â responseTime |
|---|
| | | â difficulty |
|---|
| | | |
|---|
| | | æè¯æ¥é¾ |
|---|
| | | â correctness |
|---|
| | | â responseTime |
|---|
| | | â continuity |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 26. Metric System |
|---|
| | | |
|---|
| | | ææç»ææç»ç»ä¸ä¸ºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Metric |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | accuracy |
|---|
| | | response_time |
|---|
| | | pause_count |
|---|
| | | speech_rate |
|---|
| | | pronunciation |
|---|
| | | fluency |
|---|
| | | completion |
|---|
| | | participation |
|---|
| | | engagement |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ° App å¯ä»¥æ³¨åæ°ç Metricã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 27. Behavioral Analysis |
|---|
| | | |
|---|
| | | Server 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User Baseline |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | éè¿ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 7 Days |
|---|
| | | 30 Days |
|---|
| | | 90 Days |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å½¢æå¨æåºçº¿ã |
|---|
| | | |
|---|
| | | ç¶åæ¯è¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Today |
|---|
| | | vs |
|---|
| | | Baseline |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 28. Emotion Analysis |
|---|
| | | |
|---|
| | | ç³»ç»å¯ä»¥æä¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Emotion Signal |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä½å¿
é¡»å®ä¹ä¸ºï¼ |
|---|
| | | |
|---|
| | | > **åºäºè¯é³ãè¯è¨åè¡ä¸ºæ°æ®çæ
绪表ç°ä¿¡å·ã** |
|---|
| | | |
|---|
| | | ä¸è½ç´æ¥å®ä¹ä¸ºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å¿çç¾ç
è¯æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¾åºï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | state |
|---|
| | | confidence |
|---|
| | | trend |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 29. Engagement |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Engagement Score |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 综åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ååº |
|---|
| | | å®æ |
|---|
| | | 主å¨æ§ |
|---|
| | | æç»æ§ |
|---|
| | | äºå¨ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å½¢æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 0 ~ 100 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 30. Parent Dashboard |
|---|
| | | |
|---|
| | | 家é¿æç»çå°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä»æ¥ä»»å¡ |
|---|
| | | å®ææ
åµ |
|---|
| | | Task Score |
|---|
| | | è½åææ |
|---|
| | | åä¸åº¦ |
|---|
| | | è¶å¿ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯å¤æææ¯ææ ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 31. Long-term Profile |
|---|
| | | |
|---|
| | | ç³»ç»æç»å½¢æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User Profile |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å
æ¬ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Learning |
|---|
| | | Memory |
|---|
| | | Speech |
|---|
| | | Task Performance |
|---|
| | | Participation |
|---|
| | | Behavior Trend |
|---|
| | | Preference |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ææ Profile é½åºè¯¥æ¥èªï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Event |
|---|
| | | â |
|---|
| | | Metric |
|---|
| | | â |
|---|
| | | Aggregation |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯æ工维æ¤ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 32. Developer Platform |
|---|
| | | |
|---|
| | | æç»å¼æ¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Developer |
|---|
| | | â |
|---|
| | | App |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¼åè
ä¸åºè¯¥ç´æ¥ä¿®æ¹ Device Firmwareã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 33. Developer Sandbox |
|---|
| | | |
|---|
| | | æä¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App SDK |
|---|
| | | Task SDK |
|---|
| | | Assessment SDK |
|---|
| | | Resource SDK |
|---|
| | | API |
|---|
| | | Simulator |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¼åè
å¯ä»¥å¨ PC ä¸æ¨¡æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Xiaozhi Device |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æµè¯ Taskã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 34. App Store |
|---|
| | | |
|---|
| | | æç»å¹³å°å½¢æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App Store |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç¨æ·å¯ä»¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æµè§ App |
|---|
| | | æ¥çä»ç» |
|---|
| | | è¯ç¨ |
|---|
| | | 订é
|
|---|
| | | å®è£
|
|---|
| | | è¯ä»· |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è±è¯å¦ä¹ |
|---|
| | | æ°å¦è®ç» |
|---|
| | | è®°å¿è®ç» |
|---|
| | | æè¯æ¸¸æ |
|---|
| | | AI 主æ人 |
|---|
| | | åå¿å½ |
|---|
| | | æ
äº |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 35. Subscription |
|---|
| | | |
|---|
| | | 订é
å
³ç³»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User |
|---|
| | | â |
|---|
| | | Subscription |
|---|
| | | â |
|---|
| | | App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User |
|---|
| | | â |
|---|
| | | Firmware |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 36. Developer Revenue |
|---|
| | | |
|---|
| | | å¹³å°è´è´£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 订å |
|---|
| | | æ¯ä» |
|---|
| | | 订é
|
|---|
| | | é款 |
|---|
| | | ç»ç® |
|---|
| | | åæ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¼åè
è·å¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Revenue Share |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 37. Permission Architecture |
|---|
| | | |
|---|
| | | å¿
须建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User |
|---|
| | | Child |
|---|
| | | Parent |
|---|
| | | Developer |
|---|
| | | App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¹é´çæéä½ç³»ã |
|---|
| | | |
|---|
| | | å°¤å
¶æ¯å¿ç«¥æ°æ®ã |
|---|
| | | |
|---|
| | | å¼åè
ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | é»è®¤ä¸è½è¯»åå
¶ä» App æ°æ® |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 家é¿ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å¯ä»¥æ¥çå©åæ°æ® |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¹³å°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è´è´£æéæ§å¶ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 38. Privacy |
|---|
| | | |
|---|
| | | å¿ç«¥è¯é³å¿
é¡»éç¹ä¿æ¤ã |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Consent |
|---|
| | | Encryption |
|---|
| | | Access Control |
|---|
| | | Audit Log |
|---|
| | | Retention Policy |
|---|
| | | Data Deletion |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 39. æ°æ®æ ¸å¿é¾è·¯ |
|---|
| | | |
|---|
| | | æç»ç»ä¸æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | TaskInstance |
|---|
| | | â |
|---|
| | | StepInstance |
|---|
| | | â |
|---|
| | | Device Event |
|---|
| | | â |
|---|
| | | Audio |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Voice Feature |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | â |
|---|
| | | Metric |
|---|
| | | â |
|---|
| | | Aggregation |
|---|
| | | â |
|---|
| | | Profile |
|---|
| | | â |
|---|
| | | Report |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ¯æ´ä¸ªç³»ç»æéè¦çæ°æ®é¾è·¯ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 40. Event-Driven Architecture |
|---|
| | | |
|---|
| | | æ ¸å¿ç¶æå°½éç± Event 驱å¨ã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | TaskStarted |
|---|
| | | StepStarted |
|---|
| | | AudioPlayed |
|---|
| | | SpeechStarted |
|---|
| | | SpeechStopped |
|---|
| | | ASRCompleted |
|---|
| | | AssessmentCompleted |
|---|
| | | StepCompleted |
|---|
| | | TaskCompleted |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Event æ为系ç»çéè¦äºå®æ¥æºã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 41. å¼æ¥åæ |
|---|
| | | |
|---|
| | | å®æ¶é¾è·¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device |
|---|
| | | â |
|---|
| | | Server |
|---|
| | | â |
|---|
| | | Task Runtime |
|---|
| | | â |
|---|
| | | å®æ¶åé¦ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¼æ¥é¾è·¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Audio |
|---|
| | | â |
|---|
| | | Queue |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Voice Analysis |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | â |
|---|
| | | Aggregation |
|---|
| | | â |
|---|
| | | Report |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¤æ AI åæä¸è½é»å¡ Task Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 42. 离线è½å |
|---|
| | | |
|---|
| | | Device ç½ç»æå¼æ¶ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Local Cache |
|---|
| | | â |
|---|
| | | æ§è¡ |
|---|
| | | â |
|---|
| | | Recording |
|---|
| | | â |
|---|
| | | SD Card |
|---|
| | | â |
|---|
| | | ç½ç»æ¢å¤ |
|---|
| | | â |
|---|
| | | Sync |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ï¼ |
|---|
| | | |
|---|
| | | > Server æ¯ä¸å¿æ§å¶å¨ï¼ä½ Device å¿
é¡»å
·å¤ä¸å®ç¨åº¦ç Offline Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 43. çæ¬ç®¡ç |
|---|
| | | |
|---|
| | | ææè¿äºå¯¹è±¡å¿
é¡»çæ¬åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Firmware |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | Step |
|---|
| | | Assessment |
|---|
| | | Model |
|---|
| | | Resource |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App v1.2 |
|---|
| | | Task v3 |
|---|
| | | Assessment v2 |
|---|
| | | Model v5 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åå²æ°æ®å¿
é¡»è½å¤è¿½æº¯å°å
·ä½çæ¬ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 44. ä¸è¦çåå²ç»æ |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Pronunciation Model v1 |
|---|
| | | Score = 82 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 以åï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Pronunciation Model v2 |
|---|
| | | Score = 87 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 两个ç»æé½ä¿çã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 45. æ¨èçæ°æ®åºå
³ç³» |
|---|
| | | |
|---|
| | | æ ¸å¿ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | User |
|---|
| | | âââ Child |
|---|
| | | |
|---|
| | | Child |
|---|
| | | âââ Subscription |
|---|
| | | âââ Plan |
|---|
| | | âââ Schedule |
|---|
| | | âââ TaskInstance |
|---|
| | | |
|---|
| | | App |
|---|
| | | âââ Task |
|---|
| | | âââ TaskStep |
|---|
| | | |
|---|
| | | TaskInstance |
|---|
| | | âââ TaskStepInstance |
|---|
| | | âââ DeviceEvent |
|---|
| | | âââ AudioSegment |
|---|
| | | â âââ ASR |
|---|
| | | â âââ VoiceFeature |
|---|
| | | âââ Assessment |
|---|
| | | âââ Metric |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 46. 第ä¸é¶æ®µå¼åç®æ |
|---|
| | | |
|---|
| | | ä¸è¦ä¸å¼å§å¼åå®æ´å¹³å°ã |
|---|
| | | |
|---|
| | | é¦å
建ç«æå°éç¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å®¶é¿ |
|---|
| | | â |
|---|
| | | é
ç½® Task |
|---|
| | | â |
|---|
| | | Server Scheduler |
|---|
| | | â |
|---|
| | | TaskInstance |
|---|
| | | â |
|---|
| | | Xiaozhi |
|---|
| | | â |
|---|
| | | æ§è¡ |
|---|
| | | â |
|---|
| | | å½é³ |
|---|
| | | â |
|---|
| | | Server |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | â |
|---|
| | | 家é¿æ¥çç»æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¿
é¡»å®æ´è·éã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 47. MVP Task |
|---|
| | | |
|---|
| | | 第ä¸é¶æ®µåªå®ç°ä¸ä¸ª Taskï¼ |
|---|
| | | |
|---|
| | | ## Task Aï¼è±è¯å¤è¯» |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææ¾ |
|---|
| | | â |
|---|
| | | çå¾
|
|---|
| | | â |
|---|
| | | ææ¾ |
|---|
| | | â |
|---|
| | | ... |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## Task Bï¼è±è¯è·è¯» |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææ¾æ åé³é¢ |
|---|
| | | â |
|---|
| | | å½é³ |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | åç¡®ç |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## Task Cï¼å£ç® |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææ¾é®é¢ |
|---|
| | | â |
|---|
| | | çå¾
åç |
|---|
| | | â |
|---|
| | | ASR |
|---|
| | | â |
|---|
| | | çæ¡è§£æ |
|---|
| | | â |
|---|
| | | æ£ç¡®ç |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 48. 第äºé¶æ®µ |
|---|
| | | |
|---|
| | | å¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ°ååè |
|---|
| | | æè¯æ¥é¾ |
|---|
| | | ç®åè®°å¿æ¸¸æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 并ç»ä¸ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Engine |
|---|
| | | Step Engine |
|---|
| | | Assessment Engine |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 49. 第ä¸é¶æ®µ |
|---|
| | | |
|---|
| | | å å
¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Voice Feature |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ç°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Pause |
|---|
| | | Speech Rate |
|---|
| | | Response Latency |
|---|
| | | Volume |
|---|
| | | Pitch |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 50. 第åé¶æ®µ |
|---|
| | | |
|---|
| | | å å
¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Baseline |
|---|
| | | Behavior |
|---|
| | | Engagement |
|---|
| | | Participation |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ç°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Today |
|---|
| | | 7 Days |
|---|
| | | 30 Days |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¶å¿ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 51. 第äºé¶æ®µ |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Emotion Signal |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿è¡å®éªæ§åæã |
|---|
| | | |
|---|
| | | è¦æ±ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 模åçæ¬å |
|---|
| | | 置信度 |
|---|
| | | è¶å¿ |
|---|
| | | 人工éªè¯ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸å
许ç´æ¥åå»çè¯æã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 52. 第å
é¶æ®µ |
|---|
| | | |
|---|
| | | å¼æ¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App Manifest |
|---|
| | | Task DSL |
|---|
| | | Assessment SDK |
|---|
| | | Developer API |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ç°ç¬¬ä¸æ¹ Appã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 53. 第ä¸é¶æ®µ |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App Store |
|---|
| | | Subscription |
|---|
| | | Payment |
|---|
| | | Revenue Share |
|---|
| | | Developer Console |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 54. 第å
«é¶æ®µ |
|---|
| | | |
|---|
| | | 建ç«ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Parent AI Assistant |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 家é¿å¯ä»¥è¯¢é®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å©åæè¿å¦ä¹ æä¹æ ·ï¼ |
|---|
| | | |
|---|
| | | åªä¸ª App æåæ¬¢ï¼ |
|---|
| | | |
|---|
| | | è±è¯æè¿æ没æè¿æ¥ï¼ |
|---|
| | | |
|---|
| | | æè¿åä¸åº¦æ没æä¸éï¼ |
|---|
| | | |
|---|
| | | è¿å»ä¸ä¸ªææä»ä¹ææ¾ååï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | AI ä»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Metrics |
|---|
| | | Assessment |
|---|
| | | Trend |
|---|
| | | Profile |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸çæ解éã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 55. AI Coding æ»ä½å®æ½é¡ºåº |
|---|
| | | |
|---|
| | | AI Coding ä¸å
许ä¸æ¬¡æ§éåæ´ä¸ªé¡¹ç®ã |
|---|
| | | |
|---|
| | | å¿
é¡»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Phase 0 |
|---|
| | | 代ç 审计 |
|---|
| | | |
|---|
| | | Phase 1 |
|---|
| | | 建ç«æ ¸å¿ Domain Model |
|---|
| | | |
|---|
| | | Phase 2 |
|---|
| | | Task Runtime |
|---|
| | | |
|---|
| | | Phase 3 |
|---|
| | | Device Runtime |
|---|
| | | |
|---|
| | | Phase 4 |
|---|
| | | App / Resource Sync |
|---|
| | | |
|---|
| | | Phase 5 |
|---|
| | | Audio / ASR |
|---|
| | | |
|---|
| | | Phase 6 |
|---|
| | | Assessment |
|---|
| | | |
|---|
| | | Phase 7 |
|---|
| | | Parent / Schedule |
|---|
| | | |
|---|
| | | Phase 8 |
|---|
| | | Behavior / Metrics |
|---|
| | | |
|---|
| | | Phase 9 |
|---|
| | | Developer Platform |
|---|
| | | |
|---|
| | | Phase 10 |
|---|
| | | Subscription / Marketplace |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ¯ä¸ª Phaseï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åæ |
|---|
| | | â |
|---|
| | | 设计 |
|---|
| | | â |
|---|
| | | å®ç° |
|---|
| | | â |
|---|
| | | æµè¯ |
|---|
| | | â |
|---|
| | | åå½ |
|---|
| | | â |
|---|
| | | æ交 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 56. AI Coding 第ä¸åå |
|---|
| | | |
|---|
| | | AI ç¼ç¨ Agent å¿
é¡»ï¼ |
|---|
| | | |
|---|
| | | > **ä¼å
ç解ç°æ代ç ï¼åä¿®æ¹ä»£ç ã** |
|---|
| | | |
|---|
| | | ç¦æ¢ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 没æé
读代ç |
|---|
| | | â |
|---|
| | | ç´æ¥éæ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¿
é¡»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Repository Audit |
|---|
| | | â |
|---|
| | | Architecture Mapping |
|---|
| | | â |
|---|
| | | Dependency Analysis |
|---|
| | | â |
|---|
| | | Minimal Change |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 57. ä¸å
许æ çç±éåç°æåè½ |
|---|
| | | |
|---|
| | | ç°æ Xiaozhi çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Audio |
|---|
| | | ASR |
|---|
| | | TTS |
|---|
| | | LLM |
|---|
| | | MQTT / WebSocket |
|---|
| | | Device Communication |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è½å¤å¤ç¨çå¿
é¡»å¤ç¨ã |
|---|
| | | |
|---|
| | | æ°æ¶æåºè¯¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å
裹ç°æè½å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èä¸æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å
¨é¨æ¨åéæ¥ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 58. æ°æ§ç³»ç»è¿ç§»çç¥ |
|---|
| | | |
|---|
| | | æ§ç³»ç»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Conversation |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¿çã |
|---|
| | | |
|---|
| | | å¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Runtime |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Runtime |
|---|
| | | âââ Chat Task |
|---|
| | | âââ Repeat Task |
|---|
| | | âââ Follow Read Task |
|---|
| | | âââ Translation Task |
|---|
| | | âââ Game Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å æ¤ï¼ |
|---|
| | | |
|---|
| | | > Chat æç»åªæ¯ Task Runtime çä¸ç§ Taskï¼èä¸æ¯æ´ä¸ªç³»ç»çæ ¸å¿ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 59. API 设计åå |
|---|
| | | |
|---|
| | | ææ API å¿
é¡»å´ç»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | TaskInstance |
|---|
| | | StepInstance |
|---|
| | | Event |
|---|
| | | Audio |
|---|
| | | Assessment |
|---|
| | | Metric |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 设计ã |
|---|
| | | |
|---|
| | | é¿å
大éï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | /english/xxx |
|---|
| | | /math/xxx |
|---|
| | | /game/xxx |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿ç§ App-specific APIã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 60. Task DSL åå |
|---|
| | | |
|---|
| | | æªæ¥å¼åè
åºè¯¥è½å¤éè¿ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | JSON / YAML / SDK |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ä¹ Taskã |
|---|
| | | |
|---|
| | | ä¾å¦ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task |
|---|
| | | âââ parameters |
|---|
| | | âââ steps |
|---|
| | | â âââ action |
|---|
| | | â âââ condition |
|---|
| | | â âââ wait |
|---|
| | | â âââ loop |
|---|
| | | â âââ branch |
|---|
| | | âââ assessment |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ ·ï¼ |
|---|
| | | |
|---|
| | | > æ°å¢ Task å°½éä¸éè¦ä¿®æ¹æ ¸å¿ Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 61. Runtime ç稳å®è¾¹ç |
|---|
| | | |
|---|
| | | å¿
须建ç«ç¨³å®çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Task Runtime API |
|---|
| | | Skill API |
|---|
| | | Device API |
|---|
| | | Assessment API |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æªæ¥ App è¶å¤ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Runtime ä¸å |
|---|
| | | App å¢å |
|---|
| | | Task å¢å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è¿æ¯å¼æ¾å¹³å°æç«çå
³é®ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 62. æç»æ¶æå¾ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Xiaozhi Platform |
|---|
| | | â |
|---|
| | | ââââââââââââââââââââ¼âââââââââââââââââââ |
|---|
| | | â â â |
|---|
| | | Parent Developer Device |
|---|
| | | â â â |
|---|
| | | â â â |
|---|
| | | Plan App Store Device Runtime |
|---|
| | | â â â |
|---|
| | | Schedule App Interaction |
|---|
| | | â â â |
|---|
| | | ââââââââââââââââ Task âââââââââââââââââ |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | â |
|---|
| | | Device Capability |
|---|
| | | â |
|---|
| | | â |
|---|
| | | Audio / Event |
|---|
| | | â |
|---|
| | | â |
|---|
| | | Server Pipeline |
|---|
| | | â |
|---|
| | | ââââââââââââââââââââ¼ââââââââââââââââââ |
|---|
| | | â â â |
|---|
| | | ASR Voice Analysis Assessment |
|---|
| | | â â â |
|---|
| | | ââââââââââââââââââââ¼ââââââââââââââââââ |
|---|
| | | â |
|---|
| | | Metric |
|---|
| | | â |
|---|
| | | Aggregation |
|---|
| | | â |
|---|
| | | User Profile |
|---|
| | | â |
|---|
| | | Parent Report |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 63. æç»å¹³å°çæ ¸å¿æ½è±¡ |
|---|
| | | |
|---|
| | | æ´ä¸ªé¡¹ç®æç»åªéè¦ç¢ç¢æä½ä»¥ä¸ 8 ä¸ªæ ¸å¿æ¦å¿µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | Step |
|---|
| | | Skill |
|---|
| | | Interaction Mode |
|---|
| | | Event |
|---|
| | | Assessment |
|---|
| | | Metric |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å®ä»¬ä¹é´ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | App |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Step |
|---|
| | | â |
|---|
| | | Skill |
|---|
| | | â |
|---|
| | | Device |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æ§è¡è¿ç¨ä¸äº§çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Event |
|---|
| | | â |
|---|
| | | Audio |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | â |
|---|
| | | Metric |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 64. æç»äº§åé»è¾ |
|---|
| | | |
|---|
| | | å æ¤æç» Xiaozhi çé»è¾ä¸æ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¨æ· |
|---|
| | | â |
|---|
| | | è¯´è¯ |
|---|
| | | â |
|---|
| | | AI åå¤ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | èæ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å®¶é¿ |
|---|
| | | â |
|---|
| | | 订é
App |
|---|
| | | â |
|---|
| | | é
ç½® Plan |
|---|
| | | â |
|---|
| | | é
ç½® Schedule |
|---|
| | | â |
|---|
| | | Server |
|---|
| | | â |
|---|
| | | Task |
|---|
| | | â |
|---|
| | | Xiaozhi |
|---|
| | | â |
|---|
| | | æ§è¡ Step |
|---|
| | | â |
|---|
| | | éé Audio / Event |
|---|
| | | â |
|---|
| | | Server åæ |
|---|
| | | â |
|---|
| | | Assessment |
|---|
| | | â |
|---|
| | | Metric |
|---|
| | | â |
|---|
| | | é¿æ Profile |
|---|
| | | â |
|---|
| | | å®¶é¿ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 65. 项ç®çæç»æç¥ç®æ |
|---|
| | | |
|---|
| | | 第ä¸é¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让 Xiaozhi è½æ§è¡ Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第äºé¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让 Xiaozhi è½è®°å½ Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第ä¸é¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让 Xiaozhi è½è¯ä¼° Task |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第åé¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让 Xiaozhi è½ç解é¿æåå |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第äºé¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让任ä½å¼åè
é½è½å建 Task / App |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 第å
é¶æ®µï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 让 App å¯ä»¥è®¢é
ååä¸å |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æç»ï¼ |
|---|
| | | |
|---|
| | | > **Xiaozhi Device æ为ç»ç«¯ï¼Task Runtime æ为æ§è¡å¼æï¼Server æ为 AI æ°æ®ä¸è°åº¦ä¸æ¢ï¼App Store æ为çæå
¥å£ã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 66. ç» AI Coding Agent çæé«çº§æ令 |
|---|
| | | |
|---|
| | | åç»å°æ¬ Plan æä¾ç» Cursor / Codex ç Coding Agent æ¶ï¼åºéµå®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä½ æ£å¨åä¸çæ¯ Xiaozhi AI ä»âAI 对è¯è®¾å¤â |
|---|
| | | åâAI Application Terminal Platformâçæ¶æå级ã |
|---|
| | | |
|---|
| | | ä¸è¦æ Xiaozhi 设计æåªæ Chat çæºå¨äººã |
|---|
| | | |
|---|
| | | ç³»ç»å¿
é¡»æ¯æï¼ |
|---|
| | | |
|---|
| | | App |
|---|
| | | Task |
|---|
| | | Step |
|---|
| | | Skill |
|---|
| | | Interaction Mode |
|---|
| | | Task Instance |
|---|
| | | Event |
|---|
| | | Audio |
|---|
| | | ASR |
|---|
| | | Assessment |
|---|
| | | Metric |
|---|
| | | Schedule |
|---|
| | | User Profile |
|---|
| | | |
|---|
| | | Task æ¯ç³»ç»æ ¸å¿ä¸å¡æ§è¡åä½ã |
|---|
| | | |
|---|
| | | Interaction Mode æ¯è®¾å¤äº¤äºæ¹å¼ï¼ |
|---|
| | | ä¸æ¯ä¸å¡å±çº§ã |
|---|
| | | |
|---|
| | | App æ¯ Task çä¸å¡å®¹å¨ã |
|---|
| | | |
|---|
| | | Step æ¯ Task çæ§è¡æ¥éª¤ã |
|---|
| | | |
|---|
| | | Skill æ¯åºå±ååè½åã |
|---|
| | | |
|---|
| | | Device è´è´£æ§è¡åééã |
|---|
| | | |
|---|
| | | Server è´è´£è°åº¦ãç解ãåæãè¯ä¼°åé¿ææ°æ®ç®¡çã |
|---|
| | | |
|---|
| | | æ°å¢ App / Task / Audio Resource å°½ééè¿é
ç½®åèµæºåæ¥å®ç°ï¼ |
|---|
| | | ä¸åºè¯¥ä¾èµ Firmware OTAã |
|---|
| | | |
|---|
| | | å¿
é¡»ä¿ææ§ Chat è½å继ç»å·¥ä½ã |
|---|
| | | |
|---|
| | | æææ°æ¶æå¿
须尽å¯è½å
¼å®¹åå¤ç¨ç°æ代ç ã |
|---|
| | | |
|---|
| | | ç¦æ¢å¨æªå®æ代ç 审计ä¹å大è§æ¨¡éæã |
|---|
| | | |
|---|
| | | æ¯ä¸ª Phase å¿
é¡»ï¼ |
|---|
| | | |
|---|
| | | 1. åæç°æ代ç |
|---|
| | | 2. æ确修æ¹èå´ |
|---|
| | | 3. ç»åºæ¶ææ¹æ¡ |
|---|
| | | 4. å®æ½æå°å¿
è¦ä¿®æ¹ |
|---|
| | | 5. ç¼åæµè¯ |
|---|
| | | 6. æ§è¡æµè¯ |
|---|
| | | 7. éªè¯æ§åè½æ²¡æåå½ |
|---|
| | | 8. è¾åºä¿®æ¹æ»ç» |
|---|
| | | |
|---|
| | | ä¸è¦ä¸æ¬¡æ§å®ç°æ´ä¸ªç³»ç»ã |
|---|
| | | |
|---|
| | | å¿
é¡»æç
§ Phase éæ¥æ¼è¿ã |
|---|
| | | |
|---|
| | | ä¼å
建ç«ç¨³å®ç Domain Model å Runtime Boundaryã |
|---|
| | | |
|---|
| | | æªæ¥ç¬¬ä¸æ¹å¼åè
åºè¯¥è½å¤å¨ä¸ä¿®æ¹ Xiaozhi æ ¸å¿ Runtime çæ
åµä¸å建æ°ç App å Taskã |
|---|
| | | |
|---|
| | | æç»ç®æ æ¯ï¼ |
|---|
| | | |
|---|
| | | âæ ¸å¿ Runtime 稳å®ï¼App / Task / Resource / Assessment å¯æ éæ©å±ãâ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 67. Master Plan çæ§è¡æ¹å¼ |
|---|
| | | |
|---|
| | | 以åææ Coding Plan é½åºè¯¥ä»æ¬ Master Plan æ´¾çã |
|---|
| | | |
|---|
| | | ç»æï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Master Plan |
|---|
| | | â |
|---|
| | | âââ Device Plan |
|---|
| | | â |
|---|
| | | âââ Server Plan |
|---|
| | | â |
|---|
| | | âââ Mini Program Plan |
|---|
| | | â |
|---|
| | | âââ Developer Platform Plan |
|---|
| | | â |
|---|
| | | âââ Assessment Plan |
|---|
| | | â |
|---|
| | | âââ Marketplace Plan |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç¶ååè¿ä¸æ¥ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Architecture Plan |
|---|
| | | â |
|---|
| | | Module Plan |
|---|
| | | â |
|---|
| | | Implementation Plan |
|---|
| | | â |
|---|
| | | Coding Task |
|---|
| | | â |
|---|
| | | Code |
|---|
| | | â |
|---|
| | | Test |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | # 68. æç»å¤ææ å |
|---|
| | | |
|---|
| | | 以åä»»ä½ä¸ä¸ªæ°éæ±ï¼é½å
é®ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | è¿æ¯ Appï¼ |
|---|
| | | æ¯ Taskï¼ |
|---|
| | | æ¯ Stepï¼ |
|---|
| | | æ¯ Skillï¼ |
|---|
| | | æ¯ Interaction Modeï¼ |
|---|
| | | æ¯ Eventï¼ |
|---|
| | | æ¯ Assessmentï¼ |
|---|
| | | è¿æ¯ Metricï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¦æä¸ä¸ªæ°éæ±æ æ³å½å
¥è¿äºæ ¸å¿æ½è±¡ï¼ |
|---|
| | | |
|---|
| | | > ä¸è¦ç«å³å代ç ã |
|---|
| | | |
|---|
| | | å
éæ°å®¡è§æ¶æã |
|---|
| | | |
|---|
| | | è¿å¥ååå°å³å® Xiaozhi æç»è½ä¸è½ä»ä¸ä¸ªäº§åï¼çæ£æ¼åæä¸ä¸ªå¼æ¾å¹³å°ã |
|---|
| New file |
| | |
|---|
| | | # å°æº AI Client Architecture Plan |
|---|
| | | |
|---|
| | | > ææ¡£ï¼å°æº AI 硬件å¼æ¾åºç¨å¹³å° ââ æ»ä½æ¶æä¸å¼å Plan |
|---|
| | | > Version: V1.1 |
|---|
| | | > Date: 2026-08-30 |
|---|
| | | > æ¬æå¨ V1.0 æ¦å¿µç¨¿ä¸ï¼ä¾æ®ãXiaozhi AI å¼æ¾åºç¨å¹³å° Master Planãä¸ãArchitecture Auditã修订ã |
|---|
| | | > **åªè®¾è®¡ï¼ä¸ç¼ç ã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 0. æ¬æå®ä½ |
|---|
| | | |
|---|
| | | æ¬æåªå®ä¹ **Device / Client**ï¼ä»åº `xiaozhi-esp32-2.2.6`ï¼å¦ä½ä»ãè¯é³å¯¹è¯åºä»¶ãæ¼è¿ä¸ºãå¯æ§è¡å¤ç§ InteractionMode ç Device Runtimeãã |
|---|
| | | |
|---|
| | | ä¸å¨æ¬æå±å¼ãç± Server Plan è´è´£çå
å®¹ï¼ |
|---|
| | | |
|---|
| | | - App / Task Definition çæå¨åå¨ |
|---|
| | | - TaskInstance / StepInstance æä¹
å |
|---|
| | | - Scheduler / Plan / Subscription |
|---|
| | | - ASR / LLM / Assessment / 家é¿æ¥å |
|---|
| | | - 管çåå°ä¸å¼åè
å¹³å° |
|---|
| | | |
|---|
| | | **ç¸å¯¹ V1.0 çå
³é®ä¿®æ£ï¼** |
|---|
| | | |
|---|
| | | | V1.0 | V1.1ï¼æ¬æ¬¡ï¼ | |
|---|
| | | |------|----------------| |
|---|
| | | | 设å¤ä¸æå®æ´ Task Runtime | æå¨ Task Runtime å¨ Serverï¼è®¾å¤åªæ **Task Executor** | |
|---|
| | | | Phase 1 åå¤è¯» + è·è¯» + å£ç® + æè¯æ¥é¾ | Phase 1 åªå `CHAT` + `PLAY_AUDIO`ï¼è±è¯å¤è¯»ï¼ï¼ä¸ç ´åç°æ Chat | |
|---|
| | | | æ°é ç»ä¸ Command Protocol | **æ©å±ç°æ JSON `type` + MCP**ï¼ä¸æ°å¼æ»çº¿ | |
|---|
| | | | Skill åè¡¨å« `math.calculate` / `llm.evaluate` | Skill å¿
é¡»æ å°å°**å·²æ代ç **ï¼æ²¡æçæ 为æ°å»º | |
|---|
| | | | 设å¤ç»´æ¤ App Manager å
¨å¥çå½å¨æ | Phase 1â2 ä¸å¨åºä»¶å®ç° App ååºï¼èµæºåæ¥å¯å¤ç¨æ¿çº§ä¸è½½ | |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 1. å½å Client æ¯ä»ä¹ï¼Audit äºå®ï¼ |
|---|
| | | |
|---|
| | | å½ååºä»¶**ä¸æ¯** App å¹³å°ï¼èæ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Application åä¾ |
|---|
| | | + DeviceState ç¶ææº |
|---|
| | | + Opus å®æ¶åå·¥ |
|---|
| | | + MCP å·¥å
Ꭲ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | | 项 | çå®ä½ç½® | |
|---|
| | | |----|----------| |
|---|
| | | | å
¥å£ | `main/main.cc` â `Application::Initialize` â `Application::Run` | |
|---|
| | | | ä¸»å¾ªç¯ | `application.cc`ï¼FreeRTOS EventGroupï¼æ°¸ä¸è¿å | |
|---|
| | | | 对è¯æ | `device_state.h`ï¼Idle / Connecting / Listening / Speaking ⦠| |
|---|
| | | | é³é¢ç®¡é | `audio/audio_service.{h,cc}`ï¼MicâOpusâServerï¼ServerâOpusâSpeaker | |
|---|
| | | | åè®® | `protocols/protocol.{h,cc}` + `websocket_protocol` / `mqtt_protocol` | |
|---|
| | | | Skill é¢ | `mcp_server.cc` + å board `InitializeTools()` | |
|---|
| | | | èµæº OTA | `ota.{h,cc}`ï¼åºä»¶/é
ç½®ï¼ã`assets.{h,cc}`ï¼è¯é³æ¨¡åçï¼ | |
|---|
| | | | æ¿çº§æè·¯ | Seningï¼`self.mp3.*`ã`BoardMqtt`ï¼`snlp/v1` DOWNLOAD / PLAY_MEDIAï¼ | |
|---|
| | | |
|---|
| | | å®æ´å¯¹è¯é¾ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Mic â AudioService â Opus 16k/60ms â Protocol.SendAudio |
|---|
| | | â Server ASR/LLM/TTS |
|---|
| | | â type=tts/stt + binary Opus â AudioService decode â Speaker |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **没æï¼** InteractionMode æ举ãTask/Stepãå¯å½æ¡£å½é³æ件ãæ ¸å¿ `play_audio`ï¼ä»
Sening æ¿çº§ï¼ãç»ä¸ Event Busã |
|---|
| | | |
|---|
| | | `ListeningMode`ï¼auto/manual/realtimeï¼åªæ¯æ¾é³çç¥ï¼**ä¸æ¯** Master Plan ç InteractionModeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 2. ç®æ Device Runtime |
|---|
| | | |
|---|
| | | ç®æ ä¸æ¯æææº OS æ¬å° ESP32ï¼èæ¯ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Device Runtime â ç°æ Application äºä»¶ç¯å级ï¼ä¸æ¯æ¨ç¿» |
|---|
| | | âââ Chat Runtime â ç°æ Listening â Speakingï¼é为 Mode=CHAT |
|---|
| | | âââ Task Executor â èæ§è¡å¨ï¼æ¶æ令ãè· Skillãå Event |
|---|
| | | âââ InteractionMode åæ¢ |
|---|
| | | âââ Skill â Device Capability |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | åä¸æ¶é´åªè¿è¡ä¸ä¸ª InteractionModeï¼Chat ä¸ Task é³é¢äºæ¥ï¼ã |
|---|
| | | |
|---|
| | | 设å¤**ä¸ç解**ä¸å¡ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ç¦æ¢åºä»¶åºç°ï¼if (app == english) / if (task == idiom) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 设å¤åªçè§£ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | mode + skill + resource_ref + å®ææ¡ä»¶ï¼è¶
æ¶ / ææ¾ç»æ / VAD éé³ï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 3. è责边ç |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Serverï¼æå¨ï¼ |
|---|
| | | å建 TaskInstance |
|---|
| | | æ¨è¿ Step |
|---|
| | | æå® InteractionMode |
|---|
| | | ASR / LLM / Assessment |
|---|
| | | æä¹
åç¶æä¸é³é¢å¯¹è±¡ |
|---|
| | | |
|---|
| | | Clientï¼æ§è¡ï¼ |
|---|
| | | ä¿æ Chat çè·¯å¾ |
|---|
| | | æ mode 使ç¨éº¦/æ¬å£°å¨/å± |
|---|
| | | æ§è¡ Skillï¼æãå¬ãæ¾ç¤ºãä¸è½½ï¼ |
|---|
| | | ä¸æ¥ Eventï¼å¼å§/ç»æ/å¤±è´¥ï¼ |
|---|
| | | ç¼åå½å stepï¼æ线 UIï¼ï¼ä¸ä»¥æ¬å°ä¸ºæå¨ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | å¯¹åº Master Planï¼Device è´è´£æ§è¡ãææ¾ãééãç¼åãéä¿¡ï¼Server è´è´£è°åº¦ä¸è¯ä¼°ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 4. åå±ï¼ä¿ç V1.0ï¼è½å°ç°æ代ç ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ä¸å¡å± App / Task â ä¸ä¸åå°åºä»¶ä¸å¡ä»£ç ï¼åªä¸å JSON + èµæº |
|---|
| | | 交äºå± Interaction Mode â æ°å¢ï¼æå¨ Applicationï¼ä¸ DeviceState æ£äº¤ |
|---|
| | | è½åå± Skill â ç°æ MCP tools å级为æ£å¼ Skill ç®å½ |
|---|
| | | 设å¤å± Device Capability â AudioCodec / Display / Button / SD / LED |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | **DeviceState ä¸ InteractionMode å¿
é¡»åå¼ï¼** |
|---|
| | | |
|---|
| | | | | DeviceStateï¼å·²æï¼ | InteractionModeï¼æ°å¢ï¼ | |
|---|
| | | |--|---------------------|-------------------------| |
|---|
| | | | åç | 设å¤ç°å¨è½ä¸è½å¬/说/å级 | è¿ä¸æ¥ç¨ä»ä¹äº¤äºæ¹å¼ | |
|---|
| | | | ä¾å | Idle, Listening, Speaking | CHAT, PLAY_AUDIO, FOLLOW_READ | |
|---|
| | | | è°æ¹ | é³é¢ééå ç¨ | Server æå® / Client æ§è¡ | |
|---|
| | | |
|---|
| | | ä¸è¦æ `PLAY_AUDIO` åæä¸ä¸ªæ°ç `kDeviceStatePlayingEnglishLesson`ãææ¾æ¶å¯ä¿æ Idle æå¢å **é对è¯**å ç¨æ è®°ï¼ä½ **ç¦æ¢**æ Task å¡è¿ Listening/Speaking è¿ç§»è¡¨ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 5. Chat Runtimeï¼ä¿ç并é级 |
|---|
| | | |
|---|
| | | ç°æ Chat **å¿
须继ç»ä½ä¸ºé»è®¤è½å**ãæ Appãæ Task æ¶ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å¤é / æé® â Connecting â Listening â Speaking â Idle |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è·¯å¾ãåè®® `hello/listen/abort/tts/stt/mcp`ãOpus çè·¯å¾ **Phase 1â2 ç¦æ¢æ¹è¯ä¹**ã |
|---|
| | | |
|---|
| | | é级æ¹å¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | InteractionMode = CHAT |
|---|
| | | â èµ°ç°æ Application 对è¯ç¶ææº |
|---|
| | | |
|---|
| | | InteractionMode = PLAY_AUDIO |
|---|
| | | â ä¸è¿å
¥ LLM |
|---|
| | | â æ¬å£°å¨ç» play_audio |
|---|
| | | â 麦å¯å
³æåªç¨äºææ/å¤ééåº |
|---|
| | | |
|---|
| | | ä»»æ Mode ä¸å¤éè¯ / éåº |
|---|
| | | â åå° CHAT æ Idleï¼å¯é
ç½®ï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Chat ä¸æ¯ Task Runtime çç¶å®¹å¨ï¼èæ¯ Device Runtime ä¸ä¸ Task Executor **并å**çå Runtimeã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 6. Task Executorï¼ä¸æ¯å®æ´ Task Runtimeï¼ |
|---|
| | | |
|---|
| | | V1.0 æ Task Runtime ç»å¨è®¾å¤ä¸ãAudit ç»è®ºï¼ç¼æå·²å¨ Server `ConnectionHandler`ã设å¤ä¾§åªå¢å ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | æ¶å° skill.execute / æ¿çº§ PLAY_MEDIA / MCP tools/call |
|---|
| | | â æ§è¡ |
|---|
| | | â ä¸æ¥ completed / failed |
|---|
| | | â çå¾
ä¸ä¸æ¥ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è®¾å¤ **ä¸** è´è´£ï¼ |
|---|
| | | |
|---|
| | | - å建 TaskInstance |
|---|
| | | - å¤æãè·è¯»æ¯å¦åæ ¼åæ¯å¦éè¯ãï¼å¯æ§è¡ Server ä¸åçãåæä¸æ¬¡ãï¼ |
|---|
| | | - è¯åãASRãLLM |
|---|
| | | |
|---|
| | | è®¾å¤ **å¯ä»¥** æ¬å°å¤æçå®ææ¡ä»¶ä»
é硬件å¯è§æµï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | ææ¾ç»æ |
|---|
| | | VAD éé³è¶
æ¶ |
|---|
| | | ç¨æ·æé®åæ¶ |
|---|
| | | ä¸è½½å®æ / 失败 |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ç¶ææºï¼æ§è¡å¨çº§ï¼ç¼åç¨ï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | IDLE â EXECUTING â WAITING_USER? â COMPLETED | FAILED | CANCELLED |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | æå¨ç¶æä»å¨ Server ç TaskInstance / StepInstanceã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 7. Interaction Mode |
|---|
| | | |
|---|
| | | > Interaction Mode ä¸æ¯ä¸å¡åºæ¯ï¼ä¸æ¯ãè±è¯è¯¾ãï¼ï¼åªæ述硬件æä¹ä¸äººäº¤äºã |
|---|
| | | |
|---|
| | | ### 7.1 åé¶æ®µæ¯æ |
|---|
| | | |
|---|
| | | | Mode | Phase | ç°æ代ç åºç¡ | 设å¤è¦åç | |
|---|
| | | |------|-------|--------------|------------| |
|---|
| | | | `CHAT` | 0ï¼å·²æï¼ | Listening/Speaking | ä¿æ | |
|---|
| | | | `PLAY_AUDIO` | 1 | Sening `self.mp3.play` / `PLAY_MEDIA`ï¼`PauseSdCardMp3` | æ½æå¯è°ç¨ Skillï¼ä¸ TTS äºæ¥ | |
|---|
| | | | `FOLLOW_READ` | 2 | listen æµ + ææ¾ | å
æåå¬ï¼å¬å®ä¸æ¥ï¼ä¸è¿ LLM | |
|---|
| | | | `LISTEN` | 2 | `SendStartListening` | å¯åç¬å¬ï¼ä¸èªå¨ Chat | |
|---|
| | | | `QUIZ` / `TRANSLATION` / `GAME` | 3 | æ ç¬ç« Mode | å
ç¨ CHAT+Skill ç»åï¼ååºå Mode | |
|---|
| | | |
|---|
| | | V1.0 è¦æ±ç¬¬ä¸é¶æ®µå°±æ¯æ QUIZ/TRANSLATION/GAME ââ **åæ¶**ãé£äºæ¯ Server ç¼æ + CHAT/LISTEN å³å¯å
åï¼ä¸å¼å¾å
æ¹åºä»¶ç¶ææºã |
|---|
| | | |
|---|
| | | ### 7.2 PLAY_AUDIOï¼è±è¯å¤è¯»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Serverï¼éå¤ N 次çç¼æï¼æå¨ï¼ |
|---|
| | | Clientï¼play_audio(resource) â æå® event â çä¸ä¸æ¬¡ play æ stop |
|---|
| | | LLM = falseï¼ASR = false |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | TTS æ¥æ¶æ²¿ç¨ç°æ `PauseSdCardMp3` / `resume_sdcard_mp3_after_tts_`ï¼**ä¸è¦**å¦åä¸å¥æ··é³ã |
|---|
| | | |
|---|
| | | ### 7.3 FOLLOW_READï¼è±è¯è·è¯»ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | play æ åé³é¢ |
|---|
| | | â play.completed |
|---|
| | | listenï¼å¤ç¨ä¸è¡ Opusï¼ |
|---|
| | | â speech.stopped / listen.completed |
|---|
| | | Server ASR + v0 ææ¬è¯ä¼° |
|---|
| | | â speak åé¦ï¼å¯èµ°ç°æ TTSï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 设å¤ä¸è¯åã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 8. Skillï¼åªåºäºçå®è½å |
|---|
| | | |
|---|
| | | Skill = 对 Device Capability ç稳å®å°è£
ã第ä¸æ¹ App ç¦æ¢ç´æ¥ GPIO/I2Sã |
|---|
| | | |
|---|
| | | ### 8.1 å·²æï¼å¯ç´æ¥å½ä½ Skill |
|---|
| | | |
|---|
| | | | Skill | å½åå®ç° | èå´ | |
|---|
| | | |-------|----------|------| |
|---|
| | | | `device.get_status` | `self.get_device_status` | å
¨å± MCP | |
|---|
| | | | `audio.set_volume` | `self.audio_speaker.set_volume` | å
¨å± | |
|---|
| | | | `screen.set_brightness` / `set_theme` | `self.screen.*` | æå± | |
|---|
| | | | `camera.take_photo` | `self.camera.take_photo` | ææå头 | |
|---|
| | | | `audio.play_sound` | `AudioService::PlaySound`ï¼OGG æ示é³ï¼ | å
¨å±ï¼ä¸è½å½è¯¾ç¨é³é¢ | |
|---|
| | | | `mp3.play/stop/list` | `self.mp3.*` | **Sening** | |
|---|
| | | | `media.play` | `BoardMqtt` `PLAY_MEDIA` | **Sening snlp** | |
|---|
| | | | `resource.download` | `DOWNLOAD` / Assets | Sening / Assets | |
|---|
| | | | `alarm.*` | `self.alarm.*` | Seningï¼â å®¶é¿ Schedule | |
|---|
| | | | `led` / è¿å¨ç±» | å board MCP | æ¿ç¸å
³ | |
|---|
| | | |
|---|
| | | ### 8.2 Phase 1 éè¦ãæåãç |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | play_audio |
|---|
| | | è¾å
¥ï¼SD è·¯å¾æå·²ä¸è½½ resource_id |
|---|
| | | å®ç°ï¼ä¼å
å¤ç¨ self.mp3.play / PLAY_MEDIA |
|---|
| | | ç®æ ï¼ä» board ç§æå·¥å
·æå为 Runtime Skillï¼å¯ä» ifdef æ¿åï¼ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸è¦æ±æææ¿ç¬¬ä¸å¤©é½è½ play MP3ãæ SD çæ¿ Phase 1 å¯ç¨ Server æ¨æµï¼ä¸æ¨èå TTS ééåå¤è¯»å¾ªç¯ï¼ã |
|---|
| | | |
|---|
| | | ### 8.3 Phase 2 éè¦æ°å»ºç |
|---|
| | | |
|---|
| | | | Skill | ç°ç¶ | 设计 | |
|---|
| | | |-------|------|------| |
|---|
| | | | `record_audio`ï¼å¯å½æ¡£ï¼ | **ä¸åå¨**ï¼åªææµå¼ Opus | ä¼å
å¤ç¨ listen æµï¼ç± Server å¥çº§è½ WAVï¼è®¾å¤ç«¯æ件å½é³ä¸ºå¢å¼ºï¼SD `/recordings/pending`ï¼ | |
|---|
| | | | `listen`ï¼ä¸è¿ Chatï¼ | listen æ¶æ¯ä¼è¿å
¥å¯¹è¯ | Mode=FOLLOW_READ/LISTEN æ¶ Client ä»å Opusï¼Server **è·³è¿ LLM** | |
|---|
| | | | `wait` | æ | å¯ç¨ Server 计æ¶ï¼è®¾å¤åªéæå®/è¶
æ¶äºä»¶ | |
|---|
| | | | `display.show` | æå± MCP / `SetChatMessage` | å¯æ©å±ææ¬/å¾çèµæº | |
|---|
| | | |
|---|
| | | ### 8.4 ä¸è¦åæè®¾å¤ Skill ç |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | asr.transcribe â Server |
|---|
| | | llm.chat/evaluate â Server |
|---|
| | | score.evaluate â Server |
|---|
| | | translation.translate â Server |
|---|
| | | math.calculate â Server |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | V1.0 Skill 表éçè¿äºè½åæ¹æ Server Planã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 9. é³é¢ãå½é³ãäºæ¥ |
|---|
| | | |
|---|
| | | ### 9.1 ç°æé³é¢çå½å¨æï¼å¿
é¡»ä¿çï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | 对è¯ï¼PCM 16k â Opus 60ms æµ â ä¸åç â Server å¯é转 WAV å
¥åº |
|---|
| | | æ示é³ï¼OGG PlaySound |
|---|
| | | Sening 课ç¨ï¼SD MP3 ç¬ç«ææ¾ï¼TTS æ¶ Pause |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | çè·¯å¾ç¦æ¢æ¹ï¼`MAIN_EVENT_SEND_AUDIO`ã`AudioInputTask`ã`OpusCodecTask`ã`FinishTtsResponse`ã |
|---|
| | | |
|---|
| | | ### 9.2 äºæ¥è§å |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | åä¸æ¶å»æ¬å£°å¨åªæå¡ä¸ä¸ªæºï¼TTS | SD/MP3 | PlaySound |
|---|
| | | CHAT ç Listening/Speaking ä¼å
äº PLAY_AUDIOï¼å¯æå课ç¨é³é¢ï¼ |
|---|
| | | Task éè¦éº¦æ¶ï¼è¦ä¹ Mode ç¬å listenï¼è¦ä¹æç¡®ä¸ Chat äºæ¥ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 9.3 è·è¯»é³é¢æå°æ¹æ¡ï¼Phase 2ï¼ |
|---|
| | | |
|---|
| | | **ä¸å
å**设å¤å®æ´ wav æ件ä¸ä¼ ã |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | å¤ç¨ç°æä¸è¡ Opus |
|---|
| | | Server æå¥ PCMâWAVï¼å·²æ chat_history_conf=2ï¼ |
|---|
| | | æ°å¢å
³èï¼audio_id â step_instance_id |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | 离线ç¼å `/recordings/pending` æ¾å° Phase 3ï¼é¤éè·è¯»å¨å¼±ç½æ为é»å¡ã |
|---|
| | | |
|---|
| | | 设å¤ä¾§ **没æ** audio_idï¼åªæ `session_id`ï¼è¿æ¥çº§ï¼ãaudio_id ç± Server çæååä¼ ï¼å¯éï¼ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 10. App èµæº vs Firmware OTA |
|---|
| | | |
|---|
| | | ååä¿ç V1.0 / Master Planï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | OTA = 驱å¨ãåè®®ãAudio EngineãRuntimeãSkill 宿主 |
|---|
| | | App Sync = manifestãtask JSONãMP3ãå¾çãé
ç½® |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | App **ååä¸ä¸ OTA**ãä½ä¸åè½åå¿
é¡»å¨ Firmware éï¼å¯ä¸æ¬¡ OTA è¡¥é½ï¼ä¹åæ° App ä¸åæ¹ C++ï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | Codec / VAD / å¤é |
|---|
| | | WS æ MQTT+UDP åè®®æ |
|---|
| | | MCP / Skill 宿主 |
|---|
| | | SD æä»¶ç³»ç» |
|---|
| | | HTTP ä¸è½½ï¼æçä»·ï¼ |
|---|
| | | é³é¢äºæ¥ |
|---|
| | | InteractionMode å¼å
³ |
|---|
| | | ï¼Phase 2ï¼é Chat ç listen è¯ä¹ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### 10.1 SD å¸å±ï¼ç®æ ï¼Phase 2+ è½å°ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | /sdcard/ |
|---|
| | | apps/<appId>/ # manifest + tasks JSON |
|---|
| | | resources/<appId>/ # mp3 / å¾ |
|---|
| | | recordings/pending/ # Phase 3 |
|---|
| | | cache/ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Sening å½åä¸è½½è·¯å¾å `/data/courses/...`ï¼smartlearningï¼ãæ¼è¿æ¶ **æ å°å°ç»ä¸å¸å±**ï¼ä¸è¦å¼ºè¿«ç«å³æ¹ç»ç«¯å·²ææ件ã |
|---|
| | | |
|---|
| | | ### 10.2 å MQTT æ¶æ |
|---|
| | | |
|---|
| | | å½ååå¨ä¸¤å¥ï¼ |
|---|
| | | |
|---|
| | | | éé | ç¨é | |
|---|
| | | |------|------| |
|---|
| | | | Xiaozhi `Protocol` | å¯¹è¯ JSON + Opus | |
|---|
| | | | Sening `snlp/v1` | å¿è·³ãDOWNLOADãPLAY_MEDIAãack/progress | |
|---|
| | | |
|---|
| | | **Phase 1ï¼** å
许并åï¼å¤è¯»ä¼å
èµ°å·²è½å·¥ä½ç MCP/`PLAY_MEDIA`ã |
|---|
| | | **Phase 2â3ï¼** èµæºåæ¥ä¸ææ¾æ§å¶éæ¥æ¶å° Xiaozhi JSON `type` æ MCPï¼snlp é为æ¿çº§å®ç°ç»èï¼é¿å
ä¸å¡äºé¿æååã |
|---|
| | | **ç¦æ¢ Phase 1 åå¹¶ä¸¤å¥ MQTTã** |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 11. åè®®ï¼æ©å±ï¼ä¸æ¿æ¢ |
|---|
| | | |
|---|
| | | ç°æ设å¤âäºï¼`hello / listen / abort / iot / mcp / ping / server` |
|---|
| | | ç°æäºâ设å¤ï¼`hello / stt / tts / llm / mcp / system / alert / custom` |
|---|
| | | |
|---|
| | | ### Phase 1 |
|---|
| | | |
|---|
| | | ä¸å¼ºå¶æ° `type`ãå¤ç¨ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | type=mcp tools/call self.mp3.play | stop |
|---|
| | | æ snlp PLAY_MEDIA |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ### Phase 2 æ°å¢ï¼ä¸ tts/stt/mcp 并åï¼ |
|---|
| | | |
|---|
| | | ä¸è¡ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | task.start / task.stop / task.pause / task.resume |
|---|
| | | step.start |
|---|
| | | skill.execute |
|---|
| | | mode.set # CHAT | PLAY_AUDIO | FOLLOW_READ | LISTEN |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸è¡ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | task.started / task.completed / task.failed |
|---|
| | | step.started / step.completed / step.failed |
|---|
| | | skill.completed / skill.failed |
|---|
| | | audio.play.completed |
|---|
| | | speech.started / speech.stopped # å¯ä»ç°æ VAD äºä»¶å¯¼åº |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | è½½è·å¿
须带ï¼`session_id`ï¼Phase 2 起带 `task_instance_id` / `step_id`ã |
|---|
| | | |
|---|
| | | **ä¸è¦**æ°å»ºç¬¬åå¥ EventBusãSening `rpt/event` å¯ä½å®ç°ï¼ä½è¯ä¹åºå¯¹é½ä¸è¡¨ã |
|---|
| | | |
|---|
| | | æå
¥ç¹ï¼`Application` ä¸ `protocol_->OnIncomingJson`ï¼`application.cc` 约 647â778 è¡ï¼ï¼`unknown type` åæ¯æå¢å `task` / `mode`ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 12. Event ä¸ä¸»å¾ªç¯æå
¥ç¹ |
|---|
| | | |
|---|
| | | ç°ææºå¶ï¼`MAIN_EVENT_*` + åè®®åè° + MCP JSON-RPC +ï¼Seningï¼rpt/ackã |
|---|
| | | |
|---|
| | | Task Executor æ载顺åºï¼ä¾µå
¥æ§ä»ä½å°é«ï¼ï¼ |
|---|
| | | |
|---|
| | | 1. **MCP `tools/call`** â Phase 1 é¦éï¼Chat è·¯å¾ä¸å¨ |
|---|
| | | 2. **`Application::Schedule`** â 线ç¨å®å
¨æ¨è¿ UI / éé³é¢å¯ä½ç¨ |
|---|
| | | 3. **OnIncomingJson æ° type** â Phase 2 |
|---|
| | | 4. **ç¦æ¢**æ Task ç¶æåå
¥ `DeviceStateMachine::IsValidTransition` ç Listening/Speaking è¾¹ |
|---|
| | | |
|---|
| | | æé®ï¼ä»æ¥ç»å® `ToggleChatState` / Start/Stop ListeningãPhase 2 èµ·å
许ãåæ¶å½å Task / è·³è¿ãèä¸ææ Chat æé®ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 13. æ¬å°ç¶æï¼èï¼ |
|---|
| | | |
|---|
| | | 设å¤åªæä¹
åå¿
è¦é¡¹ï¼NVS / å
åï¼ï¼ |
|---|
| | | |
|---|
| | | ```text |
|---|
| | | activeMode |
|---|
| | | activeTaskInstanceId # ç¼åï¼æå¨å¨ Server |
|---|
| | | activeStepId # ç¼å |
|---|
| | | sdcard play ç¶æ # å·²æ |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | ä¸å¨è®¾å¤ä¿åå®æ´ Step åå²ãè¯åãASR ææ¬ã |
|---|
| | | |
|---|
| | | App çå½å¨æ V1.0 ç INSTALL/LAUNCH/ACTIVE å
¨å¥ **Phase 3** ååãPhase 1 çè³ä¸éè¦ `activeAppId`ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 14. åé¶æ®µï¼Clientï¼ |
|---|
| | | |
|---|
| | | ä¸ Server Plan é¶æ®µç¼å·å¯¹é½ã |
|---|
| | | |
|---|
| | | ### Phase 1 â å模å¼è¯æï¼å¿
é¡»ä¸ç ´å Chatï¼ |
|---|
| | | |
|---|
| | | ç®æ ï¼è±è¯å¤è¯» N éã |
|---|
| | | |
|---|
| | | Client å·¥ä½ï¼ |
|---|
| | | |
|---|
| | | - ææ¡£å并å¤ç¨ `self.mp3.play` / `PLAY_MEDIA` + `PauseSdCardMp3` |
|---|
| | | - 约å®ï¼ææ¾æé´å¤é/æé®å¯åå° Chatï¼ç°æè¡ä¸ºä¼å
ï¼ |
|---|
| | | - **ä¸æ¹** Opus çè·¯å¾ãDeviceState è¿ç§»ãhello/listen/tts è¯ä¹ |
|---|
| | | - ä¸å®ç° Task ç¶ææºãApp Managerãè·è¯»å½é³æ件 |
|---|
| | | |
|---|
| | | éªæ¶ï¼ |
|---|
| | | |
|---|
| | | 1. æ Task æ¶ Chat ä¸ç°ç½ä¸è´ |
|---|
| | | 2. è½å¾ªç¯ææ¾æå® SD/å·²ä¸è½½é³é¢ N 次 |
|---|
| | | 3. TTS/å¤éä»è½ææææåææ¾ |
|---|
| | | |
|---|
| | | ### Phase 2 â Task Executor + è·è¯»éé |
|---|
| | | |
|---|
| | | - `mode.set` + `type=task` / `skill.execute` |
|---|
| | | - FOLLOW_READï¼æå®å listenï¼Server è·³è¿ LLM |
|---|
| | | - ä¸æ¥ play/listen å®æäºä»¶ï¼å¸¦ task/step idï¼ |
|---|
| | | - å½é³æå°æ¹æ¡ï¼æµå¼ä¸ä¼ + Server å½æ¡£å
³èï¼ä¸å¼ºå¶è®¾å¤ wav æ件 |
|---|
| | | |
|---|
| | | éªæ¶ï¼ä¸æ¬¡è·è¯» Step è½å¨ Server çå° audio_id + asr_textã |
|---|
| | | |
|---|
| | | ### Phase 3 â Runtime å¹³å°å |
|---|
| | | |
|---|
| | | - æ ¸å¿ `play_audio` / ä¸è½½è¿é Sening æ½è±¡ï¼ææç¡®ä»
æ¯æ SD æ¿ï¼ |
|---|
| | | - App èµæºç®å½ä¸å¢é sync |
|---|
| | | - snlp ä¸ Xiaozhi åè®®æ¶æ |
|---|
| | | - å¯éï¼æ¬å° pending å½é³ãQUIZ/GAME ä½ä¸º Mode èä¸æ¯ç¡¬ç¼ç ä¸å¡ |
|---|
| | | - æé®å¯ç»å® Task åæ¶ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 15. ç¦æ¢äºé¡¹ï¼Clientï¼ |
|---|
| | | |
|---|
| | | 1. éå AudioService çè·¯å¾æãä¸ºäº Task æ¢ä¸å¥ç¼è§£ç ã |
|---|
| | | 2. å¨åºä»¶åè±è¯/å£ç®/æè¯ä¸å¡åæ¯ |
|---|
| | | 3. ç¨ OTA åè¯¾ç¨ App |
|---|
| | | 4. Phase 1 åå¹¶ä¸¤å¥ MQTT |
|---|
| | | 5. æ Task å¡è¿ Listening/Speaking |
|---|
| | | 6. 设å¤ä¸å ASR/è¯å/æ
绪åæ |
|---|
| | | 7. 为 Demo å `EnglishRepeatManager` è¿ç§ä¸æ¬¡æ§ç±»ï¼å¯ç¨è `TaskExecutor` + Skillï¼ |
|---|
| | | 8. å¼å
¥éç¨ EventBus / Redux |
|---|
| | | 9. ä¿®æ¹ç°æ Chat åè®®å段å«ä¹ |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 16. AI Coding 约æï¼ä¿ç并æ¶ç´§ V1.0ï¼ |
|---|
| | | |
|---|
| | | 1. å
读ç°æ `Application` / `AudioService` / `Protocol` / `mcp_server`ï¼åæ¹ã |
|---|
| | | 2. æ App/Task æ¶ Chat å¿
é¡»å¯ç¨ã |
|---|
| | | 3. ä¸å¡ä¸ä¸æ²å° Clientã |
|---|
| | | 4. é
ç½®/JSON é©±å¨ InteractionMode ä¸ Skillï¼ä¸åæ» Appã |
|---|
| | | 5. æå°æ¹å¨ï¼è½ MCP 解å³ç Phase 1 ä¸æ¹åè®®ã |
|---|
| | | |
|---|
| | | --- |
|---|
| | | |
|---|
| | | ## 17. ä¸ Server Plan çå¥çº¦ |
|---|
| | | |
|---|
| | | Client ä¿è¯ï¼ |
|---|
| | | |
|---|
| | | - CHAT è¡ä¸ºååå
¼å®¹ |
|---|
| | | - è½æ§è¡ `play_audio`ï¼è³å° Sening è·¯å¾ï¼ |
|---|
| | | - Phase 2 è½æ mode å¬èä¸èªå¨å½ Chat åå¤å¼æ |
|---|
| | | - äºä»¶å¯å
³è `session_id`ï¼ååç» instance idï¼ |
|---|
| | | |
|---|
| | | Client è¦æ± Serverï¼ |
|---|
| | | |
|---|
| | | - æå¨æ¨è¿ Stepï¼ä¸è¦ææ设å¤è®°ä½æ´å¼ Workflow |
|---|
| | | - PLAY_AUDIO / FOLLOW_READ ä¸è¦è¯¯å
¥ `ConnectionHandler.chat()` |
|---|
| | | - é³é¢å¯¹è±¡ä¸ step å
³èå¨äºç«¯å®æ |
|---|
| | | - App JSON ä¸èµæº URL ç±äºä¸åï¼ä¸è¦æ±åºä»¶å
ç½®è¯¾ç¨ |
|---|
| | | |
|---|
| | | ä¸ä¸ä»½å¯æ§è¡ææ¡£åºæ¯ **Phase 1 Coding Planï¼Client åçï¼**ï¼èä¸æ¯ç»§ç»å Modeã |
|---|
