# UPDATE_MANIFEST Batch Download Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Make `UPDATE_MANIFEST` save the course manifest and asynchronously download every listed asset, overwriting existing files safely, with web UI support for uploading multiple MP3 files and generating the manifest command. **Architecture:** Reuse the existing single-file `DOWNLOAD` pipeline rather than introducing a second downloader. `mqtt.c` will add a manifest job that validates assets, stores a compact copy of the manifest, then runs a FreeRTOS worker to save the manifest and download each asset sequentially via `Board_Download_File_To_Path()`, reporting aggregate progress using the existing MQTT progress topic. The web test server will add a multi-MP3 upload endpoint and UI that builds an `UPDATE_MANIFEST` payload with local HTTP URLs, MD5s, and target paths. **Tech Stack:** ESP-IDF C/C++, FreeRTOS tasks, cJSON, SD card filesystem helpers in `system.c`, Python `http.server`/`cgi`, browser JavaScript. --- ## File Structure - Modify `main/boards/sening-xiaozhi-s3/mqtt.c` - Add manifest job structs. - Extend `BoardMqtt_HandleUpdateManifest()` from save-only to async save + batch download. - Reuse `s_task_busy` so `DOWNLOAD`, `OTA_UPDATE`, and `UPDATE_MANIFEST` are mutually exclusive. - Reuse `Board_Download_File_To_Path()`, `SDCard_Prepare_Business_File()`, `SDCard_Commit_Part_File()`, and `SDCard_Remove_File_If_Exists()`. - Modify `main/boards/sening-xiaozhi-s3/MQTT_Web/local_mqtt_test_server.py` - Add multi-file MP3 upload API. - Add UI controls for course ID and multiple MP3 import. - Generate an `UPDATE_MANIFEST` payload from uploaded files. - Ensure progress panel tracks `UPDATE_MANIFEST` tasks. - No new firmware source files. - No changes to `system.c` expected because existing helpers already overwrite via commit/rename path. --- ### Task 1: Firmware manifest job data structures and helpers **Files:** - Modify: `main/boards/sening-xiaozhi-s3/mqtt.c` - [ ] **Step 1: Add asset/job structs near existing download and OTA job structs** Insert after `board_mqtt_ota_job_t`: ```c typedef struct { char url[BOARD_MQTT_DOWNLOAD_FIELD_MAX_LEN]; char target_path[SDCARD_PATH_MAX_LEN]; char md5[33]; } board_mqtt_manifest_asset_t; typedef struct { char req_id[BOARD_MQTT_DOWNLOAD_FIELD_MAX_LEN]; char course_id[ALARM_ID_MAX_LEN]; char *manifest_json; size_t asset_count; board_mqtt_manifest_asset_t assets[]; } board_mqtt_manifest_job_t; ``` - [ ] **Step 2: Add aggregate manifest progress context after `board_mqtt_download_progress_ctx_t`** ```c typedef struct { const char *req_id; size_t asset_index; size_t asset_count; int64_t last_publish_us; int last_progress; } board_mqtt_manifest_progress_ctx_t; ``` - [ ] **Step 3: Add manifest progress callback after `BoardMqtt_DownloadProgressCallback()`** ```c static void BoardMqtt_ManifestProgressCallback(int progress, const char *info, void *ctx) { board_mqtt_manifest_progress_ctx_t *progress_ctx = (board_mqtt_manifest_progress_ctx_t *)ctx; if (!progress_ctx || progress_ctx->asset_count == 0) { return; } int aggregate = (int)(((progress_ctx->asset_index * 100) + progress) / progress_ctx->asset_count); if (aggregate < 0) { aggregate = 0; } else if (aggregate > 100) { aggregate = 100; } int64_t now = esp_timer_get_time(); bool force = aggregate <= 0 || aggregate >= 100; if (!force && aggregate == progress_ctx->last_progress) { return; } if (!force && progress_ctx->last_publish_us > 0 && now - progress_ctx->last_publish_us < 2000000) { return; } progress_ctx->last_publish_us = now; progress_ctx->last_progress = aggregate; char progress_info[BOARD_MQTT_DOWNLOAD_INFO_MAX_LEN]; snprintf(progress_info, sizeof(progress_info), "asset %u/%u %s", (unsigned)(progress_ctx->asset_index + 1), (unsigned)progress_ctx->asset_count, info ? info : "downloading"); BoardMqtt_PublishProgress(progress_ctx->req_id, aggregate, progress_info); } ``` - [ ] **Step 4: Build firmware to catch type errors** Run: ```bash cmd.exe //c "set MSYSTEM=& idf.py build" ``` Expected: build succeeds or fails only because later tasks are not implemented yet if references are added prematurely. At this task, it should succeed. --- ### Task 2: Firmware asynchronous UPDATE_MANIFEST download task **Files:** - Modify: `main/boards/sening-xiaozhi-s3/mqtt.c` - [ ] **Step 1: Add manifest worker task before `BoardMqtt_DownloadTask()`** ```c static void BoardMqtt_ManifestTask(void *arg) { board_mqtt_manifest_job_t *job = (board_mqtt_manifest_job_t *)arg; if (!job) { vTaskDelete(NULL); return; } Board_Acquire_Sleep_Inhibit(); BoardMqtt_PublishProgress(job->req_id, 0, "saving manifest"); esp_err_t err = SDCard_Save_Manifest(job->course_id, job->manifest_json); char error_info[BOARD_MQTT_DOWNLOAD_INFO_MAX_LEN] = {0}; if (err != ESP_OK) { snprintf(error_info, sizeof(error_info), "%s", esp_err_to_name(err)); } for (size_t i = 0; err == ESP_OK && i < job->asset_count; ++i) { char final_path[SDCARD_PATH_MAX_LEN] = {0}; char part_path[SDCARD_PATH_MAX_LEN] = {0}; board_mqtt_manifest_progress_ctx_t progress_ctx = { .req_id = job->req_id, .asset_index = i, .asset_count = job->asset_count, .last_publish_us = 0, .last_progress = -1, }; err = SDCard_Prepare_Business_File(job->assets[i].target_path, final_path, sizeof(final_path), part_path, sizeof(part_path)); if (err == ESP_OK) { err = Board_Download_File_To_Path(job->assets[i].url, part_path, job->assets[i].md5, BoardMqtt_ManifestProgressCallback, &progress_ctx, error_info, sizeof(error_info)); } else { snprintf(error_info, sizeof(error_info), "%s", esp_err_to_name(err)); } if (err == ESP_OK) { err = SDCard_Commit_Part_File(part_path, final_path); if (err != ESP_OK) { snprintf(error_info, sizeof(error_info), "%s", esp_err_to_name(err)); } } if (err != ESP_OK) { SDCard_Remove_File_If_Exists(part_path); break; } int completed_progress = (int)(((i + 1) * 100) / job->asset_count); char info[BOARD_MQTT_DOWNLOAD_INFO_MAX_LEN]; snprintf(info, sizeof(info), "asset %u/%u completed", (unsigned)(i + 1), (unsigned)job->asset_count); BoardMqtt_PublishProgress(job->req_id, completed_progress, info); } if (err == ESP_OK) { BoardMqtt_PublishProgress(job->req_id, 100, "manifest assets downloaded"); BoardMqtt_PublishAck(job->req_id, true, 0, "manifest saved and assets downloaded"); } else { BoardMqtt_PublishAck(job->req_id, false, BOARD_MQTT_ERR_EXEC_FAILED, error_info[0] ? error_info : esp_err_to_name(err)); } s_task_busy = false; Board_Release_Sleep_Inhibit(); free(job->manifest_json); free(job); vTaskDelete(NULL); } ``` - [ ] **Step 2: Build to catch missing symbol or declaration issues** Run: ```bash cmd.exe //c "set MSYSTEM=& idf.py build" ``` Expected: build succeeds after Task 1 is complete. --- ### Task 3: Firmware UPDATE_MANIFEST handler validation and async dispatch **Files:** - Modify: `main/boards/sening-xiaozhi-s3/mqtt.c:703-730` - [ ] **Step 1: Replace `BoardMqtt_HandleUpdateManifest()` with async implementation** Replace the whole function with: ```c static board_mqtt_action_result_t BoardMqtt_HandleUpdateManifest(const char *req_id, cJSON *payload) { cJSON *course_id = cJSON_GetObjectItem(payload, "course_id"); cJSON *assets = cJSON_GetObjectItem(payload, "assets"); if (!cJSON_IsString(course_id) || !cJSON_IsArray(assets)) { return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "invalid manifest payload"); } int asset_count = cJSON_GetArraySize(assets); if (asset_count <= 0) { return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "empty manifest assets"); } if (s_task_busy) { return BoardMqtt_ResultFail(BOARD_MQTT_ERR_BUSY, "task busy"); } char *manifest_json = cJSON_PrintUnformatted((cJSON *)payload); if (!manifest_json) { return BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, "failed to format manifest"); } size_t job_size = sizeof(board_mqtt_manifest_job_t) + sizeof(board_mqtt_manifest_asset_t) * (size_t)asset_count; board_mqtt_manifest_job_t *job = (board_mqtt_manifest_job_t *)calloc(1, job_size); if (!job) { cJSON_free(manifest_json); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, "no memory for manifest job"); } job->manifest_json = manifest_json; job->asset_count = (size_t)asset_count; if (!BoardMqtt_CopyString(job->req_id, sizeof(job->req_id), req_id) || !BoardMqtt_CopyString(job->course_id, sizeof(job->course_id), course_id->valuestring)) { free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "manifest field too long"); } int index = 0; cJSON *asset = NULL; cJSON_ArrayForEach(asset, assets) { cJSON *url = cJSON_IsObject(asset) ? cJSON_GetObjectItem(asset, "url") : NULL; cJSON *target_path = cJSON_IsObject(asset) ? cJSON_GetObjectItem(asset, "target_path") : NULL; cJSON *md5 = cJSON_IsObject(asset) ? cJSON_GetObjectItem(asset, "md5") : NULL; if (!cJSON_IsString(url) || !cJSON_IsString(target_path)) { free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "invalid manifest asset"); } if (!BoardMqtt_HasPrefix(url->valuestring, "http://") && !BoardMqtt_HasPrefix(url->valuestring, "https://")) { free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "invalid manifest asset url"); } char sd_path[SDCARD_PATH_MAX_LEN]; if (SDCard_Map_Business_Path(target_path->valuestring, sd_path, sizeof(sd_path)) != ESP_OK) { free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "invalid manifest target path"); } if (!BoardMqtt_CopyString(job->assets[index].url, sizeof(job->assets[index].url), url->valuestring) || !BoardMqtt_CopyString(job->assets[index].target_path, sizeof(job->assets[index].target_path), target_path->valuestring)) { free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_INVALID_PARAM, "manifest asset field too long"); } if (cJSON_IsString(md5) && md5->valuestring[0] != '\0') { BoardMqtt_CopyString(job->assets[index].md5, sizeof(job->assets[index].md5), md5->valuestring); } index++; } s_task_busy = true; BaseType_t ret = xTaskCreate(BoardMqtt_ManifestTask, "mqtt_manifest", 8192, job, 5, NULL); if (ret != pdPASS) { s_task_busy = false; free(job->manifest_json); free(job); return BoardMqtt_ResultFail(BOARD_MQTT_ERR_EXEC_FAILED, "failed to create manifest task"); } return BoardMqtt_ResultOk("manifest accepted"); } ``` - [ ] **Step 2: Update dispatch call signature** Change: ```c if (strcmp(action, "UPDATE_MANIFEST") == 0) return BoardMqtt_HandleUpdateManifest(payload); ``` To: ```c if (strcmp(action, "UPDATE_MANIFEST") == 0) return BoardMqtt_HandleUpdateManifest(req_id, payload); ``` - [ ] **Step 3: Build firmware** Run: ```bash cmd.exe //c "set MSYSTEM=& idf.py build" ``` Expected: build succeeds and generates `D:/Project/xiaozhi-esp32/build/xiaozhi.bin`. --- ### Task 4: Web server multi-MP3 upload API **Files:** - Modify: `main/boards/sening-xiaozhi-s3/MQTT_Web/local_mqtt_test_server.py` - [ ] **Step 1: Add POST route** In `do_POST()`, after `/api/upload-mp3` route, add: ```python elif path == "/api/upload-mp3-batch": self._handle_upload_mp3_batch() ``` - [ ] **Step 2: Add batch upload handler after `_handle_upload_mp3()`** ```python def _handle_upload_mp3_batch(self): ctype, pdict = cgi.parse_header(self.headers.get("Content-Type", "")) if ctype != "multipart/form-data": self._send_json(400, {"error": "multipart/form-data required"}) return form = cgi.FieldStorage(fp=self.rfile, headers=self.headers, environ={ "REQUEST_METHOD": "POST", "CONTENT_TYPE": self.headers.get("Content-Type", ""), "CONTENT_LENGTH": self.headers.get("Content-Length", "0"), }) fields = form["file"] if "file" in form else [] if not isinstance(fields, list): fields = [fields] uploads = [] for field in fields: if not field.filename: continue original = safe_filename(field.filename) if not original.lower().endswith(".mp3"): self._send_json(400, {"error": f"only mp3 is supported: {original}"}) return stored_name = f"{int(time.time())}-{uuid.uuid4().hex[:8]}-{original}" full_path = os.path.join(self.state.upload_dir, stored_name) with open(full_path, "wb") as fp: while True: chunk = field.file.read(1024 * 1024) if not chunk: break fp.write(chunk) meta = { "filename": stored_name, "original_name": original, "size": os.path.getsize(full_path), "md5": md5_file(full_path), "url": f"{self.state.lan_base_url}/uploads/{stored_name}", } uploads.append(meta) if not uploads: self._send_json(400, {"error": "file is required"}) return with self.state.upload_lock: self.state.uploads.extend(uploads) self._send_json(200, {"files": uploads}) ``` - [ ] **Step 3: Run Python syntax check** Run: ```bash python -m py_compile "main/boards/sening-xiaozhi-s3/MQTT_Web/local_mqtt_test_server.py" ``` Expected: no output. --- ### Task 5: Web UI for batch manifest download **Files:** - Modify: `main/boards/sening-xiaozhi-s3/MQTT_Web/local_mqtt_test_server.py` - [ ] **Step 1: Add HTML block below single MP3 upload block** Add: ```html
设备收到 UPDATE_MANIFEST 后会保存索引并逐个下载资源,同名文件会被覆盖。