v1.6.0
2026-11-11
Release notes
Added
- Startup: Optional root
startup.luaruns once per process after the database opens and before job hooks load. It is not hot-reloaded; edits require a restart. - Multi-Worker: Jobs now support a dynamic queue: instead of a static
urlsarray in config, anon_start(queue)hook can compute the batch each cycle (e.g. from a database) by mutatingqueue.urls. - Multi-Worker:
queue_push(url)lets any worker push new URLs into the live cycle queue mid-run. Re-pushes of already-visited URLs count toduplicatesinstead of re-queueing; pushes beyondMAX_BATCHare rejected. Pushed URLs are picked up by the same worker or a parked one until the queue drains. - Lua Hooks:
on_finished(queue)now receives a final stats table{ urls = {}, processed, failed, pushed, duplicates }when declared with a parameter.defer()now raises inon_start/on_finishedinstead of silently dropping cleanup. - Lua:
engine.platformexposes the build target (OS, libc on Linux, architecture), andspyweb version/checkprint it. - Lua: New
randomtable with cryptographically secure randomness backed by the OS CSPRNG (getrandom):random.bytes(n)(raw bytes),random.hex(n)(lowercase hex), andrandom.int(min, max)(unbiased, inclusive). - Security: Added opt-in
SPYWEB_BLOCK_PRIVATE_NETWORKSto block HTTP requests, redirects,http_multipart,tls_probe, webhooks, and scraper fetches to loopback, private, link-local, and reserved addresses. Disabled by default; incompatible with proxies when enabled.
Changed
- Multi-Worker: Hooks release the job's Lua lock while awaiting, workers now interleave at await; sync Lua stays exclusive.
- Dependencies: Upgrade mlua from 0.11.6 to 0.12.2 (bundled Luau 0.709 → 0.740), rename deprecated
mlua::StringtoLuaString. - Pipeline: Refactor scraper pipeline from free functions into
JobCtx/CycleRunnerstructs and aWorkenum. No behavior change. - Pipeline: Fold coordinator execution into
pipeline.rs(private queue cycle + worker loop);coordinatormodule reduced to the Lua-facing queue gate (Message/PushError/QueueState). No behavior change. - Configuration: Group optional job settings under the defaultable
JobOptionsstruct while preserving the existing flat TOML configuration format reducing files needs to be touch when new config field is added. - Updater: now supports musl/aarch64 from deploy repo assets matching artifact per platform (OS, libc, arch).
Fixed
- CLI:
spyweb debugrunson_startonce and inspects only the first URL fromon_start/urlsinstead of always usingurl. It also writes response/fields artifacts whenoverride_extractis used. - Updater (Windows):
-ono longer fails when the running executable cannot be deleted;.oldfiles are cleaned on the next startup, warning if another process still holds the old version. - CDP:
cdp.launchnow resolvesuser_data_dirto an absolute path before handing it to the browser, fixing Chrome profile errors when a relative path is passed (e.g..browser-profileon Windows). - CDP:
cdp.launchfailures now report the browser's exit status and captured stderr instead of the generic "failed to start within 5s timeout" message.