Bounded DHT metadata probe

An exploratory Mainline DHT probe with terminal output and an optional live localhost visualization. Samples infohashes (BEP 51), looks up peers without announcing, retrieves only BEP 9 metadata over outbound IPv4 TCP, verifies the original v1 SHA-1 identity, then emits only torrents with .mp4/.mkv files whose whatsthis.File title matches an IMDb movie/show in ./imdb.sqlite (unique exact or conservatively normalized primary/original/alias title, media type, and guessed year when present). The displayed video files are limited to those IMDb matches, excluding titles explicitly flagged adult by IMDb. Hybrid torrents are accepted by their v1 identity; v2-only IDs are not supported. Guesses (including movie for non-video input) are raw parser output, not a media classification or confidence score.

Building from source requires Go >= 1.26.1, a C compiler (cgo SQLite), Bun, and tsc. Frontend builds are tested with Bun 1.4.2 and TypeScript 6.0.3; no npm packages or dependency installation are needed. Running the crawler requires outbound IPv4 UDP with replies and outbound IPv4 TCP. Build browser assets before any Go command that compiles the application:

bun scripts/build-ui.mjs
GOTOOLCHAIN=local go test ./...
GOTOOLCHAIN=local go vet ./...
GOTOOLCHAIN=local go test -race ./...
GOTOOLCHAIN=local go run ./cmd/crawler -duration=24h -ui-addr=localhost:8080 -imdb-db=./imdb.sqlite
GOTOOLCHAIN=local go run ./cmd/crawler -duration=0 -ui-addr=localhost:8080 -imdb-db=./imdb.sqlite
GOTOOLCHAIN=local go run ./cmd/crawler -duration=0 -interface mullvad -ui-addr=0.0.0.0:8080
GOTOOLCHAIN=local go run github.com/odwrtw/libimdb/cmd/imdb update -db=./imdb.sqlite

IMDb catalog creation, refresh, and resolution use libimdb. Go resolves the pinned module version from go.mod; no sibling checkout is required. The library streams title.basics, title.akas, and title.ratings directly from HTTP/gzip/TSV into one atomic SQLite update, without dataset temporary files. It retains movies, TV movies, series, and mini-series with primary/original titles, aliases, years, and other IMDb facts. Episodes and shorts are excluded. The IMDb library/CLI is CGO-free; the crawler still requires cgo and a C compiler for its existing state/findings SQLite driver.

Run the library CLI directly as shown above. Use -input-dir=/path/to/datasets for offline imports of all three gzip files; -progress=always reports progress on redirected stderr. Ctrl-C cancels the update, and a failed refresh preserves the previous committed catalog.

Migration: the old unversioned imdb.sqlite is incompatible and is never overwritten by the library. An existing libimdb catalog can be selected directly with -imdb-db=/path/to/imdb.sqlite. Otherwise build a new catalog, then explicitly point the crawler at it (leave existing state/findings paths unchanged):

GOTOOLCHAIN=local go run github.com/odwrtw/libimdb/cmd/imdb update -db=./imdb-lib.sqlite
GOTOOLCHAIN=local go run ./cmd/crawler -imdb-db=./imdb-lib.sqlite

The crawler uses Catalog.Resolve with ExcludeAdult: true, not fuzzy candidates or popularity. Titles explicitly flagged adult are excluded before ambiguity detection; unknown adult flags remain eligible. This downstream filter does not change verified-resolution counts. Unicode case/NFC and whitespace normalization are allowed; punctuation and accents are not removed. A specified year is a hard constraint; absent years remain unknown. Ambiguous titles are filtered as IMDb-unmatched rather than choosing the lowest ID. Invalid parser guesses are also unmatched; catalog/operational failures stop the crawler. Primary IMDb titles are retained in output even when an original title or alias matched. IMDb datasets remain subject to their non-commercial terms and attribution requirements, separate from the library code license.

Build and frontend development

sh scripts/build.sh                     # fully static Linux executable: .cache/build/tapis
sh scripts/build.sh .cache/build/crawler # optional output path, relative to the repository
sh scripts/check.sh                     # type checks, frontend tests, Go tests/vet/race

The release script type-checks TypeScript, bundles the three browser entrypoints with Bun, then embeds all HTML, CSS and JavaScript into the Go executable. It statically links cgo with the installed C toolchain (CC, default gcc), using netgo,osusergo,sqlite_omit_load_extension. It requires static libc archives plus file and readelf, and rejects an artifact with an ELF interpreter or shared-library dependencies. This target is Linux; cross-compilation needs a matching static C toolchain. The scripts never install missing tools. Do not use CGO_ENABLED=0: the existing SQLite driver would become a nonfunctional stub.

Bun, TypeScript, frontend source files and an asset directory are not needed at runtime. Databases remain external mutable data; DNS configuration and system data such as CA certificates are not embedded. GOTOOLCHAIN=local only prevents automatic Go toolchain downloads/switching—it does not control static linking or asset embedding. A normal Go development build remains supported after bun scripts/build-ui.mjs.

Frontend source ownership:

• internal/webui/frontend/: strict TypeScript; app.ts, results.ts, and dashboard.ts are the browser entrypoints. live/ separates retained state, observed events, stream lifecycle, display/layout policy, Canvas rendering, and text views. results/ separates navigation, cancellable requests, the controller and DOM rendering. dashboard/ owns saved-library snapshot requests and rendering. JSON contracts are in contracts.ts; keep them aligned with the Go JSON types and API tests.
• internal/webui/static/: authored HTML and readable CSS. navbar.html is shared by all three pages via their <!-- navbar --> slots; the UI build inserts it and marks the current page. Preserve responsive/full-screen behavior when editing layout.
• internal/webui/assets/: ignored, generated HTML and JavaScript; do not edit or commit it. Explicit go:embed inputs make missing pages or bundles fail compilation. Rebuild assets and the serving Go binary after source changes.
• .cache/ui-tests/: generated test-only bundles. bun scripts/build-ui.mjs --test builds these plus production assets. Node graph/activity tests use explicit controller interfaces; browser checks intercept /app.js with the test entrypoint. Production assets expose no controller globals or test hooks.

The shared UI palette is in internal/webui/static/live-palette.css. Edit that one file to recolor Live probe, Results, Dashboard, the legend, graph and activity animations; layout and typography stay in app.css and the page stylesheets. The --live-* tokens are shared across pages despite their original names. Graph colors must be opaque #RRGGBB values, with opacity applied by the renderer. Rebuild/restart the Go app and refresh after editing: CSS is embedded and the graph reads the palette once at startup.

For focused frontend checks:

bun scripts/build-ui.mjs --test
node --test scripts/ui-graph.test.mjs scripts/ui-activity.test.mjs scripts/ui-results.test.mjs scripts/ui-dashboard.test.mjs

The build/check scripts use only already available tools. Keep the tool versions consistent when comparing generated assets or browser performance. Bun transpilation is not type checking; build-ui.mjs runs tsc first. It stages all production bundles before updating embed inputs; builds with unchanged inputs/toolchain produce identical JavaScript. No development server, React, router or runtime frontend dependency is required.

These commands use the normal Go module cache unless you explicitly set GOMODCACHE/GOCACHE. For an opt-in, smaller live smoke test use -duration=2m -max-results=1 -workers=2 -peer-concurrency=1 -peer-attempts=2 -sample-concurrency=1 -dht-rate=5 -dial-timeout=5s. Never run live tests in quick succession against the same DHT contacts, especially with different or deleted state files: their sampling intervals can be up to six hours. A single, reused state database persists cooldowns across restarts. Regular go test ./... uses local fixtures and does not contact public DHT nodes.

The default run budget is 24 hours including DNS/bootstrap, with no result-count stop. -duration=0 runs until interrupted; -max-results=0 disables the output limit. Durations and result limits must be nonnegative.

-ui-addr=HOST:PORT defaults to localhost:8080; 0.0.0.0:8080 binds all IPv4 interfaces. Wildcard listeners accept only Host headers naming locally assigned IPs. Binding beyond loopback exposes observed IPs, infohashes, torrent names and findings; use trusted networks only. -interface=NAME binds crawler IPv4 traffic to the first IPv4 address assigned to that up interface; otherwise system routing chooses it. Keep the same -state-db (default ./.cache/tapis-state.sqlite) across restarts to preserve cooldowns and pending work. IMDb input (./imdb.sqlite), state and findings (./db.sqlite) must be distinct files; state failures stop collection.

Protocol bounds: 10,000 active DHT endpoints, 50,000 recent hash outcomes, 45 seconds/job, 16 lookup queries and 16 peer attempts/hash, 256 peer candidates/hash, 20 seconds/peer attempt, 4 MiB raw metadata/attempt, 64 KiB frames, 1,000 accepted files, 100 displayed files, and bounded bencode depth/count. The DHT socket is read-only (ro=1, no unsolicited replies). No trackers, PEX, announcements, inbound peer listener, torrent payload requests/uploads or payload storage are used. Unsolicited payload-looking bytes are discarded under bounded frame/message handling. The BEP 10/9-only transport enforces limits before allocating/decoding metadata.

Findings dashboard

Use Dashboard in the page navigation, or open /dashboard, for saved IMDb matches across runs. GET /api/dashboard takes no query parameters and returns one consistent read-only SQLite snapshot. The page loads on entry; Refresh snapshot updates it manually. It does not subscribe to live telemetry or contact DHT nodes/peers.

• Totals count distinct IMDb titles, saved infohashes, and currently retained matched-file entries (not unique content or episode counts).
• Whole-torrent sizes count each torrent once; matched-file sizes sum only saved matched files. Neither measures downloaded data. API byte sums are decimal strings to preserve exact integers in JavaScript; integer overflow fails the request rather than silently rounding.
• Type and release-year breakdowns show exact unique-title counts and percentages of all saved IMDb titles, using the same latest-observation metadata and deterministic tie-breaks as Results. Missing/unknown values remain explicit; shares are unavailable when there are no titles.
• The top ten titles are ranked by distinct torrent count, then IMDb ID, with links into Results. Mixed-title torrents can contribute to multiple titles.
• The 30-day discovery chart counts first saved observations per torrent, in UTC, including zero-count days; today is partial. Repeat observations preserve the original discovery date.

The findings database is a mutable saved-library snapshot, not a sightings log: replaced file lists and prior title metadata cannot be reconstructed. Verified metadata without IMDb matches is not retained here. Protocol success remains metadata verified against its infohash, independent of IMDb output; use the live page and telemetry for crawler performance. Dashboard reads have a ten-second deadline and use the existing read-only WAL connection, not the crawler's writer connection. SQLite returns grouped breakdowns and the top ten titles rather than transferring the entire title set to Go. Failed library API queries log their underlying errors to stderr; browser responses retain generic error messages.

For browser validation, use the offline demo documented in AGENTS.md. The browser check also covers dashboard totals, refresh, links, and mobile layout; screenshots are written under .cache/ui-smoke/.

Throughput settings

High-throughput settings are now the ordinary defaults; the redundant -fast flag has been removed. Existing explicit flags still override defaults.

Flag	Default	Range
-workers	64	1–64 hash jobs
-peer-concurrency	2	1–4 attempts/hash
-peer-attempts	128	1–128 global attempts; 0 equals workers
-sample-concurrency	4	1–4 sampling RPCs; 0 uses 1
-dht-rate	100	Positive finite datagrams/s
-dht-burst	2	Positive integer
-dial-timeout	1s	Whole seconds, 1s–20s; 0 uses 1s

The default conservative raw-metadata retention bound is 1 GiB, plus parsing, socket and runtime overhead. Startup diagnostics and config.raw_metadata_mib report it. The 1s dial timeout can skip slow reachable peers. For the former low-traffic settings, explicitly use -workers=4 -peer-concurrency=1 -peer-attempts=4 -sample-concurrency=1 -dht-rate=5 -dial-timeout=5s. Preserve your interface, database paths and UI binding; never reset cooldowns or run a competing crawler. Measure verified resolutions/minute, not IMDb output.

Readable, grouped records stream to stdout as they are verified; parser guesses show labeled fields rather than raw JSON. Colors are enabled only for terminal output (and disabled by NO_COLOR). Untrusted text remains quoted/escaped. Limits, dependency revisions, and the grouped final summary table go to stderr. Only .mp4/.mkv files with IMDb title matches are listed; total_size remains the size of the entire torrent. ./imdb.sqlite must first be created with the libimdb CLI; -imdb-db selects a different database path. File order is metadata order and files: N (showing M, omitted K) makes truncation explicit. UTF-8 name/path fields take precedence over legacy fields when present, without title normalization. Path components, not host filesystem paths, supply basenames. A root guess never overwrites a file guess. Invalid/oversized metadata is rejected; verified torrents without an .mp4 or .mkv file are filtered from output, and other file entries are omitted. The state database retains per-endpoint next-eligible sampling times, bounded provenance, durable pending/staged/retry work, leases, and a bounded outcome history; it stores no payload, raw metadata, or UI record. The separate findings database retains each matched torrent's v1 infohash, name, total size, first/last seen times, and all IMDb-matched video paths, sizes, IDs, titles, kinds, and years; it does not store payload or raw torrent metadata. Findings are upserted by infohash in one transaction, independently of state history eviction; they are not automatically pruned, so db.sqlite may grow over time. A findings write failure stops the crawler rather than silently losing a match. A valid BEP 51 interval is a minimum wait, not a revisit deadline: samples are revisited no earlier than that interval, with a project-imposed ten-minute minimum when the node reports zero. A query reserved before sending but lacking an interval conservatively waits six hours; intervals over six hours are rejected as samples but their longer reported cooldown is still honored. Matched, filtered, and invalid-metadata outcomes suppress repeat work for 24 hours; terminal retrieval failures suppress for one hour. The completed/suppression history keeps at most 50,000 records, so evictions can permit repeats sooner. Pending work survives restart; committed staged batches recover after restart, while a UDP reply lost before its staging transaction commits cannot be recovered. Findings are saved before stdout; findings, stdout, and state acknowledgement do not share a transaction. A crash after a finding write or successful stdout but before durable acknowledgement can repeat output after lease recovery; findings upsert by hash, but stdout is at-least-once across that boundary. A failed stdout write can leave a finding already persisted, and the job remains recoverable. Repeats update the existing finding rather than duplicating it. Sampled endpoints older than 24 hours are pruned once their cooldown has expired. A restart starts a new run with fresh counters and UI, while retaining cooldown and recent-hash records. Deleting or switching the state database removes that cooldown protection (but does not erase findings). A zero-result clean exit is not evidence of feasibility; inspect summary counters. Result limit, deadline, or Ctrl-C cancels probe work and closes its sockets; stdout errors fail the command. When the UI is enabled, a normally completed probe leaves the web listener open with its final state until Ctrl-C (but performs no more DHT queries). Output to a permanently blocked pipe can delay termination because terminal writes are synchronous. IPv6, uTP, v2-only identifiers, raw torrent metadata/UI history persistence, and representativeness are out of scope.

Discovery pauses before issuing another sampling RPC while its one-response durable staging slot is occupied. The scheduler commits a validated response and its learned sampling interval together, then drains it incrementally into at most 128 fresh durable jobs; the ready handoff is 32 entries, the retry-wait budget is 32 hashes, and active leases are bounded by worker count (at most 64). A staged response is bounded by the accepted 16 KiB UDP datagram ceiling. Parallel discovery reserves each endpoint's cooldown before sending and processes replies as they arrive under one heap/staging owner. At most sample-concurrency RPCs/reply handoffs are outstanding (up to four datagrams, 64 KiB of accepted wire data, plus decoding overhead); in-flight endpoints cannot be admitted again by referrals. Replies already in flight may wait in memory for the single durable staging slot. A crash before their staging transaction commits can lose those replies, while their conservative sampling reservations survive. Committed staging and queued/leased work retain the existing recovery guarantees. Saturation pauses sampling rather than dropping admitted work; once the retry share or three-total-attempt budget is exhausted, a visible retry_budget_exhausted terminal history outcome is recorded. Initial retry delay is at least one hour, followed by exponential delay and up to one minute of positive jitter. hash_evictions counts bounded history entries removed to make room (including in-memory and persistent history); expiry alone is not an eviction. The summary reads pending, leased, retry-wait and staged backlog from durable state after shutdown. It does not subtract job attempts from admissions: retries and restored work can make attempts exceed this run's admissions. Canceled leases remain recoverable after expiry. jobs_reported counts results delivered to the summary; lookup and peer-attempt metrics cover those reported jobs only (some started jobs can exit on cancellation without reporting). lookup_queries and lookup_replies count get_peers RPCs; lookup_peers counts addresses returned across jobs, and peer_attempts counts outbound connection attempts. On the first usable peer from get_peers, a worker starts a metadata attempt before querying more DHT nodes; after a failure it tries buffered peers, then resumes the same bounded lookup (up to 16 DHT queries and 16 distinct peers per hash within the 45-second job budget). With parallel peers enabled, each job can have one cancellable lookup in flight while attempts run. Completed attempts are consumed independently of lookup and global-permit waits; the first success cancels unnecessary work, and all in-flight work is joined before reporting the job. Permits remain owned until their attempt results are consumed, and late sibling failures cannot replace a successful result. lookup_time and peer_time sum their respective phases across reported jobs, even when interleaved; they are not wall-clock durations. no_peers includes lookups with no successful RPC replies; lookup_no_reply is that subset (including jobs with no known contacts). timeout is split into job_timeout and peer_timeout for terminal jobs. Peer-stage failure counters count failed attempts (including attempts before an eventual success), not terminal jobs; a job's terminal peer error is classified from its last attempt.

Pinned versions: github.com/odwrtw/whatsthis v0.0.0-20260923161841-ae97c8336c10 (show-season), github.com/anacrolix/dht/v2 v2.23.0, and github.com/anacrolix/torrent v1.60.0 (bencode/v1 metainfo decoding only). The full torrent client is intentionally not used: its 16 MiB advertised-metadata limit does not guard its automatic hybrid file-tree/piece allocations before exposing raw info bytes. The protocol transport and test fixture exercise metadata retrieval without that unsafe decode path.

The live SSE metric schema, bounded distributions, effective configuration and reconciliation rules are defined in docs/telemetry.md. New RPC and peer-attempt totals count work at its ownership boundary, including work from jobs whose results were not delivered; legacy terminal-summary lookup and peer counts still cover reported jobs only.

Read-only measurement

With Node (built-in fetch, TextDecoder, and node:test), observe an existing run using only GET /api/events:

node --check scripts/measure-crawler.mjs
node --test scripts/measure-crawler.test.mjs
node scripts/measure-crawler.mjs --url http://127.0.0.1:8080/ --seconds 1200 --revision YOUR_REVISION --format text
# Optional machine-readable, sanitized aggregate report; choose an ignored local path:
mkdir -p .cache/measurements
node scripts/measure-crawler.mjs --url http://127.0.0.1:8080/ --seconds 1200 --revision YOUR_REVISION --format json > .cache/measurements/baseline.json

The default URL is http://127.0.0.1:8080/, window 1200 seconds (maximum 86400); --format is text or json. The supplied URL must be an HTTP(S) base origin (no path, query or credentials); redirects are not followed. The script does not start, configure or stop a crawler, touch SQLite, fetch findings, or contact peers/DHT. It uses bounded SSE framing and correlations, up to three reconnects with backoff, and aborts the request when the window expires or on Ctrl-C. It stores only aggregate counts in memory; neither JSON nor text contains endpoint, hash or name data. JSON uses schema_version: 1, observer_version, observation, revision, units, and per-run segments with config, counters, rates, DHT/peer ratios, RPC outcomes, gauge coverage, cumulative-histogram deltas, observed-subset durations and diagnostics/warnings. See telemetry semantics.

Do not replay snapshot.recent: window totals come from authoritative counter/histogram differences, not update event counts. Queue depth and active-work means are elapsed-time-weighted over covered intervals (gaps >10 seconds and disconnects excluded). Histogram percentiles are bucket upper-bound estimates (the +Inf bucket gives null); job duration comes only from observed events, with its authoritative completed-job denominator shown separately. Per-stage times sum across concurrent attempts and are not wall time. Ratios of independent window deltas are pipeline rates, not cohort conversion probabilities. A zero denominator yields null. Loss, disconnects, active work at the boundaries, missing telemetry fields and short windows limit inference; servers without the required histograms fail with an explicit unsupported-schema error. Run identity, not a provisional counter correction, determines segmentation. Stale same-run snapshots are ignored and counted in diagnostics. Live sampling should use the same state database/binding, never reset cooldowns or run a competing crawler. If deployment restarted the process, record restart/warm-up. Use a 20–30 minute initial window, extend if sparse, and compare one subsequent change at a time; do not infer causality or statistical significance from a short run.

Live visualization

By default, open http://localhost:8080/. With -ui-addr=192.168.1.20:8080, open http://192.168.1.20:8080/ from a client that can reach that address. The Canvas displays at most 1,600 DHT dots and 128 TCP diamonds, prioritizing selected and hovered contacts' retained referral chains, active/recent attempts, and newly observed contacts over a stable background sample. The UI still retains up to 12,000 DHT endpoints and 1,024 TCP peers for inspection. Displayed and retained counts are separate: hiding a contact to keep the graph responsive is not lost telemetry. Selecting a hidden retained contact through the Inspector brings it into view.

The organic layout groups recorded referrals using bounded attraction and local repulsion, not geography or measured DHT connectivity. Layout work is time-sliced, updates at most about 15 times/second, and stops when settled; query-state changes alone do not restart it. Graph rendering is capped around 30 frames/second during activity, with at most 16 illustrative activity animations. Journeys stay visible until arrival, take 1.2–3 seconds depending on screen distance, and finish with a brief highlight before fading. Animated endpoints stay displayed and stationary through arrival; repeated activity is coalesced and excess bursts summarized rather than interrupting existing journeys. The legend's animation-summary count is separate from telemetry loss; counters and observations update without waiting for animations. Hidden tabs suspend visual work, and reduced-motion mode disables layout animation and pulses. There are no frontend runtime dependencies or third-party graph services.

The crawler links to its bootstrap addresses and contacts restored from saved state (starting points, not replies); separate thin links are observed referrals: a find_node, sample_infohashes, or get_peers reply from A supplied B's contact address. Hover a DHT dot to reveal and highlight its retained first-observed referral chain back to a recorded starting point; hover temporarily takes priority over a clicked selection. Click to inspect the chain or its reported contacts. A starting link does not mean the contact replied; a referral does not prove B replied, was queried, or holds a torrent. Links are drawn only when both endpoints are displayed; missing hops are never bridged. History eviction and telemetry overload can leave gaps. Request journeys represent instrumented DHT and TCP metadata attempts, not measured packet latency (a rate-limited DHT attempt may wait or time out before sending a datagram). Successful completions can travel inward; failures and metadata progress use endpoint highlights, not fictitious returning packets. Click a DHT dot or TCP diamond to inspect its recent observations and timings. The workflow shows BEP 51 sampling → hash admission → peer lookup → BEP 10/9 metadata only → v1 verification → video/IMDb filtering. Returning a hash does not imply the reporting DHT node holds its torrent. This bounded crawl cannot estimate the size of the whole DHT.

The Full screen button fills the browser display and expands the graph; Esc exits. If the browser blocks the Fullscreen API, the button still expands the layout within its tab. The IMDb matches panel retains the last 100 emitted records in memory; select one to see its matched video files, filename-parser guesses, IMDb title/ID/kind/year (when listed), and sizes. At most 20 matched files per record are detailed in the UI and long text is shortened; the terminal output remains the reference for the full result.

The UI and findings API are read-only and make no additional DHT or peer requests. The default localhost address is local-only; -ui-addr can expose them to network clients, so bind only to a trusted interface. The UI shows run elapsed/remaining time (or endless mode), sampling attempts versus revisits, waiting/next-eligible times, and a selected endpoint's reported interval and next sample time. Its peer-lookup/connection counters include observed in-flight attempts; the terminal's job metrics count only reported jobs. Periodic crawler-owned status snapshots restore aggregate counts if the bounded event collector drops telemetry; the graph and Inspector observations can still be incomplete. Its embedded assets and SSE stream have no third-party services; no UI events are persisted. Events, displayed nodes, and recent history are bounded. A slow browser may be reconnected for a new snapshot; if the crawler's non-blocking event collector drops telemetry, the Live contacts header warns that its graph and momentary event details may be incomplete; periodic snapshots restore aggregate counters. The terminal summary remains authoritative. The HTTP listener exposes observed IP addresses, infohashes, and matched torrent names to clients that can reach its configured address; do not expose it to untrusted users. No public live runs are needed for UI development.

Offline UI stress check

Run the opt-in fixture instead of the crawler (no DHT, peers, IMDb DB, or public network):

bun scripts/build-ui.mjs --test
GOTOOLCHAIN=local go run ./internal/webui/testdata/demo -stress -port=8085
# In another terminal, start Chromium on a blank page with remote debugging:
mkdir -p .cache/ui-smoke
chromium --headless --no-sandbox --disable-gpu --disable-background-networking --no-first-run --disable-extensions --disable-dev-shm-usage --user-data-dir="$PWD/.cache/ui-smoke/profile" --remote-debugging-port=9229 --window-size=1600,1080 about:blank
# In a third terminal:
node scripts/ui-stress-check.mjs

--no-sandbox is for restricted headless sandboxes only; omit it for ordinary browsing. Create .cache/ui-smoke/ before starting Chromium and remove its generated profile afterward. The fixture prints when seeding is complete; the check also waits for 12,500 synthetic DHT addresses and 1,100 peers to reach the hub before opening the browser late. It checks the 12,000-node/1,024-peer history, the 1,600-dot/128-diamond display, inspectors, sustained traffic (some intentionally missing completion events), and an EventSource reconnect after the fixture closes SSE every 15 seconds. The fixture defaults to five synthetic DHT requests/second; add -stress-rate=100 for faster traffic with six-contact referral bursts. The check reports first-snapshot time, heap usage, counters, telemetry loss and reconnects; these are diagnostics, not a memory-leak guarantee. It runs for 60 seconds by default; set TAPIS_STRESS_SECONDS=3600 for a longer observation or TAPIS_DEMO_PORT for another fixture port. Stop Chromium and the fixture with Ctrl-C. The default demo (without -stress) remains the screenshot fixture.

With Chromium on about:blank and CDP port 9229, the performance check can instead serve repository assets inside the browser and replace EventSource with a synthetic observer. This needs no fixture server, database, or public network:

bun scripts/build-ui.mjs --test
node --test scripts/ui-graph.test.mjs scripts/ui-activity.test.mjs
TAPIS_UI_OFFLINE=1 TAPIS_UI_CHECK=1 TAPIS_STRESS_SECONDS=30 node scripts/ui-performance-check.mjs

The replay defaults to 100 synthetic DHT requests/second; TAPIS_STRESS_RATE accepts 1–1000. It reports snapshot-handler durations, draw/layout/text timings, long tasks, frame gaps and heap usage. Optional interaction checks cover hidden-contact/path promotion, pointer hit testing, match details, TCP inspection, full-screen fallback/Escape, reduced motion, idle rendering, simulated hidden-tab recovery, and laptop/DPR resizing. Set TAPIS_UI_SCREENSHOTS=1 to save screenshots under .cache/ui-smoke/. Offline replay defaults to .cache/ui-tests/. TAPIS_UI_ASSETS can point to saved previous assets for an identical-workload comparison (omit interaction checks for old versions). Without the current test entrypoint, per-function/layout metrics are unavailable; snapshot timings, frame gaps and event-stream counters are still collected. In native live mode retained counts come from the latest snapshot, and display counts are null. Timings and heap samples are machine-dependent; compare matching viewport, DPR, browser flags, duration and workload, and do not interpret one heap sample as a leak.

For an existing crawler, omit TAPIS_UI_OFFLINE and TAPIS_UI_CHECK:

TAPIS_UI_URL=http://localhost:8080/ TAPIS_STRESS_SECONDS=30 node scripts/ui-performance-check.mjs

Live mode only loads the UI and reads its event stream; it does not inject traffic, change crawler settings, or restart anything. After bun scripts/build-ui.mjs --test, set TAPIS_UI_ASSETS=.cache/ui-tests to preview repository assets against the existing server's native SSE stream without relaunching it; only static asset requests are intercepted. Each check returns the browser to about:blank so it can be repeated. Embedded asset changes require rebuilding/relaunching the program and refreshing the page; preserve the same state database, interface and other crawler arguments.

Found movies & shows

Open /results or use Results in the dashboard navigation. This read-only page browses persisted matches from the findings database across runs, with one entry per IMDb ID. Multiple releases and show episodes are grouped beneath their matched movie/show. Browse newest title discoveries first, filter All / Movies / Shows, or search IMDb primary titles using partial names and small typos. Search ignores case, accents, punctuation and whitespace differences; exact matches rank before prefixes/partials and typo matches. Tokens shorter than four characters are literal, four-to-seven-character tokens allow one edit, and longer tokens allow two (including adjacent transpositions). Search does not include catalog aliases, original titles, torrent names, or titles that have not been found. It never changes the crawler's strict IMDb identity matching.

Select a title to page through its saved torrents, then select a torrent to inspect its infohash, total size, discovery timestamps and files matching that title. Mixed-title torrents show selected-title versus whole-torrent matched-file counts separately. Show more files makes all persisted matching files accessible; the live dashboard's 20-file detail cap does not apply. Filename-parser guesses are not persisted and therefore are not shown here. Search, filters, pagination and selection are retained in the URL for reload and Back/Forward navigation. Refresh resets pagination and reads current findings; this page does not subscribe to crawler events or make DHT/peer requests. Pages are best-effort during collection, not a frozen snapshot. Existing findings are immediately available without a migration or catalog refresh.

GET /api/titles returns {"titles": [...]} with an opaque next_cursor when more matches remain. Each title includes imdb_id, imdb_title, imdb_kind, optional imdb_year, distinct torrents, matched_files, and first_seen/last_seen across its findings. Display metadata comes from one latest observation. Parameters: q (up to 128 Unicode characters, 512 bytes and 16 normalized tokens), kind=movie|show (omit for all), limit=1..100 (default 20), and cursor from the preceding response. A cursor belongs to its normalized query/filter and ranking version; reset it when these change. Browsing without a search query paginates in SQLite; unfiltered browsing fetches canonical metadata only for the selected page. Search ranks the complete found-title set before pagination, retains only a bounded page, and uses a separate read-only SQLite connection with one active title query and a ten-second deadline including reader waits. An incomplete/timed-out search fails explicitly rather than returning partial results. Empty results contain "titles":[]; unavailable findings return 503. The existing findings endpoints below provide torrent lists and full details.

For an offline browser check, first run bun scripts/build-ui.mjs --test, then start GOTOOLCHAIN=local go run ./internal/webui/testdata/demo -port=8085 and open http://127.0.0.1:8085/results. The default demo seeds synthetic titles, multiple releases, mixed-title files and a 55-file show collection in a disposable .cache/ui-demo-*/ database, removed on normal shutdown; it never opens the real findings or state databases. The -stress fixture remains database-free. Run node --test scripts/ui-results.test.mjs for frontend request/navigation tests. With the default demo and Chromium's CDP port 9229 running as described in AGENTS.md, node scripts/ui-browser-check.mjs also checks results pagination, typo/accent search, filters, title/torrent/file navigation and mobile overflow, saving .cache/ui-smoke/results.png and results-mobile.png. These checks use only offline fixtures; remove their generated screenshots/profile afterward. Embedded asset changes require rebuilding/restarting the serving instance with the same state database and bindings.

Findings API

With the default -ui-addr=localhost:8080, GET /api/findings (for example, curl 'http://localhost:8080/api/findings?imdb_id=tt0000042') returns persisted matches from ./db.sqlite across runs, newest first (by first discovery). ?limit=20 defaults to 20 (maximum 100), ?imdb_id=tt... filters by matched IMDb ID, and ?cursor=... accepts the returned next_cursor to continue a page. GET /api/findings/{40-character-v1-infohash} returns one finding (404 if absent). List entries include the torrent's infohash, name, total size, first/last seen timestamps, matched-file count, and IMDb IDs. The detail endpoint additionally includes all matched files with their paths, sizes, IMDb IDs/titles/kinds/years; the live UI's 20-file detail cap does not apply. An empty list has "findings":[]; the last page omits next_cursor. The API is available only when the HTTP server is running (including after the probe finishes with the UI still open). Resetting the state DB does not erase findings. Existing stdout output and recent-hash history are not backfilled into db.sqlite; only matches processed after this change are saved. ./imdb.sqlite remains a read-only input for the crawler.

Verification status

Offline tests, vet and race checks cover the protocol, durable scheduling and 64-job/128-peer pipeline. A user-reported run with the high-throughput settings lasted 9h47m8.386s and recorded 43,047 verified resolutions and 9,425 IMDb-matched emissions: approximately 73.3 verified/minute. This supports the chosen defaults, not a controlled attribution to individual settings. Hash-history evictions can permit repeat resolutions, so these totals are not guaranteed unique hashes. Use offline fixtures for development and preserve sampling cooldowns.