From 449a0e11c3369f305c368ad380d12b6b7c61edae Mon Sep 17 00:00:00 2001 From: DebaucheryLibrarian Date: Thu, 5 Nov 2026 22:18:25 +0100 Subject: [PATCH] Added CLAUDE.md documenting the public reddit API rewrite. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d0d50a8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,49 @@ +# ripunzel + +Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads. + +## Branches + +- `master` is maintained by hand. **Do not commit to or merge into `master`** unless explicitly asked. +- `public-reddit-api` was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into `master` (see commit `2e4ba27`). + +## Running + +- Node `>=24.19.0` (`.nvmrc`). `npm install`, then `npm start -- --user ` (see `src/cli.js` or README for all flags). +- ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via `dist-exiftool`. +- No test suite. Lint with `npx eslint src` (airbnb-base, 4-space indent, CommonJS, `'use strict'`). +- Config uses the `config` package: `config/default.js` is committed; `config/local.js` overrides it and is gitignored, as is `config/cookies.js` (a Netscape cookie-jar dump, exported as a string and pulled in by `local.js`). **Never print, commit or log cookie contents.** +- `output/`, `logs/`, `users*`, `ignore` are local data and gitignored. + +## Architecture + +The pipeline runs from `src/app.js`: + +1. **Sources** (`src/sources/`): `getUserPosts` / `getPosts` fetch the user profile, social links and submissions through the reddit client, plus the existing index (`getIndex`, YAML at `library.index.file`). +2. **Curate** (`src/curate/`): `posts.js` normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs `dissectLink` to choose a host method. `user.js` normalizes the profile. +3. **Info** (`src/fetch/info.js`): calls `methods[host.method]` (in `src/methods/`) to resolve each post into `{ album, items[] }`. If that fails, it falls back to reddit preview images (`redditPreview`). Some methods export `{ fetchInfo, fetchPredata }`; predata (e.g. a RedGifs temporary token) is fetched once per run. +4. **Content** (`src/fetch/content.js`): downloads items (`src/fetch/item.js`, using bhttp), dedupes by blake2 hash, saves to a path built by `src/interpolate.js` from `config.library` patterns, muxes video+audio, writes EXIF and finally updates the index (`src/save/writeToIndex.js`). + +### Reddit client (`src/reddit/client.js`), the core of this branch + +- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth. +- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. If reddit returns HTML instead of JSON, the cookies have most likely expired. The client warns when the soonest cookie expires within 48h. +- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup. +- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 6/min, `minTime` 1500ms (`config.reddit.throttle`). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way. +- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts). + +### Networking (`src/proxy.js`, `src/limiter.js`) + +- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged). +- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons. +- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter. +- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads. + +## Known caveats / review notes + +- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user. +- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway. +- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`. +- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist. +- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch. +- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.