ripunzel/CLAUDE.md

# ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

## Branches

- `master` is maintained by hand. **Do not commit to or merge into `master`** unless explicitly asked.
- `public-reddit-api` was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into `master` (see commit `2e4ba27`).

## Running

- Node `>=24.19.0` (`.nvmrc`). `npm install`, then `npm start -- --user <name>` (see `src/cli.js` or README for all flags).
- ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via `dist-exiftool`.
- No test suite. Lint with `npx eslint src` (airbnb-base, 4-space indent, CommonJS, `'use strict'`).
- Config uses the `config` package: `config/default.js` is committed; `config/local.js` overrides it and is gitignored, as is `config/cookies.js` (a Netscape cookie-jar dump, exported as a string and pulled in by `local.js`). **Never print, commit or log cookie contents.**
- `output/`, `logs/`, `users*`, `ignore` are local data and gitignored.

## Architecture

The pipeline runs from `src/app.js`:

1. **Sources** (`src/sources/`): `getUserPosts` / `getPosts` fetch the user profile, social links and submissions through the reddit client, plus the existing index (`getIndex`, YAML at `library.index.file`).
2. **Curate** (`src/curate/`): `posts.js` normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs `dissectLink` to choose a host method. `user.js` normalizes the profile.
3. **Info** (`src/fetch/info.js`): calls `methods[host.method]` (in `src/methods/`) to resolve each post into `{ album, items[] }`. If that fails, it falls back to reddit preview images (`redditPreview`). Some methods export `{ fetchInfo, fetchPredata }`; predata (e.g. a RedGifs temporary token) is fetched once per run.
4. **Content** (`src/fetch/content.js`): downloads items (`src/fetch/item.js`, using bhttp), dedupes by blake2 hash, saves to a path built by `src/interpolate.js` from `config.library` patterns, muxes video+audio, writes EXIF and finally updates the index (`src/save/writeToIndex.js`).

### Reddit client (`src/reddit/client.js`), the core of this branch

- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. If reddit returns HTML instead of JSON, the cookies have most likely expired. The client warns when the soonest cookie expires within 48h.
- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 6/min, `minTime` 1500ms (`config.reddit.throttle`). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).

### Networking (`src/proxy.js`, `src/limiter.js`)

- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged).
- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.

## Known caveats / review notes

- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user.
- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`.
- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist.
- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch.
- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.