# ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

## Branches

- `master` is maintained by hand. **Do not commit to or merge into `master`** unless explicitly asked.
- `public-reddit-api` was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into `master` (see commit `2e4ba27`).

## Running

- Node `>=24.19.0` (`.nvmrc`). `npm install`, then `npm start -- --user <name>` (see `src/cli.js` or README for all flags).
- ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via `dist-exiftool`.
- No test suite. Lint with `npx eslint src` (airbnb-base, 4-space indent, CommonJS, `'use strict'`).
- Config uses the `config` package: `config/default.js` is committed; `config/local.js` overrides it and is gitignored, as is `config/cookies.js` (a Netscape cookie-jar dump, exported as a string and pulled in by `local.js`). **Never print, commit or log cookie contents.**
- `output/`, `logs/`, `users*`, `ignore` are local data and gitignored.

## Architecture

The pipeline runs from `src/app.js`:

1. **Sources** (`src/sources/`): `getUserPosts` / `getPosts` fetch the user profile, social links and submissions through the reddit client, plus the existing index (`getIndex`, YAML at `library.index.file`).
2. **Curate** (`src/curate/`): `posts.js` normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs `dissectLink` to choose a host method. `user.js` normalizes the profile.
3. **Info** (`src/fetch/info.js`): calls `methods[host.method]` (in `src/methods/`) to resolve each post into `{ album, items[] }`. If that fails, it falls back to reddit preview images (`redditPreview`). Some methods export `{ fetchInfo, fetchPredata }`; predata (e.g. a RedGifs temporary token) is fetched once per run.
4. **Content** (`src/fetch/content.js`): downloads items (`src/fetch/item.js`, using bhttp), dedupes by blake2 hash, saves to a path built by `src/interpolate.js` from `config.library` patterns, muxes video+audio, writes EXIF and finally updates the index (`src/save/writeToIndex.js`).

### Reddit client (`src/reddit/client.js`), the core of this branch

- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h.
- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).

### Networking (`src/proxy.js`, `src/limiter.js`)

- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.

## Account status and cleanup

- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`).
- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. Files are only written at the end of a run.
- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false.
- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set.

## Fetching strategy

- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.

## Known caveats / review notes

- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
- The README's "API keys" section and date-format notes predate this branch.
- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.