# ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

## Branches

- `master` is maintained by hand. **Do not commit to or merge into `master`** unless explicitly asked.
- `public-reddit-api` was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into `master` (see commit `2e4ba27`).

## Running

- Node `>=24.19.0` (`.nvmrc`). `npm install`, then `npm start -- --user <name>` (see `src/cli.js` or README for all flags).
- ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via `dist-exiftool`.
- No test suite. Lint with `npx eslint src` (airbnb-base, 4-space indent, CommonJS, `'use strict'`).
- Config uses the `config` package: `config/default.js` is committed; `config/local.js` overrides it and is gitignored, as is `config/cookies.js` (a Netscape cookie-jar dump, exported as a string and pulled in by `local.js`). **Never print, commit or log cookie contents.**
- `output/`, `logs/`, `users*`, `ignore` are local data and gitignored.

## Architecture

`src/app.js` processes users **one at a time through the whole pipeline**, via `runUsers()`. While one user's media downloads, the next user is already being prepared (lookahead of one). Each user's index is saved as soon as that user is done. Do not go back to processing stage by stage across all users: that meant hours of reddit requests before the first download, everything lost on a crash, and no way to queue new users.

Per user:

1. **Sources** (`src/sources/`): `getUserPosts(username, tracker)` resolves to `{ status, user }`. Status is `active`, `empty`, `suspended`, `deleted` or `error`. It fetches the profile (`user.js`), the existing index (`getIndex`, YAML at `library.index.file`), social links if needed, and submissions.
2. **Curate** (`src/curate/`): `posts.js` normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs `dissectLink` to choose a host method. `user.js` normalizes the profile.
3. **Info** (`src/fetch/info.js`): calls `methods[host.method]` (in `src/methods/`) to resolve each post into `{ album, items[] }`. If that fails, it falls back to reddit preview images (`redditPreview`). Some methods export `{ fetchInfo, fetchPredata }`. Predata (e.g. a RedGifs temporary token) is cached in `app.js` for `fetch.predataTtl`.
4. **Content** (`src/fetch/content.js`): downloads items (`src/fetch/item.js`, using bhttp), dedupes by blake2 hash, saves to a path built by `src/interpolate.js` from `config.library` patterns, muxes video+audio, writes EXIF and finally updates the index (`src/save/writeToIndex.js`).

Where users come from: `src/queue.js` provides `createListSource` (`--users`/`--file-users`) and `createQueueSource` (`--queue [file]`, default `queue`). `--posts` and `--fetch` still run as small batches.

### Queue (`--queue [file]`, `--wait`)

- The queue file is reread before each next user. A user is removed after finishing, including when suspended or deleted. Users with status `error` stay queued and are skipped for the rest of the run, so they can't loop.
- `<queue>.lock` (holding a pid, created with `wx`) ensures only one process runs a queue. A second `--queue --users …` process only appends its users and exits. Without `--queue`, the lock is ignored.
- At the end of a run, the lock is released first and the queue checked once more, which closes the race with a process that appended while it still saw the lock.
- List files are edited with `src/utils/listFile.js`: reread right before writing, only the matching lines touched, replaced atomically via rename.

### Reddit client (`src/reddit/client.js`), the core of this branch

- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h.
- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).

### Networking (`src/proxy.js`, `src/limiter.js`)

- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.

## Account status and cleanup

- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`).
- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. The record is saved after every user.
- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false.
- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set.

## Fetching strategy

- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.

## Known caveats / review notes

- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
- The README's "API keys" section and date-format notes predate this branch.
- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.