ripunzel/CLAUDE.md

ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

Branches

• master is maintained by hand. Do not commit to or merge into master unless explicitly asked.
• public-reddit-api was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into master (see commit 2e4ba27).

Running

• Node >=24.19.0 (.nvmrc). npm install, then npm start -- --user <name> (see src/cli.js or README for all flags).
• ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via dist-exiftool.
• No test suite. Lint with npx eslint src (airbnb-base, 4-space indent, CommonJS, 'use strict').
• Config uses the config package: config/default.js is committed; config/local.js overrides it and is gitignored, as is config/cookies.js (a Netscape cookie-jar dump, exported as a string and pulled in by local.js). Never print, commit or log cookie contents.
• output/, logs/, users*, ignore are local data and gitignored.

Architecture

src/app.js processes users one at a time through the whole pipeline, via runUsers(). While one user's media downloads, the next user is already being prepared (lookahead of one). Each user's index is saved as soon as that user is done. Do not go back to processing stage by stage across all users: that meant hours of reddit requests before the first download, everything lost on a crash, and no way to queue new users.

Per user:

1. Sources (src/sources/): getUserPosts(username, tracker) resolves to { status, user }. Status is active, empty, suspended, deleted or error. It fetches the profile (user.js), the existing index (getIndex, YAML at library.index.file), social links if needed, and submissions.
2. Curate (src/curate/): posts.js normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs dissectLink to choose a host method. user.js normalizes the profile.
3. Info (src/fetch/info.js): calls methods[host.method] (in src/methods/) to resolve each post into { album, items[] }. If that fails, it falls back to reddit preview images (redditPreview). Some methods export { fetchInfo, fetchPredata }. Predata (e.g. a RedGifs temporary token) is cached in app.js for fetch.predataTtl.
4. Content (src/fetch/content.js): downloads items (src/fetch/item.js, using bhttp), dedupes by blake2 hash, saves to a path built by src/interpolate.js from config.library patterns, muxes video+audio, writes EXIF and finally updates the index (src/save/writeToIndex.js).

Where users come from: src/queue.js provides createListSource (--users/--file-users) and createQueueSource (--queue [file], default queue). --posts and --fetch still run as small batches.

Queue (--queue [file], --wait)

• The queue file is reread before each next user. A user is removed after finishing, including when suspended or deleted. Users with status error stay queued and are skipped for the rest of the run, so they can't loop.
• <queue>.lock (holding a pid, created with wx) ensures only one process runs a queue. A second --queue --users … process only appends its users and exits. Without --queue, the lock is ignored.
• At the end of a run, the lock is released first and the queue checked once more, which closes the race with a process that appended while it still saw the lock.
• List files are edited with src/utils/listFile.js: reread right before writing, only the matching lines touched, replaced atomically via rename.

Reddit client (src/reddit/client.js), the core of this branch

• Calls reddit's public .json endpoints on www.reddit.com (/user/x/about.json, /user/x/submitted.json, /comments/id.json, /api/morechildren.json) with raw_json=1. There is no OAuth.
• Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (config.reddit.cookies, a cookie-jar string, or config.reddit.cookie, a raw header) are sent. Live testing (2026-11) showed that only reddit_session (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when reddit_session is missing or expires within 48h.
• Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (data-faceplate-tracking-context attributes), which is fragile and breaks whenever reddit changes its markup.
• All reddit requests go through one Bottleneck limiter: maxConcurrent: 1, reservoir 15/min, minTime 1500ms (config.reddit.throttle). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen outside limiter.schedule(). Retrying inside it previously caused a deadlock; keep it that way.
• Avoid unnecessary reddit calls: gallery posts reuse media_metadata/gallery_data from the submission listing (curate/posts.js → methods/redditAlbum.js). getSubmission (post only, cheap) is separate from getSubmissionWithComments (full comment tree including more stubs, which is expensive and only used for self posts).

Networking (src/proxy.js, src/limiter.js)

• proxy.js exports a wrapped fetch (undici; per-host proxy from config.proxy.hosts with suffix matching, plus a hard config.fetch.timeout) and agentFor() for bhttp. Both cover the reddit API when reddit.com is configured. fetch() supports http(s) and SOCKS5 (undici's experimental Socks5ProxyAgent; socks5h maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
• config.proxy.retryFlipped retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
• limiter.js provides per-host Bottleneck limiters for media downloads only (config.limiter.hosts). It is separate from the reddit API limiter.
• bhttp has no timeout of its own. responseTimeout in fetch/item.js makes it abort the socket. Don't replace this with a Promise.race, which would leak connections and eventually freeze downloads.

Account status and cleanup

• src/sources/user.js fetchUser() classifies accounts as active, suspended (about.json returns 200 with is_suspended: true; there is no date), deleted (JSON 404, which may also be a shadowban) or error. error (network failure, 5xx, HTML bot block) must never count as dead; only genuine JSON API errors carry error.status (see requestJson).
• Suspended and deleted users are skipped after that one request. src/sources/accountStatus.js records them in users_dead.yml (config.users.dead) with firstSeen. Once they have been dead for retainDays[status] (default 90), they are removed from the --file-users file, never from --users. A record is dropped if the account turns out active again. The record is saved after every user.
• --cleanup runs only this check (one request per user) without archiving. Regular archive runs do it too unless users.dead.duringArchive is false.
• Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or --redownload-profile is set.

Fetching strategy

• By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so gaps anywhere in the archive are backfilled.
• With --after <date> or --after-indexed oldest|latest (sort new only), getSubmissions stops paging at the first page that crosses the cutoff, ignoring pinned posts. --after-indexed latest is the cheap "only fetch what's new" mode, and it does not backfill gaps.
• The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
• Media data for videos and galleries comes from the listing, including crossposts via crosspost_parent_list[0]. A post is only fetched again if that data is missing.

Known caveats / review notes

• Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (arctic-shift.photon-reddit.com/api/posts/search?author=…) returns full reddit post objects and is the candidate replacement for the abandoned --archives code (src/archives/, whose sites are gone).
• getUserPosts processes users with concurrency 10, but the reddit client is serial anyway.
• Self posts fetch their full comment tree, which takes at least one request per post (more with more stubs).
• gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label gfycat. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
• The README's "API keys" section and date-format notes predate this branch.
• Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
• No test suite. To test reddit logic offline, stub require('./src/proxy').fetch before requiring the client, and raise reddit.throttle via NODE_CONFIG, or the 6/min limiter applies.