ripunzel/CLAUDE.md

ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

Branches

• master is maintained by hand. Do not commit to or merge into master unless explicitly asked.
• public-reddit-api was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into master (see commit 2e4ba27).

Running

• Node >=24.19.0 (.nvmrc). npm install, then npm start -- --user <name> (see src/cli.js or README for all flags).
• ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via dist-exiftool.
• No test suite. Lint with npx eslint src (airbnb-base, 4-space indent, CommonJS, 'use strict').
• Config uses the config package: config/default.js is committed; config/local.js overrides it and is gitignored, as is config/cookies.js (a Netscape cookie-jar dump, exported as a string and pulled in by local.js). Never print, commit or log cookie contents.
• output/, logs/, users*, ignore are local data and gitignored.

Architecture

The pipeline runs from src/app.js:

1. Sources (src/sources/): getUserPosts / getPosts fetch the user profile, social links and submissions through the reddit client, plus the existing index (getIndex, YAML at library.index.file).
2. Curate (src/curate/): posts.js normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs dissectLink to choose a host method. user.js normalizes the profile.
3. Info (src/fetch/info.js): calls methods[host.method] (in src/methods/) to resolve each post into { album, items[] }. If that fails, it falls back to reddit preview images (redditPreview). Some methods export { fetchInfo, fetchPredata }; predata (e.g. a RedGifs temporary token) is fetched once per run.
4. Content (src/fetch/content.js): downloads items (src/fetch/item.js, using bhttp), dedupes by blake2 hash, saves to a path built by src/interpolate.js from config.library patterns, muxes video+audio, writes EXIF and finally updates the index (src/save/writeToIndex.js).

Reddit client (src/reddit/client.js), the core of this branch

• Calls reddit's public .json endpoints on www.reddit.com (/user/x/about.json, /user/x/submitted.json, /comments/id.json, /api/morechildren.json) with raw_json=1. There is no OAuth.
• Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (config.reddit.cookies, a cookie-jar string, or config.reddit.cookie, a raw header) are sent. If reddit returns HTML instead of JSON, the cookies have most likely expired. The client warns when the soonest cookie expires within 48h.
• Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (data-faceplate-tracking-context attributes), which is fragile and breaks whenever reddit changes its markup.
• All reddit requests go through one Bottleneck limiter: maxConcurrent: 1, reservoir 6/min, minTime 1500ms (config.reddit.throttle). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen outside limiter.schedule(). Retrying inside it previously caused a deadlock; keep it that way.
• Avoid unnecessary reddit calls: gallery posts reuse media_metadata/gallery_data from the submission listing (curate/posts.js → methods/redditAlbum.js). getSubmission (post only, cheap) is separate from getSubmissionWithComments (full comment tree including more stubs, which is expensive and only used for self posts).

Networking (src/proxy.js, src/limiter.js)

• proxy.js exports a wrapped fetch (undici; per-host proxy from config.proxy.hosts with suffix matching, plus a hard config.fetch.timeout) and agentFor() for bhttp. SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly (a warning is logged).
• config.proxy.retryFlipped retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
• limiter.js provides per-host Bottleneck limiters for media downloads only (config.limiter.hosts). It is separate from the reddit API limiter.
• bhttp has no timeout of its own. responseTimeout in fetch/item.js makes it abort the socket. Don't replace this with a Promise.race, which would leak connections and eventually freeze downloads.

Known caveats / review notes

• getUserPosts fetches every submission (limit: Infinity, up to reddit's ~1000-item listing cap); --limit is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user.
• getUserPosts processes users with concurrency 10, but the reddit client is serial anyway.
• redditVideo always re-fetches the submission even though the listing already contains media.reddit_video.
• src/archives/ip.js (the --archives source) calls cheerio as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist.
• Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch.
• Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.