ripunzel/CLAUDE.md

ripunzel

Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.

Branches

• master is maintained by hand. Do not commit to or merge into master unless explicitly asked.
• public-reddit-api was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into master (see commit 2e4ba27).

Running

• Node >=24.19.0 (.nvmrc). npm install, then npm start -- --user <name> (see src/cli.js or README for all flags).
• ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via dist-exiftool.
• No test suite. Lint with npx eslint src (airbnb-base, 4-space indent, CommonJS, 'use strict').
• Config uses the config package: config/default.js is committed; config/local.js overrides it and is gitignored, as is config/cookies.js (a Netscape cookie-jar dump, exported as a string and pulled in by local.js). Never print, commit or log cookie contents.
• output/, logs/, users*, ignore are local data and gitignored.

Architecture

The pipeline runs from src/app.js:

1. Sources (src/sources/): getUserPosts / getPosts fetch the user profile, social links and submissions through the reddit client, plus the existing index (getIndex, YAML at library.index.file).
2. Curate (src/curate/): posts.js normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs dissectLink to choose a host method. user.js normalizes the profile.
3. Info (src/fetch/info.js): calls methods[host.method] (in src/methods/) to resolve each post into { album, items[] }. If that fails, it falls back to reddit preview images (redditPreview). Some methods export { fetchInfo, fetchPredata }; predata (e.g. a RedGifs temporary token) is fetched once per run.
4. Content (src/fetch/content.js): downloads items (src/fetch/item.js, using bhttp), dedupes by blake2 hash, saves to a path built by src/interpolate.js from config.library patterns, muxes video+audio, writes EXIF and finally updates the index (src/save/writeToIndex.js).

Reddit client (src/reddit/client.js), the core of this branch

• Calls reddit's public .json endpoints on www.reddit.com (/user/x/about.json, /user/x/submitted.json, /comments/id.json, /api/morechildren.json) with raw_json=1. There is no OAuth.
• Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (config.reddit.cookies, a cookie-jar string, or config.reddit.cookie, a raw header) are sent. If reddit returns HTML instead of JSON, the cookies have most likely expired. The client warns when the soonest cookie expires within 48h.
• Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (data-faceplate-tracking-context attributes), which is fragile and breaks whenever reddit changes its markup.
• All reddit requests go through one Bottleneck limiter: maxConcurrent: 1, reservoir 6/min, minTime 1500ms (config.reddit.throttle). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen outside limiter.schedule(). Retrying inside it previously caused a deadlock; keep it that way.
• Avoid unnecessary reddit calls: gallery posts reuse media_metadata/gallery_data from the submission listing (curate/posts.js → methods/redditAlbum.js). getSubmission (post only, cheap) is separate from getSubmissionWithComments (full comment tree including more stubs, which is expensive and only used for self posts).

Networking (src/proxy.js, src/limiter.js)

• proxy.js exports a wrapped fetch (undici; per-host proxy from config.proxy.hosts with suffix matching, plus a hard config.fetch.timeout) and agentFor() for bhttp. Both cover the reddit API when reddit.com is configured. fetch() supports http(s) and SOCKS5 (undici's experimental Socks5ProxyAgent; socks5h maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
• config.proxy.retryFlipped retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
• limiter.js provides per-host Bottleneck limiters for media downloads only (config.limiter.hosts). It is separate from the reddit API limiter.
• bhttp has no timeout of its own. responseTimeout in fetch/item.js makes it abort the socket. Don't replace this with a Promise.race, which would leak connections and eventually freeze downloads.

Fetching strategy

• By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so gaps anywhere in the archive are backfilled.
• With --after <date> or --after-indexed oldest|latest (sort new only), getSubmissions stops paging at the first page that crosses the cutoff, ignoring pinned posts. --after-indexed latest is the cheap "only fetch what's new" mode, and it does not backfill gaps.
• The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
• Media data for videos and galleries comes from the listing, including crossposts via crosspost_parent_list[0]. A post is only fetched again if that data is missing.

Known caveats / review notes

• Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (arctic-shift.photon-reddit.com/api/posts/search?author=…) returns full reddit post objects and is the candidate replacement for the abandoned --archives code (src/archives/, whose sites are gone).
• getUserPosts processes users with concurrency 10, but the reddit client is serial anyway.
• Self posts fetch their full comment tree, which takes at least one request per post (more with more stubs).
• gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label gfycat. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
• The README's "API keys" section and date-format notes predate this branch.
• Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
• No test suite. To test reddit logic offline, stub require('./src/proxy').fetch before requiring the client, and raise reddit.throttle via NODE_CONFIG, or the 6/min limiter applies.