DebaucheryLibrarian/ripunzel
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Reduced reddit requests: stop paging at --after(-indexed) cutoff, reuse listing media for videos, SOCKS5 support for reddit/fetch requests, routed defunct gfycat links to RedGifs.

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DebaucheryLibrarian
2026-11-05 22:33:28 +01:00
co-authored by Claude Opus 5.5
parent 449a0e11c3
commit 309a99755c
10 changed files with 129 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+13 -5
View File
@@ -34,16 +34,24 @@ The pipeline runs from `src/app.js`:
### Networking (`src/proxy.js`, `src/limiter.js`)
- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged).
- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
## Fetching strategy
- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.
## Known caveats / review notes
- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user.
- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`.
- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist.
- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch.
- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
- The README's "API keys" section and date-format notes predate this branch.
- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.