From 309a99755ca663284bb6e40d4321dbd4bab2f419 Mon Sep 17 00:00:00 2001 From: DebaucheryLibrarian Date: Thu, 5 Nov 2026 22:33:28 +0100 Subject: [PATCH] Reduced reddit requests: stop paging at --after(-indexed) cutoff, reuse listing media for videos, SOCKS5 support for reddit/fetch requests, routed defunct gfycat links to RedGifs. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 18 +++++++++++---- README.md | 2 +- src/curate/posts.js | 13 +++++++++-- src/dissectLink.js | 6 ++++- src/methods/gfycat.js | 46 ------------------------------------- src/methods/methods.js | 2 -- src/methods/redditVideo.js | 23 +++++++++++++++++-- src/proxy.js | 38 ++++++++++++++++-------------- src/reddit/client.js | 27 +++++++++++++++++++++- src/sources/getUserPosts.js | 38 ++++++++++++++++++++++++------ 10 files changed, 129 insertions(+), 84 deletions(-) delete mode 100644 src/methods/gfycat.js diff --git a/CLAUDE.md b/CLAUDE.md index d0d50a8..50a9b31 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,16 +34,24 @@ The pipeline runs from `src/app.js`: ### Networking (`src/proxy.js`, `src/limiter.js`) -- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged). +- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants. - `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons. - `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter. - bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads. +## Fetching strategy + +- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**. +- With `--after ` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps. +- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff. +- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing. + ## Known caveats / review notes -- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user. +- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone). - `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway. -- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`. -- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist. -- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch. +- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs). +- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online. +- The README's "API keys" section and date-format notes predate this branch. - Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior. +- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies. diff --git a/README.md b/README.md index 1703cee..75f74cd 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Most features are optional and can easily be disabled! ### Supported hosts * Reddit text/self, images, albums and videos[\*](#reddit-videos) * Imgur (requires API key as of late 2019) -* Gfycat +* Gfycat (via RedGifs, since Gfycat shut down in 2023) * RedGifs * YouTube * PornHub, YouPorn, xHamster, RedTube, xnxx, YouJizz diff --git a/src/curate/posts.js b/src/curate/posts.js index 7680c91..85c456d 100644 --- a/src/curate/posts.js +++ b/src/curate/posts.js @@ -44,6 +44,12 @@ function report(curatedPosts, indexed, user, args) { } } +// a crosspost's own media fields are empty - the video/gallery data lives on the original +// post, which reddit embeds in full as crosspost_parent_list[0] +function mediaSource(post) { + return post.crosspost_parent_list?.[0] || post; +} + function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args) { const host = dissectLink(unprint.prefixUrl(post.url, 'https://reddit.com')); const permalink = `https://reddit.com${post.permalink}`; @@ -66,8 +72,11 @@ function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args) // front) already embeds these for gallery posts - carrying them through lets // methods/redditAlbum.js use them directly instead of re-fetching the same post // individually through reddit's own separately-throttled client (see there) - mediaMetadata: post.media_metadata || null, - galleryData: post.gallery_data || null, + mediaMetadata: post.media_metadata || mediaSource(post).media_metadata || null, + galleryData: post.gallery_data || mediaSource(post).gallery_data || null, + // same idea for v.redd.it videos (methods/redditVideo.js): the listing already has + // media.reddit_video (fallback_url, dash_url, has_audio), no need to re-fetch the post + media: post.secure_media || post.media || mediaSource(post).secure_media || mediaSource(post).media || null, // count only - the full thread (when needed) is fetched separately via methods/self.js comments: post.num_comments, hash: hashPost(post), diff --git a/src/dissectLink.js b/src/dissectLink.js index f440bec..4109d86 100644 --- a/src/dissectLink.js +++ b/src/dissectLink.js @@ -56,7 +56,11 @@ const hosts = [ pattern: new UrlPattern('http(s)\\://(www.)vidble.com/album/:id(?*)'), }, { - method: 'gfycat', + // gfycat shut down on 2023-09-01. Its NSFW content had already been migrated to + // RedGifs under the same (lowercased) ID, which methods/redgifs.js already lowercases - + // so an old gfycat link is resolved through RedGifs directly. Label stays 'gfycat' so + // --include/--exclude and {host.name} paths behave as before. + method: 'redgifs', label: 'gfycat', pattern: new UrlPattern('http(s)\\://(:server.)gfycat.com/(gifs/detail/)(:id-mobile)(:id-size_restricted)(:id)(.:ext)(?*)'), }, diff --git a/src/methods/gfycat.js b/src/methods/gfycat.js deleted file mode 100644 index d622557..0000000 --- a/src/methods/gfycat.js +++ /dev/null @@ -1,46 +0,0 @@ -'use strict'; - -const bhttp = require('bhttp'); - -const { agentFor } = require('../proxy'); -const redgifs = require('./redgifs'); - -async function gfycat(host) { - const infoUrl = `https://api.gfycat.com/v1/gfycats/${host.id}`; - const res = await bhttp.get(infoUrl, { agent: agentFor(infoUrl) }); - const data = await res.body; - - if (data.errorMessage) { - const redirectRes = await bhttp.head(host.url, { - followRedirects: false, - agent: agentFor(host.url), - }); - - if (redirectRes.statusCode === 301) { - // Gfycat redirects all NSFW gifs to RedGifs, likely the case - return redgifs({ - ...host, - url: `https://www.redgifs.com/watch/${host.id}`, - method: 'redgifs', - label: 'redgifs', - }); - } - - throw new Error(`Gfycat API returned error for source '${host.url}' (${res.status}): ${data.errorMessage}`); - } - - return { - album: null, - items: [{ - id: data.gfyItem.gfyName, - url: data.gfyItem.webmUrl, - title: data.gfyItem.title, - description: data.gfyItem.description, - type: 'video/webm', - datetime: new Date(data.gfyItem.createDate * 1000), - original: data.gfyItem, - }], - }; -} - -module.exports = gfycat; diff --git a/src/methods/methods.js b/src/methods/methods.js index 171a9c9..164d198 100644 --- a/src/methods/methods.js +++ b/src/methods/methods.js @@ -3,7 +3,6 @@ const erome = require('./erome'); const eroshareAlbum = require('./eroshareAlbum'); const eroshareItem = require('./eroshareItem'); -const gfycat = require('./gfycat'); const imgurAlbum = require('./imgurAlbum'); const imgurImage = require('./imgurImage'); const redditImage = require('./redditImage'); @@ -21,7 +20,6 @@ module.exports = { erome, eroshareAlbum, eroshareItem, - gfycat, imgurAlbum, imgurImage, redditImage, diff --git a/src/methods/redditVideo.js b/src/methods/redditVideo.js index b46e910..5faa697 100644 --- a/src/methods/redditVideo.js +++ b/src/methods/redditVideo.js @@ -33,11 +33,30 @@ async function findAudio(video, videoId) { return findAudioUrl(await res.text(), videoId); } -async function redditVideo(host, post, { reddit }) { +async function getVideo(post, reddit) { + // the submission listing already includes media.reddit_video for video posts, and + // curate/posts.js carries it through as post.media (also for crossposts) - re-fetching each + // video post individually costs one full request against reddit's 6/min budget. Only + // fall back to that if the listing didn't have it (e.g. a post still processing). + if (post.media?.reddit_video) { + return post.media.reddit_video; + } + // fetched through the authenticated/throttled client, not a raw fetch() - reddit's // anonymous-traffic bot challenge blocks plain unauthenticated .json requests const data = await reddit.getSubmission(post.id); - const video = data.media.reddit_video; + const source = data.crosspost_parent_list?.[0] || data; + const video = (source.secure_media || source.media)?.reddit_video; + + if (!video) { + throw new Error(`No reddit video data available for post '${post.id}'`); + } + + return video; +} + +async function redditVideo(host, post, { reddit }) { + const video = await getVideo(post, reddit); const videoId = post.host.id || post.id; const item = { diff --git a/src/proxy.js b/src/proxy.js index 7624fd9..a44330a 100644 --- a/src/proxy.js +++ b/src/proxy.js @@ -1,16 +1,13 @@ 'use strict'; const config = require('config'); -const { ProxyAgent } = require('undici'); +const { ProxyAgent, Socks5ProxyAgent } = require('undici'); const { HttpProxyAgent } = require('http-proxy-agent'); const { HttpsProxyAgent } = require('https-proxy-agent'); const { SocksProxyAgent } = require('socks-proxy-agent'); -const logger = require('./logger')(__filename); - const dispatcherCache = new Map(); const nodeAgentCache = new Map(); -const warnedNoSocksForFetch = new Set(); // same suffix matching as config.reddit.cookies' domain filter (src/reddit/client.js): a // bare hostname also matches its subdomains, so 'reddit.com' catches 'www.reddit.com' but @@ -65,9 +62,25 @@ function isSocks(proxyUrl) { return /^socks(4a?|5h?):/i.test(proxyUrl); } -// undici (which node's built-in fetch() is implemented with) only tunnels through http(s) -// proxies (CONNECT) via ProxyAgent - it has no built-in SOCKS support. agentFor() below can -// still do SOCKS, since bhttp goes through node's plain http/https modules instead. +// fetch() goes through undici: http(s) proxies via ProxyAgent (CONNECT), SOCKS5 via +// Socks5ProxyAgent (undici >= 7.x, still flagged experimental - node prints a one-time +// ExperimentalWarning). Socks5ProxyAgent always sends the target as a hostname (address type +// DOMAIN), i.e. it resolves on the proxy side like 'socks5h', so both spellings map onto it. +// SOCKS4 isn't supported by undici at all - rather than silently sending what's presumably +// meant to be private traffic out directly, that's an error (agentFor() below, for bhttp +// downloads, does support it). +function createDispatcher(proxyUrl, hostname) { + if (/^socks5h?:/i.test(proxyUrl)) { + return new Socks5ProxyAgent(proxyUrl.replace(/^socks5h:/i, 'socks5:')); + } + + if (isSocks(proxyUrl)) { + throw new Error(`SOCKS4 proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (reddit API, host info lookups) - use a socks5:// or http(s):// proxy for this host instead`); + } + + return new ProxyAgent(proxyUrl); +} + function dispatcherFor(targetUrl, { flip = false } = {}) { const { hostname } = new URL(targetUrl); const proxyUrl = resolveProxyUrl(hostname, { flip }); @@ -76,17 +89,8 @@ function dispatcherFor(targetUrl, { flip = false } = {}) { return undefined; } - if (isSocks(proxyUrl)) { - if (!warnedNoSocksForFetch.has(proxyUrl)) { - warnedNoSocksForFetch.add(proxyUrl); - logger.warn(`SOCKS proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (only http(s) proxies can) - these will go out directly instead. Item downloads (which use bhttp) can still use it.`); - } - - return undefined; - } - if (!dispatcherCache.has(proxyUrl)) { - dispatcherCache.set(proxyUrl, new ProxyAgent(proxyUrl)); + dispatcherCache.set(proxyUrl, createDispatcher(proxyUrl, hostname)); } return dispatcherCache.get(proxyUrl); diff --git a/src/reddit/client.js b/src/reddit/client.js index 0725757..335aa51 100644 --- a/src/reddit/client.js +++ b/src/reddit/client.js @@ -244,9 +244,27 @@ async function getUser(username) { return data.data; } -async function getSubmissions(username, { sort = 'new', limit = Infinity } = {}) { +// `until` (a Date, only honoured with sort 'new'): stop paging once a page reaches a post at +// or before this moment - everything on later pages is older still, so there's no point +// spending more of reddit's throttled budget (1 page = 1 request at 6/min) on posts the +// caller is going to discard anyway. The page that crosses the line is still returned in full; +// filtering out the older posts on it is left to curate/posts.js as before. Pinned posts are +// ignored for this check, since reddit lists them first on a user's profile regardless of age. +function reachedUntil(page, until) { + if (!until) { + return false; + } + + const cutoff = until.getTime() / 1000; + + return page.some((post) => !post.pinned && !post.stickied && post.created_utc <= cutoff); +} + +async function getSubmissions(username, { sort = 'new', limit = Infinity, until = null } = {}) { const submissions = []; + const stopAt = sort === 'new' ? until : null; let after = null; + let pages = 0; do { // eslint-disable-next-line no-await-in-loop @@ -260,6 +278,13 @@ async function getSubmissions(username, { sort = 'new', limit = Infinity } = {}) submissions.push(...page); after = data.data.after; + pages += 1; + + if (after && reachedUntil(page, stopAt)) { + logger.verbose(`Stopped fetching submissions for '${username}' after ${pages} page(s), reached ${stopAt.toISOString()}`); + + break; + } } while (after && submissions.length < limit); return submissions; diff --git a/src/sources/getUserPosts.js b/src/sources/getUserPosts.js index 3264a05..fdbdf69 100644 --- a/src/sources/getUserPosts.js +++ b/src/sources/getUserPosts.js @@ -39,11 +39,35 @@ async function getUser(username, reddit) { } } -async function getPosts(username, reddit, args) { +// the moment before which posts will be discarded by curate/posts.js anyway (--after, or +// --after-indexed against the existing index) - lets getSubmissions() stop paging there +// instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag, +// every page is still fetched, so gaps anywhere in the archive keep getting backfilled. +function getCutoff(indexedPosts, args) { + const cutoffs = []; + + if (args.after) { + cutoffs.push(args.after); + } + + if (args.afterIndexed && indexedPosts?.length) { + const dates = indexedPosts.map((entry) => new Date(entry.date).getTime()).filter((time) => !Number.isNaN(time)); + + if (dates.length) { + cutoffs.push(new Date(args.afterIndexed === 'latest' ? Math.max(...dates) : Math.min(...dates))); + } + } + + // with both set, the later of the two discards the most, so stop there + return cutoffs.length ? new Date(Math.max(...cutoffs.map((date) => date.getTime()))) : null; +} + +async function getPosts(username, reddit, args, until) { try { const submissions = await reddit.getSubmissions(username, { sort: args.sort, limit: Infinity, + until, }); logger.info(`Fetched ${submissions.length} submissions for '${username}' (https://reddit.com/user/${username})`); @@ -67,12 +91,12 @@ function getUserPostsWrap(reddit, args) { try { const users = await Promise.map(usernames, async (username) => { try { - const [user, posts] = await Promise.all([ - getUser(username, reddit), - getPosts(username, reddit, args), - ]); - + // sequential rather than parallel: the index (whose path may depend on the + // user's details) decides how far back to page, and every reddit request + // goes through the same single-concurrency limiter anyway + const user = await getUser(username, reddit); const { profile, posts: indexed } = await getIndex(user); + const posts = await getPosts(username, reddit, args, getCutoff(indexed, args)); if (args.archives) { posts.push(...await getArchivedPosts(username, posts, reddit)); @@ -92,7 +116,7 @@ function getUserPostsWrap(reddit, args) { return null; } catch (error) { - logger.error(`Failed to fetch posts from 'username': ${error.message}`); + logger.error(`Failed to fetch posts from '${username}': ${error.message}`); return null; }