Reduced reddit requests: stop paging at --after(-indexed) cutoff, reuse listing media for videos, SOCKS5 support for reddit/fetch requests, routed defunct gfycat links to RedGifs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -34,16 +34,24 @@ The pipeline runs from `src/app.js`:
 
 ### Networking (`src/proxy.js`, `src/limiter.js`)
 
-- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged).
+- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
 - `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
 - `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
 - bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
 
+## Fetching strategy
+
+- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
+- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
+- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
+- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.
+
 ## Known caveats / review notes
 
-- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user.
+- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
 - `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
-- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`.
+- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
-- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist.
+- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
-- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch.
+- The README's "API keys" section and date-format notes predate this branch.
 - Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
+- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.

README.md

@@ -14,7 +14,7 @@ Most features are optional and can easily be disabled!
 ### Supported hosts
 * Reddit text/self, images, albums and videos[\*](#reddit-videos)
 * Imgur (requires API key as of late 2019)
-* Gfycat
+* Gfycat (via RedGifs, since Gfycat shut down in 2023)
 * RedGifs
 * YouTube
 * PornHub, YouPorn, xHamster, RedTube, xnxx, YouJizz

src/curate/posts.js

@@ -44,6 +44,12 @@ function report(curatedPosts, indexed, user, args) {
     }
 }
 
+// a crosspost's own media fields are empty - the video/gallery data lives on the original
+// post, which reddit embeds in full as crosspost_parent_list[0]
+function mediaSource(post) {
+    return post.crosspost_parent_list?.[0] || post;
+}
+
 function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args) {
     const host = dissectLink(unprint.prefixUrl(post.url, 'https://reddit.com'));
     const permalink = `https://reddit.com${post.permalink}`;
@@ -66,8 +72,11 @@ function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args)
         // front) already embeds these for gallery posts - carrying them through lets
         // methods/redditAlbum.js use them directly instead of re-fetching the same post
         // individually through reddit's own separately-throttled client (see there)
-        mediaMetadata: post.media_metadata || null,
+        mediaMetadata: post.media_metadata || mediaSource(post).media_metadata || null,
-        galleryData: post.gallery_data || null,
+        galleryData: post.gallery_data || mediaSource(post).gallery_data || null,
+        // same idea for v.redd.it videos (methods/redditVideo.js): the listing already has
+        // media.reddit_video (fallback_url, dash_url, has_audio), no need to re-fetch the post
+        media: post.secure_media || post.media || mediaSource(post).secure_media || mediaSource(post).media || null,
         // count only - the full thread (when needed) is fetched separately via methods/self.js
         comments: post.num_comments,
         hash: hashPost(post),

src/dissectLink.js

@@ -56,7 +56,11 @@ const hosts = [
         pattern: new UrlPattern('http(s)\\://(www.)vidble.com/album/:id(?*)'),
     },
     {
-        method: 'gfycat',
+        // gfycat shut down on 2023-09-01. Its NSFW content had already been migrated to
+        // RedGifs under the same (lowercased) ID, which methods/redgifs.js already lowercases -
+        // so an old gfycat link is resolved through RedGifs directly. Label stays 'gfycat' so
+        // --include/--exclude and {host.name} paths behave as before.
+        method: 'redgifs',
         label: 'gfycat',
         pattern: new UrlPattern('http(s)\\://(:server.)gfycat.com/(gifs/detail/)(:id-mobile)(:id-size_restricted)(:id)(.:ext)(?*)'),
     },

src/methods/gfycat.js

@@ -1,46 +0,0 @@
-'use strict';
-
-const bhttp = require('bhttp');
-
-const { agentFor } = require('../proxy');
-const redgifs = require('./redgifs');
-
-async function gfycat(host) {
-    const infoUrl = `https://api.gfycat.com/v1/gfycats/${host.id}`;
-    const res = await bhttp.get(infoUrl, { agent: agentFor(infoUrl) });
-    const data = await res.body;
-
-    if (data.errorMessage) {
-        const redirectRes = await bhttp.head(host.url, {
-            followRedirects: false,
-            agent: agentFor(host.url),
-        });
-
-        if (redirectRes.statusCode === 301) {
-            // Gfycat redirects all NSFW gifs to RedGifs, likely the case
-            return redgifs({
-                ...host,
-                url: `https://www.redgifs.com/watch/${host.id}`,
-                method: 'redgifs',
-                label: 'redgifs',
-            });
-        }
-
-        throw new Error(`Gfycat API returned error for source '${host.url}' (${res.status}): ${data.errorMessage}`);
-    }
-
-    return {
-        album: null,
-        items: [{
-            id: data.gfyItem.gfyName,
-            url: data.gfyItem.webmUrl,
-            title: data.gfyItem.title,
-            description: data.gfyItem.description,
-            type: 'video/webm',
-            datetime: new Date(data.gfyItem.createDate * 1000),
-            original: data.gfyItem,
-        }],
-    };
-}
-
-module.exports = gfycat;

src/methods/methods.js

@@ -3,7 +3,6 @@
 const erome = require('./erome');
 const eroshareAlbum = require('./eroshareAlbum');
 const eroshareItem = require('./eroshareItem');
-const gfycat = require('./gfycat');
 const imgurAlbum = require('./imgurAlbum');
 const imgurImage = require('./imgurImage');
 const redditImage = require('./redditImage');
@@ -21,7 +20,6 @@ module.exports = {
     erome,
     eroshareAlbum,
     eroshareItem,
-    gfycat,
     imgurAlbum,
     imgurImage,
     redditImage,

src/methods/redditVideo.js

@@ -33,11 +33,30 @@ async function findAudio(video, videoId) {
     return findAudioUrl(await res.text(), videoId);
 }
 
-async function redditVideo(host, post, { reddit }) {
+async function getVideo(post, reddit) {
+    // the submission listing already includes media.reddit_video for video posts, and
+    // curate/posts.js carries it through as post.media (also for crossposts) - re-fetching each
+    // video post individually costs one full request against reddit's 6/min budget. Only
+    // fall back to that if the listing didn't have it (e.g. a post still processing).
+    if (post.media?.reddit_video) {
+        return post.media.reddit_video;
+    }
+
     // fetched through the authenticated/throttled client, not a raw fetch() - reddit's
     // anonymous-traffic bot challenge blocks plain unauthenticated .json requests
     const data = await reddit.getSubmission(post.id);
-    const video = data.media.reddit_video;
+    const source = data.crosspost_parent_list?.[0] || data;
+    const video = (source.secure_media || source.media)?.reddit_video;
+
+    if (!video) {
+        throw new Error(`No reddit video data available for post '${post.id}'`);
+    }
+
+    return video;
+}
+
+async function redditVideo(host, post, { reddit }) {
+    const video = await getVideo(post, reddit);
     const videoId = post.host.id || post.id;
 
     const item = {

src/proxy.js

@@ -1,16 +1,13 @@
 'use strict';
 
 const config = require('config');
-const { ProxyAgent } = require('undici');
+const { ProxyAgent, Socks5ProxyAgent } = require('undici');
 const { HttpProxyAgent } = require('http-proxy-agent');
 const { HttpsProxyAgent } = require('https-proxy-agent');
 const { SocksProxyAgent } = require('socks-proxy-agent');
 
-const logger = require('./logger')(__filename);
-
 const dispatcherCache = new Map();
 const nodeAgentCache = new Map();
-const warnedNoSocksForFetch = new Set();
 
 // same suffix matching as config.reddit.cookies' domain filter (src/reddit/client.js): a
 // bare hostname also matches its subdomains, so 'reddit.com' catches 'www.reddit.com' but
@@ -65,9 +62,25 @@ function isSocks(proxyUrl) {
     return /^socks(4a?|5h?):/i.test(proxyUrl);
 }
 
-// undici (which node's built-in fetch() is implemented with) only tunnels through http(s)
+// fetch() goes through undici: http(s) proxies via ProxyAgent (CONNECT), SOCKS5 via
-// proxies (CONNECT) via ProxyAgent - it has no built-in SOCKS support. agentFor() below can
+// Socks5ProxyAgent (undici >= 7.x, still flagged experimental - node prints a one-time
-// still do SOCKS, since bhttp goes through node's plain http/https modules instead.
+// ExperimentalWarning). Socks5ProxyAgent always sends the target as a hostname (address type
+// DOMAIN), i.e. it resolves on the proxy side like 'socks5h', so both spellings map onto it.
+// SOCKS4 isn't supported by undici at all - rather than silently sending what's presumably
+// meant to be private traffic out directly, that's an error (agentFor() below, for bhttp
+// downloads, does support it).
+function createDispatcher(proxyUrl, hostname) {
+    if (/^socks5h?:/i.test(proxyUrl)) {
+        return new Socks5ProxyAgent(proxyUrl.replace(/^socks5h:/i, 'socks5:'));
+    }
+
+    if (isSocks(proxyUrl)) {
+        throw new Error(`SOCKS4 proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (reddit API, host info lookups) - use a socks5:// or http(s):// proxy for this host instead`);
+    }
+
+    return new ProxyAgent(proxyUrl);
+}
+
 function dispatcherFor(targetUrl, { flip = false } = {}) {
     const { hostname } = new URL(targetUrl);
     const proxyUrl = resolveProxyUrl(hostname, { flip });
@@ -76,17 +89,8 @@ function dispatcherFor(targetUrl, { flip = false } = {}) {
         return undefined;
     }
 
-    if (isSocks(proxyUrl)) {
-        if (!warnedNoSocksForFetch.has(proxyUrl)) {
-            warnedNoSocksForFetch.add(proxyUrl);
-            logger.warn(`SOCKS proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (only http(s) proxies can) - these will go out directly instead. Item downloads (which use bhttp) can still use it.`);
-        }
-
-        return undefined;
-    }
-
     if (!dispatcherCache.has(proxyUrl)) {
-        dispatcherCache.set(proxyUrl, new ProxyAgent(proxyUrl));
+        dispatcherCache.set(proxyUrl, createDispatcher(proxyUrl, hostname));
     }
 
     return dispatcherCache.get(proxyUrl);

src/reddit/client.js

@@ -244,9 +244,27 @@ async function getUser(username) {
     return data.data;
 }
 
-async function getSubmissions(username, { sort = 'new', limit = Infinity } = {}) {
+// `until` (a Date, only honoured with sort 'new'): stop paging once a page reaches a post at
+// or before this moment - everything on later pages is older still, so there's no point
+// spending more of reddit's throttled budget (1 page = 1 request at 6/min) on posts the
+// caller is going to discard anyway. The page that crosses the line is still returned in full;
+// filtering out the older posts on it is left to curate/posts.js as before. Pinned posts are
+// ignored for this check, since reddit lists them first on a user's profile regardless of age.
+function reachedUntil(page, until) {
+    if (!until) {
+        return false;
+    }
+
+    const cutoff = until.getTime() / 1000;
+
+    return page.some((post) => !post.pinned && !post.stickied && post.created_utc <= cutoff);
+}
+
+async function getSubmissions(username, { sort = 'new', limit = Infinity, until = null } = {}) {
     const submissions = [];
+    const stopAt = sort === 'new' ? until : null;
     let after = null;
+    let pages = 0;
 
     do {
         // eslint-disable-next-line no-await-in-loop
@@ -260,6 +278,13 @@ async function getSubmissions(username, { sort = 'new', limit = Infinity } = {})
 
         submissions.push(...page);
         after = data.data.after;
+        pages += 1;
+
+        if (after && reachedUntil(page, stopAt)) {
+            logger.verbose(`Stopped fetching submissions for '${username}' after ${pages} page(s), reached ${stopAt.toISOString()}`);
+
+            break;
+        }
     } while (after && submissions.length < limit);
 
     return submissions;

src/sources/getUserPosts.js

@@ -39,11 +39,35 @@ async function getUser(username, reddit) {
     }
 }
 
-async function getPosts(username, reddit, args) {
+// the moment before which posts will be discarded by curate/posts.js anyway (--after, or
+// --after-indexed against the existing index) - lets getSubmissions() stop paging there
+// instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag,
+// every page is still fetched, so gaps anywhere in the archive keep getting backfilled.
+function getCutoff(indexedPosts, args) {
+    const cutoffs = [];
+
+    if (args.after) {
+        cutoffs.push(args.after);
+    }
+
+    if (args.afterIndexed && indexedPosts?.length) {
+        const dates = indexedPosts.map((entry) => new Date(entry.date).getTime()).filter((time) => !Number.isNaN(time));
+
+        if (dates.length) {
+            cutoffs.push(new Date(args.afterIndexed === 'latest' ? Math.max(...dates) : Math.min(...dates)));
+        }
+    }
+
+    // with both set, the later of the two discards the most, so stop there
+    return cutoffs.length ? new Date(Math.max(...cutoffs.map((date) => date.getTime()))) : null;
+}
+
+async function getPosts(username, reddit, args, until) {
     try {
         const submissions = await reddit.getSubmissions(username, {
             sort: args.sort,
             limit: Infinity,
+            until,
         });
 
         logger.info(`Fetched ${submissions.length} submissions for '${username}' (https://reddit.com/user/${username})`);
@@ -67,12 +91,12 @@ function getUserPostsWrap(reddit, args) {
         try {
             const users = await Promise.map(usernames, async (username) => {
                 try {
-                    const [user, posts] = await Promise.all([
+                    // sequential rather than parallel: the index (whose path may depend on the
-                        getUser(username, reddit),
+                    // user's details) decides how far back to page, and every reddit request
-                        getPosts(username, reddit, args),
+                    // goes through the same single-concurrency limiter anyway
-                    ]);
+                    const user = await getUser(username, reddit);
-
                     const { profile, posts: indexed } = await getIndex(user);
+                    const posts = await getPosts(username, reddit, args, getCutoff(indexed, args));
 
                     if (args.archives) {
                         posts.push(...await getArchivedPosts(username, posts, reddit));
@@ -92,7 +116,7 @@ function getUserPostsWrap(reddit, args) {
 
                     return null;
                 } catch (error) {
-                    logger.error(`Failed to fetch posts from 'username': ${error.message}`);
+                    logger.error(`Failed to fetch posts from '${username}': ${error.message}`);
 
                     return null;
                 }