Reduced reddit requests: stop paging at --after(-indexed) cutoff, reuse listing media for videos, SOCKS5 support for reddit/fetch requests, routed defunct gfycat links to RedGifs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -34,16 +34,24 @@ The pipeline runs from `src/app.js`:
|
||||
|
||||
### Networking (`src/proxy.js`, `src/limiter.js`)
|
||||
|
||||
- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. **SOCKS proxies only work for bhttp item downloads. fetch()-based calls, including the reddit API, then go out directly** (a warning is logged).
|
||||
- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
|
||||
- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
|
||||
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
|
||||
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
|
||||
|
||||
## Fetching strategy
|
||||
|
||||
- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
|
||||
- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
|
||||
- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
|
||||
- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.
|
||||
|
||||
## Known caveats / review notes
|
||||
|
||||
- `getUserPosts` fetches **every** submission (`limit: Infinity`, up to reddit's ~1000-item listing cap); `--limit` is only applied later in curation. With 100 posts per page at 6 requests/min, large accounts take minutes per user.
|
||||
- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
|
||||
- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
|
||||
- `redditVideo` always re-fetches the submission even though the listing already contains `media.reddit_video`.
|
||||
- `src/archives/ip.js` (the `--archives` source) calls `cheerio` as a function, which cheerio 1.x no longer supports. It is broken and the target site may no longer exist.
|
||||
- Dead or legacy hosts are still present: gfycat, eroshare, vidble. The README's "API keys" section and date-format notes predate this branch.
|
||||
- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
|
||||
- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
|
||||
- The README's "API keys" section and date-format notes predate this branch.
|
||||
- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
|
||||
- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.
|
||||
|
||||
@@ -14,7 +14,7 @@ Most features are optional and can easily be disabled!
|
||||
### Supported hosts
|
||||
* Reddit text/self, images, albums and videos[\*](#reddit-videos)
|
||||
* Imgur (requires API key as of late 2019)
|
||||
* Gfycat
|
||||
* Gfycat (via RedGifs, since Gfycat shut down in 2023)
|
||||
* RedGifs
|
||||
* YouTube
|
||||
* PornHub, YouPorn, xHamster, RedTube, xnxx, YouJizz
|
||||
|
||||
+11
-2
@@ -44,6 +44,12 @@ function report(curatedPosts, indexed, user, args) {
|
||||
}
|
||||
}
|
||||
|
||||
// a crosspost's own media fields are empty - the video/gallery data lives on the original
|
||||
// post, which reddit embeds in full as crosspost_parent_list[0]
|
||||
function mediaSource(post) {
|
||||
return post.crosspost_parent_list?.[0] || post;
|
||||
}
|
||||
|
||||
function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args) {
|
||||
const host = dissectLink(unprint.prefixUrl(post.url, 'https://reddit.com'));
|
||||
const permalink = `https://reddit.com${post.permalink}`;
|
||||
@@ -66,8 +72,11 @@ function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args)
|
||||
// front) already embeds these for gallery posts - carrying them through lets
|
||||
// methods/redditAlbum.js use them directly instead of re-fetching the same post
|
||||
// individually through reddit's own separately-throttled client (see there)
|
||||
mediaMetadata: post.media_metadata || null,
|
||||
galleryData: post.gallery_data || null,
|
||||
mediaMetadata: post.media_metadata || mediaSource(post).media_metadata || null,
|
||||
galleryData: post.gallery_data || mediaSource(post).gallery_data || null,
|
||||
// same idea for v.redd.it videos (methods/redditVideo.js): the listing already has
|
||||
// media.reddit_video (fallback_url, dash_url, has_audio), no need to re-fetch the post
|
||||
media: post.secure_media || post.media || mediaSource(post).secure_media || mediaSource(post).media || null,
|
||||
// count only - the full thread (when needed) is fetched separately via methods/self.js
|
||||
comments: post.num_comments,
|
||||
hash: hashPost(post),
|
||||
|
||||
+5
-1
@@ -56,7 +56,11 @@ const hosts = [
|
||||
pattern: new UrlPattern('http(s)\\://(www.)vidble.com/album/:id(?*)'),
|
||||
},
|
||||
{
|
||||
method: 'gfycat',
|
||||
// gfycat shut down on 2023-09-01. Its NSFW content had already been migrated to
|
||||
// RedGifs under the same (lowercased) ID, which methods/redgifs.js already lowercases -
|
||||
// so an old gfycat link is resolved through RedGifs directly. Label stays 'gfycat' so
|
||||
// --include/--exclude and {host.name} paths behave as before.
|
||||
method: 'redgifs',
|
||||
label: 'gfycat',
|
||||
pattern: new UrlPattern('http(s)\\://(:server.)gfycat.com/(gifs/detail/)(:id-mobile)(:id-size_restricted)(:id)(.:ext)(?*)'),
|
||||
},
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
const bhttp = require('bhttp');
|
||||
|
||||
const { agentFor } = require('../proxy');
|
||||
const redgifs = require('./redgifs');
|
||||
|
||||
async function gfycat(host) {
|
||||
const infoUrl = `https://api.gfycat.com/v1/gfycats/${host.id}`;
|
||||
const res = await bhttp.get(infoUrl, { agent: agentFor(infoUrl) });
|
||||
const data = await res.body;
|
||||
|
||||
if (data.errorMessage) {
|
||||
const redirectRes = await bhttp.head(host.url, {
|
||||
followRedirects: false,
|
||||
agent: agentFor(host.url),
|
||||
});
|
||||
|
||||
if (redirectRes.statusCode === 301) {
|
||||
// Gfycat redirects all NSFW gifs to RedGifs, likely the case
|
||||
return redgifs({
|
||||
...host,
|
||||
url: `https://www.redgifs.com/watch/${host.id}`,
|
||||
method: 'redgifs',
|
||||
label: 'redgifs',
|
||||
});
|
||||
}
|
||||
|
||||
throw new Error(`Gfycat API returned error for source '${host.url}' (${res.status}): ${data.errorMessage}`);
|
||||
}
|
||||
|
||||
return {
|
||||
album: null,
|
||||
items: [{
|
||||
id: data.gfyItem.gfyName,
|
||||
url: data.gfyItem.webmUrl,
|
||||
title: data.gfyItem.title,
|
||||
description: data.gfyItem.description,
|
||||
type: 'video/webm',
|
||||
datetime: new Date(data.gfyItem.createDate * 1000),
|
||||
original: data.gfyItem,
|
||||
}],
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = gfycat;
|
||||
@@ -3,7 +3,6 @@
|
||||
const erome = require('./erome');
|
||||
const eroshareAlbum = require('./eroshareAlbum');
|
||||
const eroshareItem = require('./eroshareItem');
|
||||
const gfycat = require('./gfycat');
|
||||
const imgurAlbum = require('./imgurAlbum');
|
||||
const imgurImage = require('./imgurImage');
|
||||
const redditImage = require('./redditImage');
|
||||
@@ -21,7 +20,6 @@ module.exports = {
|
||||
erome,
|
||||
eroshareAlbum,
|
||||
eroshareItem,
|
||||
gfycat,
|
||||
imgurAlbum,
|
||||
imgurImage,
|
||||
redditImage,
|
||||
|
||||
@@ -33,11 +33,30 @@ async function findAudio(video, videoId) {
|
||||
return findAudioUrl(await res.text(), videoId);
|
||||
}
|
||||
|
||||
async function redditVideo(host, post, { reddit }) {
|
||||
async function getVideo(post, reddit) {
|
||||
// the submission listing already includes media.reddit_video for video posts, and
|
||||
// curate/posts.js carries it through as post.media (also for crossposts) - re-fetching each
|
||||
// video post individually costs one full request against reddit's 6/min budget. Only
|
||||
// fall back to that if the listing didn't have it (e.g. a post still processing).
|
||||
if (post.media?.reddit_video) {
|
||||
return post.media.reddit_video;
|
||||
}
|
||||
|
||||
// fetched through the authenticated/throttled client, not a raw fetch() - reddit's
|
||||
// anonymous-traffic bot challenge blocks plain unauthenticated .json requests
|
||||
const data = await reddit.getSubmission(post.id);
|
||||
const video = data.media.reddit_video;
|
||||
const source = data.crosspost_parent_list?.[0] || data;
|
||||
const video = (source.secure_media || source.media)?.reddit_video;
|
||||
|
||||
if (!video) {
|
||||
throw new Error(`No reddit video data available for post '${post.id}'`);
|
||||
}
|
||||
|
||||
return video;
|
||||
}
|
||||
|
||||
async function redditVideo(host, post, { reddit }) {
|
||||
const video = await getVideo(post, reddit);
|
||||
const videoId = post.host.id || post.id;
|
||||
|
||||
const item = {
|
||||
|
||||
+21
-17
@@ -1,16 +1,13 @@
|
||||
'use strict';
|
||||
|
||||
const config = require('config');
|
||||
const { ProxyAgent } = require('undici');
|
||||
const { ProxyAgent, Socks5ProxyAgent } = require('undici');
|
||||
const { HttpProxyAgent } = require('http-proxy-agent');
|
||||
const { HttpsProxyAgent } = require('https-proxy-agent');
|
||||
const { SocksProxyAgent } = require('socks-proxy-agent');
|
||||
|
||||
const logger = require('./logger')(__filename);
|
||||
|
||||
const dispatcherCache = new Map();
|
||||
const nodeAgentCache = new Map();
|
||||
const warnedNoSocksForFetch = new Set();
|
||||
|
||||
// same suffix matching as config.reddit.cookies' domain filter (src/reddit/client.js): a
|
||||
// bare hostname also matches its subdomains, so 'reddit.com' catches 'www.reddit.com' but
|
||||
@@ -65,9 +62,25 @@ function isSocks(proxyUrl) {
|
||||
return /^socks(4a?|5h?):/i.test(proxyUrl);
|
||||
}
|
||||
|
||||
// undici (which node's built-in fetch() is implemented with) only tunnels through http(s)
|
||||
// proxies (CONNECT) via ProxyAgent - it has no built-in SOCKS support. agentFor() below can
|
||||
// still do SOCKS, since bhttp goes through node's plain http/https modules instead.
|
||||
// fetch() goes through undici: http(s) proxies via ProxyAgent (CONNECT), SOCKS5 via
|
||||
// Socks5ProxyAgent (undici >= 7.x, still flagged experimental - node prints a one-time
|
||||
// ExperimentalWarning). Socks5ProxyAgent always sends the target as a hostname (address type
|
||||
// DOMAIN), i.e. it resolves on the proxy side like 'socks5h', so both spellings map onto it.
|
||||
// SOCKS4 isn't supported by undici at all - rather than silently sending what's presumably
|
||||
// meant to be private traffic out directly, that's an error (agentFor() below, for bhttp
|
||||
// downloads, does support it).
|
||||
function createDispatcher(proxyUrl, hostname) {
|
||||
if (/^socks5h?:/i.test(proxyUrl)) {
|
||||
return new Socks5ProxyAgent(proxyUrl.replace(/^socks5h:/i, 'socks5:'));
|
||||
}
|
||||
|
||||
if (isSocks(proxyUrl)) {
|
||||
throw new Error(`SOCKS4 proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (reddit API, host info lookups) - use a socks5:// or http(s):// proxy for this host instead`);
|
||||
}
|
||||
|
||||
return new ProxyAgent(proxyUrl);
|
||||
}
|
||||
|
||||
function dispatcherFor(targetUrl, { flip = false } = {}) {
|
||||
const { hostname } = new URL(targetUrl);
|
||||
const proxyUrl = resolveProxyUrl(hostname, { flip });
|
||||
@@ -76,17 +89,8 @@ function dispatcherFor(targetUrl, { flip = false } = {}) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (isSocks(proxyUrl)) {
|
||||
if (!warnedNoSocksForFetch.has(proxyUrl)) {
|
||||
warnedNoSocksForFetch.add(proxyUrl);
|
||||
logger.warn(`SOCKS proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (only http(s) proxies can) - these will go out directly instead. Item downloads (which use bhttp) can still use it.`);
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (!dispatcherCache.has(proxyUrl)) {
|
||||
dispatcherCache.set(proxyUrl, new ProxyAgent(proxyUrl));
|
||||
dispatcherCache.set(proxyUrl, createDispatcher(proxyUrl, hostname));
|
||||
}
|
||||
|
||||
return dispatcherCache.get(proxyUrl);
|
||||
|
||||
+26
-1
@@ -244,9 +244,27 @@ async function getUser(username) {
|
||||
return data.data;
|
||||
}
|
||||
|
||||
async function getSubmissions(username, { sort = 'new', limit = Infinity } = {}) {
|
||||
// `until` (a Date, only honoured with sort 'new'): stop paging once a page reaches a post at
|
||||
// or before this moment - everything on later pages is older still, so there's no point
|
||||
// spending more of reddit's throttled budget (1 page = 1 request at 6/min) on posts the
|
||||
// caller is going to discard anyway. The page that crosses the line is still returned in full;
|
||||
// filtering out the older posts on it is left to curate/posts.js as before. Pinned posts are
|
||||
// ignored for this check, since reddit lists them first on a user's profile regardless of age.
|
||||
function reachedUntil(page, until) {
|
||||
if (!until) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const cutoff = until.getTime() / 1000;
|
||||
|
||||
return page.some((post) => !post.pinned && !post.stickied && post.created_utc <= cutoff);
|
||||
}
|
||||
|
||||
async function getSubmissions(username, { sort = 'new', limit = Infinity, until = null } = {}) {
|
||||
const submissions = [];
|
||||
const stopAt = sort === 'new' ? until : null;
|
||||
let after = null;
|
||||
let pages = 0;
|
||||
|
||||
do {
|
||||
// eslint-disable-next-line no-await-in-loop
|
||||
@@ -260,6 +278,13 @@ async function getSubmissions(username, { sort = 'new', limit = Infinity } = {})
|
||||
|
||||
submissions.push(...page);
|
||||
after = data.data.after;
|
||||
pages += 1;
|
||||
|
||||
if (after && reachedUntil(page, stopAt)) {
|
||||
logger.verbose(`Stopped fetching submissions for '${username}' after ${pages} page(s), reached ${stopAt.toISOString()}`);
|
||||
|
||||
break;
|
||||
}
|
||||
} while (after && submissions.length < limit);
|
||||
|
||||
return submissions;
|
||||
|
||||
@@ -39,11 +39,35 @@ async function getUser(username, reddit) {
|
||||
}
|
||||
}
|
||||
|
||||
async function getPosts(username, reddit, args) {
|
||||
// the moment before which posts will be discarded by curate/posts.js anyway (--after, or
|
||||
// --after-indexed against the existing index) - lets getSubmissions() stop paging there
|
||||
// instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag,
|
||||
// every page is still fetched, so gaps anywhere in the archive keep getting backfilled.
|
||||
function getCutoff(indexedPosts, args) {
|
||||
const cutoffs = [];
|
||||
|
||||
if (args.after) {
|
||||
cutoffs.push(args.after);
|
||||
}
|
||||
|
||||
if (args.afterIndexed && indexedPosts?.length) {
|
||||
const dates = indexedPosts.map((entry) => new Date(entry.date).getTime()).filter((time) => !Number.isNaN(time));
|
||||
|
||||
if (dates.length) {
|
||||
cutoffs.push(new Date(args.afterIndexed === 'latest' ? Math.max(...dates) : Math.min(...dates)));
|
||||
}
|
||||
}
|
||||
|
||||
// with both set, the later of the two discards the most, so stop there
|
||||
return cutoffs.length ? new Date(Math.max(...cutoffs.map((date) => date.getTime()))) : null;
|
||||
}
|
||||
|
||||
async function getPosts(username, reddit, args, until) {
|
||||
try {
|
||||
const submissions = await reddit.getSubmissions(username, {
|
||||
sort: args.sort,
|
||||
limit: Infinity,
|
||||
until,
|
||||
});
|
||||
|
||||
logger.info(`Fetched ${submissions.length} submissions for '${username}' (https://reddit.com/user/${username})`);
|
||||
@@ -67,12 +91,12 @@ function getUserPostsWrap(reddit, args) {
|
||||
try {
|
||||
const users = await Promise.map(usernames, async (username) => {
|
||||
try {
|
||||
const [user, posts] = await Promise.all([
|
||||
getUser(username, reddit),
|
||||
getPosts(username, reddit, args),
|
||||
]);
|
||||
|
||||
// sequential rather than parallel: the index (whose path may depend on the
|
||||
// user's details) decides how far back to page, and every reddit request
|
||||
// goes through the same single-concurrency limiter anyway
|
||||
const user = await getUser(username, reddit);
|
||||
const { profile, posts: indexed } = await getIndex(user);
|
||||
const posts = await getPosts(username, reddit, args, getCutoff(indexed, args));
|
||||
|
||||
if (args.archives) {
|
||||
posts.push(...await getArchivedPosts(username, posts, reddit));
|
||||
@@ -92,7 +116,7 @@ function getUserPostsWrap(reddit, args) {
|
||||
|
||||
return null;
|
||||
} catch (error) {
|
||||
logger.error(`Failed to fetch posts from 'username': ${error.message}`);
|
||||
logger.error(`Failed to fetch posts from '${username}': ${error.message}`);
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user