diff --git a/CLAUDE.md b/CLAUDE.md index 746ba24..fc09af0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,7 @@ The pipeline runs from `src/app.js`: - Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth. - Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h. - Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup. -- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 6/min, `minTime` 1500ms (`config.reddit.throttle`). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way. +- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way. - Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts). ### Networking (`src/proxy.js`, `src/limiter.js`) @@ -39,6 +39,13 @@ The pipeline runs from `src/app.js`: - `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter. - bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads. +## Account status and cleanup + +- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`). +- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. Files are only written at the end of a run. +- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false. +- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set. + ## Fetching strategy - By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**. diff --git a/README.md b/README.md index 75f74cd..d40f867 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ ripunzel requires a arbitrarily recent version of Node.js. Before use, dependenc * `--posts [...]`: Fetch multiple posts by supplying a space-separated list of post IDs to `--posts`. * `--fetch `: Fetch content directly from an URL to an album or image on one of the supported hosts * `--file-users `: Fetch posts from multiple users by supplying a file with newline separated usernames +* `--cleanup`: Instead of archiving, only check the supplied users (`--users`/`--file-users`) for suspended or deleted accounts. These are recorded in `users_dead.yml` with the date they were first seen, and removed from the `--file-users` file once they've stayed dead for longer than `users.dead.retainDays` (90 days by default). Regular archive runs do the same unless `users.dead.duringArchive` is disabled. * `--file-posts `: Fetch multiple posts by supplying a file with newline separated post IDs * `--file-fetch `: Fetch content directly from multiple sources by supplying a file with newline separated URLs * `--limit `: Maximum amount posts per user to fetch content from. Limit is applied after filtering out ignored, cross- and reposts. Posts requested directly by ID may be discarded as duplicates, but are not otherwise affected by the limit. diff --git a/config/default.js b/config/default.js index 274e792..d2179b4 100644 --- a/config/default.js +++ b/config/default.js @@ -106,12 +106,32 @@ module.exports = { cookie: null, userAgent: 'ripunzel', throttle: { - // deliberately conservative - this is unauthenticated-looking public access, - // not an approved API key, getting flagged as a bot defeats the point - reservoir: 6, + // requests per minute (and the minimum gap between two, in ms) for everything sent + // to reddit itself - all of it is serialized through one limiter. reddit sends no + // rate-limit headers for cookie sessions, so there's no documented ceiling; this is + // public-endpoint access, not an approved API key, and getting flagged as a bot + // defeats the point. 15/min held up in testing (2026-11), raise with care. + reservoir: 15, minTime: 1500, }, }, + users: { + // record of suspended/deleted accounts, and pruning them from the --file-users list - see + // src/sources/accountStatus.js. Also run on its own with --cleanup. + dead: { + file: 'users_dead.yml', + // also record and prune during regular archive runs (always done with --cleanup) + duringArchive: true, + // days an account must stay suspended/deleted (counted from the first run that saw + // it, reddit exposes no date) before it's removed from the users file - until then + // it's re-checked every run, so a lifted suspension is noticed. 'deleted' is really + // "profile 404s", which reportedly includes shadowbanned accounts, hence not 0. + retainDays: { + suspended: 90, + deleted: 90, + }, + }, + }, methods: { imgur: { clientId: null, diff --git a/src/app.js b/src/app.js index ce0833d..a192a01 100644 --- a/src/app.js +++ b/src/app.js @@ -20,6 +20,9 @@ const methods = require('./methods/methods'); const { attachContentInfo, getInfo } = require('./fetch/info'); const { fetchSaveUserContent, fetchSaveDirectContent } = require('./fetch/content'); +const createAccountTracker = require('./sources/accountStatus'); +const { fetchUser } = require('./sources/user'); + const getPosts = require('./sources/getPosts')(reddit, args); const getUserPosts = require('./sources/getUserPosts')(reddit, args); @@ -61,16 +64,56 @@ async function fetchPredata(hosts) { }, Promise.resolve({})); } +async function getUsernames() { + const fileUsernames = args.fileUsers ? await getFileContents(args.fileUsers, 'username') : []; + + return { + usernames: (args.users || []).concat(fileUsernames), + fileUsernames, + }; +} + +// --cleanup: only check each user's account status (one request per user, instead of a full +// archive run's several) to record suspended/deleted accounts and prune the users file - see +// sources/accountStatus.js +async function cleanupUsers() { + const { usernames, fileUsernames } = await getUsernames(); + + if (!usernames.length) { + logger.warn('Nothing to clean up, supply --users and/or --file-users'); + + return; + } + + const tracker = await createAccountTracker({ usersFile: args.fileUsers, fileUsernames }); + const statuses = { + active: 0, suspended: 0, deleted: 0, error: 0, + }; + + logger.info(`Checking ${usernames.length} accounts...`); + + await Promise.each(usernames, async (username, index) => { + const { status } = await fetchUser(username, reddit); + + statuses[status] += 1; + tracker.record(username, status); + + if ((index + 1) % 50 === 0) { + logger.info(`Checked ${index + 1}/${usernames.length} accounts`); + } + }); + + logger.info(`Checked ${usernames.length} accounts: ${statuses.active} active, ${statuses.suspended} suspended, ${statuses.deleted} deleted, ${statuses.error} failed to check`); + + await tracker.save(); +} + async function getCompletePosts() { let userPosts = {}; let ignoreIds = []; - let usernames = args.users || []; + const { usernames, fileUsernames } = await getUsernames(); let postIds = args.posts || []; - if (args.fileUsers) { - usernames = usernames.concat(await getFileContents(args.fileUsers, 'username')); - } - if (args.filePosts) { postIds = postIds.concat(await getFileContents(args.filePosts, 'post ID')); } @@ -80,7 +123,15 @@ async function getCompletePosts() { } if (usernames.length) { - userPosts = await getUserPosts(usernames); + const tracker = await createAccountTracker({ + usersFile: args.fileUsers, + fileUsernames, + enabled: config.users.dead.duringArchive, + }); + + userPosts = await getUserPosts(usernames, tracker); + + await tracker.save(); } if (postIds.length) { @@ -143,6 +194,12 @@ async function initApp() { let ep; try { + if (args.cleanup) { + await cleanupUsers(); + + return; + } + ep = new exiftool.ExiftoolProcess(exiftoolBin); await ep.open(); diff --git a/src/cli.js b/src/cli.js index b6939a2..2dc4323 100644 --- a/src/cli.js +++ b/src/cli.js @@ -31,6 +31,10 @@ function getArgs() { describe: 'Load reddit usernames from file', type: 'string', }) + .option('cleanup', { + describe: 'Only check the supplied users for suspended/deleted accounts: record them, and remove them from --file-users once dead for longer than configured. Does not archive anything', + type: 'boolean', + }) .option('posts', { alias: 'post', describe: 'Reddit post IDs to fetch', diff --git a/src/reddit/client.js b/src/reddit/client.js index c19916c..a9945c9 100644 --- a/src/reddit/client.js +++ b/src/reddit/client.js @@ -195,7 +195,13 @@ async function requestJson(path, params = {}, attempt = 0) { if (!res.ok) { const message = data?.message || data?.reason || JSON.stringify(data).slice(0, 200); - throw new Error(`Reddit returned ${res.status} for '${path}': ${message}`); + const error = new Error(`Reddit returned ${res.status} for '${path}': ${message}`); + + // only set for genuine JSON API errors - sources/user.js relies on a 404 here meaning + // the account is really gone, as opposed to e.g. the HTML bot-challenge page above + error.status = res.status; + + throw error; } return data; diff --git a/src/sources/accountStatus.js b/src/sources/accountStatus.js new file mode 100644 index 0000000..fa73bf8 --- /dev/null +++ b/src/sources/accountStatus.js @@ -0,0 +1,137 @@ +'use strict'; + +const config = require('config'); +const fs = require('fs-extra'); +const yaml = require('js-yaml'); + +const logger = require('../logger')(__filename); + +const DAY = 24 * 60 * 60 * 1000; + +// Keeps a YAML record (config.users.dead.file) of accounts found suspended or deleted, keyed by +// username: +// +// SomeUser: +// status: suspended # or 'deleted' +// firstSeen: 2026-11-05T… # first run that observed this status - reddit exposes no date +// lastSeen: 2026-11-05T… +// removed: 2026-02-03T… # when it was dropped from the users file, if it was +// +// An account stays in the users file (and keeps being re-checked, at the cost of one request +// per run) until it's been dead for longer than config.users.dead.retainDays[status], so a +// lifted suspension or shadowban is noticed in time. After that it's removed from the users +// file - but only if it came from one (--file-users), never from --users on the command line. +// An account that turns out active again has its record dropped. Nothing is written until +// save(), at the end of the run, since users are processed concurrently. +async function loadRecords(file) { + try { + return yaml.load(await fs.readFile(file, 'utf8')) || {}; + } catch (error) { + if (error.code !== 'ENOENT') { + throw new Error(`Could not read dead account record '${file}': ${error.message}`); + } + + return {}; + } +} + +function findKey(records, username) { + const lower = username.toLowerCase(); + + return Object.keys(records).find((key) => key.toLowerCase() === lower); +} + +async function removeFromUsersFile(usersFile, usernames) { + const lower = new Set([...usernames].map((name) => name.toLowerCase())); + const contents = await fs.readFile(usersFile, 'utf8'); + // keeps comments, blank lines and order intact - only drops the exact matching entries + const kept = contents.split('\n').filter((line) => !lower.has(line.trim().toLowerCase())); + const tmpFile = `${usersFile}.tmp`; + + await fs.writeFile(tmpFile, kept.join('\n')); + await fs.rename(tmpFile, usersFile); +} + +async function createAccountTracker({ usersFile = null, fileUsernames = [], enabled = true } = {}) { + const deadConfig = config.users?.dead || {}; + const file = deadConfig.file || 'users_dead.yml'; + const retainDays = deadConfig.retainDays || {}; + const fromFile = new Set(fileUsernames.map((name) => name.toLowerCase())); + const records = enabled ? await loadRecords(file) : {}; + const removals = new Set(); + const counts = { + suspended: 0, deleted: 0, restored: 0, removed: 0, + }; + let changed = false; + + function record(username, status, date = new Date()) { + const now = date.toISOString(); + + if (!enabled || status === 'error') { + return; + } + + const key = findKey(records, username); + + if (status === 'active') { + if (key) { + logger.info(`Account '${username}' is active again (was ${records[key].status} since ${new Date(records[key].firstSeen).toISOString()}), dropped from dead account record`); + + delete records[key]; + counts.restored += 1; + changed = true; + } + + return; + } + + counts[status] += 1; + + const entry = key && records[key].status === status + ? { ...records[key], lastSeen: now } + // new, or changed status (e.g. suspended -> deleted): the clock restarts + : { status, firstSeen: now, lastSeen: now }; + + if (key && key !== username) { + delete records[key]; + } + + records[username] = entry; + changed = true; + + const deadDays = (date - new Date(entry.firstSeen)) / DAY; + const retain = retainDays[status] ?? 90; + + if (deadDays >= retain && fromFile.has(username.toLowerCase())) { + removals.add(username); + entry.removed = now; + logger.info(`Account '${username}' has been ${status} since ${new Date(entry.firstSeen).toISOString()} (${Math.floor(deadDays)} days, retaining ${retain}), removing it from '${usersFile}'`); + } else { + // removed earlier, but (manually) put back in the users file since + if (fromFile.has(username.toLowerCase())) { + delete entry.removed; + } + + logger.info(`Account '${username}' is ${status} (first seen ${new Date(entry.firstSeen).toISOString()}, ${Math.floor(deadDays)} of ${retain} days before removal${fromFile.has(username.toLowerCase()) ? '' : ', not from a users file so never removed automatically'})`); + } + } + + async function save() { + if (!enabled || !changed) { + return; + } + + if (removals.size && usersFile) { + await removeFromUsersFile(usersFile, removals); + counts.removed = removals.size; + } + + await fs.writeFile(file, yaml.dump(records, { sortKeys: true })); + + logger.info(`Dead account record '${file}' updated: ${counts.suspended} suspended, ${counts.deleted} deleted, ${counts.restored} active again, ${counts.removed} removed from users file`); + } + + return { record, save }; +} + +module.exports = createAccountTracker; diff --git a/src/sources/getPosts.js b/src/sources/getPosts.js index 593abaa..a8217ee 100644 --- a/src/sources/getPosts.js +++ b/src/sources/getPosts.js @@ -3,40 +3,9 @@ const Promise = require('bluebird'); const getIndex = require('./getIndex'); -const curateUser = require('../curate/user'); +const { fetchUser, indexUser, resolveUser } = require('./user'); -const logger = require('../logger')(__filename); - -async function getSocialLinks(username, reddit) { - try { - return await reddit.getSocialLinks(username); - } catch (error) { - // supplementary, not worth failing the whole user fetch over - fall back to none - logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`); - - return []; - } -} - -async function getUser(username, reddit) { - try { - const [user, socialLinks] = await Promise.all([ - reddit.getUser(username), - getSocialLinks(username, reddit), - ]); - - return curateUser(user, socialLinks); - } catch (error) { - logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`); - - return { - name: username, - fallback: true, - }; - } -} - -const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) { +const getPostsWrap = (reddit, args) => function getPosts(postIds, userPosts = {}) { return Promise.reduce(postIds, (accUserPosts, postId) => Promise.resolve().then(async () => { const post = await reddit.getSubmission(postId); @@ -69,8 +38,10 @@ const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) { }; } - const user = await getUser(post.author, reddit); - const { profile, posts: indexed } = await getIndex(user); + // a suspended/deleted author still gets their post saved, under the fallback user + const fetched = await fetchUser(post.author, reddit); + const { profile, posts: indexed } = await getIndex(indexUser(post.author, fetched)); + const user = await resolveUser(post.author, fetched, profile, reddit, args); return { ...accUserPosts, diff --git a/src/sources/getUserPosts.js b/src/sources/getUserPosts.js index fdbdf69..8fb87a1 100644 --- a/src/sources/getUserPosts.js +++ b/src/sources/getUserPosts.js @@ -4,41 +4,10 @@ const Promise = require('bluebird'); const getIndex = require('./getIndex'); const getArchivePostIds = require('../archives/getArchivePostIds'); -const curateUser = require('../curate/user'); +const { fetchUser, indexUser, resolveUser } = require('./user'); const logger = require('../logger')(__filename); -async function getSocialLinks(username, reddit) { - try { - return await reddit.getSocialLinks(username); - } catch (error) { - // supplementary, not worth failing the whole user fetch over - fall back to none - logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`); - - return []; - } -} - -async function getUser(username, reddit) { - try { - const [user, socialLinks] = await Promise.all([ - reddit.getUser(username), - getSocialLinks(username, reddit), - ]); - - logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`); - - return curateUser(user, socialLinks); - } catch (error) { - logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`); - - return { - name: username, - fallback: true, - }; - } -} - // the moment before which posts will be discarded by curate/posts.js anyway (--after, or // --after-indexed against the existing index) - lets getSubmissions() stop paging there // instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag, @@ -87,15 +56,32 @@ async function getArchivedPosts(username, posts, reddit) { } function getUserPostsWrap(reddit, args) { - return async function getUserPosts(usernames) { + // tracker: see accountStatus.js - records suspended/deleted accounts (and drops them from the + // users file once they've been dead long enough), optional + return async function getUserPosts(usernames, tracker = null) { try { const users = await Promise.map(usernames, async (username) => { try { - // sequential rather than parallel: the index (whose path may depend on the - // user's details) decides how far back to page, and every reddit request - // goes through the same single-concurrency limiter anyway - const user = await getUser(username, reddit); - const { profile, posts: indexed } = await getIndex(user); + // sequential rather than parallel: the user decides whether there's anything to + // fetch at all, the index (whose path may depend on the user's details) + // decides how far back to page and whether social links are needed, and + // every reddit request goes through the same single-concurrency limiter anyway + const fetched = await fetchUser(username, reddit); + + tracker?.record(username, fetched.status); + + // nothing more to get from a suspended or deleted account - its listing is a + // 403/404 too, and skipping it saves the rest of this user's requests + if (fetched.status === 'suspended' || fetched.status === 'deleted') { + return null; + } + + if (fetched.status === 'active') { + logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`); + } + + const { profile, posts: indexed } = await getIndex(indexUser(username, fetched)); + const user = await resolveUser(username, fetched, profile, reddit, args); const posts = await getPosts(username, reddit, args, getCutoff(indexed, args)); if (args.archives) { diff --git a/src/sources/user.js b/src/sources/user.js new file mode 100644 index 0000000..d6ff59a --- /dev/null +++ b/src/sources/user.js @@ -0,0 +1,79 @@ +'use strict'; + +const config = require('config'); + +const curateUser = require('../curate/user'); +const logger = require('../logger')(__filename); + +// reddit tells deleted and suspended accounts apart (verified live, 2026-11): +// - deleted: /user/x/about.json returns a JSON 404 (a shadowbanned account reportedly looks +// the same, so a 404 isn't necessarily permanent - see accountStatus.js's grace period) +// - suspended: about.json returns 200 with only { name, is_suspended: true, ... } - no date of +// suspension is exposed anywhere +// Anything else (network errors, 5xx, or the HTML bot-challenge page, which carries no +// `status` - see reddit/client.js's requestJson()) is 'error', which must never be treated as +// dead: a blocked or expired session would otherwise get the entire users file cleaned out. +async function fetchUser(username, reddit) { + try { + const user = await reddit.getUser(username); + + if (user.is_suspended) { + return { status: 'suspended', user: null }; + } + + return { status: 'active', user }; + } catch (error) { + if (error.status === 404) { + return { status: 'deleted', user: null }; + } + + logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`); + + return { status: 'error', user: null, error }; + } +} + +async function getSocialLinks(username, reddit) { + try { + return await reddit.getSocialLinks(username); + } catch (error) { + // supplementary, not worth failing the whole user fetch over - fall back to none + logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`); + + return []; + } +} + +// social links are only ever used for the profile description file (save/profileDetails.js), +// which is skipped once it's in the index (unless --redownload-profile) - scraping the profile +// page for them regardless costs one throttled reddit request per user per run for nothing +function needsSocialLinks(indexedProfile, args) { + return Boolean(config.library.profile.description && (args.redownloadProfile || !indexedProfile?.description)); +} + +// the user as getIndex() needs it to resolve the index path - the full curated user (minus social +// links, which depend on the index) in case a custom library.index.file pattern uses more than +// the name, otherwise the same fallback as resolveUser() below +function indexUser(username, fetched) { + return fetched.status === 'active' ? curateUser(fetched.user) : { name: username, fallback: true }; +} + +// curated user, or the same minimal fallback as before for an account that couldn't be fetched +// (or is suspended/deleted, when the caller still wants to proceed - e.g. a --post by them) +async function resolveUser(username, fetched, indexedProfile, reddit, args) { + if (fetched.status !== 'active') { + return { name: username, fallback: true }; + } + + const socialLinks = fetched.user.subreddit && needsSocialLinks(indexedProfile, args) + ? await getSocialLinks(username, reddit) + : []; + + return curateUser(fetched.user, socialLinks); +} + +module.exports = { + fetchUser, + indexUser, + resolveUser, +};