Track suspended/deleted accounts and prune them from the users file after a grace period, also as standalone --cleanup. Only fetch social links when the profile description is saved, raised reddit throttle to 15/min.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -29,7 +29,7 @@ The pipeline runs from `src/app.js`:
 - Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
 - Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h.
 - Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
-- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 6/min, `minTime` 1500ms (`config.reddit.throttle`). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
+- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
 - Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).
 
 ### Networking (`src/proxy.js`, `src/limiter.js`)
@@ -39,6 +39,13 @@ The pipeline runs from `src/app.js`:
 - `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
 - bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
 
+## Account status and cleanup
+
+- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`).
+- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. Files are only written at the end of a run.
+- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false.
+- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set.
+
 ## Fetching strategy
 
 - By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.

README.md

@@ -46,6 +46,7 @@ ripunzel requires a arbitrarily recent version of Node.js. Before use, dependenc
 * `--posts <post-id> [<post-id>...]`: Fetch multiple posts by supplying a space-separated list of post IDs to `--posts`.
 * `--fetch <content-url>`: Fetch content directly from an URL to an album or image on one of the supported hosts
 * `--file-users <filepath>`: Fetch posts from multiple users by supplying a file with newline separated usernames
+* `--cleanup`: Instead of archiving, only check the supplied users (`--users`/`--file-users`) for suspended or deleted accounts. These are recorded in `users_dead.yml` with the date they were first seen, and removed from the `--file-users` file once they've stayed dead for longer than `users.dead.retainDays` (90 days by default). Regular archive runs do the same unless `users.dead.duringArchive` is disabled.
 * `--file-posts <filepath>`: Fetch multiple posts by supplying a file with newline separated post IDs
 * `--file-fetch <filepath>`: Fetch content directly from multiple sources by supplying a file with newline separated URLs
 * `--limit <number>`: Maximum amount posts per user to fetch content from. Limit is applied after filtering out ignored, cross- and reposts. Posts requested directly by ID may be discarded as duplicates, but are not otherwise affected by the limit.

config/default.js

@@ -106,12 +106,32 @@ module.exports = {
         cookie: null,
         userAgent: 'ripunzel',
         throttle: {
-            // deliberately conservative - this is unauthenticated-looking public access,
+            // requests per minute (and the minimum gap between two, in ms) for everything sent
-            // not an approved API key, getting flagged as a bot defeats the point
+            // to reddit itself - all of it is serialized through one limiter. reddit sends no
-            reservoir: 6,
+            // rate-limit headers for cookie sessions, so there's no documented ceiling; this is
+            // public-endpoint access, not an approved API key, and getting flagged as a bot
+            // defeats the point. 15/min held up in testing (2026-11), raise with care.
+            reservoir: 15,
             minTime: 1500,
         },
     },
+    users: {
+        // record of suspended/deleted accounts, and pruning them from the --file-users list - see
+        // src/sources/accountStatus.js. Also run on its own with --cleanup.
+        dead: {
+            file: 'users_dead.yml',
+            // also record and prune during regular archive runs (always done with --cleanup)
+            duringArchive: true,
+            // days an account must stay suspended/deleted (counted from the first run that saw
+            // it, reddit exposes no date) before it's removed from the users file - until then
+            // it's re-checked every run, so a lifted suspension is noticed. 'deleted' is really
+            // "profile 404s", which reportedly includes shadowbanned accounts, hence not 0.
+            retainDays: {
+                suspended: 90,
+                deleted: 90,
+            },
+        },
+    },
     methods: {
         imgur: {
             clientId: null,

src/app.js

@@ -20,6 +20,9 @@ const methods = require('./methods/methods');
 const { attachContentInfo, getInfo } = require('./fetch/info');
 const { fetchSaveUserContent, fetchSaveDirectContent } = require('./fetch/content');
 
+const createAccountTracker = require('./sources/accountStatus');
+const { fetchUser } = require('./sources/user');
+
 const getPosts = require('./sources/getPosts')(reddit, args);
 const getUserPosts = require('./sources/getUserPosts')(reddit, args);
 
@@ -61,16 +64,56 @@ async function fetchPredata(hosts) {
     }, Promise.resolve({}));
 }
 
+async function getUsernames() {
+    const fileUsernames = args.fileUsers ? await getFileContents(args.fileUsers, 'username') : [];
+
+    return {
+        usernames: (args.users || []).concat(fileUsernames),
+        fileUsernames,
+    };
+}
+
+// --cleanup: only check each user's account status (one request per user, instead of a full
+// archive run's several) to record suspended/deleted accounts and prune the users file - see
+// sources/accountStatus.js
+async function cleanupUsers() {
+    const { usernames, fileUsernames } = await getUsernames();
+
+    if (!usernames.length) {
+        logger.warn('Nothing to clean up, supply --users and/or --file-users');
+
+        return;
+    }
+
+    const tracker = await createAccountTracker({ usersFile: args.fileUsers, fileUsernames });
+    const statuses = {
+        active: 0, suspended: 0, deleted: 0, error: 0,
+    };
+
+    logger.info(`Checking ${usernames.length} accounts...`);
+
+    await Promise.each(usernames, async (username, index) => {
+        const { status } = await fetchUser(username, reddit);
+
+        statuses[status] += 1;
+        tracker.record(username, status);
+
+        if ((index + 1) % 50 === 0) {
+            logger.info(`Checked ${index + 1}/${usernames.length} accounts`);
+        }
+    });
+
+    logger.info(`Checked ${usernames.length} accounts: ${statuses.active} active, ${statuses.suspended} suspended, ${statuses.deleted} deleted, ${statuses.error} failed to check`);
+
+    await tracker.save();
+}
+
 async function getCompletePosts() {
     let userPosts = {};
     let ignoreIds = [];
-    let usernames = args.users || [];
+    const { usernames, fileUsernames } = await getUsernames();
     let postIds = args.posts || [];
 
-    if (args.fileUsers) {
-        usernames = usernames.concat(await getFileContents(args.fileUsers, 'username'));
-    }
-
     if (args.filePosts) {
         postIds = postIds.concat(await getFileContents(args.filePosts, 'post ID'));
     }
@@ -80,7 +123,15 @@ async function getCompletePosts() {
     }
 
     if (usernames.length) {
-        userPosts = await getUserPosts(usernames);
+        const tracker = await createAccountTracker({
+            usersFile: args.fileUsers,
+            fileUsernames,
+            enabled: config.users.dead.duringArchive,
+        });
+
+        userPosts = await getUserPosts(usernames, tracker);
+
+        await tracker.save();
     }
 
     if (postIds.length) {
@@ -143,6 +194,12 @@ async function initApp() {
     let ep;
 
     try {
+        if (args.cleanup) {
+            await cleanupUsers();
+
+            return;
+        }
+
         ep = new exiftool.ExiftoolProcess(exiftoolBin);
         await ep.open();
 

src/cli.js

@@ -31,6 +31,10 @@ function getArgs() {
             describe: 'Load reddit usernames from file',
             type: 'string',
         })
+        .option('cleanup', {
+            describe: 'Only check the supplied users for suspended/deleted accounts: record them, and remove them from --file-users once dead for longer than configured. Does not archive anything',
+            type: 'boolean',
+        })
         .option('posts', {
             alias: 'post',
             describe: 'Reddit post IDs to fetch',

src/reddit/client.js

@@ -195,7 +195,13 @@ async function requestJson(path, params = {}, attempt = 0) {
     if (!res.ok) {
         const message = data?.message || data?.reason || JSON.stringify(data).slice(0, 200);
 
-        throw new Error(`Reddit returned ${res.status} for '${path}': ${message}`);
+        const error = new Error(`Reddit returned ${res.status} for '${path}': ${message}`);
+
+        // only set for genuine JSON API errors - sources/user.js relies on a 404 here meaning
+        // the account is really gone, as opposed to e.g. the HTML bot-challenge page above
+        error.status = res.status;
+
+        throw error;
     }
 
     return data;

src/sources/accountStatus.js

@@ -0,0 +1,137 @@
+'use strict';
+
+const config = require('config');
+const fs = require('fs-extra');
+const yaml = require('js-yaml');
+
+const logger = require('../logger')(__filename);
+
+const DAY = 24 * 60 * 60 * 1000;
+
+// Keeps a YAML record (config.users.dead.file) of accounts found suspended or deleted, keyed by
+// username:
+//
+//   SomeUser:
+//     status: suspended        # or 'deleted'
+//     firstSeen: 2026-11-05T…  # first run that observed this status - reddit exposes no date
+//     lastSeen: 2026-11-05T…
+//     removed: 2026-02-03T…    # when it was dropped from the users file, if it was
+//
+// An account stays in the users file (and keeps being re-checked, at the cost of one request
+// per run) until it's been dead for longer than config.users.dead.retainDays[status], so a
+// lifted suspension or shadowban is noticed in time. After that it's removed from the users
+// file - but only if it came from one (--file-users), never from --users on the command line.
+// An account that turns out active again has its record dropped. Nothing is written until
+// save(), at the end of the run, since users are processed concurrently.
+async function loadRecords(file) {
+    try {
+        return yaml.load(await fs.readFile(file, 'utf8')) || {};
+    } catch (error) {
+        if (error.code !== 'ENOENT') {
+            throw new Error(`Could not read dead account record '${file}': ${error.message}`);
+        }
+
+        return {};
+    }
+}
+
+function findKey(records, username) {
+    const lower = username.toLowerCase();
+
+    return Object.keys(records).find((key) => key.toLowerCase() === lower);
+}
+
+async function removeFromUsersFile(usersFile, usernames) {
+    const lower = new Set([...usernames].map((name) => name.toLowerCase()));
+    const contents = await fs.readFile(usersFile, 'utf8');
+    // keeps comments, blank lines and order intact - only drops the exact matching entries
+    const kept = contents.split('\n').filter((line) => !lower.has(line.trim().toLowerCase()));
+    const tmpFile = `${usersFile}.tmp`;
+
+    await fs.writeFile(tmpFile, kept.join('\n'));
+    await fs.rename(tmpFile, usersFile);
+}
+
+async function createAccountTracker({ usersFile = null, fileUsernames = [], enabled = true } = {}) {
+    const deadConfig = config.users?.dead || {};
+    const file = deadConfig.file || 'users_dead.yml';
+    const retainDays = deadConfig.retainDays || {};
+    const fromFile = new Set(fileUsernames.map((name) => name.toLowerCase()));
+    const records = enabled ? await loadRecords(file) : {};
+    const removals = new Set();
+    const counts = {
+        suspended: 0, deleted: 0, restored: 0, removed: 0,
+    };
+    let changed = false;
+
+    function record(username, status, date = new Date()) {
+        const now = date.toISOString();
+
+        if (!enabled || status === 'error') {
+            return;
+        }
+
+        const key = findKey(records, username);
+
+        if (status === 'active') {
+            if (key) {
+                logger.info(`Account '${username}' is active again (was ${records[key].status} since ${new Date(records[key].firstSeen).toISOString()}), dropped from dead account record`);
+
+                delete records[key];
+                counts.restored += 1;
+                changed = true;
+            }
+
+            return;
+        }
+
+        counts[status] += 1;
+
+        const entry = key && records[key].status === status
+            ? { ...records[key], lastSeen: now }
+            // new, or changed status (e.g. suspended -> deleted): the clock restarts
+            : { status, firstSeen: now, lastSeen: now };
+
+        if (key && key !== username) {
+            delete records[key];
+        }
+
+        records[username] = entry;
+        changed = true;
+
+        const deadDays = (date - new Date(entry.firstSeen)) / DAY;
+        const retain = retainDays[status] ?? 90;
+
+        if (deadDays >= retain && fromFile.has(username.toLowerCase())) {
+            removals.add(username);
+            entry.removed = now;
+            logger.info(`Account '${username}' has been ${status} since ${new Date(entry.firstSeen).toISOString()} (${Math.floor(deadDays)} days, retaining ${retain}), removing it from '${usersFile}'`);
+        } else {
+            // removed earlier, but (manually) put back in the users file since
+            if (fromFile.has(username.toLowerCase())) {
+                delete entry.removed;
+            }
+
+            logger.info(`Account '${username}' is ${status} (first seen ${new Date(entry.firstSeen).toISOString()}, ${Math.floor(deadDays)} of ${retain} days before removal${fromFile.has(username.toLowerCase()) ? '' : ', not from a users file so never removed automatically'})`);
+        }
+    }
+
+    async function save() {
+        if (!enabled || !changed) {
+            return;
+        }
+
+        if (removals.size && usersFile) {
+            await removeFromUsersFile(usersFile, removals);
+            counts.removed = removals.size;
+        }
+
+        await fs.writeFile(file, yaml.dump(records, { sortKeys: true }));
+
+        logger.info(`Dead account record '${file}' updated: ${counts.suspended} suspended, ${counts.deleted} deleted, ${counts.restored} active again, ${counts.removed} removed from users file`);
+    }
+
+    return { record, save };
+}
+
+module.exports = createAccountTracker;

src/sources/getPosts.js

@@ -3,40 +3,9 @@
 const Promise = require('bluebird');
 
 const getIndex = require('./getIndex');
-const curateUser = require('../curate/user');
+const { fetchUser, indexUser, resolveUser } = require('./user');
 
-const logger = require('../logger')(__filename);
+const getPostsWrap = (reddit, args) => function getPosts(postIds, userPosts = {}) {
-
-async function getSocialLinks(username, reddit) {
-    try {
-        return await reddit.getSocialLinks(username);
-    } catch (error) {
-        // supplementary, not worth failing the whole user fetch over - fall back to none
-        logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return [];
-    }
-}
-
-async function getUser(username, reddit) {
-    try {
-        const [user, socialLinks] = await Promise.all([
-            reddit.getUser(username),
-            getSocialLinks(username, reddit),
-        ]);
-
-        return curateUser(user, socialLinks);
-    } catch (error) {
-        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return {
-            name: username,
-            fallback: true,
-        };
-    }
-}
-
-const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) {
     return Promise.reduce(postIds, (accUserPosts, postId) => Promise.resolve().then(async () => {
         const post = await reddit.getSubmission(postId);
 
@@ -69,8 +38,10 @@ const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) {
             };
         }
 
-        const user = await getUser(post.author, reddit);
+        // a suspended/deleted author still gets their post saved, under the fallback user
-        const { profile, posts: indexed } = await getIndex(user);
+        const fetched = await fetchUser(post.author, reddit);
+        const { profile, posts: indexed } = await getIndex(indexUser(post.author, fetched));
+        const user = await resolveUser(post.author, fetched, profile, reddit, args);
 
         return {
             ...accUserPosts,

src/sources/getUserPosts.js

@@ -4,41 +4,10 @@ const Promise = require('bluebird');
 
 const getIndex = require('./getIndex');
 const getArchivePostIds = require('../archives/getArchivePostIds');
-const curateUser = require('../curate/user');
+const { fetchUser, indexUser, resolveUser } = require('./user');
 
 const logger = require('../logger')(__filename);
 
-async function getSocialLinks(username, reddit) {
-    try {
-        return await reddit.getSocialLinks(username);
-    } catch (error) {
-        // supplementary, not worth failing the whole user fetch over - fall back to none
-        logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return [];
-    }
-}
-
-async function getUser(username, reddit) {
-    try {
-        const [user, socialLinks] = await Promise.all([
-            reddit.getUser(username),
-            getSocialLinks(username, reddit),
-        ]);
-
-        logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`);
-
-        return curateUser(user, socialLinks);
-    } catch (error) {
-        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return {
-            name: username,
-            fallback: true,
-        };
-    }
-}
-
 // the moment before which posts will be discarded by curate/posts.js anyway (--after, or
 // --after-indexed against the existing index) - lets getSubmissions() stop paging there
 // instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag,
@@ -87,15 +56,32 @@ async function getArchivedPosts(username, posts, reddit) {
 }
 
 function getUserPostsWrap(reddit, args) {
-    return async function getUserPosts(usernames) {
+    // tracker: see accountStatus.js - records suspended/deleted accounts (and drops them from the
+    // users file once they've been dead long enough), optional
+    return async function getUserPosts(usernames, tracker = null) {
         try {
             const users = await Promise.map(usernames, async (username) => {
                 try {
-                    // sequential rather than parallel: the index (whose path may depend on the
+                    // sequential rather than parallel: the user decides whether there's anything to
-                    // user's details) decides how far back to page, and every reddit request
+                    // fetch at all, the index (whose path may depend on the user's details)
-                    // goes through the same single-concurrency limiter anyway
+                    // decides how far back to page and whether social links are needed, and
-                    const user = await getUser(username, reddit);
+                    // every reddit request goes through the same single-concurrency limiter anyway
-                    const { profile, posts: indexed } = await getIndex(user);
+                    const fetched = await fetchUser(username, reddit);
+
+                    tracker?.record(username, fetched.status);
+
+                    // nothing more to get from a suspended or deleted account - its listing is a
+                    // 403/404 too, and skipping it saves the rest of this user's requests
+                    if (fetched.status === 'suspended' || fetched.status === 'deleted') {
+                        return null;
+                    }
+
+                    if (fetched.status === 'active') {
+                        logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`);
+                    }
+
+                    const { profile, posts: indexed } = await getIndex(indexUser(username, fetched));
+                    const user = await resolveUser(username, fetched, profile, reddit, args);
                     const posts = await getPosts(username, reddit, args, getCutoff(indexed, args));
 
                     if (args.archives) {

src/sources/user.js

@@ -0,0 +1,79 @@
+'use strict';
+
+const config = require('config');
+
+const curateUser = require('../curate/user');
+const logger = require('../logger')(__filename);
+
+// reddit tells deleted and suspended accounts apart (verified live, 2026-11):
+// - deleted: /user/x/about.json returns a JSON 404 (a shadowbanned account reportedly looks
+//   the same, so a 404 isn't necessarily permanent - see accountStatus.js's grace period)
+// - suspended: about.json returns 200 with only { name, is_suspended: true, ... } - no date of
+//   suspension is exposed anywhere
+// Anything else (network errors, 5xx, or the HTML bot-challenge page, which carries no
+// `status` - see reddit/client.js's requestJson()) is 'error', which must never be treated as
+// dead: a blocked or expired session would otherwise get the entire users file cleaned out.
+async function fetchUser(username, reddit) {
+    try {
+        const user = await reddit.getUser(username);
+
+        if (user.is_suspended) {
+            return { status: 'suspended', user: null };
+        }
+
+        return { status: 'active', user };
+    } catch (error) {
+        if (error.status === 404) {
+            return { status: 'deleted', user: null };
+        }
+
+        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
+
+        return { status: 'error', user: null, error };
+    }
+}
+
+async function getSocialLinks(username, reddit) {
+    try {
+        return await reddit.getSocialLinks(username);
+    } catch (error) {
+        // supplementary, not worth failing the whole user fetch over - fall back to none
+        logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`);
+
+        return [];
+    }
+}
+
+// social links are only ever used for the profile description file (save/profileDetails.js),
+// which is skipped once it's in the index (unless --redownload-profile) - scraping the profile
+// page for them regardless costs one throttled reddit request per user per run for nothing
+function needsSocialLinks(indexedProfile, args) {
+    return Boolean(config.library.profile.description && (args.redownloadProfile || !indexedProfile?.description));
+}
+
+// the user as getIndex() needs it to resolve the index path - the full curated user (minus social
+// links, which depend on the index) in case a custom library.index.file pattern uses more than
+// the name, otherwise the same fallback as resolveUser() below
+function indexUser(username, fetched) {
+    return fetched.status === 'active' ? curateUser(fetched.user) : { name: username, fallback: true };
+}
+
+// curated user, or the same minimal fallback as before for an account that couldn't be fetched
+// (or is suspended/deleted, when the caller still wants to proceed - e.g. a --post by them)
+async function resolveUser(username, fetched, indexedProfile, reddit, args) {
+    if (fetched.status !== 'active') {
+        return { name: username, fallback: true };
+    }
+
+    const socialLinks = fetched.user.subreddit && needsSocialLinks(indexedProfile, args)
+        ? await getSocialLinks(username, reddit)
+        : [];
+
+    return curateUser(fetched.user, socialLinks);
+}
+
+module.exports = {
+    fetchUser,
+    indexUser,
+    resolveUser,
+};