Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.

Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -7,3 +7,4 @@ dist/
 users*
 posts
 ignore
+/queue*

CLAUDE.md

@@ -0,0 +1,75 @@
+# ripunzel
+
+Node.js CLI that archives reddit users' posts (and direct links to supported media hosts) to disk, with configurable filename patterns and a per-user YAML index to avoid redownloads.
+
+## Branches
+
+- `master` is maintained by hand. **Do not commit to or merge into `master`** unless explicitly asked.
+- `public-reddit-api` was largely written with Claude Code to get the app working again after reddit revoked API keys (Snoowrap/OAuth no longer usable). It needs a thorough manual review and a major version bump before it may be merged into `master` (see commit `2e4ba27`).
+
+## Running
+
+- Node `>=24.19.0` (`.nvmrc`). `npm install`, then `npm start -- --user <name>` (see `src/cli.js` or README for all flags).
+- ffmpeg must be on the PATH to mux v.redd.it video+audio; exiftool ships via `dist-exiftool`.
+- No test suite. Lint with `npx eslint src` (airbnb-base, 4-space indent, CommonJS, `'use strict'`).
+- Config uses the `config` package: `config/default.js` is committed; `config/local.js` overrides it and is gitignored, as is `config/cookies.js` (a Netscape cookie-jar dump, exported as a string and pulled in by `local.js`). **Never print, commit or log cookie contents.**
+- `output/`, `logs/`, `users*`, `ignore` are local data and gitignored.
+
+## Architecture
+
+`src/app.js` processes users **one at a time through the whole pipeline**, via `runUsers()`. While one user's media downloads, the next user is already being prepared (lookahead of one). Each user's index is saved as soon as that user is done. Do not go back to processing stage by stage across all users: that meant hours of reddit requests before the first download, everything lost on a crash, and no way to queue new users.
+
+Per user:
+
+1. **Sources** (`src/sources/`): `getUserPosts(username, tracker)` resolves to `{ status, user }`. Status is `active`, `empty`, `suspended`, `deleted` or `error`. It fetches the profile (`user.js`), the existing index (`getIndex`, YAML at `library.index.file`), social links if needed, and submissions.
+2. **Curate** (`src/curate/`): `posts.js` normalizes reddit post objects, applies filters (index, dates, limit, ignore, include/exclude, duplicates) and runs `dissectLink` to choose a host method. `user.js` normalizes the profile.
+3. **Info** (`src/fetch/info.js`): calls `methods[host.method]` (in `src/methods/`) to resolve each post into `{ album, items[] }`. If that fails, it falls back to reddit preview images (`redditPreview`). Some methods export `{ fetchInfo, fetchPredata }`. Predata (e.g. a RedGifs temporary token) is cached in `app.js` for `fetch.predataTtl`.
+4. **Content** (`src/fetch/content.js`): downloads items (`src/fetch/item.js`, using bhttp), dedupes by blake2 hash, saves to a path built by `src/interpolate.js` from `config.library` patterns, muxes video+audio, writes EXIF and finally updates the index (`src/save/writeToIndex.js`).
+
+Where users come from: `src/queue.js` provides `createListSource` (`--users`/`--file-users`) and `createQueueSource` (`--queue [file]`, default `queue`). `--posts` and `--fetch` still run as small batches.
+
+### Queue (`--queue [file]`, `--wait`)
+
+- The queue file is reread before each next user. A user is removed after finishing, including when suspended or deleted. Users with status `error` stay queued and are skipped for the rest of the run, so they can't loop.
+- `<queue>.lock` (holding a pid, created with `wx`) ensures only one process runs a queue. A second `--queue --users …` process only appends its users and exits. Without `--queue`, the lock is ignored.
+- At the end of a run, the lock is released first and the queue checked once more, which closes the race with a process that appended while it still saw the lock.
+- List files are edited with `src/utils/listFile.js`: reread right before writing, only the matching lines touched, replaced atomically via rename.
+
+### Reddit client (`src/reddit/client.js`), the core of this branch
+
+- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
+- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h.
+- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
+- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
+- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).
+
+### Networking (`src/proxy.js`, `src/limiter.js`)
+
+- `proxy.js` exports a wrapped `fetch` (undici; per-host proxy from `config.proxy.hosts` with suffix matching, plus a hard `config.fetch.timeout`) and `agentFor()` for bhttp. Both cover the reddit API when `reddit.com` is configured. fetch() supports http(s) and SOCKS5 (undici's experimental `Socks5ProxyAgent`; `socks5h` maps to it and hostnames are resolved on the proxy). SOCKS4 throws for fetch() rather than going out directly; bhttp supports all SOCKS variants.
+- `config.proxy.retryFlipped` retries a failed request via the opposite route (proxied ↔ direct). It is off by default for privacy reasons.
+- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
+- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
+
+## Account status and cleanup
+
+- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`).
+- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. The record is saved after every user.
+- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false.
+- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set.
+
+## Fetching strategy
+
+- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.
+- With `--after <date>` or `--after-indexed oldest|latest` (sort `new` only), `getSubmissions` stops paging at the first page that crosses the cutoff, ignoring pinned posts. `--after-indexed latest` is the cheap "only fetch what's new" mode, and it does not backfill gaps.
+- The user, then the index, then the posts are fetched in that order, because the index decides the cutoff.
+- Media data for videos and galleries comes from the listing, including crossposts via `crosspost_parent_list[0]`. A post is only fetched again if that data is missing.
+
+## Known caveats / review notes
+
+- Posts beyond reddit's 1000-post listing cap can't be reached through reddit. Arctic Shift (`arctic-shift.photon-reddit.com/api/posts/search?author=…`) returns full reddit post objects and is the candidate replacement for the abandoned `--archives` code (`src/archives/`, whose sites are gone).
+- `getUserPosts` processes users with concurrency 10, but the reddit client is serial anyway.
+- Self posts fetch their full comment tree, which takes at least one request per post (more with `more` stubs).
+- gfycat (shut down in 2023) links are routed to the RedGifs method under the same ID, keeping the label `gfycat`. eroshare is recovered from a fixed Wayback Machine snapshot. vidble is still online.
+- The README's "API keys" section and date-format notes predate this branch.
+- Several files have long explanatory comments from the rewrite. Keep them accurate when changing behavior.
+- No test suite. To test reddit logic offline, stub `require('./src/proxy').fetch` before requiring the client, and raise `reddit.throttle` via `NODE_CONFIG`, or the 6/min limiter applies.

README.md

@@ -14,7 +14,7 @@ Most features are optional and can easily be disabled!
 ### Supported hosts
 * Reddit text/self, images, albums and videos[\*](#reddit-videos)
 * Imgur (requires API key as of late 2019)
-* Gfycat
+* Gfycat (via RedGifs, since Gfycat shut down in 2023)
 * RedGifs
 * YouTube
 * PornHub, YouPorn, xHamster, RedTube, xnxx, YouJizz
@@ -46,6 +46,9 @@ ripunzel requires a arbitrarily recent version of Node.js. Before use, dependenc
 * `--posts <post-id> [<post-id>...]`: Fetch multiple posts by supplying a space-separated list of post IDs to `--posts`.
 * `--fetch <content-url>`: Fetch content directly from an URL to an album or image on one of the supported hosts
 * `--file-users <filepath>`: Fetch posts from multiple users by supplying a file with newline separated usernames
+* `--queue [<filepath>]`: Work through a queue file (default `queue`, see `queue.file`) of newline separated usernames. Each user is removed from the file once archived (or found suspended/deleted), and the file is re-read before every next user, so users added while ripunzel is running are picked up along the way. Users given with `--users`/`--file-users` are added to the queue first. If another ripunzel process is already working through the same queue, they are only added for that process to pick up, and the new process exits. Users that fail (e.g. network errors) stay in the queue for the next run.
+* `--wait`: With `--queue`, keep running once the queue is empty and check it for new users every `queue.interval` milliseconds (default 60 seconds).
+* `--cleanup`: Instead of archiving, only check the supplied users (`--users`/`--file-users`) for suspended or deleted accounts. These are recorded in `users_dead.yml` with the date they were first seen, and removed from the `--file-users` file once they've stayed dead for longer than `users.dead.retainDays` (90 days by default). Regular archive runs do the same unless `users.dead.duringArchive` is disabled.
 * `--file-posts <filepath>`: Fetch multiple posts by supplying a file with newline separated post IDs
 * `--file-fetch <filepath>`: Fetch content directly from multiple sources by supplying a file with newline separated URLs
 * `--limit <number>`: Maximum amount posts per user to fetch content from. Limit is applied after filtering out ignored, cross- and reposts. Posts requested directly by ID may be discarded as duplicates, but are not otherwise affected by the limit.

config/default.js

@@ -60,9 +60,19 @@ module.exports = {
         avoidDuplicates: true,
         retries: 3,
         concurrency: 10,
+        // wall-clock ceiling (ms) for a single item-download attempt (src/fetch/item.js).
+        // bhttp has no timeout of its own - a request that connects but then just stops
+        // responding (a stalled/misbehaving proxy tunnel is a common cause) would otherwise
+        // hang forever, and since the limiter's concurrency is capped, that one stuck request
+        // blocks every other download queued behind it indefinitely. This bounds each attempt
+        // so a stall fails (and retries, per `retries` above) instead of hanging forever.
+        timeout: 60000,
         watch: {
             schedule: '*/30 * * * *',
         },
+        // how long (ms) per-method predata (e.g. RedGifs' temporary API token) is reused before
+        // being fetched again - matters for long --queue --wait runs
+        predataTtl: 60 * 60 * 1000,
         archives: {
             search: false,
             preview: true,
@@ -74,8 +84,19 @@ module.exports = {
         level: 'info',
     },
     limiter: {
+        // default concurrency/interval for item/media downloads (src/fetch/item.js) - applies
+        // to any host without its own entry in `hosts` below
         concurrency: 10,
         interval: 100,
+        hosts: {
+            // per-host overrides of `concurrency`/`interval` above, for hosts that can safely
+            // be hit harder (most media CDNs) or need to be treated more cautiously than the
+            // default. Hostnames are matched by suffix, same as proxy.hosts above - a bare
+            // 'redgifs.com' also covers 'api.redgifs.com' and 'media.redgifs.com'. Unlike a
+            // single shared limiter, each entry here runs independently: a host that's
+            // currently stalling or being throttled doesn't block any other host's downloads.
+            // 'redgifs.com': { concurrency: 20, interval: 50 },
+        },
     },
     reddit: {
         // Netscape/Mozilla cookie-jar dump (string) of a logged-in reddit session, used to
@@ -88,15 +109,63 @@ module.exports = {
         cookie: null,
         userAgent: 'ripunzel',
         throttle: {
-            // deliberately conservative - this is unauthenticated-looking public access,
-            // not an approved API key, getting flagged as a bot defeats the point
-            reservoir: 6,
+            // requests per minute (and the minimum gap between two, in ms) for everything sent
+            // to reddit itself - all of it is serialized through one limiter. reddit sends no
+            // rate-limit headers for cookie sessions, so there's no documented ceiling; this is
+            // public-endpoint access, not an approved API key, and getting flagged as a bot
+            // defeats the point. 15/min held up in testing (2026-11), raise with care.
+            reservoir: 15,
             minTime: 1500,
         },
     },
+    queue: {
+        // used by --queue when no filename is given
+        file: 'queue',
+        // with --queue --wait, how often (ms) an empty queue is checked for new users
+        interval: 60 * 1000,
+    },
+    users: {
+        // record of suspended/deleted accounts, and pruning them from the --file-users list - see
+        // src/sources/accountStatus.js. Also run on its own with --cleanup.
+        dead: {
+            file: 'users_dead.yml',
+            // also record and prune during regular archive runs (always done with --cleanup)
+            duringArchive: true,
+            // days an account must stay suspended/deleted (counted from the first run that saw
+            // it, reddit exposes no date) before it's removed from the users file - until then
+            // it's re-checked every run, so a lifted suspension is noticed. 'deleted' is really
+            // "profile 404s", which reportedly includes shadowbanned accounts, hence not 0.
+            retainDays: {
+                suspended: 90,
+                deleted: 90,
+            },
+        },
+    },
     methods: {
         imgur: {
             clientId: null,
         },
     },
+    proxy: {
+        use: true,
+        default: null,
+        // if a request fails (including timing out - see fetch.timeout above) and is about to
+        // be retried, retry it using whichever of proxied/direct is NOT what would normally
+        // apply for that host - e.g. the proxy itself is down or being blocked by the target
+        // site, or (the reverse) a host reachable directly is refusing unproxied traffic.
+        // Off by default: for a privacy-sensitive proxy (e.g. Tor) silently falling back to a
+        // direct connection on failure is very much not what you want.
+        retryFlipped: false,
+        hosts: {
+            // hostnames are matched by suffix, so e.g. 'reddit.com' also covers
+            // 'www.reddit.com', but not 'v.redd.it' or 'redd.it' - those need their own
+            // entry, or fall back to `default`. A host entry can be:
+            // - a proxy URL, to override `default` for that host
+            // - `true`, to explicitly use `default` for that host
+            // - `null`/`false`, to explicitly exclude that host from `default`
+            // 'reddit.com': true,
+            // 'v.redd.it': null,
+            // 'imgur.com': 'socks5://127.0.0.1:9050',
+        },
+    },
 };

package-lock.json

@@ -20,6 +20,8 @@
         "dist-exiftool": "^10.53.0",
         "fluent-ffmpeg": "^2.1.3",
         "fs-extra": "^11.4.0",
+        "http-proxy-agent": "^7.0.2",
+        "https-proxy-agent": "^7.0.6",
         "js-yaml": "^5.2.3",
         "mime": "^3.0.0",
         "mime-types": "^3.0.2",
@@ -28,7 +30,9 @@
         "object.omit": "^3.0.0",
         "object.pick": "^1.3.0",
         "sharp-phash": "^2.2.0",
+        "socks-proxy-agent": "^8.0.5",
         "template-format": "^1.2.5",
+        "undici": "^7.29.0",
         "unprint": "^0.19.20",
         "url-pattern": "^1.0.3",
         "winston": "^3.19.0",
@@ -597,9 +601,6 @@
       "cpu": [
         "arm"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -617,9 +618,6 @@
       "cpu": [
         "arm64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -637,9 +635,6 @@
       "cpu": [
         "ppc64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -657,9 +652,6 @@
       "cpu": [
         "riscv64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -677,9 +669,6 @@
       "cpu": [
         "s390x"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -697,9 +686,6 @@
       "cpu": [
         "x64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -717,9 +703,6 @@
       "cpu": [
         "arm64"
       ],
-      "libc": [
-        "musl"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -737,9 +720,6 @@
       "cpu": [
         "x64"
       ],
-      "libc": [
-        "musl"
-      ],
       "license": "LGPL-3.0-or-later",
       "optional": true,
       "os": [
@@ -757,9 +737,6 @@
       "cpu": [
         "arm"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -783,9 +760,6 @@
       "cpu": [
         "arm64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -809,9 +783,6 @@
       "cpu": [
         "ppc64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -835,9 +806,6 @@
       "cpu": [
         "riscv64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -861,9 +829,6 @@
       "cpu": [
         "s390x"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -887,9 +852,6 @@
       "cpu": [
         "x64"
       ],
-      "libc": [
-        "glibc"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -913,9 +875,6 @@
       "cpu": [
         "arm64"
       ],
-      "libc": [
-        "musl"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -939,9 +898,6 @@
       "cpu": [
         "x64"
       ],
-      "libc": [
-        "musl"
-      ],
       "license": "Apache-2.0",
       "optional": true,
       "os": [
@@ -1279,6 +1235,15 @@
         "node": ">=0.4.0"
       }
     },
+    "node_modules/agent-base": {
+      "version": "7.1.4",
+      "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
+      "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 14"
+      }
+    },
     "node_modules/ajv": {
       "version": "6.15.0",
       "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz",
@@ -4223,6 +4188,32 @@
         "url": "https://opencollective.com/express"
       }
     },
+    "node_modules/http-proxy-agent": {
+      "version": "7.0.2",
+      "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz",
+      "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==",
+      "license": "MIT",
+      "dependencies": {
+        "agent-base": "^7.1.0",
+        "debug": "^4.3.4"
+      },
+      "engines": {
+        "node": ">= 14"
+      }
+    },
+    "node_modules/https-proxy-agent": {
+      "version": "7.0.6",
+      "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz",
+      "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==",
+      "license": "MIT",
+      "dependencies": {
+        "agent-base": "^7.1.2",
+        "debug": "4"
+      },
+      "engines": {
+        "node": ">= 14"
+      }
+    },
     "node_modules/human-readable-ids": {
       "version": "1.0.4",
       "resolved": "https://registry.npmjs.org/human-readable-ids/-/human-readable-ids-1.0.4.tgz",
@@ -4318,6 +4309,15 @@
         "node": ">= 0.4"
       }
     },
+    "node_modules/ip-address": {
+      "version": "10.5.0",
+      "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.5.0.tgz",
+      "integrity": "sha512-R5SnVLJmgYYvf2F2ZgwSBnelz5G4q5AxIC277GDfUaNbrZKNANcBC7RHqYYePlszf4kBolVkJauG0ZjHHFh55g==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 12"
+      }
+    },
     "node_modules/ipaddr.js": {
       "version": "1.9.1",
       "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
@@ -6601,6 +6601,44 @@
         "url": "https://github.com/sponsors/isaacs"
       }
     },
+    "node_modules/smart-buffer": {
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz",
+      "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 6.0.0",
+        "npm": ">= 3.0.0"
+      }
+    },
+    "node_modules/socks": {
+      "version": "2.8.9",
+      "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.9.tgz",
+      "integrity": "sha512-LJhUYUvItdQ0LkJTmPeaEObWXAqFyfmP85x0tch/ez9cahmhlBBLbIqDFnvBnUJGagb0JbIQrkBs1wJ+yRYpEw==",
+      "license": "MIT",
+      "dependencies": {
+        "ip-address": "^10.1.1",
+        "smart-buffer": "^4.2.0"
+      },
+      "engines": {
+        "node": ">= 10.0.0",
+        "npm": ">= 3.0.0"
+      }
+    },
+    "node_modules/socks-proxy-agent": {
+      "version": "8.0.5",
+      "resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz",
+      "integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==",
+      "license": "MIT",
+      "dependencies": {
+        "agent-base": "^7.1.2",
+        "debug": "^4.3.4",
+        "socks": "^2.8.3"
+      },
+      "engines": {
+        "node": ">= 14"
+      }
+    },
     "node_modules/source-map": {
       "version": "0.6.1",
       "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",

package.json

@@ -42,6 +42,8 @@
     "dist-exiftool": "^10.53.0",
     "fluent-ffmpeg": "^2.1.3",
     "fs-extra": "^11.4.0",
+    "http-proxy-agent": "^7.0.2",
+    "https-proxy-agent": "^7.0.6",
     "js-yaml": "^5.2.3",
     "mime": "^3.0.0",
     "mime-types": "^3.0.2",
@@ -50,7 +52,9 @@
     "object.omit": "^3.0.0",
     "object.pick": "^1.3.0",
     "sharp-phash": "^2.2.0",
+    "socks-proxy-agent": "^8.0.5",
     "template-format": "^1.2.5",
+    "undici": "^7.29.0",
     "unprint": "^0.19.20",
     "url-pattern": "^1.0.3",
     "winston": "^3.19.0",
@@ -63,5 +67,10 @@
     "eslint": "^8.57.1",
     "eslint-config-airbnb-base": "^15.0.0",
     "eslint-plugin-import": "^2.32.0"
+  },
+  "allowScripts": {
+    "blake2@5.0.1": true,
+    "dist-exiftool@10.53.0": true,
+    "youtube-dl-exec@3.1.12": true
   }
 }

src/app.js

@@ -20,6 +20,13 @@ const methods = require('./methods/methods');
 const { attachContentInfo, getInfo } = require('./fetch/info');
 const { fetchSaveUserContent, fetchSaveDirectContent } = require('./fetch/content');
 
+const createAccountTracker = require('./sources/accountStatus');
+const { fetchUser } = require('./sources/user');
+const {
+    acquireLock, releaseLock, createQueueSource, createListSource,
+} = require('./queue');
+const { appendToList } = require('./utils/listFile');
+
 const getPosts = require('./sources/getPosts')(reddit, args);
 const getUserPosts = require('./sources/getUserPosts')(reddit, args);
 
@@ -27,7 +34,7 @@ async function getFileContents(location, label) {
     try {
         const fileContents = await fs.readFile(location, 'utf8');
 
-        return fileContents.split('\n').filter((entry) => entry && entry.slice(0, 1) !== '#');
+        return fileContents.split('\n').map((entry) => entry.trim()).filter((entry) => entry && entry.slice(0, 1) !== '#');
     } catch (error) {
         logger.error(`Could not read ${label} file '${location}': ${error}.`);
 
@@ -36,65 +43,288 @@ async function getFileContents(location, label) {
 }
 
 function getPostHosts(posts) {
-    // const hosts = Array.from(new Set(Object.values(posts).flatMap((user) => user.posts.map((post) => post.host?.method)))).filter(Boolean);
     const hosts = Object.values(Object.fromEntries(Object.values(posts).flatMap((user) => user.posts.map((post) => post.host && [post.host?.method, post.host])).filter(Boolean)));
 
     return hosts;
 }
 
+// predata (e.g. RedGifs' temporary token, imgur's remaining credits) used to be fetched once per
+// run, for all users at once. Users are now processed one at a time, and a --queue --wait run
+// can go on indefinitely, so it's fetched when a method first needs it and cached for
+// config.fetch.predataTtl - long enough not to refetch it for every user, short enough that
+// a temporary token doesn't expire mid-run.
+const predataCache = new Map();
+
 async function fetchPredata(hosts) {
-    return hosts.reduce(async (chain, host) => {
-        const acc = await chain;
+    const hostMethods = [...new Set(hosts.map((host) => host?.method))].filter((method) => methods[method]?.fetchPredata);
 
-        if (methods[host?.method]?.fetchPredata) {
-            const data = await methods[host.method].fetchPredata();
+    return Promise.reduce(hostMethods, async (acc, method) => {
+        const cached = predataCache.get(method);
 
-            logger.info(`Fetched predata for ${host.method}`);
-
-            return {
-                ...acc,
-                [host.method]: data,
-            };
+        if (cached && Date.now() - cached.fetchedAt < config.fetch.predataTtl) {
+            return { ...acc, [method]: cached.data };
         }
 
-        return acc;
-    }, Promise.resolve({}));
+        const data = await methods[method].fetchPredata();
+
+        predataCache.set(method, { data, fetchedAt: Date.now() });
+        logger.info(`Fetched predata for ${method}`);
+
+        return { ...acc, [method]: data };
+    }, {});
 }
 
-async function getCompletePosts() {
-    let userPosts = {};
-    let ignoreIds = [];
-    let usernames = args.users || [];
-    let postIds = args.posts || [];
+async function getUsernames() {
+    const fileUsernames = args.fileUsers ? await getFileContents(args.fileUsers, 'username') : [];
 
-    if (args.fileUsers) {
-        usernames = usernames.concat(await getFileContents(args.fileUsers, 'username'));
+    return {
+        usernames: (args.users || []).concat(fileUsernames),
+        fileUsernames,
+    };
+}
+
+// --cleanup: only check each user's account status (one request per user, instead of a full
+// archive run's several) to record suspended/deleted accounts and prune the users file - see
+// sources/accountStatus.js
+async function cleanupUsers() {
+    const { usernames, fileUsernames } = await getUsernames();
+
+    if (!usernames.length) {
+        logger.warn('Nothing to clean up, supply --users and/or --file-users');
+
+        return;
     }
 
-    if (args.filePosts) {
-        postIds = postIds.concat(await getFileContents(args.filePosts, 'post ID'));
+    const tracker = await createAccountTracker({ usersFile: args.fileUsers, fileUsernames });
+    const statuses = {
+        active: 0, suspended: 0, deleted: 0, error: 0,
+    };
+
+    logger.info(`Checking ${usernames.length} accounts...`);
+
+    await Promise.each(usernames, async (username, index) => {
+        const { status } = await fetchUser(username, reddit);
+
+        statuses[status] += 1;
+        tracker.record(username, status);
+
+        if ((index + 1) % 50 === 0) {
+            logger.info(`Checked ${index + 1}/${usernames.length} accounts`);
+            await tracker.save();
+        }
+    });
+
+    logger.info(`Checked ${usernames.length} accounts: ${statuses.active} active, ${statuses.suspended} suspended, ${statuses.deleted} deleted, ${statuses.error} failed to check`);
+
+    await tracker.save();
+    tracker.summarize();
+}
+
+let ignoreIdsCache = null;
+
+async function getIgnoreIds() {
+    if (!ignoreIdsCache) {
+        ignoreIdsCache = args.fileIgnore ? await getFileContents(args.fileIgnore, 'ignore') : [];
     }
 
-    if (!usernames.length && !postIds.length) {
-        return null;
+    return ignoreIdsCache;
+}
+
+// everything up to (not including) downloading for one user: profile, posts, curation and
+// content info. This is where nearly all the reddit requests happen.
+async function prepareUser(username, tracker) {
+    try {
+        const { status, user } = await getUserPosts(username, tracker);
+
+        if (!user) {
+            return { username, status, users: null };
         }
 
+        const curated = curatePosts({ [user.name]: user }, await getIgnoreIds(), args);
+        const predata = await fetchPredata(getPostHosts(curated));
+        const users = await attachContentInfo(curated, { reddit, predata });
+
+        return { username, status, users };
+    } catch (error) {
+        logger.error(`Failed to fetch '${username}': ${args.debug ? error.stack : error.message}`);
+
+        return { username, status: 'error', users: null };
+    }
+}
+
+async function saveUser(prepared, ep) {
+    if (!prepared.users) {
+        return prepared.status;
+    }
+
+    try {
+        await Promise.each(Object.values(prepared.users), (user) => fetchSaveUserContent(user, ep, args));
+
+        return prepared.status;
+    } catch (error) {
+        logger.error(`Failed to save '${prepared.username}': ${args.debug ? error.stack : error.message}`);
+
+        return 'error';
+    }
+}
+
+function wait(ms) {
+    return new Promise((resolve) => {
+        setTimeout(resolve, ms);
+    });
+}
+
+// Processes users one at a time through the whole pipeline - profile, posts, content info,
+// downloads, index - instead of the old stage-by-stage approach (every user's posts first,
+// then every user's content info, then all downloads), which meant hours of reddit requests
+// before the first download on a large users list, all of it lost on a crash, and no way to
+// take on new users mid-run. Each user's index is now saved as soon as that user is done.
+//
+// While one user's media downloads (mostly bound by the hosting sites), the next user is
+// already being prepared (mostly bound by reddit's throttle) - a lookahead of one, so the two
+// kinds of waiting overlap without ever having more than two users in flight.
+//
+// source: see queue.js - next() hands out the next username (or null if there's none right
+// now), done() is told how each one went. With `waitInterval`, an empty source is polled
+// again after that many ms instead of ending the run (--queue --wait).
+async function runUsers(source, ep, tracker, { waitInterval = null } = {}) {
+    const counts = {};
+    let pending = null;
+    let waiting = false;
+
+    const startNext = async () => {
+        const username = await source.next();
+
+        pending = username ? prepareUser(username, tracker) : null;
+    };
+
+    await startNext();
+
+    // eslint-disable-next-line no-constant-condition
+    while (true) {
+        if (!pending) {
+            if (!waitInterval) {
+                break;
+            }
+
+            if (!waiting) {
+                logger.info(`Nothing left in ${source.label}, checking again every ${Math.round(waitInterval / 1000)}s`);
+                waiting = true;
+            }
+
+            // eslint-disable-next-line no-await-in-loop
+            await wait(waitInterval);
+            // eslint-disable-next-line no-await-in-loop
+            await startNext();
+
+            // eslint-disable-next-line no-continue
+            continue;
+        }
+
+        waiting = false;
+
+        // eslint-disable-next-line no-await-in-loop
+        const prepared = await pending;
+
+        // start preparing the next user before downloading this one's media
+        // eslint-disable-next-line no-await-in-loop
+        await startNext();
+
+        // eslint-disable-next-line no-await-in-loop
+        const status = await saveUser(prepared, ep);
+
+        counts[status] = (counts[status] || 0) + 1;
+
+        // eslint-disable-next-line no-await-in-loop
+        await source.done(prepared.username, status);
+        // eslint-disable-next-line no-await-in-loop
+        await tracker.save();
+    }
+
+    const total = Object.values(counts).reduce((acc, count) => acc + count, 0);
+
+    if (total) {
+        logger.info(`Processed ${total} user(s) from ${source.label}: ${Object.entries(counts).map(([status, count]) => `${count} ${status}`).join(', ')}`);
+    }
+
+    tracker.summarize();
+}
+
+// --queue [file]: queue the given users (if any), then work through the queue - unless another
+// process already is, in which case it'll pick up the added users itself
+async function runQueue(ep) {
+    const queueFile = args.queue || config.queue.file;
+    const { usernames, fileUsernames } = await getUsernames();
+
     if (usernames.length) {
-        userPosts = await getUserPosts(usernames);
+        const added = await appendToList(queueFile, usernames);
+
+        logger.info(`Added ${added.length} user(s) to queue '${queueFile}'${added.length < usernames.length ? ` (${usernames.length - added.length} already queued)` : ''}`);
     }
 
-    if (postIds.length) {
-        userPosts = await getPosts(postIds, userPosts);
+    const owner = await acquireLock(queueFile);
+
+    if (owner) {
+        logger.info(`Queue '${queueFile}' is already being processed by another ripunzel process (pid ${owner})${usernames.length ? ', which will pick up the added users' : ''}`);
+
+        return;
     }
 
-    if (args.fileIgnore) {
-        ignoreIds = await getFileContents(args.fileIgnore, 'ignore');
+    const tracker = await createAccountTracker({
+        usersFile: args.fileUsers,
+        fileUsernames,
+        enabled: config.users.dead.duringArchive,
+    });
+
+    const source = createQueueSource(queueFile);
+
+    logger.info(`Processing queue '${queueFile}'${args.wait ? ', waiting for new users when empty' : ''}`);
+
+    await runUsers(source, ep, tracker, { waitInterval: args.wait ? config.queue.interval : null });
+
+    // another process may have appended users after this one's last look at the queue, but
+    // while it still held the lock - so that process left them to this one. Release first,
+    // then look again: anything appended before this check is seen here, anything after it
+    // finds no lock, so its own process takes over.
+    releaseLock();
+
+    if (await source.hasPending() && !(await acquireLock(queueFile))) {
+        await runUsers(source, ep, tracker);
+        releaseLock();
+    }
+}
+
+async function runUserList(ep) {
+    const { usernames, fileUsernames } = await getUsernames();
+
+    if (!usernames.length) {
+        return;
     }
 
-    const curatedUserPosts = curatePosts(userPosts, ignoreIds, args);
-    const predata = await fetchPredata(getPostHosts(curatedUserPosts));
+    const tracker = await createAccountTracker({
+        usersFile: args.fileUsers,
+        fileUsernames,
+        enabled: config.users.dead.duringArchive,
+    });
 
-    return attachContentInfo(curatedUserPosts, { reddit, predata });
+    await runUsers(createListSource(usernames), ep, tracker);
+}
+
+// --posts/--file-posts: individual posts, grouped by author - small enough in practice to still
+// run as one batch
+async function runPosts(ep) {
+    const postIds = (args.posts || []).concat(args.filePosts ? await getFileContents(args.filePosts, 'post ID') : []);
+
+    if (!postIds.length) {
+        return;
+    }
+
+    const userPosts = await getPosts(postIds, {});
+    const curated = curatePosts(userPosts, await getIgnoreIds(), args);
+    const predata = await fetchPredata(getPostHosts(curated));
+    const users = await attachContentInfo(curated, { reddit, predata });
+
+    await Promise.each(Object.values(users), (user) => fetchSaveUserContent(user, ep, args));
 }
 
 async function getDirectContent(links, ep) {
@@ -134,15 +364,16 @@ async function getCompleteContents(ep) {
     return null;
 }
 
-function fetchSavePosts(userPosts, ep) {
-    // don't map to apply concurrency limit and reduce network stress
-    return Promise.reduce(Object.values(userPosts), (acc, user) => fetchSaveUserContent(user, ep, args), null);
-}
-
 async function initApp() {
     let ep;
 
     try {
+        if (args.cleanup) {
+            await cleanupUsers();
+
+            return;
+        }
+
         ep = new exiftool.ExiftoolProcess(exiftoolBin);
         await ep.open();
 
@@ -150,12 +381,14 @@ async function initApp() {
             await getCompleteContents(ep);
         }
 
-        const userPosts = await getCompletePosts();
-
-        if (userPosts) {
-            await fetchSavePosts(userPosts, ep);
+        if (args.queue !== undefined) {
+            await runQueue(ep);
+        } else {
+            await runUserList(ep);
         }
 
+        await runPosts(ep);
+
         if (args.watch) {
             logger.info(`Watch-mode enabled, checking again for new posts according to crontab '${config.fetch.watch.schedule}'.`);
         }

src/archives/ip.js

@@ -2,6 +2,8 @@
 
 const $ = require('cheerio');
 
+const { fetch } = require('../proxy');
+
 function findOnIp(username, page = 1, acc = []) {
     return Promise.resolve().then(() => {
         return fetch(`https://www.imageporn.net/user/${username}/all/${page}`);

src/cli.js

@@ -31,6 +31,18 @@ function getArgs() {
             describe: 'Load reddit usernames from file',
             type: 'string',
         })
+        .option('queue', {
+            describe: 'Work through a queue file of usernames (default: queue.file in config, \'queue\'), removing each user once archived and re-reading the file for new users as it goes. Users given with --users/--file-users are added to the queue. If another process is already working through the queue, only adds them and exits',
+            type: 'string',
+        })
+        .option('wait', {
+            describe: 'With --queue: keep running when the queue is empty, checking it for new users every queue.interval ms',
+            type: 'boolean',
+        })
+        .option('cleanup', {
+            describe: 'Only check the supplied users for suspended/deleted accounts: record them, and remove them from --file-users once dead for longer than configured. Does not archive anything',
+            type: 'boolean',
+        })
         .option('posts', {
             alias: 'post',
             describe: 'Reddit post IDs to fetch',

src/curate/posts.js

@@ -44,6 +44,12 @@ function report(curatedPosts, indexed, user, args) {
     }
 }
 
+// a crosspost's own media fields are empty - the video/gallery data lives on the original
+// post, which reddit embeds in full as crosspost_parent_list[0]
+function mediaSource(post) {
+    return post.crosspost_parent_list?.[0] || post;
+}
+
 function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args) {
     const host = dissectLink(unprint.prefixUrl(post.url, 'https://reddit.com'));
     const permalink = `https://reddit.com${post.permalink}`;
@@ -62,6 +68,15 @@ function curatePost(acc, post, user, index, indexed, ignoreIds, processed, args)
         preview: post.preview ? post.preview.images.map((image) => image.source) : null,
         host,
         direct: post.direct,
+        // reddit's submission listing (src/reddit/client.js's getSubmissions, fetched once up
+        // front) already embeds these for gallery posts - carrying them through lets
+        // methods/redditAlbum.js use them directly instead of re-fetching the same post
+        // individually through reddit's own separately-throttled client (see there)
+        mediaMetadata: post.media_metadata || mediaSource(post).media_metadata || null,
+        galleryData: post.gallery_data || mediaSource(post).gallery_data || null,
+        // same idea for v.redd.it videos (methods/redditVideo.js): the listing already has
+        // media.reddit_video (fallback_url, dash_url, has_audio), no need to re-fetch the post
+        media: post.secure_media || post.media || mediaSource(post).secure_media || mediaSource(post).media || null,
         // count only - the full thread (when needed) is fetched separately via methods/self.js
         comments: post.num_comments,
         hash: hashPost(post),

src/curate/user.js

@@ -1,6 +1,6 @@
 'use strict';
 
-function curateUser(user) {
+function curateUser(user, socialLinks = []) {
     const curatedUser = {
         id: user.id,
         name: user.name,
@@ -21,6 +21,9 @@ function curateUser(user) {
                 banner: user.subreddit.banner_img,
                 description: user.subreddit.public_description,
                 over18: user.subreddit.over_18,
+                // the "Social Links" widget on their profile (Twitter/Instagram/OnlyFans/etc.) -
+                // see reddit/client.js's getSocialLinks()
+                links: socialLinks,
             },
         });
     }

src/dissectLink.js

@@ -56,7 +56,11 @@ const hosts = [
         pattern: new UrlPattern('http(s)\\://(www.)vidble.com/album/:id(?*)'),
     },
     {
-        method: 'gfycat',
+        // gfycat shut down on 2023-09-01. Its NSFW content had already been migrated to
+        // RedGifs under the same (lowercased) ID, which methods/redgifs.js already lowercases -
+        // so an old gfycat link is resolved through RedGifs directly. Label stays 'gfycat' so
+        // --include/--exclude and {host.name} paths behave as before.
+        method: 'redgifs',
         label: 'gfycat',
         pattern: new UrlPattern('http(s)\\://(:server.)gfycat.com/(gifs/detail/)(:id-mobile)(:id-size_restricted)(:id)(.:ext)(?*)'),
     },

src/fetch/content.js

@@ -51,8 +51,10 @@ function selfPostToText(item, post) {
 
 async function getBuffers(item, context) {
     if (item.self) {
+        // `buffer:`, not a spread - spreading a Buffer copies its bytes into numbered keys,
+        // leaving `.buffer` undefined, so self posts were silently never saved
         return [{
-            ...Buffer.from(selfPostToText(item, context.post), 'utf8'),
+            buffer: Buffer.from(selfPostToText(item, context.post), 'utf8'),
             hash: context.post.hash,
         }];
     }

src/fetch/item.js

@@ -6,8 +6,9 @@ const blake2 = require('blake2');
 const phash = require('sharp-phash');
 // const phashDistance = require('sharp-phash/distance');
 
+const { agentFor } = require('../proxy');
 const logger = require('../logger')(__filename);
-const limiter = require('../limiter').items;
+const limiterFor = require('../limiter');
 
 async function fetchItem(url, attempt, context) {
     async function retry(error) {
@@ -22,7 +23,25 @@ async function fetchItem(url, attempt, context) {
 
     try {
         // throw new Error('Failed it!');
-        const res = await limiter.schedule(async () => bhttp.get(url, { headers: context.headers }));
+        // config.proxy.retryFlipped: once this url has already failed once, retry it via
+        // whichever of proxied/direct is NOT what it'd normally get - see resolveProxyUrl()
+        const flip = config.proxy.retryFlipped && attempt > 0;
+
+        const res = await limiterFor(url).schedule(() => bhttp.get(url, {
+            headers: context.headers,
+            agent: agentFor(url, { flip }),
+            // bhttp has no timeout at all by default - a server (or, more likely, a proxy
+            // tunnel) that connects fine but then just stops responding would otherwise hang
+            // forever. This isn't just a wall-clock ceiling on the *promise* (an external
+            // Promise.race achieves that, but leaves the real connection dangling) - bhttp
+            // calls req.abort() itself when this fires, actually releasing the socket. Without
+            // that, every stalled attempt leaks one connection onto the shared agent for that
+            // host; enough of those piling up (e.g. after several retries against a host that's
+            // currently black-holing everything, as seen with i.redd.it) exhausts it, and every
+            // *future* request to that host silently queues forever waiting for a socket that's
+            // never coming back - a freeze with no further log lines, not just a slow retry.
+            responseTimeout: config.fetch.timeout,
+        }));
 
         if (res.statusCode !== 200) {
             throw new Error(`Response not OK for ${url} (${res.statusCode})`);

src/interpolate.js

@@ -27,6 +27,17 @@ function safeFormat(date, dateFormat, label) {
     return dateFns.format(date, dateFormat);
 }
 
+// mime.extension() returns the boolean `false` (not undefined/null) for a mime type it
+// doesn't recognize - reddit's gallery API is a known offender, sending the non-standard
+// 'image/jpg' instead of 'image/jpeg' - which without this check ends up interpolated
+// verbatim as the literal filename extension '.false'. Fall back to whatever extension the
+// URL itself has instead, same as when there's no type at all.
+function resolveExt(item) {
+    const extFromType = item.type && mime.extension(item.type);
+
+    return extFromType ? `.${extFromType}` : path.extname(url.parse(item.url).pathname);
+}
+
 function interpolate(pattern, item = null, content = null, host = null, post = null, user = null, strip = true, dateFormat = config.library.dateFormat) {
     const data = {
         tags: {},
@@ -41,9 +52,7 @@ function interpolate(pattern, item = null, content = null, host = null, post = n
                 date: safeFormat(item.datetime, dateFormat, 'item'),
                 index: item.index + config.library.indexOffset,
             },
-            ext: item.type
-                ? `.${mime.extension(item.type)}`
-                : path.extname(url.parse(item.url).pathname),
+            ext: resolveExt(item),
             tags: {
                 ...data.tags,
                 extracted: item.extracted && config.library.tags.extracted,

src/limiter.js

@@ -1,13 +1,50 @@
 'use strict';
 
 const Bottleneck = require('bottleneck');
+const config = require('config');
 
 const args = require('./cli')();
 
-module.exports = {
-    // reddit requests are throttled inside src/reddit/client.js itself (config.reddit.throttle)
-    items: new Bottleneck({
+// same suffix matching as config.proxy.hosts (src/proxy.js): a bare hostname also matches its
+// subdomains, so 'redgifs.com' catches 'media.redgifs.com' but not the reverse
+function hostMatches(hostname, pattern) {
+    const normalized = pattern.replace(/^\./, '').toLowerCase();
+
+    return hostname === normalized || hostname.endsWith(`.${normalized}`);
+}
+
+// reddit's own API requests are throttled separately, inside src/reddit/client.js
+// (config.reddit.throttle) - this only governs item/media downloads (src/fetch/item.js)
+const defaultLimiter = new Bottleneck({
     maxConcurrent: args.concurrency,
     minTime: args.interval,
-    }),
-};
+});
+
+const hostLimiters = new Map();
+
+// config.limiter.hosts: most media CDNs (redgifs, imgur, i.redd.it, ...) don't need to be
+// treated as cautiously as reddit's own API, but a single shared limiter forces every host
+// down to whatever pace the most conservative one requires - worse, it means one host that's
+// currently stalling/throttling (see src/fetch/item.js's timeout handling) occupies the one
+// shared slot and blocks every *other* host's downloads behind it too, even though they have
+// nothing to do with each other. Giving a host its own limiter here lets it run at its own
+// concurrency/interval, independently of every other host's.
+function limiterFor(url) {
+    const { hostname } = new URL(url);
+    const hosts = config.limiter?.hosts || {};
+    const matchedPattern = Object.keys(hosts).find((pattern) => hostMatches(hostname, pattern));
+
+    if (!matchedPattern) {
+        return defaultLimiter;
+    }
+
+    if (!hostLimiters.has(matchedPattern)) {
+        const { concurrency = args.concurrency, interval = args.interval } = hosts[matchedPattern];
+
+        hostLimiters.set(matchedPattern, new Bottleneck({ maxConcurrent: concurrency, minTime: interval }));
+    }
+
+    return hostLimiters.get(matchedPattern);
+}
+
+module.exports = limiterFor;

src/methods/erome.js

@@ -3,6 +3,7 @@
 const config = require('config');
 const cheerio = require('cheerio');
 
+const { fetch } = require('../proxy');
 const logger = require('../logger')(__filename);
 
 const base = 'https://www.erome.com/';

src/methods/eroshareAlbum.js

@@ -2,6 +2,8 @@
 
 const config = require('config');
 
+const { fetch } = require('../proxy');
+
 async function eroshareAlbum(host) {
     const res = await fetch(`https://web.archive.org/web/20170630040157im_/https://eroshare.com/${host.id}`);
 

src/methods/eroshareItem.js

@@ -2,6 +2,8 @@
 
 const cheerio = require('cheerio');
 
+const { fetch } = require('../proxy');
+
 async function eroshareItem(host, post) {
     const res = await fetch(`https://web.archive.org/web/20170630040157im_/https://eroshare.com/i/${host.id}`);
     if (!res.ok) {

src/methods/gfycat.js

@@ -1,42 +0,0 @@
-'use strict';
-
-const bhttp = require('bhttp');
-const redgifs = require('./redgifs');
-
-async function gfycat(host) {
-    const res = await bhttp.get(`https://api.gfycat.com/v1/gfycats/${host.id}`);
-    const data = await res.body;
-
-    if (data.errorMessage) {
-        const redirectRes = await bhttp.head(host.url, {
-            followRedirects: false,
-        });
-
-        if (redirectRes.statusCode === 301) {
-            // Gfycat redirects all NSFW gifs to RedGifs, likely the case
-            return redgifs({
-                ...host,
-                url: `https://www.redgifs.com/watch/${host.id}`,
-                method: 'redgifs',
-                label: 'redgifs',
-            });
-        }
-
-        throw new Error(`Gfycat API returned error for source '${host.url}' (${res.status}): ${data.errorMessage}`);
-    }
-
-    return {
-        album: null,
-        items: [{
-            id: data.gfyItem.gfyName,
-            url: data.gfyItem.webmUrl,
-            title: data.gfyItem.title,
-            description: data.gfyItem.description,
-            type: 'video/webm',
-            datetime: new Date(data.gfyItem.createDate * 1000),
-            original: data.gfyItem,
-        }],
-    };
-}
-
-module.exports = gfycat;

src/methods/imgurAlbum.js

@@ -2,6 +2,7 @@
 
 const config = require('config');
 
+const { fetch } = require('../proxy');
 const logger = require('../logger')(__filename);
 const { fetchPredata } = require('./imgurImage');
 

src/methods/imgurImage.js

@@ -2,6 +2,8 @@
 
 const config = require('config');
 
+const { fetch } = require('../proxy');
+
 async function fetchPredata() {
     const data = {
         limit: 0,

src/methods/methods.js

@@ -3,7 +3,6 @@
 const erome = require('./erome');
 const eroshareAlbum = require('./eroshareAlbum');
 const eroshareItem = require('./eroshareItem');
-const gfycat = require('./gfycat');
 const imgurAlbum = require('./imgurAlbum');
 const imgurImage = require('./imgurImage');
 const redditImage = require('./redditImage');
@@ -21,7 +20,6 @@ module.exports = {
     erome,
     eroshareAlbum,
     eroshareItem,
-    gfycat,
     imgurAlbum,
     imgurImage,
     redditImage,

src/methods/redditAlbum.js

@@ -8,22 +8,47 @@ function resolveMediaUrl(media) {
     return url ? url.replace(/&/g, '&') : null;
 }
 
-async function redditAlbum(host, post, { reddit }) {
-    // fetched through the authenticated/throttled client, not a raw fetch()/bhttp request -
-    // reddit's anonymous-traffic bot challenge blocks plain unauthenticated .json requests
-    const data = await reddit.getSubmission(post.id);
+// reddit sends the non-standard 'image/jpg' instead of the real IANA 'image/jpeg' for
+// gallery items - left as-is, this silently breaks anything matching against real mime
+// types downstream (interpolate.js's extension lookup ends up with a '.false' extension,
+// fetch/content.js's addMeta() skips embedding metadata since it only matches 'image/jpeg')
+function normalizeType(type) {
+    return type === 'image/jpg' ? 'image/jpeg' : type;
+}
 
-    const items = (data.gallery_data?.items || [])
+async function redditAlbum(host, post, { reddit }) {
+    // reddit's submission listing already embeds media_metadata/gallery_data for gallery posts,
+    // and curate/posts.js carries them through as post.mediaMetadata/post.galleryData - reuse
+    // those instead of re-fetching the same post individually through reddit's own client
+    // (reservoir: 6/min, maxConcurrent: 1 - see src/reddit/client.js), which would otherwise cost
+    // one full throttled round trip per gallery post. Only fall back to a fresh fetch if the
+    // listing didn't have it for some reason (e.g. gallery_data null on the listing too).
+    const data = post.mediaMetadata
+        ? { media_metadata: post.mediaMetadata, gallery_data: post.galleryData }
+        : await reddit.getSubmission(post.id);
+    const mediaMetadata = data.media_metadata || {};
+
+    // gallery_data (the ordered {media_id, is_deleted, ...} list for this gallery) has been
+    // seen coming back null on a freshly-posted gallery even though media_metadata itself
+    // was already fully populated - fall back to media_metadata's own keys, in the order the
+    // JSON provided them (reddit sends them in gallery order, and object key order is
+    // preserved through JSON.parse), skipping anything not (yet, or any longer) valid
+    const galleryItems = data.gallery_data?.items
+        || Object.entries(mediaMetadata)
+            .filter(([, media]) => media.status === 'valid')
+            .map(([mediaId]) => ({ media_id: mediaId }));
+
+    const items = galleryItems
         .filter((item) => !item.is_deleted)
         .map((item) => {
-            const media = data.media_metadata?.[item.media_id];
+            const media = mediaMetadata[item.media_id];
             const url = resolveMediaUrl(media);
 
             return url && {
                 id: item.media_id,
                 url,
                 datetime: post.datetime,
-                type: media.m || 'image/jpeg',
+                type: normalizeType(media.m) || 'image/jpeg',
             };
         })
         .filter(Boolean);

src/methods/redditVideo.js

@@ -1,5 +1,7 @@
 'use strict';
 
+const { fetch } = require('../proxy');
+
 // the DASH manifest lists the actual audio track filename - reddit has changed this naming
 // scheme before (DASH_audio.mp4 -> /audio -> CMAF_AUDIO_128.mp4), so read it from the
 // manifest rather than guessing a hardcoded suffix off the video URL again
@@ -31,11 +33,30 @@ async function findAudio(video, videoId) {
     return findAudioUrl(await res.text(), videoId);
 }
 
-async function redditVideo(host, post, { reddit }) {
+async function getVideo(post, reddit) {
+    // the submission listing already includes media.reddit_video for video posts, and
+    // curate/posts.js carries it through as post.media (also for crossposts) - re-fetching each
+    // video post individually costs one full request against reddit's 6/min budget. Only
+    // fall back to that if the listing didn't have it (e.g. a post still processing).
+    if (post.media?.reddit_video) {
+        return post.media.reddit_video;
+    }
+
     // fetched through the authenticated/throttled client, not a raw fetch() - reddit's
     // anonymous-traffic bot challenge blocks plain unauthenticated .json requests
     const data = await reddit.getSubmission(post.id);
-    const video = data.media.reddit_video;
+    const source = data.crosspost_parent_list?.[0] || data;
+    const video = (source.secure_media || source.media)?.reddit_video;
+
+    if (!video) {
+        throw new Error(`No reddit video data available for post '${post.id}'`);
+    }
+
+    return video;
+}
+
+async function redditVideo(host, post, { reddit }) {
+    const video = await getVideo(post, reddit);
     const videoId = post.host.id || post.id;
 
     const item = {

src/methods/redgifs.js

@@ -3,6 +3,7 @@
 const mime = require('mime');
 // const unprint = require('unprint');
 
+const { fetch } = require('../proxy');
 const { version } = require('../../package.json');
 
 async function fetchPredata() {

src/methods/vidbleAlbum.js

@@ -5,6 +5,7 @@ const UrlPattern = require('url-pattern');
 const cheerio = require('cheerio');
 const mime = require('mime-types');
 
+const { fetch } = require('../proxy');
 const logger = require('../logger')(__filename);
 
 const pattern = new UrlPattern('https\\://(www.)vidble.com/:id(_med)(.:ext)');

src/methods/vidbleImage.js

@@ -3,6 +3,8 @@
 const $ = require('cheerio');
 const mime = require('mime-types');
 
+const { fetch } = require('../proxy');
+
 async function vidbleImage(host, post) {
     const res = await fetch(`https://vidble.com/${host.id}`);
 

src/methods/vidbleVideo.js

@@ -2,6 +2,8 @@
 
 const cheerio = require('cheerio');
 
+const { fetch } = require('../proxy');
+
 async function vidbleVideo(host, post) {
     const res = await fetch(`https://www.vidble.com/watch?v=${host.id}`);
 

src/proxy.js

@@ -0,0 +1,150 @@
+'use strict';
+
+const config = require('config');
+const { ProxyAgent, Socks5ProxyAgent } = require('undici');
+const { HttpProxyAgent } = require('http-proxy-agent');
+const { HttpsProxyAgent } = require('https-proxy-agent');
+const { SocksProxyAgent } = require('socks-proxy-agent');
+
+const dispatcherCache = new Map();
+const nodeAgentCache = new Map();
+
+// same suffix matching as config.reddit.cookies' domain filter (src/reddit/client.js): a
+// bare hostname also matches its subdomains, so 'reddit.com' catches 'www.reddit.com' but
+// not 'v.redd.it' or 'redd.it' - those need their own entry, or fall back to `default`
+function hostMatches(hostname, pattern) {
+    const normalized = pattern.replace(/^\./, '').toLowerCase();
+
+    return hostname === normalized || hostname.endsWith(`.${normalized}`);
+}
+
+function resolveProxyUrl(hostname, { flip = false } = {}) {
+    if (config.proxy?.use === false) {
+        return null;
+    }
+
+    const hosts = config.proxy?.hosts || {};
+    const matchedPattern = Object.keys(hosts).find((pattern) => hostMatches(hostname, pattern));
+
+    const normal = (() => {
+        if (!matchedPattern) {
+            return config.proxy?.default || null;
+        }
+
+        const matched = hosts[matchedPattern];
+
+        // `true` explicitly opts this host into `default`, `null`/`false`/'' explicitly opts it
+        // out (rather than falling through to `default`), anything else is a per-host override
+        if (matched === true) {
+            return config.proxy?.default || null;
+        }
+
+        return matched || null;
+    })();
+
+    if (!flip) {
+        return normal;
+    }
+
+    // used for config.proxy.retryFlipped: retry via whichever of proxied/direct is NOT what
+    // this host would normally get. A request that would normally be proxied (whether via
+    // `default` or a per-host override) retries direct; a request that would normally go
+    // direct (no proxy configured for this host, or explicitly excluded from `default`)
+    // retries through `default`, if one is set.
+    return normal ? null : (config.proxy?.default || null);
+}
+
+// 'socks4a'/'socks5h' (as opposed to plain 'socks4'/'socks5') tell socks-proxy-agent to
+// resolve the target hostname on the proxy side rather than locally - the same convention
+// curl uses, and worth keeping since it's the whole point of routing through SOCKS/Tor for
+// most people
+function isSocks(proxyUrl) {
+    return /^socks(4a?|5h?):/i.test(proxyUrl);
+}
+
+// fetch() goes through undici: http(s) proxies via ProxyAgent (CONNECT), SOCKS5 via
+// Socks5ProxyAgent (undici >= 7.x, still flagged experimental - node prints a one-time
+// ExperimentalWarning). Socks5ProxyAgent always sends the target as a hostname (address type
+// DOMAIN), i.e. it resolves on the proxy side like 'socks5h', so both spellings map onto it.
+// SOCKS4 isn't supported by undici at all - rather than silently sending what's presumably
+// meant to be private traffic out directly, that's an error (agentFor() below, for bhttp
+// downloads, does support it).
+function createDispatcher(proxyUrl, hostname) {
+    if (/^socks5h?:/i.test(proxyUrl)) {
+        return new Socks5ProxyAgent(proxyUrl.replace(/^socks5h:/i, 'socks5:'));
+    }
+
+    if (isSocks(proxyUrl)) {
+        throw new Error(`SOCKS4 proxy '${proxyUrl}' configured for '${hostname}' can't be used for fetch()-based requests (reddit API, host info lookups) - use a socks5:// or http(s):// proxy for this host instead`);
+    }
+
+    return new ProxyAgent(proxyUrl);
+}
+
+function dispatcherFor(targetUrl, { flip = false } = {}) {
+    const { hostname } = new URL(targetUrl);
+    const proxyUrl = resolveProxyUrl(hostname, { flip });
+
+    if (!proxyUrl) {
+        return undefined;
+    }
+
+    if (!dispatcherCache.has(proxyUrl)) {
+        dispatcherCache.set(proxyUrl, createDispatcher(proxyUrl, hostname));
+    }
+
+    return dispatcherCache.get(proxyUrl);
+}
+
+// bhttp uses node's raw http/https modules under the hood, which take a plain Agent
+// instance - unlike fetch() above, this does support SOCKS
+function agentFor(targetUrl, { flip = false } = {}) {
+    const url = new URL(targetUrl);
+    const proxyUrl = resolveProxyUrl(url.hostname, { flip });
+
+    if (!proxyUrl) {
+        return undefined;
+    }
+
+    const cacheKey = `${proxyUrl}|${isSocks(proxyUrl) ? 'socks' : url.protocol}`;
+
+    if (!nodeAgentCache.has(cacheKey)) {
+        if (isSocks(proxyUrl)) {
+            nodeAgentCache.set(cacheKey, new SocksProxyAgent(proxyUrl));
+        } else if (url.protocol === 'https:') {
+            nodeAgentCache.set(cacheKey, new HttpsProxyAgent(proxyUrl));
+        } else {
+            nodeAgentCache.set(cacheKey, new HttpProxyAgent(proxyUrl));
+        }
+    }
+
+    return nodeAgentCache.get(cacheKey);
+}
+
+// drop-in replacement for the global fetch() - importing this as `fetch` transparently
+// proxies requests per config.proxy, if one is configured for the request's host. Also
+// applies a hard wall-clock ceiling (config.fetch.timeout) to every request: unlike bhttp,
+// fetch() doesn't hang *forever* on a stalled/black-holed connection (undici defaults to a
+// 300s headers/body timeout), but that default is high enough - and, per live testing, a
+// proxy CONNECT tunnel that never completes can still take well over 20s to surface any
+// error - to look like a freeze, especially since reddit/client.js's requestJson() serializes
+// all reddit API calls behind a single-concurrency limiter. Bounding it here covers every
+// fetch()-based call site (reddit's API, and every hosting site's info-lookup) at once.
+// Pass `proxyFlip: true` in options (stripped before reaching fetch()) to use whichever of
+// proxied/direct is NOT what this host would normally get - see resolveProxyUrl().
+async function proxiedFetch(url, options = {}) {
+    const { proxyFlip, signal, ...fetchOptions } = options;
+    const dispatcher = dispatcherFor(String(url), { flip: proxyFlip });
+    const timeoutSignal = AbortSignal.timeout(config.fetch.timeout);
+
+    return fetch(url, {
+        ...fetchOptions,
+        ...(dispatcher ? { dispatcher } : {}),
+        signal: signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal,
+    });
+}
+
+module.exports = {
+    fetch: proxiedFetch,
+    agentFor,
+};

src/queue.js

@@ -0,0 +1,171 @@
+'use strict';
+
+const fs = require('fs-extra');
+
+const { readList, removeFromList } = require('./utils/listFile');
+const logger = require('./logger')(__filename);
+
+// --queue [file]: a users list that's consumed as it's processed - each user is removed once
+// archived (or found suspended/deleted), and the file is re-read before every next user, so
+// users added while ripunzel is running are picked up without restarting it. A lock file
+// (<queue>.lock, holding the pid) makes sure only one process works through a queue: another
+// `--queue --users ...` invocation just appends its users and leaves them to the running one.
+
+function lockPath(queueFile) {
+    return `${queueFile}.lock`;
+}
+
+function isAlive(pid) {
+    try {
+        process.kill(pid, 0);
+
+        return true;
+    } catch (error) {
+        // EPERM: exists, but owned by someone else
+        return error.code === 'EPERM';
+    }
+}
+
+async function lockOwner(queueFile) {
+    try {
+        const pid = Number(await fs.readFile(lockPath(queueFile), 'utf8'));
+
+        return pid && isAlive(pid) ? pid : null;
+    } catch (error) {
+        return null;
+    }
+}
+
+let heldLock = null;
+
+function releaseLock() {
+    if (!heldLock) {
+        return;
+    }
+
+    try {
+        // only remove it if it's still ours
+        if (Number(fs.readFileSync(heldLock, 'utf8')) === process.pid) {
+            fs.unlinkSync(heldLock);
+        }
+    } catch (error) {
+        // already gone
+    }
+
+    heldLock = null;
+}
+
+process.on('exit', releaseLock);
+
+['SIGINT', 'SIGTERM'].forEach((signal) => {
+    process.once(signal, () => {
+        releaseLock();
+        process.exit(signal === 'SIGINT' ? 130 : 143);
+    });
+});
+
+// returns null when acquired, or the pid of the live process holding it
+async function acquireLock(queueFile) {
+    const file = lockPath(queueFile);
+
+    try {
+        // 'wx' fails if it exists - atomic, so two processes starting at once can't both get it
+        await fs.writeFile(file, String(process.pid), { flag: 'wx' });
+    } catch (error) {
+        if (error.code !== 'EEXIST') {
+            throw error;
+        }
+
+        const owner = await lockOwner(queueFile);
+
+        if (owner) {
+            return owner;
+        }
+
+        logger.warn(`Removing stale queue lock '${file}' (process no longer running)`);
+
+        await fs.remove(file);
+
+        return acquireLock(queueFile);
+    }
+
+    heldLock = file;
+
+    return null;
+}
+
+// a source of usernames for app.js's runUsers(): next() hands out the next user not yet taken
+// in this run (re-reading the file every time), done() removes it from the queue unless it
+// failed - a failed user stays queued for the next run, but isn't retried in this one, so a
+// persistent failure can't loop forever
+function createQueueSource(queueFile) {
+    const taken = new Set();
+
+    return {
+        label: `queue '${queueFile}'`,
+        async next() {
+            const entries = await readList(queueFile);
+            const username = entries.find((entry) => !taken.has(entry.toLowerCase()));
+
+            if (username) {
+                taken.add(username.toLowerCase());
+            }
+
+            return username || null;
+        },
+        async done(username, status) {
+            if (status === 'error') {
+                logger.warn(`Leaving '${username}' in ${this.label} to retry on the next run`);
+
+                return;
+            }
+
+            await removeFromList(queueFile, [username]);
+
+            const remaining = (await readList(queueFile)).filter((entry) => !taken.has(entry.toLowerCase()));
+
+            logger.info(`Finished '${username}' (${status}), ${remaining.length} user(s) left in ${this.label}`);
+        },
+        // see app.js's runQueue(): closes the gap between deciding the queue is empty and
+        // another process appending to it while it still saw this one's lock
+        async hasPending() {
+            const entries = await readList(queueFile);
+
+            return entries.some((entry) => !taken.has(entry.toLowerCase()));
+        },
+    };
+}
+
+// plain --users/--file-users: a fixed list, nothing to remove afterwards (the dead account
+// tracker takes care of the users file)
+function createListSource(usernames) {
+    const seen = new Set();
+    const pending = usernames.filter((username) => {
+        const lower = username.toLowerCase();
+
+        if (seen.has(lower)) {
+            return false;
+        }
+
+        seen.add(lower);
+
+        return true;
+    });
+
+    return {
+        label: 'users',
+        async next() {
+            return pending.shift() || null;
+        },
+        async done(username, status) {
+            logger.verbose(`Finished '${username}' (${status}), ${pending.length} user(s) left`);
+        },
+    };
+}
+
+module.exports = {
+    acquireLock,
+    releaseLock,
+    createQueueSource,
+    createListSource,
+};

src/reddit/client.js

@@ -1,8 +1,10 @@
 'use strict';
 
 const Bottleneck = require('bottleneck');
+const cheerio = require('cheerio');
 const config = require('config');
 
+const { fetch } = require('../proxy');
 const { version } = require('../../package.json');
 const logger = require('../logger')(__filename);
 
@@ -36,17 +38,29 @@ function parseCookieJar(raw) {
         .join('; ');
 }
 
-function soonestExpiry(raw) {
-    const expiries = raw
+// per live testing (2026-11), reddit_session (the login itself, valid for ~6 months) is the
+// only cookie needed to get past the bot challenge - everything else in a browser export
+// (token_v2 ~24h, seeker_session a few hours, ...) is irrelevant here, and warning about those
+// expiring just cries wolf. Only fall back to the soonest-expiring cookie if there's no
+// reddit_session in the jar at all.
+function loginExpiry(raw) {
+    const cookies = raw
         .split('\n')
         .map((line) => line.trim())
         .filter((line) => line && !line.startsWith('#'))
         .map((line) => line.split('\t'))
         .filter((fields) => fields.length === 7)
-        .map(([, , , , expiry]) => Number(expiry))
-        .filter((expiry) => expiry > 0);
+        .map(([, , , , expiry, name]) => ({ name, expiry: Number(expiry) }));
 
-    return expiries.length ? Math.min(...expiries) : null;
+    const session = cookies.find(({ name }) => name === 'reddit_session');
+
+    if (session) {
+        return { name: session.name, expiry: session.expiry > 0 ? session.expiry : null };
+    }
+
+    const expiring = cookies.filter(({ expiry }) => expiry > 0).sort((a, b) => a.expiry - b.expiry);
+
+    return expiring[0] || { name: null, expiry: null };
 }
 
 let cachedCookieHeader;
@@ -62,11 +76,15 @@ function cookieHeader() {
             : 'No reddit cookies configured, requesting unauthenticated');
 
         if (config.reddit.cookies) {
-            const expiry = soonestExpiry(config.reddit.cookies);
+            const { name, expiry } = loginExpiry(config.reddit.cookies);
             const hoursLeft = expiry ? (expiry - Date.now() / 1000) / 3600 : null;
 
+            if (name !== 'reddit_session') {
+                logger.warn('No reddit_session cookie in config.reddit.cookies - reddit will likely block requests');
+            }
+
             if (hoursLeft !== null && hoursLeft < 48) {
-                logger.warn(`Soonest-expiring reddit cookie expires in ${hoursLeft.toFixed(1)}h (${new Date(expiry * 1000).toISOString()}) - refresh config/cookies.js before then`);
+                logger.warn(`Reddit cookie '${name}' expires in ${hoursLeft.toFixed(1)}h (${new Date(expiry * 1000).toISOString()}) - refresh config/cookies.js before then`);
             }
         }
     }
@@ -74,10 +92,10 @@ function cookieHeader() {
     return cachedCookieHeader;
 }
 
-function headers() {
+function headers(accept = 'application/json') {
     const requestHeaders = {
         'user-agent': config.reddit.userAgent || `ripunzel/${version}`,
-        accept: 'application/json',
+        accept,
     };
 
     const cookie = cookieHeader();
@@ -107,12 +125,40 @@ async function wait(seconds) {
     });
 }
 
-async function requestJson(path, params = {}, attempt = 0) {
+// shared by requestJson() and requestHtml() below - handles the actual throttled network
+// call plus retrying on a network error or a 429/503, regardless of what kind of response
+// the caller ultimately wants back.
+//
+// Only the actual throttled network call belongs inside limiter.schedule() - this has
+// maxConcurrent: 1, and every retry path below used to recurse back into this function
+// (and so back into limiter.schedule()) from *inside* that same callback. Bottleneck won't
+// start a second job while the first is still running, and the first can't finish until
+// its awaited recursive call starts - a guaranteed deadlock the instant a retry was ever
+// needed (e.g. a single 429), freezing every reddit request behind it for good. Retrying
+// now happens here, after the scheduled call has actually returned and freed its slot.
+async function fetchWithRetry(path, params, attempt, accept) {
     const url = buildUrl(path, params);
+    let res;
 
-    return limiter.schedule(async () => {
-        const res = await fetch(url.toString(), { headers: headers() });
-        const contentType = res.headers.get('content-type') || '';
+    try {
+        // config.proxy.retryFlipped: once this request has already failed once, retry it via
+        // whichever of proxied/direct is NOT what it'd normally get - see proxy.js
+        res = await limiter.schedule(async () => fetch(url.toString(), {
+            headers: headers(accept),
+            proxyFlip: config.proxy.retryFlipped && attempt > 0,
+        }));
+    } catch (error) {
+        // fetch() has no timeout of its own beyond undici's fairly generous defaults (see
+        // proxy.js) - this catches both that and genuine network errors (DNS, connection
+        // refused/reset, etc.), neither of which was retried before
+        if (attempt >= MAX_RETRIES) {
+            throw new Error(`Failed to reach reddit after ${attempt} retries for '${path}': ${error.message}`);
+        }
+
+        logger.warn(`Failed to reach reddit for '${path}', retrying: ${error.message}`);
+
+        return fetchWithRetry(path, params, attempt + 1, accept);
+    }
 
     if (res.status === 429 || res.status === 503) {
         if (attempt >= MAX_RETRIES) {
@@ -125,9 +171,16 @@ async function requestJson(path, params = {}, attempt = 0) {
 
         await wait(retryAfter);
 
-            return requestJson(path, params, attempt + 1);
+        return fetchWithRetry(path, params, attempt + 1, accept);
     }
 
+    return res;
+}
+
+async function requestJson(path, params = {}, attempt = 0) {
+    const res = await fetchWithRetry(path, params, attempt, 'application/json');
+    const contentType = res.headers.get('content-type') || '';
+
     if (!contentType.includes('json')) {
         // reddit serves an HTML interstitial/block page instead of JSON when it decides
         // the request looks like a bot - this is a distinct failure mode from a genuine
@@ -142,11 +195,69 @@ async function requestJson(path, params = {}, attempt = 0) {
     if (!res.ok) {
         const message = data?.message || data?.reason || JSON.stringify(data).slice(0, 200);
 
-            throw new Error(`Reddit returned ${res.status} for '${path}': ${message}`);
+        const error = new Error(`Reddit returned ${res.status} for '${path}': ${message}`);
+
+        // only set for genuine JSON API errors - sources/user.js relies on a 404 here meaning
+        // the account is really gone, as opposed to e.g. the HTML bot-challenge page above
+        error.status = res.status;
+
+        throw error;
     }
 
     return data;
+}
+
+// unlike requestJson(), this is for reddit's rendered profile pages - there's no public JSON
+// endpoint for a user's social links (the "Social Links" widget on their profile), they're
+// only present in the page's HTML (see getSocialLinks() below)
+async function requestHtml(path, params = {}, attempt = 0) {
+    const res = await fetchWithRetry(path, params, attempt, 'text/html');
+    const contentType = res.headers.get('content-type') || '';
+
+    if (!contentType.includes('html')) {
+        const hint = cookieHeader() ? ' - session cookie may have expired, refresh config/cookies.js' : ' - likely blocked or interstitial page (consider setting config.reddit.cookies)';
+
+        throw new Error(`Reddit returned non-HTML response (${res.status}, content-type '${contentType || 'none'}') for '${path}'${hint}`);
+    }
+
+    if (!res.ok) {
+        throw new Error(`Reddit returned ${res.status} for '${path}'`);
+    }
+
+    return res.text();
+}
+
+// reddit's "Social Links" profile widget (the row of Twitter/Instagram/OnlyFans/etc. buttons
+// under a user's bio) has no public JSON endpoint - the structured API for it
+// (/r/u_{username}/api/widgets.json) requires OAuth, which this project deliberately doesn't
+// use (see reddit.throttle above). The rendered profile page embeds the same data anyway, as
+// JSON inside each link's `data-faceplate-tracking-context` attribute, and that page is
+// accessible with the same cookies as everything else here.
+function parseSocialLinks(html) {
+    const $ = cheerio.load(html);
+    const links = [];
+
+    $('[data-faceplate-tracking-context]').each((index, el) => {
+        try {
+            const { social_link: link } = JSON.parse($(el).attr('data-faceplate-tracking-context'));
+
+            if (link) {
+                links.push(link);
+            }
+        } catch (error) {
+            // not a social-link tracker, or a shape we don't recognize - ignore
+        }
     });
+
+    return links
+        .sort((a, b) => a.position - b.position)
+        .map(({ type, name, url }) => ({ type, name, url }));
+}
+
+async function getSocialLinks(username) {
+    const html = await requestHtml(`/user/${username}/`);
+
+    return parseSocialLinks(html);
 }
 
 async function getUser(username) {
@@ -155,9 +266,27 @@ async function getUser(username) {
     return data.data;
 }
 
-async function getSubmissions(username, { sort = 'new', limit = Infinity } = {}) {
+// `until` (a Date, only honoured with sort 'new'): stop paging once a page reaches a post at
+// or before this moment - everything on later pages is older still, so there's no point
+// spending more of reddit's throttled budget (1 page = 1 request at 6/min) on posts the
+// caller is going to discard anyway. The page that crosses the line is still returned in full;
+// filtering out the older posts on it is left to curate/posts.js as before. Pinned posts are
+// ignored for this check, since reddit lists them first on a user's profile regardless of age.
+function reachedUntil(page, until) {
+    if (!until) {
+        return false;
+    }
+
+    const cutoff = until.getTime() / 1000;
+
+    return page.some((post) => !post.pinned && !post.stickied && post.created_utc <= cutoff);
+}
+
+async function getSubmissions(username, { sort = 'new', limit = Infinity, until = null } = {}) {
     const submissions = [];
+    const stopAt = sort === 'new' ? until : null;
     let after = null;
+    let pages = 0;
 
     do {
         // eslint-disable-next-line no-await-in-loop
@@ -171,6 +300,13 @@ async function getSubmissions(username, { sort = 'new', limit = Infinity } = {})
 
         submissions.push(...page);
         after = data.data.after;
+        pages += 1;
+
+        if (after && reachedUntil(page, stopAt)) {
+            logger.verbose(`Stopped fetching submissions for '${username}' after ${pages} page(s), reached ${stopAt.toISOString()}`);
+
+            break;
+        }
     } while (after && submissions.length < limit);
 
     return submissions;
@@ -277,6 +413,7 @@ async function getSubmissionWithComments(id) {
 module.exports = {
     requestJson,
     getUser,
+    getSocialLinks,
     getSubmissions,
     getSubmission,
     getSubmissionWithComments,

src/save/profileDetails.js

@@ -3,6 +3,7 @@
 const config = require('config');
 const Promise = require('bluebird');
 const UrlPattern = require('url-pattern');
+const yaml = require('js-yaml');
 
 const interpolate = require('../interpolate');
 const fetchItem = require('../fetch/item');
@@ -62,9 +63,15 @@ async function saveProfileDescription(user, args) {
     }
 
     if (config.library.profile.description && !user.fallback && !user.deleted) {
-        if (user.profile && user.profile.description) {
+        const links = user.profile?.links || [];
+
+        // YAML rather than plain text (like index.js) so the free-form bio and the structured
+        // social links (reddit/client.js's getSocialLinks()) stay separately parseable instead
+        // of just being smushed into one text blob
+        if (user.profile && (user.profile.description || links.length)) {
             const filepath = interpolate(config.library.profile.description, null, null, null, null, user);
-            const stream = Buffer.from(user.profile.description, 'utf8');
+            const data = { description: user.profile.description || null, links };
+            const stream = Buffer.from(yaml.dump(data, { skipInvalid: true }), 'utf8');
 
             try {
                 const targets = await save(filepath, stream);
@@ -77,7 +84,7 @@ async function saveProfileDescription(user, args) {
             }
         }
 
-        logger.verbose(`No profile description for '${user.name}' (https://reddit.com/user/${user.name})`);
+        logger.verbose(`No profile description or social links for '${user.name}' (https://reddit.com/user/${user.name})`);
 
         return null;
     }

src/sources/accountStatus.js

@@ -0,0 +1,135 @@
+'use strict';
+
+const config = require('config');
+const fs = require('fs-extra');
+const yaml = require('js-yaml');
+
+const { removeFromList } = require('../utils/listFile');
+
+const logger = require('../logger')(__filename);
+
+const DAY = 24 * 60 * 60 * 1000;
+
+// Keeps a YAML record (config.users.dead.file) of accounts found suspended or deleted, keyed by
+// username:
+//
+//   SomeUser:
+//     status: suspended        # or 'deleted'
+//     firstSeen: 2026-11-05T…  # first run that observed this status - reddit exposes no date
+//     lastSeen: 2026-11-05T…
+//     removed: 2026-02-03T…    # when it was dropped from the users file, if it was
+//
+// An account stays in the users file (and keeps being re-checked, at the cost of one request
+// per run) until it's been dead for longer than config.users.dead.retainDays[status], so a
+// lifted suspension or shadowban is noticed in time. After that it's removed from the users
+// file - but only if it came from one (--file-users), never from --users on the command line.
+// An account that turns out active again has its record dropped. Nothing is written until
+// save(), which is called after every user - a long (e.g. --queue) run shouldn't lose its
+// observations when interrupted.
+async function loadRecords(file) {
+    try {
+        return yaml.load(await fs.readFile(file, 'utf8')) || {};
+    } catch (error) {
+        if (error.code !== 'ENOENT') {
+            throw new Error(`Could not read dead account record '${file}': ${error.message}`);
+        }
+
+        return {};
+    }
+}
+
+function findKey(records, username) {
+    const lower = username.toLowerCase();
+
+    return Object.keys(records).find((key) => key.toLowerCase() === lower);
+}
+
+async function createAccountTracker({ usersFile = null, fileUsernames = [], enabled = true } = {}) {
+    const deadConfig = config.users?.dead || {};
+    const file = deadConfig.file || 'users_dead.yml';
+    const retainDays = deadConfig.retainDays || {};
+    const fromFile = new Set(fileUsernames.map((name) => name.toLowerCase()));
+    const records = enabled ? await loadRecords(file) : {};
+    const removals = new Set();
+    const counts = {
+        suspended: 0, deleted: 0, restored: 0, removed: 0,
+    };
+    let changed = false;
+
+    function record(username, status, date = new Date()) {
+        const now = date.toISOString();
+
+        if (!enabled || status === 'error') {
+            return;
+        }
+
+        const key = findKey(records, username);
+
+        if (status === 'active') {
+            if (key) {
+                logger.info(`Account '${username}' is active again (was ${records[key].status} since ${new Date(records[key].firstSeen).toISOString()}), dropped from dead account record`);
+
+                delete records[key];
+                counts.restored += 1;
+                changed = true;
+            }
+
+            return;
+        }
+
+        counts[status] += 1;
+
+        const entry = key && records[key].status === status
+            ? { ...records[key], lastSeen: now }
+            // new, or changed status (e.g. suspended -> deleted): the clock restarts
+            : { status, firstSeen: now, lastSeen: now };
+
+        if (key && key !== username) {
+            delete records[key];
+        }
+
+        records[username] = entry;
+        changed = true;
+
+        const deadDays = (date - new Date(entry.firstSeen)) / DAY;
+        const retain = retainDays[status] ?? 90;
+
+        if (deadDays >= retain && fromFile.has(username.toLowerCase())) {
+            removals.add(username);
+            entry.removed = now;
+            logger.info(`Account '${username}' has been ${status} since ${new Date(entry.firstSeen).toISOString()} (${Math.floor(deadDays)} days, retaining ${retain}), removing it from '${usersFile}'`);
+        } else {
+            // removed earlier, but (manually) put back in the users file since
+            if (fromFile.has(username.toLowerCase())) {
+                delete entry.removed;
+            }
+
+            logger.info(`Account '${username}' is ${status} (first seen ${new Date(entry.firstSeen).toISOString()}, ${Math.floor(deadDays)} of ${retain} days before removal${fromFile.has(username.toLowerCase()) ? '' : ', not from a users file so never removed automatically'})`);
+        }
+    }
+
+    async function save() {
+        if (!enabled || !changed) {
+            return;
+        }
+
+        if (removals.size && usersFile) {
+            await removeFromList(usersFile, removals);
+            counts.removed += removals.size;
+            removals.clear();
+        }
+
+        await fs.writeFile(file, yaml.dump(records, { sortKeys: true }));
+        changed = false;
+    }
+
+    function summarize() {
+        if (enabled && (counts.suspended || counts.deleted || counts.restored)) {
+            logger.info(`Dead account record '${file}': ${counts.suspended} suspended, ${counts.deleted} deleted, ${counts.restored} active again, ${counts.removed} removed from users file`);
+        }
+    }
+
+    return { record, save, summarize };
+}
+
+module.exports = createAccountTracker;

src/sources/getPosts.js

@@ -3,26 +3,9 @@
 const Promise = require('bluebird');
 
 const getIndex = require('./getIndex');
-const curateUser = require('../curate/user');
+const { fetchUser, indexUser, resolveUser } = require('./user');
 
-const logger = require('../logger')(__filename);
-
-async function getUser(username, reddit) {
-    try {
-        const user = await reddit.getUser(username);
-
-        return curateUser(user);
-    } catch (error) {
-        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return {
-            name: username,
-            fallback: true,
-        };
-    }
-}
-
-const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) {
+const getPostsWrap = (reddit, args) => function getPosts(postIds, userPosts = {}) {
     return Promise.reduce(postIds, (accUserPosts, postId) => Promise.resolve().then(async () => {
         const post = await reddit.getSubmission(postId);
 
@@ -55,8 +38,10 @@ const getPostsWrap = (reddit) => function getPosts(postIds, userPosts = {}) {
             };
         }
 
-        const user = await getUser(post.author, reddit);
-        const { profile, posts: indexed } = await getIndex(user);
+        // a suspended/deleted author still gets their post saved, under the fallback user
+        const fetched = await fetchUser(post.author, reddit);
+        const { profile, posts: indexed } = await getIndex(indexUser(post.author, fetched));
+        const user = await resolveUser(post.author, fetched, profile, reddit, args);
 
         return {
             ...accUserPosts,

src/sources/getUserPosts.js

@@ -1,35 +1,40 @@
 'use strict';
 
-const Promise = require('bluebird');
-
 const getIndex = require('./getIndex');
 const getArchivePostIds = require('../archives/getArchivePostIds');
-const curateUser = require('../curate/user');
+const { fetchUser, indexUser, resolveUser } = require('./user');
 
 const logger = require('../logger')(__filename);
 
-async function getUser(username, reddit) {
-    try {
-        const user = await reddit.getUser(username);
+// the moment before which posts will be discarded by curate/posts.js anyway (--after, or
+// --after-indexed against the existing index) - lets getSubmissions() stop paging there
+// instead of always walking reddit's entire (up to ~1000 post) listing. Without either flag,
+// every page is still fetched, so gaps anywhere in the archive keep getting backfilled.
+function getCutoff(indexedPosts, args) {
+    const cutoffs = [];
 
-        logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`);
-
-        return curateUser(user);
-    } catch (error) {
-        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
-
-        return {
-            name: username,
-            fallback: true,
-        };
+    if (args.after) {
+        cutoffs.push(args.after);
     }
+
+    if (args.afterIndexed && indexedPosts?.length) {
+        const dates = indexedPosts.map((entry) => new Date(entry.date).getTime()).filter((time) => !Number.isNaN(time));
+
+        if (dates.length) {
+            cutoffs.push(new Date(args.afterIndexed === 'latest' ? Math.max(...dates) : Math.min(...dates)));
+        }
+    }
+
+    // with both set, the later of the two discards the most, so stop there
+    return cutoffs.length ? new Date(Math.max(...cutoffs.map((date) => date.getTime()))) : null;
 }
 
-async function getPosts(username, reddit, args) {
+async function getPosts(username, reddit, args, until) {
     try {
         const submissions = await reddit.getSubmissions(username, {
             sort: args.sort,
             limit: Infinity,
+            until,
         });
 
         logger.info(`Fetched ${submissions.length} submissions for '${username}' (https://reddit.com/user/${username})`);
@@ -38,7 +43,8 @@ async function getPosts(username, reddit, args) {
     } catch (error) {
         logger.error(`Failed to fetch posts from reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
 
-        return [];
+        // distinct from [] (no posts), so the caller can tell a failure apart - see below
+        return null;
     }
 }
 
@@ -48,24 +54,51 @@ async function getArchivedPosts(username, posts, reddit) {
     return Promise.all(postIds.map((postId) => reddit.getSubmission(postId)));
 }
 
+// fetches a single user's profile and posts. Resolves to { status, user }:
+// - status 'active' with user (incl. posts and index) when there's something to curate,
+//   'empty' when the account has no posts, 'suspended'/'deleted' (see sources/user.js), or
+//   'error' when neither the profile nor the posts could be fetched - the latter is what
+//   e.g. the queue uses to decide a user should be retried later instead of being dropped
+// - tracker: see accountStatus.js - records suspended/deleted accounts, optional
 function getUserPostsWrap(reddit, args) {
-    return async function getUserPosts(usernames) {
-        try {
-            const users = await Promise.map(usernames, async (username) => {
-                try {
-                    const [user, posts] = await Promise.all([
-                        getUser(username, reddit),
-                        getPosts(username, reddit, args),
-                    ]);
+    return async function getUserPosts(username, tracker = null) {
+        // sequential: the user decides whether there's anything to fetch at all, the index
+        // (whose path may depend on the user's details) decides how far back to page and
+        // whether social links are needed, and every reddit request goes through the same
+        // single-concurrency limiter anyway
+        const fetched = await fetchUser(username, reddit);
 
-                    const { profile, posts: indexed } = await getIndex(user);
+        tracker?.record(username, fetched.status);
+
+        // nothing more to get from a suspended or deleted account - its listing is a 403/404
+        // too, and skipping it saves the rest of this user's requests
+        if (fetched.status === 'suspended' || fetched.status === 'deleted') {
+            return { status: fetched.status, user: null };
+        }
+
+        if (fetched.status === 'active') {
+            logger.info(`Fetched user profile for '${username}' (https://reddit.com/user/${username})`);
+        }
+
+        const { profile, posts: indexed } = await getIndex(indexUser(username, fetched));
+        const user = await resolveUser(username, fetched, profile, reddit, args);
+        const posts = await getPosts(username, reddit, args, getCutoff(indexed, args));
+
+        if (posts === null) {
+            return { status: 'error', user: null };
+        }
 
         if (args.archives) {
             posts.push(...await getArchivedPosts(username, posts, reddit));
         }
 
-                    if (posts.length) {
+        if (!posts.length) {
+            return { status: 'empty', user: null };
+        }
+
         return {
+            status: 'active',
+            user: {
                 ...user,
                 posts,
                 indexed: {
@@ -73,34 +106,8 @@ function getUserPostsWrap(reddit, args) {
                     original: indexed,
                     updated: [],
                 },
+            },
         };
-                    }
-
-                    return null;
-                } catch (error) {
-                    logger.error(`Failed to fetch posts from 'username': ${error.message}`);
-
-                    return null;
-                }
-            }, {
-                concurrency: 10,
-            });
-
-            return users.reduce(
-                (userPosts, user) => (user
-                    ? {
-                        ...userPosts,
-                        [user.name]: user,
-                    }
-                    : userPosts
-                ),
-                {},
-            );
-        } catch (error) {
-            logger.error(error);
-
-            throw error;
-        }
     };
 }
 

src/sources/user.js

@@ -0,0 +1,79 @@
+'use strict';
+
+const config = require('config');
+
+const curateUser = require('../curate/user');
+const logger = require('../logger')(__filename);
+
+// reddit tells deleted and suspended accounts apart (verified live, 2026-11):
+// - deleted: /user/x/about.json returns a JSON 404 (a shadowbanned account reportedly looks
+//   the same, so a 404 isn't necessarily permanent - see accountStatus.js's grace period)
+// - suspended: about.json returns 200 with only { name, is_suspended: true, ... } - no date of
+//   suspension is exposed anywhere
+// Anything else (network errors, 5xx, or the HTML bot-challenge page, which carries no
+// `status` - see reddit/client.js's requestJson()) is 'error', which must never be treated as
+// dead: a blocked or expired session would otherwise get the entire users file cleaned out.
+async function fetchUser(username, reddit) {
+    try {
+        const user = await reddit.getUser(username);
+
+        if (user.is_suspended) {
+            return { status: 'suspended', user: null };
+        }
+
+        return { status: 'active', user };
+    } catch (error) {
+        if (error.status === 404) {
+            return { status: 'deleted', user: null };
+        }
+
+        logger.error(`Failed to fetch reddit user '${username}': ${error.message} (https://reddit.com/user/${username})`);
+
+        return { status: 'error', user: null, error };
+    }
+}
+
+async function getSocialLinks(username, reddit) {
+    try {
+        return await reddit.getSocialLinks(username);
+    } catch (error) {
+        // supplementary, not worth failing the whole user fetch over - fall back to none
+        logger.warn(`Failed to fetch social links for '${username}': ${error.message} (https://reddit.com/user/${username})`);
+
+        return [];
+    }
+}
+
+// social links are only ever used for the profile description file (save/profileDetails.js),
+// which is skipped once it's in the index (unless --redownload-profile) - scraping the profile
+// page for them regardless costs one throttled reddit request per user per run for nothing
+function needsSocialLinks(indexedProfile, args) {
+    return Boolean(config.library.profile.description && (args.redownloadProfile || !indexedProfile?.description));
+}
+
+// the user as getIndex() needs it to resolve the index path - the full curated user (minus social
+// links, which depend on the index) in case a custom library.index.file pattern uses more than
+// the name, otherwise the same fallback as resolveUser() below
+function indexUser(username, fetched) {
+    return fetched.status === 'active' ? curateUser(fetched.user) : { name: username, fallback: true };
+}
+
+// curated user, or the same minimal fallback as before for an account that couldn't be fetched
+// (or is suspended/deleted, when the caller still wants to proceed - e.g. a --post by them)
+async function resolveUser(username, fetched, indexedProfile, reddit, args) {
+    if (fetched.status !== 'active') {
+        return { name: username, fallback: true };
+    }
+
+    const socialLinks = fetched.user.subreddit && needsSocialLinks(indexedProfile, args)
+        ? await getSocialLinks(username, reddit)
+        : [];
+
+    return curateUser(fetched.user, socialLinks);
+}
+
+module.exports = {
+    fetchUser,
+    indexUser,
+    resolveUser,
+};

src/utils/listFile.js

@@ -0,0 +1,79 @@
+'use strict';
+
+const fs = require('fs-extra');
+
+// helpers for the plain newline-separated list files (users, queue): one entry per line, '#'
+// comments and blank lines allowed. Writes only ever touch the matching entries, keep everything
+// else (order, comments) as-is, and replace the file atomically via rename - the file may be
+// edited by hand while ripunzel is running, so it's always re-read right before writing to keep
+// that window as small as possible.
+
+function parseList(contents) {
+    return contents
+        .split('\n')
+        .map((line) => line.trim())
+        .filter((line) => line && !line.startsWith('#'));
+}
+
+async function readList(file) {
+    try {
+        return parseList(await fs.readFile(file, 'utf8'));
+    } catch (error) {
+        if (error.code === 'ENOENT') {
+            return [];
+        }
+
+        throw error;
+    }
+}
+
+async function writeAtomic(file, contents) {
+    const tmpFile = `${file}.${process.pid}.tmp`;
+
+    await fs.writeFile(tmpFile, contents);
+    await fs.rename(tmpFile, file);
+}
+
+// case-insensitive, like reddit usernames
+async function removeFromList(file, entries) {
+    const remove = new Set([...entries].map((entry) => entry.toLowerCase()));
+    const contents = await fs.readFile(file, 'utf8');
+    const kept = contents.split('\n').filter((line) => !remove.has(line.trim().toLowerCase()));
+
+    await writeAtomic(file, kept.join('\n'));
+}
+
+// appends the entries not already listed, returns the ones actually added
+async function appendToList(file, entries) {
+    const contents = await fs.readFile(file, 'utf8').catch((error) => {
+        if (error.code === 'ENOENT') {
+            return '';
+        }
+
+        throw error;
+    });
+
+    const present = new Set(parseList(contents).map((entry) => entry.toLowerCase()));
+    const added = [];
+
+    entries.forEach((entry) => {
+        if (!present.has(entry.toLowerCase())) {
+            present.add(entry.toLowerCase());
+            added.push(entry);
+        }
+    });
+
+    if (added.length) {
+        const separator = contents && !contents.endsWith('\n') ? '\n' : '';
+
+        await writeAtomic(file, `${contents}${separator}${added.join('\n')}\n`);
+    }
+
+    return added;
+}
+
+module.exports = {
+    readList,
+    removeFromList,
+    appendToList,
+};