Track suspended/deleted accounts and prune them from the users file after a grace period, also as standalone --cleanup. Only fetch social links when the profile description is saved, raised reddit throttle to 15/min.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 00:05:11 +01:00
co-authored by Claude Opus 5.5
parent 6582bac049
commit 6905fb9b4a
10 changed files with 352 additions and 84 deletions
+8 -1
View File
@@ -29,7 +29,7 @@ The pipeline runs from `src/app.js`:
- Calls reddit's public `.json` endpoints on `www.reddit.com` (`/user/x/about.json`, `/user/x/submitted.json`, `/comments/id.json`, `/api/morechildren.json`) with `raw_json=1`. There is no OAuth.
- Anonymous requests run into reddit's bot challenge, so a logged-in session's cookies (`config.reddit.cookies`, a cookie-jar string, or `config.reddit.cookie`, a raw header) are sent. Live testing (2026-11) showed that only `reddit_session` (the login, valid ~6 months) is required; without it, reddit returns a 403 HTML page. The client warns when `reddit_session` is missing or expires within 48h.
- Social links have no public JSON endpoint (the widgets API needs OAuth). They are scraped from the profile page HTML (`data-faceplate-tracking-context` attributes), which is fragile and breaks whenever reddit changes its markup.
- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 6/min, `minTime` 1500ms (`config.reddit.throttle`). This is deliberately slow, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
- All reddit requests go through **one Bottleneck limiter**: `maxConcurrent: 1`, reservoir 15/min, `minTime` 1500ms (`config.reddit.throttle`). reddit sends no rate-limit headers to cookie sessions, so the ceiling is unknown. Raise it with care, because being flagged as a bot defeats the purpose. Retries for 429/503 and network errors happen **outside** `limiter.schedule()`. Retrying inside it previously caused a deadlock; keep it that way.
- Avoid unnecessary reddit calls: gallery posts reuse `media_metadata`/`gallery_data` from the submission listing (`curate/posts.js` → `methods/redditAlbum.js`). `getSubmission` (post only, cheap) is separate from `getSubmissionWithComments` (full comment tree including `more` stubs, which is expensive and only used for self posts).
### Networking (`src/proxy.js`, `src/limiter.js`)
@@ -39,6 +39,13 @@ The pipeline runs from `src/app.js`:
- `limiter.js` provides per-host Bottleneck limiters for media downloads only (`config.limiter.hosts`). It is separate from the reddit API limiter.
- bhttp has no timeout of its own. `responseTimeout` in `fetch/item.js` makes it abort the socket. Don't replace this with a `Promise.race`, which would leak connections and eventually freeze downloads.
## Account status and cleanup
- `src/sources/user.js` `fetchUser()` classifies accounts as `active`, `suspended` (about.json returns 200 with `is_suspended: true`; there is no date), `deleted` (JSON 404, which may also be a shadowban) or `error`. **`error` (network failure, 5xx, HTML bot block) must never count as dead**; only genuine JSON API errors carry `error.status` (see `requestJson`).
- Suspended and deleted users are skipped after that one request. `src/sources/accountStatus.js` records them in `users_dead.yml` (`config.users.dead`) with `firstSeen`. Once they have been dead for `retainDays[status]` (default 90), they are removed from the `--file-users` file, never from `--users`. A record is dropped if the account turns out active again. Files are only written at the end of a run.
- `--cleanup` runs only this check (one request per user) without archiving. Regular archive runs do it too unless `users.dead.duringArchive` is false.
- Social links (an extra HTML request) are only fetched when the profile description will actually be saved, i.e. it isn't indexed yet or `--redownload-profile` is set.
## Fetching strategy
- By default, the entire submission listing is paged (up to reddit's ~1000-post cap). Every post not yet in the index is downloaded, so **gaps anywhere in the archive are backfilled**.