Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.

Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 01:10:14 +01:00
co-authored by Claude Opus 5.5
parent 6905fb9b4a
commit 189e0ad227
11 changed files with 577 additions and 137 deletions
+9
View File
@@ -70,6 +70,9 @@ module.exports = {
watch: {
schedule: '*/30 * * * *',
},
// how long (ms) per-method predata (e.g. RedGifs' temporary API token) is reused before
// being fetched again - matters for long --queue --wait runs
predataTtl: 60 * 60 * 1000,
archives: {
search: false,
preview: true,
@@ -115,6 +118,12 @@ module.exports = {
minTime: 1500,
},
},
queue: {
// used by --queue when no filename is given
file: 'queue',
// with --queue --wait, how often (ms) an empty queue is checked for new users
interval: 60 * 1000,
},
users: {
// record of suspended/deleted accounts, and pruning them from the --file-users list - see
// src/sources/accountStatus.js. Also run on its own with --cleanup.