Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.

Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 01:10:14 +01:00
co-authored by Claude Opus 5.5
parent 6905fb9b4a
commit 189e0ad227
11 changed files with 577 additions and 137 deletions
+8
View File
@@ -31,6 +31,14 @@ function getArgs() {
describe: 'Load reddit usernames from file',
type: 'string',
})
.option('queue', {
describe: 'Work through a queue file of usernames (default: queue.file in config, \'queue\'), removing each user once archived and re-reading the file for new users as it goes. Users given with --users/--file-users are added to the queue. If another process is already working through the queue, only adds them and exits',
type: 'string',
})
.option('wait', {
describe: 'With --queue: keep running when the queue is empty, checking it for new users every queue.interval ms',
type: 'boolean',
})
.option('cleanup', {
describe: 'Only check the supplied users for suspended/deleted accounts: record them, and remove them from --file-users once dead for longer than configured. Does not archive anything',
type: 'boolean',