Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.
Users now go through profile, posts, content info and downloads one by one (preparing the next user while the current one downloads), saving each index as soon as that user is done. The queue is re-read between users, users are removed once archived, and a lock file lets a second --queue invocation hand its users to the running process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -70,6 +70,9 @@ module.exports = {
|
||||
watch: {
|
||||
schedule: '*/30 * * * *',
|
||||
},
|
||||
// how long (ms) per-method predata (e.g. RedGifs' temporary API token) is reused before
|
||||
// being fetched again - matters for long --queue --wait runs
|
||||
predataTtl: 60 * 60 * 1000,
|
||||
archives: {
|
||||
search: false,
|
||||
preview: true,
|
||||
@@ -115,6 +118,12 @@ module.exports = {
|
||||
minTime: 1500,
|
||||
},
|
||||
},
|
||||
queue: {
|
||||
// used by --queue when no filename is given
|
||||
file: 'queue',
|
||||
// with --queue --wait, how often (ms) an empty queue is checked for new users
|
||||
interval: 60 * 1000,
|
||||
},
|
||||
users: {
|
||||
// record of suspended/deleted accounts, and pruning them from the --file-users list - see
|
||||
// src/sources/accountStatus.js. Also run on its own with --cleanup.
|
||||
|
||||
Reference in New Issue
Block a user