DebaucheryLibrarian and Claude Opus 5.5
189e0ad227
Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.
...
Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-11-06 01:10:14 +01:00
DebaucheryLibrarian and Claude Opus 5.5
6905fb9b4a
Track suspended/deleted accounts and prune them from the users file after a grace period, also as standalone --cleanup. Only fetch social links when the profile description is saved, raised reddit throttle to 15/min.
...
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-11-06 00:05:11 +01:00
DebaucheryLibrarian
5c7a0397c9
Bumped to Node 24 and upgraded dependencies.
2026-08-09 02:03:54 +02:00
DebaucheryLibrarian
7d633c31b4
Using Imgur API rate limit feedback to prevent exceeding it.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
de50d609f3
Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
6e3bfdd175
Added support for various tube and social media sites via youtube-dl.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
5c7dca19e5
Added dynamic dividers to patterns. Fixed PornHub module. Updated README.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
443f0fc028
Completed Erome module. Added content URL list fetching.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
b9a7e4b83a
Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
9b0ef4da91
Added after/before-index alias for after/before-indexed
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
e48c00b529
Added option to index (specified) ignored posts. Saving index through save module, and it now notifies the user that an index has been written.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
c30006f361
Added byte truncate script to repo. Fixed yargs not enforcing choices.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
91577a00b1
Added support for file with host IDs to ignore.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
352b2a66d7
Allow usernames and post IDs to be read from file.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
e087e47069
No longer redownload profile image and description when indexed, unless --redownload-profile is specified.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
fcb85f57c8
Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
cdd5ed36bf
Added redownload option to ignore index file.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
24a165cf48
Added date and indexed oldest/latest limit arguments.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
029351f228
Using YAML rather than TSV for index files. Improves both readability and reindexability.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
952392e0d9
Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist).
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
84836bc8c0
Refactored post retrieval so limit is applied per-user and ignores directly requested posts, and to start utilizing async/await.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
c2cf26dde6
Refactored post fetching. Cleaner code that will now fetch archived posts even if the user no longer exists on reddit. Expanded gfycat URL detection.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
e01979a757
Added include and exclude source arguments. Improved stream fetch failure handling and improved logging details.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
f0e312f2b1
Added archive support, and the IP archive.
2024-09-11 05:16:54 +02:00
DebaucheryLibrarian
6131d0a76c
Added npm script for running application.
2024-09-11 05:16:54 +02:00
DebaucheryLibrarian
de64cc6d13
Added support for fetching individual posts. Improved use of yargs, --help now available. Refactored main app flow.
2024-09-11 05:16:54 +02:00
