DebaucheryLibrarian and Claude Opus 5.5
189e0ad227
Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved.
...
Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-11-06 01:10:14 +01:00
DebaucheryLibrarian and Claude Opus 5.5
6905fb9b4a
Track suspended/deleted accounts and prune them from the users file after a grace period, also as standalone --cleanup. Only fetch social links when the profile description is saved, raised reddit throttle to 15/min.
...
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-11-06 00:05:11 +01:00
DebaucheryLibrarian
8b52d73a65
Separated and improved request throttling.
2026-09-08 00:01:24 +02:00
DebaucheryLibrarian
3e8ed79ccc
Addressed freezing indefinitely when requests are aborted.
2026-09-02 17:27:26 +02:00
DebaucheryLibrarian
941cb280be
Added proxy.
2026-08-17 22:48:02 +02:00
DebaucheryLibrarian
5c7a0397c9
Bumped to Node 24 and upgraded dependencies.
2026-08-09 02:03:54 +02:00
DebaucheryLibrarian
2e4ba27d98
Replaced Snoowrap with public API calls in response to Reddit revoking API keys. This was largely done using Claude Code, extensive manual review and major version bump needed if merged into main.
2026-08-07 23:37:26 +02:00
DebaucheryLibrarian
bed4fe288f
Added predata step. Using RedGifs temporary API key.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
de50d609f3
Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
9140b58c70
Removed post.user variable.
2024-09-11 05:16:58 +02:00
DebaucheryLibrarian
8d72683ebc
Renamed from reddit-post-dump to ripunzel.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
5c7dca19e5
Added dynamic dividers to patterns. Fixed PornHub module. Updated README.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
b9a7e4b83a
Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
fab6d0aa1c
Reintroduced imgur API as a fallback method.
2024-09-11 05:16:57 +02:00
DebaucheryLibrarian
e950a9ac54
No longer relying on imgur API for albums. Removed config for imgur API. Fixed ?#0 URLs not being detected for imgur.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
18acb3a878
Added fetch concurrency.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
00de0d43f5
Switched from setTimeout to crontab for watch-mode.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
fcb85f57c8
Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
029351f228
Using YAML rather than TSV for index files. Improves both readability and reindexability.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
c49e1edca0
Reading index file and ignoring already indexed content.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
952392e0d9
Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist).
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
5eb2eb651a
Saving index per user as TSV. Refactoring.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
bba0f8f1d7
Fetching and curating posts per user without merge.
2024-09-11 05:16:56 +02:00
DebaucheryLibrarian
be7bc1b10d
Added filepath component length limit with truncator. Fixed double extension dot when copying original extension. Moved API keys to local config.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
89ab431a2f
Added support for Vidble (image) albums. Replaced extensions.json and reverseLookup with mime-types lib.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
e01979a757
Added include and exclude source arguments. Improved stream fetch failure handling and improved logging details.
2024-09-11 05:16:55 +02:00
DebaucheryLibrarian
7f00e5b6b2
Added support for reddit preview fallback.
2024-09-11 05:16:54 +02:00
DebaucheryLibrarian
f0e312f2b1
Added archive support, and the IP archive.
2024-09-11 05:16:54 +02:00
DebaucheryLibrarian
8a57a9a8f9
Added reddit avatar URL detection and avoid option.
2024-09-11 05:16:54 +02:00
DebaucheryLibrarian
07d157604d
Saving EXIF data for jpegs.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
7e6828bd63
Added support for reddituploads.com. Changed default sorting to 'new'.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
901029d9a1
Changed ignoreDuplicates to avoidDuplicates.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
df3421639f
Flow and modularization refactor. Added duplicates option and applying limit after fetch.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
1b0bd3a5a4
Refactoring flow. Added user and profile saving and variables.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
c3fc11d1ea
Added single item album extraction with configurable display variable, as well as a host/source name variable.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
a6ff72ae0c
Added groundwork for Eroshare support. Removed gfycat API requirement, using JSON output.
2024-09-11 05:16:53 +02:00
DebaucheryLibrarian
9cde1ce31a
Added default config to repository.
2024-09-11 05:16:53 +02:00
