DebaucheryLibrarian/ripunzel
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
260 Commits 2 Branches 0 Tags
189e0ad227da9b26bd416eb4f5e9ee586118f995
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
DebaucheryLibrarianandClaude Opus 5.5 189e0ad227 Process users one at a time through the whole pipeline, added --queue [file] and --wait. Fixed self posts never being saved. 
Users now go through profile, posts, content info and downloads one by one (preparing the next
user while the current one downloads), saving each index as soon as that user is done. The queue
is re-read between users, users are removed once archived, and a lock file lets a second
--queue invocation hand its users to the running process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 01:10:14 +01:00
DebaucheryLibrarianandClaude Opus 5.5 6905fb9b4a Track suspended/deleted accounts and prune them from the users file after a grace period, also as standalone --cleanup. Only fetch social links when the profile description is saved, raised reddit throttle to 15/min. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 00:05:11 +01:00
DebaucheryLibrarianandClaude Opus 5.5 6582bac049 Base reddit cookie expiry warning on reddit_session, the only cookie actually required. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-05 22:49:09 +01:00
DebaucheryLibrarianandClaude Opus 5.5 309a99755c Reduced reddit requests: stop paging at --after(-indexed) cutoff, reuse listing media for videos, SOCKS5 support for reddit/fetch requests, routed defunct gfycat links to RedGifs. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-05 22:33:28 +01:00
DebaucheryLibrarianandClaude Opus 5.5 449a0e11c3 Added CLAUDE.md documenting the public reddit API rewrite. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-05 22:18:25 +01:00
DebaucheryLibrarian e23d8becd4 Storing social links in profile file, changed format to YAML. 2026-09-29 17:12:16 +02:00
DebaucheryLibrarian 1496a48199 Improved timeout handling. 2026-09-14 20:08:55 +02:00
DebaucheryLibrarian 8b52d73a65 Separated and improved request throttling. 2026-09-08 00:01:24 +02:00
DebaucheryLibrarian 3e8ed79ccc Addressed freezing indefinitely when requests are aborted. 2026-09-02 17:27:26 +02:00
DebaucheryLibrarian ba922bb385 Fixed .false file extension. 2026-08-17 23:31:29 +02:00
DebaucheryLibrarian 941cb280be Added proxy. 2026-08-17 22:48:02 +02:00
DebaucheryLibrarian 5c7a0397c9 Bumped to Node 24 and upgraded dependencies. 2026-08-09 02:03:54 +02:00
DebaucheryLibrarian 2e4ba27d98 Replaced Snoowrap with public API calls in response to Reddit revoking API keys. This was largely done using Claude Code, extensive manual review and major version bump needed if merged into main. 2026-08-07 23:37:26 +02:00
DebaucheryLibrarian 51f0267133 1.27.6 2025-11-28 01:16:03 +01:00
DebaucheryLibrarian 5923bb00bc Updated Node to v20 (blake2 incompatible with newer versions), replaced youtube-dl with youtube-dl-exec. 2025-11-28 01:15:59 +01:00
DebaucheryLibrarian 508df69808 1.27.5 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian e5b1e9ea25 Removed error response body from item fetch to minimize spamming image contents. 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian b4f8605464 1.27.4 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian e642203d6c Fixed invalid URL breaking scrape, prefixing reddit URLs with origin. 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian b1e13e048c 1.27.3 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian 96a041001b Stripping query for URL matching. 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian d402916d5e 1.27.2 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian d1ac6d0068 Fixed RedGIFs gallery support. 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian cedecc273a 1.27.1 2024-09-11 05:16:59 +02:00
DebaucheryLibrarian afaa428fec Fixed YAML index file parser failing on duplicate keys. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian aec5ce292d 1.27.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian b7dc3f5fd1 Storing pHash. Fixed RedGIFs user agent. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 3931996e1b 1.26.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7d633c31b4 Using Imgur API rate limit feedback to prevent exceeding it. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7ec4143972 1.25.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian bed4fe288f Added predata step. Using RedGifs temporary API key. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 4acbe16fb8 1.24.2 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 529f84d9c2 Fixed RedGIFs module content type. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7eaaadaa2a 1.24.1 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 350a2dc1dc Fixed username in save message. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7f79a44830 1.24.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian de50d609f3 Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian bb06fe9763 Cleanup 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 58a35fe998 1.23.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 6e3bfdd175 Added support for various tube and social media sites via youtube-dl. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 17c7acd6a2 1.22.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 45123bc630 Added hash comparison to duplicate avoidance. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7c6c99c9d1 1.21.0 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 7de0c51c16 Fetching and storing comments for self posts. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 926ab1d25c 1.20.2 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 9140b58c70 Removed post.user variable. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian e1a57a07a3 1.20.1 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian f47b6cbeb8 Fixed typos in README. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian bcd5759d0f 1.20.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 8d72683ebc Renamed from reddit-post-dump to ripunzel. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 66189694fa 1.19.3 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian e00f895b05 Removed obsolete 'boolean' mention from README. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 54dd191d6c 1.19.2 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian c2c5dc8d8a Fixed header in README. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 0c50bcab32 1.19.1 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 4e827fb9a8 Improved README formatting. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 50f1621a26 1.19.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 5c7dca19e5 Added dynamic dividers to patterns. Fixed PornHub module. Updated README. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian c4d434ecdf 1.18.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 443f0fc028 Completed Erome module. Added content URL list fetching. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian aecf79fa25 1.17.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian b9a7e4b83a Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 20cb522689 1.16.3 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 8b85ec219d Removed debug log. Updated README to mention imgur API key requirement. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian da90b832a1 1.16.2 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 416a3254fb Removed HTML scraper from imgur module, no longer support due to login restriction. Preserving 'preview' property in index file. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian d5d56b83bd 1.16.1 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian c21ca0df0c Fixed gfycat fetching. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian d7916af3d5 1.16.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 60973a0e70 Added post score to index and variables. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 12268fcb05 1.15.0 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 8d75cc170d Added basic PornHub support. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian f47b99501a 1.14.9 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 9b0ef4da91 Added after/before-index alias for after/before-indexed 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian bdd79031ef 1.14.8 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian a817a07250 Always returning null for profile details when no profile was found, not undefined. Prevents YAML index write from failing. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 7711979199 1.14.7 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 246ed1576f Defaulting to jpeg if reddit image extension is not present. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 34f7766496 1.14.6 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian db9f6ed60c Added optional URL query to all external hosts in link dissection. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 82b7f38336 Removed duplicate post link from imgur logs 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian b388142d8b 1.14.5 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian e48c00b529 Added option to index (specified) ignored posts. Saving index through save module, and it now notifies the user that an index has been written. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian c1b164a5f9 1.14.4 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian c30006f361 Added byte truncate script to repo. Fixed yargs not enforcing choices. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian e44f617d9e 1.14.3 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 12e17c7c3c Calculating filename component limit in bytes rather than characters, preventing 'filename too long' errors when e.g. emojis are used 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian e5d439cbaf 1.14.2 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian fab6d0aa1c Reintroduced imgur API as a fallback method. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 0377053d5a Allowing for -mobile gfycat links, simplified URL patterns 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 91550406fe Accounting for actualid_d.jpg imgur pattern. Added debug argument to print full stack trace on error. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian e950a9ac54 No longer relying on imgur API for albums. Removed config for imgur API. Fixed ?#0 URLs not being detected for imgur. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian dbc4c45601 No longer using imgur API for individual images. Only saving EXIF data to JPEGs. Always using global exiftool instance. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian b05ae06b00 Improved missing gfycat handling. Added 'preview' marker to index. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian e48892126f 1.14.1 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian a68493d0fa Don't attempt to save index when there are no posts or profile details (complained about directory not existing) 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 44cdd37e49 1.14.0 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 18acb3a878 Added fetch concurrency. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian e71b54e828 1.13.1 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 91577a00b1 Added support for file with host IDs to ignore. 2024-09-11 05:16:56 +02:00