Commit Graph

30 Commits

Author SHA1 Message Date
DebaucheryLibrarian bed4fe288f Added predata step. Using RedGifs temporary API key. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian de50d609f3 Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 9140b58c70 Removed post.user variable. 2024-09-11 05:16:58 +02:00
DebaucheryLibrarian 8d72683ebc Renamed from reddit-post-dump to ripunzel. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian 5c7dca19e5 Added dynamic dividers to patterns. Fixed PornHub module. Updated README. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian b9a7e4b83a Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian fab6d0aa1c Reintroduced imgur API as a fallback method. 2024-09-11 05:16:57 +02:00
DebaucheryLibrarian e950a9ac54 No longer relying on imgur API for albums. Removed config for imgur API. Fixed ?#0 URLs not being detected for imgur. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 18acb3a878 Added fetch concurrency. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 00de0d43f5 Switched from setTimeout to crontab for watch-mode. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian fcb85f57c8 Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 029351f228 Using YAML rather than TSV for index files. Improves both readability and reindexability. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian c49e1edca0 Reading index file and ignoring already indexed content. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 952392e0d9 Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist). 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 5eb2eb651a Saving index per user as TSV. Refactoring. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian bba0f8f1d7 Fetching and curating posts per user without merge. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian be7bc1b10d Added filepath component length limit with truncator. Fixed double extension dot when copying original extension. Moved API keys to local config. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 89ab431a2f Added support for Vidble (image) albums. Replaced extensions.json and reverseLookup with mime-types lib. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian e01979a757 Added include and exclude source arguments. Improved stream fetch failure handling and improved logging details. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 7f00e5b6b2 Added support for reddit preview fallback. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian f0e312f2b1 Added archive support, and the IP archive. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 8a57a9a8f9 Added reddit avatar URL detection and avoid option. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 07d157604d Saving EXIF data for jpegs. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian 7e6828bd63 Added support for reddituploads.com. Changed default sorting to 'new'. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian 901029d9a1 Changed ignoreDuplicates to avoidDuplicates. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian df3421639f Flow and modularization refactor. Added duplicates option and applying limit after fetch. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian 1b0bd3a5a4 Refactoring flow. Added user and profile saving and variables. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian c3fc11d1ea Added single item album extraction with configurable display variable, as well as a host/source name variable. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian a6ff72ae0c Added groundwork for Eroshare support. Removed gfycat API requirement, using JSON output. 2024-09-11 05:16:53 +02:00
DebaucheryLibrarian 9cde1ce31a Added default config to repository. 2024-09-11 05:16:53 +02:00