DebaucheryLibrarian
/
ripunzel
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
164 Commits 1 Branch 0 Tags 450 KiB
Commit Graph

63 Commits

Author SHA1 Message Date
DebaucheryLibrarian 18acb3a878 Added fetch concurrency. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 91577a00b1 Added support for file with host IDs to ignore. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 352b2a66d7 Allow usernames and post IDs to be read from file. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 00de0d43f5 Switched from setTimeout to crontab for watch-mode. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian c4afd398dc Added timestamp to watch mode status log. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 438ba05260 Fixed profile image checking for indexed description instead of image. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian e087e47069 No longer redownload profile image and description when indexed, unless --redownload-profile is specified. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian fcb85f57c8 Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian dcf7fdd274 Fixed extension getting cut off at filename limit. Cleaned up save module. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 99c7d143f7 Fixed index file for single post fetching. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian cdd5ed36bf Added redownload option to ignore index file. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 24a165cf48 Added date and indexed oldest/latest limit arguments. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 86002ef00b Accounting for indexed posts. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 029351f228 Using YAML rather than TSV for index files. Improves both readability and reindexability. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian f41b788183 Extracting indexed posts from fetched posts to allow for index rewrite flexibility. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian c49e1edca0 Reading index file and ignoring already indexed content. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 952392e0d9 Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist). 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 5eb2eb651a Saving index per user as TSV. Refactoring. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 6696438da0 Refactored info content fetching and saving to handle users object and utilize async/await. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian bba0f8f1d7 Fetching and curating posts per user without merge. 2024-09-11 05:16:56 +02:00
DebaucheryLibrarian 20a9c01444 Remove double slashes from URLs before matching. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian c39f2eda8c Improved handling of unavailable/deleted vidble videos 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 4e0fe3e6bc Added vidble direct video (/watch?v=) support. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 2ecc2d8098 Added -size-restricted URL pattern for gfycat. Fixed missing getUser argument. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 84836bc8c0 Refactored post retrieval so limit is applied per-user and ignores directly requested posts, and to start utilizing async/await. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian be7bc1b10d Added filepath component length limit with truncator. Fixed double extension dot when copying original extension. Moved API keys to local config. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian c2cf26dde6 Refactored post fetching. Cleaner code that will now fetch archived posts even if the user no longer exists on reddit. Expanded gfycat URL detection. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian e0ca4a0822 Added new eroshare method source files. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 583b567b20 Added support for /i/ eroshare items. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian d113015eb2 Removed jpeg to jpg replace. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 5165e8612d Added vidble image support. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 2c58947548 Fixed jpeg jpg substitution. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 89ab431a2f Added support for Vidble (image) albums. Replaced extensions.json and reverseLookup with mime-types lib. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 5264020084 Fixed missing return causing no saves. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian e01979a757 Added include and exclude source arguments. Improved stream fetch failure handling and improved logging details. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 60eeca09d8 Added support for m.imgur.com links. Changed curate ignore log message details. 2024-09-11 05:16:55 +02:00
DebaucheryLibrarian 2ec097c053 Cleaned up entrypoint. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 7f00e5b6b2 Added support for reddit preview fallback. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 1ae51b18f8 Added preview property. Removed debugging intercepts. Logging post permalink on failed info fetch. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian f0e312f2b1 Added archive support, and the IP archive. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 2d050be1a5 Added support for redd.it PNG. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian f1db5728af Allow posts from [deleted] users to be fetched. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 854d55618a Fixed profile description not saving when user has avatar. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 6131d0a76c Added npm script for running application. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian de64cc6d13 Added support for fetching individual posts. Improved use of yargs, --help now available. Refactored main app flow. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 1581e780b8 Checking result status rather than ok for imgur methods. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 4af6aa85d3 Fixed obsolete eroshare extract single album item property. Throwing error when imgur album and image info fetch fails. Main catch method console log colored red. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian c250a02edf Fixed avatar property. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 767ae082fa Added ignore argument. 2024-09-11 05:16:54 +02:00
DebaucheryLibrarian 8a57a9a8f9 Added reddit avatar URL detection and avoid option. 2024-09-11 05:16:54 +02:00