Commit Graph

20 Commits

Author SHA1 Message Date
de50d609f3 Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs. 2024-09-11 05:16:58 +02:00
6e3bfdd175 Added support for various tube and social media sites via youtube-dl. 2024-09-11 05:16:58 +02:00
7de0c51c16 Fetching and storing comments for self posts. 2024-09-11 05:16:58 +02:00
b9a7e4b83a Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules. 2024-09-11 05:16:57 +02:00
416a3254fb Removed HTML scraper from imgur module, no longer support due to login restriction. Preserving 'preview' property in index file. 2024-09-11 05:16:57 +02:00
e48c00b529 Added option to index (specified) ignored posts. Saving index through save module, and it now notifies the user that an index has been written. 2024-09-11 05:16:57 +02:00
a68493d0fa Don't attempt to save index when there are no posts or profile details (complained about directory not existing) 2024-09-11 05:16:56 +02:00
352b2a66d7 Allow usernames and post IDs to be read from file. 2024-09-11 05:16:56 +02:00
e087e47069 No longer redownload profile image and description when indexed, unless --redownload-profile is specified. 2024-09-11 05:16:56 +02:00
fcb85f57c8 Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file. 2024-09-11 05:16:56 +02:00
99c7d143f7 Fixed index file for single post fetching. 2024-09-11 05:16:56 +02:00
86002ef00b Accounting for indexed posts. 2024-09-11 05:16:56 +02:00
029351f228 Using YAML rather than TSV for index files. Improves both readability and reindexability. 2024-09-11 05:16:56 +02:00
c49e1edca0 Reading index file and ignoring already indexed content. 2024-09-11 05:16:56 +02:00
952392e0d9 Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist). 2024-09-11 05:16:56 +02:00
bba0f8f1d7 Fetching and curating posts per user without merge. 2024-09-11 05:16:56 +02:00
2ecc2d8098 Added -size-restricted URL pattern for gfycat. Fixed missing getUser argument. 2024-09-11 05:16:55 +02:00
84836bc8c0 Refactored post retrieval so limit is applied per-user and ignores directly requested posts, and to start utilizing async/await. 2024-09-11 05:16:55 +02:00
c2cf26dde6 Refactored post fetching. Cleaner code that will now fetch archived posts even if the user no longer exists on reddit. Expanded gfycat URL detection. 2024-09-11 05:16:55 +02:00
2ec097c053 Cleaned up entrypoint. 2024-09-11 05:16:54 +02:00