Commit Graph

17 Commits

Author SHA1 Message Date
5c7a0397c9 Bumped to Node 24 and upgraded dependencies. 2026-08-09 02:03:54 +02:00
b7dc3f5fd1 Storing pHash. Fixed RedGIFs user agent. 2024-09-11 05:16:58 +02:00
529f84d9c2 Fixed RedGIFs module content type. 2024-09-11 05:16:58 +02:00
350a2dc1dc Fixed username in save message. 2024-09-11 05:16:58 +02:00
de50d609f3 Added support for RedGifs and Reddit albums. Improved command line logger. Added rate limiters for reddit and host URLs. 2024-09-11 05:16:58 +02:00
45123bc630 Added hash comparison to duplicate avoidance. 2024-09-11 05:16:58 +02:00
b9a7e4b83a Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules. 2024-09-11 05:16:57 +02:00
60973a0e70 Added post score to index and variables. 2024-09-11 05:16:57 +02:00
e48c00b529 Added option to index (specified) ignored posts. Saving index through save module, and it now notifies the user that an index has been written. 2024-09-11 05:16:57 +02:00
b05ae06b00 Improved missing gfycat handling. Added 'preview' marker to index. 2024-09-11 05:16:56 +02:00
a68493d0fa Don't attempt to save index when there are no posts or profile details (complained about directory not existing) 2024-09-11 05:16:56 +02:00
18acb3a878 Added fetch concurrency. 2024-09-11 05:16:56 +02:00
fcb85f57c8 Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file. 2024-09-11 05:16:56 +02:00
86002ef00b Accounting for indexed posts. 2024-09-11 05:16:56 +02:00
029351f228 Using YAML rather than TSV for index files. Improves both readability and reindexability. 2024-09-11 05:16:56 +02:00
c49e1edca0 Reading index file and ignoring already indexed content. 2024-09-11 05:16:56 +02:00
952392e0d9 Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist). 2024-09-11 05:16:56 +02:00