Commit Graph

22 Commits

Author SHA1 Message Date
ThePendulum c1f26440fd Added fetch concurrency. 2018-07-02 03:45:20 +02:00
ThePendulum 5b023adf30 Switched from setTimeout to crontab for watch-mode. 2018-07-01 23:16:03 +02:00
ThePendulum e7eef86b23 Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file. 2018-07-01 03:06:57 +02:00
ThePendulum 74e36a6826 Using YAML rather than TSV for index files. Improves both readability and reindexability. 2018-06-30 03:33:30 +02:00
ThePendulum 33ef3d2c19 Reading index file and ignoring already indexed content. 2018-06-17 03:39:12 +02:00
ThePendulum 7cf1a99915 Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist). 2018-06-17 01:11:10 +02:00
ThePendulum 3b5d886da3 Saving index per user as TSV. Refactoring. 2018-06-15 01:06:31 +02:00
ThePendulum aef4dd02c7 Fetching and curating posts per user without merge. 2018-06-10 02:48:49 +02:00
ThePendulum 569213e291 Added filepath component length limit with truncator. Fixed double extension dot when copying original extension. Moved API keys to local config. 2018-05-23 17:03:12 +02:00
ThePendulum adbbc79b16 Added support for Vidble (image) albums. Replaced extensions.json and reverseLookup with mime-types lib. 2018-05-11 01:46:08 +02:00
ThePendulum 068ffbdbd8 Added include and exclude source arguments. Improved stream fetch failure handling and improved logging details. 2018-05-05 17:08:40 +02:00
ThePendulum 9d32450864 Added support for reddit preview fallback. 2018-05-05 02:14:48 +02:00
ThePendulum ca3bdd717d Added archive support, and the IP archive. 2018-05-05 00:51:58 +02:00
ThePendulum ccfdc16ca8 Added reddit avatar URL detection and avoid option. 2018-04-27 23:14:48 +02:00
ThePendulum f4535e443b Saving EXIF data for jpegs. 2018-04-24 22:25:36 +02:00
ThePendulum dece4b2530 Added support for reddituploads.com. Changed default sorting to 'new'. 2018-04-24 00:55:58 +02:00
ThePendulum 8bec57e906 Changed ignoreDuplicates to avoidDuplicates. 2018-04-23 14:58:23 +02:00
ThePendulum c66c011ff4 Flow and modularization refactor. Added duplicates option and applying limit after fetch. 2018-04-23 01:50:07 +02:00
ThePendulum dc3f3c8440 Refactoring flow. Added user and profile saving and variables. 2018-04-22 23:46:14 +02:00
ThePendulum b7bcb79d63 Added single item album extraction with configurable display variable, as well as a host/source name variable. 2018-04-20 04:40:37 +02:00
ThePendulum 81f8f4be0d Added groundwork for Eroshare support. Removed gfycat API requirement, using JSON output. 2018-04-19 01:47:09 +02:00
ThePendulum afbd07bd1e Added default config to repository. 2018-04-18 02:46:12 +02:00