 DebaucheryLibrarian
|
5c7dca19e5
|
Added dynamic dividers to patterns. Fixed PornHub module. Updated README.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
443f0fc028
|
Completed Erome module. Added content URL list fetching.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
b9a7e4b83a
|
Added support for fetching content directly from host. Improved pattern interpolation. Refactored content modules.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
8b85ec219d
|
Removed debug log. Updated README to mention imgur API key requirement.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
416a3254fb
|
Removed HTML scraper from imgur module, no longer support due to login restriction. Preserving 'preview' property in index file.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
c21ca0df0c
|
Fixed gfycat fetching.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
60973a0e70
|
Added post score to index and variables.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
8d75cc170d
|
Added basic PornHub support.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
9b0ef4da91
|
Added after/before-index alias for after/before-indexed
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
a817a07250
|
Always returning null for profile details when no profile was found, not undefined. Prevents YAML index write from failing.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
246ed1576f
|
Defaulting to jpeg if reddit image extension is not present.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
db9f6ed60c
|
Added optional URL query to all external hosts in link dissection.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
82b7f38336
|
Removed duplicate post link from imgur logs
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
e48c00b529
|
Added option to index (specified) ignored posts. Saving index through save module, and it now notifies the user that an index has been written.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
c30006f361
|
Added byte truncate script to repo. Fixed yargs not enforcing choices.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
12e17c7c3c
|
Calculating filename component limit in bytes rather than characters, preventing 'filename too long' errors when e.g. emojis are used
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
fab6d0aa1c
|
Reintroduced imgur API as a fallback method.
|
2024-09-11 05:16:57 +02:00 |
|
DebaucheryLibrarian
|
0377053d5a
|
Allowing for -mobile gfycat links, simplified URL patterns
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
91550406fe
|
Accounting for actualid_d.jpg imgur pattern. Added debug argument to print full stack trace on error.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
e950a9ac54
|
No longer relying on imgur API for albums. Removed config for imgur API. Fixed ?#0 URLs not being detected for imgur.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
dbc4c45601
|
No longer using imgur API for individual images. Only saving EXIF data to JPEGs. Always using global exiftool instance.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
b05ae06b00
|
Improved missing gfycat handling. Added 'preview' marker to index.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
a68493d0fa
|
Don't attempt to save index when there are no posts or profile details (complained about directory not existing)
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
18acb3a878
|
Added fetch concurrency.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
91577a00b1
|
Added support for file with host IDs to ignore.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
352b2a66d7
|
Allow usernames and post IDs to be read from file.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
00de0d43f5
|
Switched from setTimeout to crontab for watch-mode.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
c4afd398dc
|
Added timestamp to watch mode status log.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
438ba05260
|
Fixed profile image checking for indexed description instead of image.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
e087e47069
|
No longer redownload profile image and description when indexed, unless --redownload-profile is specified.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
fcb85f57c8
|
Added watch-mode. Waiting for profile detail write to finalize before new watch cycle and capture details in index file.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
dcf7fdd274
|
Fixed extension getting cut off at filename limit. Cleaned up save module.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
99c7d143f7
|
Fixed index file for single post fetching.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
cdd5ed36bf
|
Added redownload option to ignore index file.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
24a165cf48
|
Added date and indexed oldest/latest limit arguments.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
86002ef00b
|
Accounting for indexed posts.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
029351f228
|
Using YAML rather than TSV for index files. Improves both readability and reindexability.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
f41b788183
|
Extracting indexed posts from fetched posts to allow for index rewrite flexibility.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
c49e1edca0
|
Reading index file and ignoring already indexed content.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
952392e0d9
|
Building user posts object after fetching user to ensure user fetched posts and directly fetched posts are added to the same user key. Refactor to make better use of functions. Moved profile detail saving call to content fetch. No longer attempting and failing to save profile details for deleted users (directory would not exist).
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
5eb2eb651a
|
Saving index per user as TSV. Refactoring.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
6696438da0
|
Refactored info content fetching and saving to handle users object and utilize async/await.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
bba0f8f1d7
|
Fetching and curating posts per user without merge.
|
2024-09-11 05:16:56 +02:00 |
|
DebaucheryLibrarian
|
20a9c01444
|
Remove double slashes from URLs before matching.
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
c39f2eda8c
|
Improved handling of unavailable/deleted vidble videos
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
4e0fe3e6bc
|
Added vidble direct video (/watch?v=) support.
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
2ecc2d8098
|
Added -size-restricted URL pattern for gfycat. Fixed missing getUser argument.
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
84836bc8c0
|
Refactored post retrieval so limit is applied per-user and ignores directly requested posts, and to start utilizing async/await.
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
be7bc1b10d
|
Added filepath component length limit with truncator. Fixed double extension dot when copying original extension. Moved API keys to local config.
|
2024-09-11 05:16:55 +02:00 |
|
DebaucheryLibrarian
|
c2cf26dde6
|
Refactored post fetching. Cleaner code that will now fetch archived posts even if the user no longer exists on reddit. Expanded gfycat URL detection.
|
2024-09-11 05:16:55 +02:00 |