DebaucheryLibrarian
|
5deba6b90f
|
Passing resolved parameters into before-scrape methods.
|
2022-02-25 22:13:41 +01:00 |
|
DebaucheryLibrarian
|
3d349c413f
|
Fixed Adult Time scraper.
|
2022-02-07 22:16:43 +01:00 |
|
DebaucheryLibrarian
|
280032937f
|
Added centralized movie page fetcher. Added memory heap dump.
|
2021-12-20 02:22:10 +01:00 |
|
DebaucheryLibrarian
|
e88554666c
|
Reduced media concurrency to assess effect on memory. Moved qu context removal to deep scrape runner. Updated movie graphql queries.
|
2021-12-05 02:54:55 +01:00 |
|
DebaucheryLibrarian
|
abd063a578
|
Removed window map from http module to prevent memory pile-up with update scraping.
|
2021-12-04 00:32:28 +01:00 |
|
DebaucheryLibrarian
|
61daf5f037
|
Added debug log for jsdom window closing.
|
2021-12-01 23:44:25 +01:00 |
|
DebaucheryLibrarian
|
60c151ff6a
|
Reduced deep scrape concurrency to 1.
|
2021-12-01 23:39:09 +01:00 |
|
DebaucheryLibrarian
|
e41f9fa937
|
Added waitImmediate to deep scrape, reduced concurrency.
|
2021-12-01 23:30:10 +01:00 |
|
DebaucheryLibrarian
|
e29cbc9fea
|
Closing JSDOM window after deep scrape in an attempt to save memory. Reduced deep scrape concurrency to 5.
|
2021-12-01 17:26:13 +01:00 |
|
DebaucheryLibrarian
|
9d7183ac69
|
Added PurgatoryX scraper.
|
2021-11-27 23:55:16 +01:00 |
|
DebaucheryLibrarian
|
26539b74a5
|
Updated dependencies. Added periodic memory logger.
|
2021-11-20 23:59:48 +01:00 |
|
DebaucheryLibrarian
|
a22c4d5679
|
Added beforeNetwork hook, used by MindGeek. Added Filthy Kings to Gamma.
|
2021-10-27 17:19:23 +02:00 |
|
DebaucheryLibrarian
|
100a35b4e8
|
Added before scene fetch method to prevent e.g. unnecessary session requests, moved scraper assignment to entity lookup. Removed channel URL hostname matching..
|
2021-10-26 23:42:32 +02:00 |
|
DebaucheryLibrarian
|
7bd858f96b
|
Generalized Adult Empire subsite scraper, added West Coast Productions.
|
2021-08-30 01:13:32 +02:00 |
|
DebaucheryLibrarian
|
b08cb46ae5
|
Removed stray console log.
|
2021-08-15 13:21:02 +02:00 |
|
DebaucheryLibrarian
|
aacfd1b29d
|
Updated object-merge-advanced API use.
|
2021-08-15 13:16:48 +02:00 |
|
DebaucheryLibrarian
|
c2a008afbe
|
Added mimetype check to teasers and trailers. Added chapters to MindGeek scraper, fixed scene ID extraction getting stuck on numbers in domain name. Ordering chapters by timestamp.
|
2021-02-27 18:05:06 +01:00 |
|
DebaucheryLibrarian
|
7ff222ce25
|
Passing recursive parameters to all scraper methods. Using throttle parameters in MindGeek scraper, fixed missing slug breaking scene and actor URLs.
|
2021-02-10 03:23:48 +01:00 |
|
DebaucheryLibrarian
|
a6c7c60290
|
Improved movie tile actor overflow. Improved deep movie error feedback.
|
2021-02-10 03:00:17 +01:00 |
|
DebaucheryLibrarian
|
ff123b99b7
|
Added WankzVR update, scene and profile scraper.
|
2021-02-04 01:13:02 +01:00 |
|
DebaucheryLibrarian
|
3bbc5a5e87
|
Hard merging covers to preserve order.
|
2021-02-02 03:10:58 +01:00 |
|
DebaucheryLibrarian
|
d5cdfb36a9
|
Selecting included networks with infinite parent depth to facilitate scraper resolve.
|
2021-02-02 01:31:12 +01:00 |
|
DebaucheryLibrarian
|
aade7490f8
|
Querying infinite parent depth for deep release entities.
|
2021-02-01 01:45:30 +01:00 |
|
DebaucheryLibrarian
|
9535ab7953
|
Regarding layout parameters from parent and grandparent entities. Removed Evil Angel and Fantasy Massage scraper wrappers.
|
2021-01-30 00:01:40 +01:00 |
|
DebaucheryLibrarian
|
b506a00e7d
|
Improved scene merging. Improved Porn World/DDF scraper for poster and title redundancy. Fixed SFW poster showing in NSFW mode.
|
2021-01-25 23:53:56 +01:00 |
|
DebaucheryLibrarian
|
e9cbf5dab2
|
Added Woodman Casting X and WUNF update and scene scraper. Fixed actor and fallback media arrays being merged in deep scraper.
|
2021-01-24 23:31:28 +01:00 |
|
DebaucheryLibrarian
|
39d149c728
|
Added default deep scrape fetch method. Added Karups scene and profile scraper. Added schoolgirl tag photo.
|
2021-01-13 21:29:05 +01:00 |
|
DebaucheryLibrarian
|
2bae6f693e
|
Removed stray console log.
|
2021-01-13 16:08:53 +01:00 |
|
DebaucheryLibrarian
|
ae0efccb04
|
Skipping Babel, updated node version. Improved deep scrape array merge.
|
2021-01-13 16:08:19 +01:00 |
|
DebaucheryLibrarian
|
77ec2d3747
|
Using advanced merge library to maximize scraped data.
|
2020-12-27 23:43:08 +01:00 |
|
DebaucheryLibrarian
|
2656e3adb0
|
Added First Anal Quest and Double View Casting latest and scene scraper.
|
2020-12-02 03:17:32 +01:00 |
|
DebaucheryLibrarian
|
980efbc93d
|
Added series as channels with logos and photo album scraping to Little Caprice. Added various tag photos.
|
2020-11-24 04:29:44 +01:00 |
|
DebaucheryLibrarian
|
d14ef90136
|
Scraping actor scenes from Hussie Pass. Adding entity to actor base releases.
|
2020-11-15 23:50:04 +01:00 |
|
DebaucheryLibrarian
|
3a5ea3dd9a
|
Returning duplicate releases from pagination.
|
2020-08-22 04:22:56 +02:00 |
|
DebaucheryLibrarian
|
4ec89e2cc8
|
Added upcoming, profile and detailed scene actor scraping to InTheCrack. Fixed clip upsert.
|
2020-08-20 23:35:18 +02:00 |
|
DebaucheryLibrarian
|
40aed1086f
|
Showing actors, tags and date on movie tiles.
|
2020-08-10 21:39:55 +02:00 |
|
DebaucheryLibrarian
|
a3d281192d
|
Reordered scraper arguments. Fixed Jules Jordan scraper for Amateur Allure.
|
2020-07-17 23:27:59 +02:00 |
 ThePendulum
|
749864e922
|
Added 'independent' property for network-like channels. Changed release tile design. Adding Brazzers timeline events as tags. Added Property Sex to MindGeek. Changed DP, DAP and DVP tag slugs. Changed Porn Pros logo. Added better BAM Visions and Mug Fucked logos.
|
2020-07-02 04:04:28 +02:00 |
|
ThePendulum
|
4959dfd14f
|
Refactored deep and store modules to use entities.
|
2020-06-25 02:26:25 +02:00 |
|
ThePendulum
|
f0a89df6ab
|
Refactoring to use entities over sites and networks.
|
2020-06-17 04:07:24 +02:00 |
|
ThePendulum
|
b1b7cd6d50
|
Fixed Whale Member posters and photos.
|
2020-05-20 02:23:45 +02:00 |
|
ThePendulum
|
4826ae8571
|
Added oil tag. Removed Fame Digital from profile config.
|
2020-05-18 04:28:38 +02:00 |
|
ThePendulum
|
985ab9d2dc
|
Added profile interpolation.
|
2020-05-17 03:00:44 +02:00 |
|
ThePendulum
|
21d4dd6bfa
|
Scraping and storing actor profiles.
|
2020-05-15 04:40:59 +02:00 |
|
ThePendulum
|
11eb66f834
|
Switched to tabs. Adding missing actor entries when scraping actors, with batch ID.
|
2020-05-14 04:26:05 +02:00 |
|
ThePendulum
|
6d9f96c5d5
|
Improved duplicate filtering, now also applying to upcoming updates. Updated Gamma fetchLatest method parameters. Added shortcut for SFW-mode.
|
2020-03-28 04:37:04 +01:00 |
|
ThePendulum
|
fdb2b132f6
|
Improved release storage module. Added new tags module. Added movie scraping.
|
2020-03-22 03:50:24 +01:00 |
|
ThePendulum
|
d765543b30
|
Improved update runner. Improved HTTP module API, added default user agent. Added PornCZ and Czechav logos.
|
2020-03-21 02:48:24 +01:00 |
|
ThePendulum
|
e4b269956e
|
Attaching channel site and studio to stored releases.
|
2020-03-17 00:58:03 +01:00 |
|
ThePendulum
|
0f09fd53eb
|
Refactoring deep scrape. Added tag posters.
|
2020-03-16 04:10:52 +01:00 |