DebaucheryLibrarian
|
e29cbc9fea
|
Closing JSDOM window after deep scrape in an attempt to save memory. Reduced deep scrape concurrency to 5.
|
2021-12-01 17:26:13 +01:00 |
|
DebaucheryLibrarian
|
87094a9498
|
Replaced cheerio with qu in Gamma scraper.
|
2021-11-29 02:49:07 +01:00 |
|
DebaucheryLibrarian
|
e81864ebde
|
Removed verbose http queued output.
|
2021-11-21 00:47:32 +01:00 |
|
DebaucheryLibrarian
|
2be3ea9bbb
|
Explicitly adding headers to http module result.
|
2021-11-21 00:41:12 +01:00 |
|
DebaucheryLibrarian
|
98c103c625
|
Fixed uuid import. Changed fs rmdir to rm.
|
2021-11-21 00:31:09 +01:00 |
|
DebaucheryLibrarian
|
f0b7678444
|
Fixed upsert failing on empty insert array due breaking Knex API change.
|
2021-11-21 00:19:10 +01:00 |
|
DebaucheryLibrarian
|
26539b74a5
|
Updated dependencies. Added periodic memory logger.
|
2021-11-20 23:59:48 +01:00 |
|
DebaucheryLibrarian
|
6c5d4389fe
|
Not parsing HTML with jsdom when using http module directly to save memory. Added loading ellipsis to release grid pages.
|
2021-10-25 02:06:24 +02:00 |
|
DebaucheryLibrarian
|
0e4fd12d70
|
Added more loggers to http module.
|
2021-10-20 01:46:56 +02:00 |
|
DebaucheryLibrarian
|
522584711b
|
Added Teen Mega World scraper.
|
2021-09-17 03:30:49 +02:00 |
|
DebaucheryLibrarian
|
7bd858f96b
|
Generalized Adult Empire subsite scraper, added West Coast Productions.
|
2021-08-30 01:13:32 +02:00 |
|
DebaucheryLibrarian
|
bed329cd8c
|
Using bulk insert utility for alert notifications to prevent duplicate errors.
|
2021-06-04 03:10:41 +02:00 |
|
DebaucheryLibrarian
|
c979173422
|
Rescraping upcoming scenes. Fixed language and scene deep scraping for Dorcel scraper.
|
2021-06-02 03:27:32 +02:00 |
|
DebaucheryLibrarian
|
c386a9098f
|
Generalized Team Skeet scraper, added MYLF network and various Team Skeet partner channels.
|
2021-03-29 22:22:56 +02:00 |
|
DebaucheryLibrarian
|
193af9bab5
|
Fixed session options in http module.
|
2021-03-23 15:25:21 +01:00 |
|
DebaucheryLibrarian
|
336b91c872
|
Refactored http timeout handling.
|
2021-03-17 02:09:34 +01:00 |
|
DebaucheryLibrarian
|
1703e9a541
|
Moved http timeout cancelation before pipeline to prevent large files from getting canceled.
|
2021-03-16 03:59:36 +01:00 |
|
DebaucheryLibrarian
|
a0be8f0aa3
|
Accounting for missing options in http utility timeout function.
|
2021-03-15 04:13:09 +01:00 |
|
DebaucheryLibrarian
|
cb447da7d0
|
Added harder timeouts to http utility. Split owner and query database users.
|
2021-03-15 04:11:14 +01:00 |
|
DebaucheryLibrarian
|
a275d0c855
|
Smaller tag logo on small displays. Centering tag tile lazy image to prevent load jumps. Including SASS breakpoint variable file automatically.
|
2021-03-07 16:39:54 +01:00 |
|
DebaucheryLibrarian
|
bb20659934
|
Refactored clips into chapters.
|
2021-02-27 00:37:22 +01:00 |
|
DebaucheryLibrarian
|
f310fec869
|
Added S3 experiment.
|
2021-02-20 01:12:44 +01:00 |
|
DebaucheryLibrarian
|
7ff222ce25
|
Passing recursive parameters to all scraper methods. Using throttle parameters in MindGeek scraper, fixed missing slug breaking scene and actor URLs.
|
2021-02-10 03:23:48 +01:00 |
|
DebaucheryLibrarian
|
79b51eca67
|
Added BaDoink profile scraper. Improved convert wrapper.
|
2021-02-03 21:03:35 +01:00 |
|
DebaucheryLibrarian
|
6d93083581
|
Removed superfluous MindGeek scrapers.
|
2021-02-03 00:46:59 +01:00 |
|
DebaucheryLibrarian
|
e3ef0a0d69
|
Added Top Web Models profile scraper.
|
2021-01-16 04:10:43 +01:00 |
|
DebaucheryLibrarian
|
12c3f0d7b9
|
Added Top Web Models framework.
|
2021-01-14 01:37:50 +01:00 |
|
DebaucheryLibrarian
|
db4e74fb99
|
Fixed Nubiles base poster query, handling trailing commas in qu source set. Added profile scene scraper to Dogfart. Added tag photo.
|
2021-01-11 16:20:01 +01:00 |
|
DebaucheryLibrarian
|
8739ec08cf
|
Fixed qu init selector. Fixed Aziani scene page scope.
|
2021-01-02 03:20:39 +01:00 |
|
DebaucheryLibrarian
|
bc944c2373
|
Restored Mike Adriano scraper.
|
2020-12-29 02:05:22 +01:00 |
|
DebaucheryLibrarian
|
8dd10f7e77
|
Adjusting tooltip arrow position, added open and close events. Fixed search tooltip layout.
|
2020-12-29 00:42:02 +01:00 |
|
DebaucheryLibrarian
|
cd8e810c35
|
Fixed various Kelly Madison scraper issues.
|
2020-12-17 02:05:01 +01:00 |
|
DebaucheryLibrarian
|
2656e3adb0
|
Added First Anal Quest and Double View Casting latest and scene scraper.
|
2020-12-02 03:17:32 +01:00 |
|
DebaucheryLibrarian
|
9a61d2305c
|
Added fixed actor age. Added male profiles to Littlr Caprice Dreams scraper. Added various tag photos.
|
2020-11-29 03:59:47 +01:00 |
|
DebaucheryLibrarian
|
980efbc93d
|
Added series as channels with logos and photo album scraping to Little Caprice. Added various tag photos.
|
2020-11-24 04:29:44 +01:00 |
|
DebaucheryLibrarian
|
711a9441a6
|
Added Little Caprice Dreams scraped (WIP).
|
2020-11-23 04:32:56 +01:00 |
|
DebaucheryLibrarian
|
ca7c8f0afd
|
Updated place resolve module to use new HTTP module.
|
2020-11-23 00:30:20 +01:00 |
|
DebaucheryLibrarian
|
0633197793
|
Removed direct bhttp usage from scrapers in favor of local http module. Deleted legacy scrapers, as old code is available via git repo history.
|
2020-11-23 00:05:02 +01:00 |
|
DebaucheryLibrarian
|
3d427f7e1d
|
Allowing HTTP rate limits to be set by configuration or argument.
|
2020-11-22 23:50:24 +01:00 |
|
DebaucheryLibrarian
|
b9b777c621
|
Using new HTTP module with a dynamic rate limiter.
|
2020-11-22 04:07:09 +01:00 |
|
DebaucheryLibrarian
|
77f9193669
|
Updated Dorcel scraper, added movie support.
|
2020-11-19 02:01:13 +01:00 |
|
DebaucheryLibrarian
|
d14ef90136
|
Scraping actor scenes from Hussie Pass. Adding entity to actor base releases.
|
2020-11-15 23:50:04 +01:00 |
|
DebaucheryLibrarian
|
df9a6eac05
|
Updated Hussie Pass scraper for new site design. Added cock size to profiles.
|
2020-11-15 04:33:24 +01:00 |
|
DebaucheryLibrarian
|
8a22ff07a6
|
Merged legacy JayRock scraper into new scraper for CosPimps.
|
2020-11-13 01:16:17 +01:00 |
|
DebaucheryLibrarian
|
39f8c037a5
|
Replaced bhttp with patched fork. Improved Jesse Loads Monster Facials scraper reliability (WIP). Added various tag photos.
|
2020-10-30 17:37:10 +01:00 |
|
DebaucheryLibrarian
|
3789ef51f2
|
(Temporarily) removed studio filter from entity query for performance reasons.
|
2020-09-18 22:43:45 +02:00 |
|
DebaucheryLibrarian
|
0190ee9531
|
Fixed Porn Doe's poster query. Checking style attribute existence in qu before attempting to use it.
|
2020-09-17 00:35:41 +02:00 |
|
DebaucheryLibrarian
|
6fb15fb591
|
Added Dorcel Club with scene and actor scraping. Added count method to qu.
|
2020-09-16 04:55:30 +02:00 |
|
DebaucheryLibrarian
|
ba7419d3b0
|
Added basic Pascals Subsluts scraper.
|
2020-09-14 02:40:27 +02:00 |
|
DebaucheryLibrarian
|
03ba35d65a
|
Fixed include object. Fixed qu's undefined URL handling.
|
2020-09-08 03:26:34 +02:00 |