boiii
/
traxxx
forked from DebaucheryLibrarian/traxxx
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
2,365 Commits 8 Branches 0 Tags 4.7 GiB
Commit Graph

184 Commits

Author SHA1 Message Date
DebaucheryLibrarian 5f2c9eb5df Fixed place cache check. 2023-08-03 00:31:07 +02:00
DebaucheryLibrarian c5d81e94e5 Updated OSM API path, added redis caching. 2023-08-02 23:12:41 +02:00
DebaucheryLibrarian 8c1f1b69ff Splitting Han titles and actors in Model Media scraper. 2023-08-02 02:14:41 +02:00
DebaucheryLibrarian 0905847ffa Updated Jules Jordan scraper. 2023-07-06 00:14:38 +02:00
DebaucheryLibrarian 205102ff90 Added Meiden van Holland and Vurig Vlaanderen. 2023-07-01 21:46:44 +02:00
DebaucheryLibrarian 80334843c9 Improved puppeteer bypass, enabled for Team Skeet. 2023-06-10 02:05:59 +02:00
DebaucheryLibrarian adda78f0c6 Refactored New Sensations scraper. 2023-06-05 02:13:36 +02:00
DebaucheryLibrarian cd98b9d147 Changed deep merge so base release photos are only used when no deep photos are available. 2022-12-29 23:07:08 +01:00
DebaucheryLibrarian 74214bc060 Added puppeteer to http module, refactored Kink scraper to utilize it. 2022-12-28 01:34:12 +01:00
DebaucheryLibrarian 6edd62c337 Refactored Bang! scraper to match new website, first use of unprint. 2022-11-27 04:22:58 +01:00
DebaucheryLibrarian 2943aea4d8 Added showcased migration. Added Love Her Films scraper. 2022-10-25 23:13:24 +02:00
DebaucheryLibrarian 0fc37e46d2 Fixed media module trying to fetch invalid source URLs. Added Accidental Gangbang to Adult Time. 2022-09-27 20:09:46 +02:00
DebaucheryLibrarian 3db8b80164 Added CF resolver to http module. Using priority lookup in tags seed. 2022-08-15 23:51:51 +02:00
DebaucheryLibrarian 527112d5da Refactored Vixen scraper, moved XEmpire to generic Gamma scraper after site update, added BiPhoria. 2022-05-15 23:28:56 +02:00
DebaucheryLibrarian 1d84830423 Supporting JSON in scenes file. Added Dogfart link finding utility. 2022-04-07 16:06:38 +02:00
DebaucheryLibrarian fffe9080f3 Added stand-alone Spizoo sites. 2022-02-07 21:58:27 +01:00
DebaucheryLibrarian dc187a9a3a Added execute method to qu, removed runScripts from Gamma's fetchMovie to observe effect on memory usage. 2021-12-19 23:03:44 +01:00
DebaucheryLibrarian 8e4be06675 Closing JSDOM window after deep scrapes in an attempt to save memory. 2021-12-18 23:22:35 +01:00
DebaucheryLibrarian abd063a578 Removed window map from http module to prevent memory pile-up with update scraping. 2021-12-04 00:32:28 +01:00
DebaucheryLibrarian 1918dd4ad5 Removing query methods from XEmpire release. 2021-12-02 00:45:22 +01:00
DebaucheryLibrarian e41f9fa937 Added waitImmediate to deep scrape, reduced concurrency. 2021-12-01 23:30:10 +01:00
DebaucheryLibrarian e29cbc9fea Closing JSDOM window after deep scrape in an attempt to save memory. Reduced deep scrape concurrency to 5. 2021-12-01 17:26:13 +01:00
DebaucheryLibrarian 87094a9498 Replaced cheerio with qu in Gamma scraper. 2021-11-29 02:49:07 +01:00
DebaucheryLibrarian e81864ebde Removed verbose http queued output. 2021-11-21 00:47:32 +01:00
DebaucheryLibrarian 2be3ea9bbb Explicitly adding headers to http module result. 2021-11-21 00:41:12 +01:00
DebaucheryLibrarian 98c103c625 Fixed uuid import. Changed fs rmdir to rm. 2021-11-21 00:31:09 +01:00
DebaucheryLibrarian f0b7678444 Fixed upsert failing on empty insert array due breaking Knex API change. 2021-11-21 00:19:10 +01:00
DebaucheryLibrarian 26539b74a5 Updated dependencies. Added periodic memory logger. 2021-11-20 23:59:48 +01:00
DebaucheryLibrarian 6c5d4389fe Not parsing HTML with jsdom when using http module directly to save memory. Added loading ellipsis to release grid pages. 2021-10-25 02:06:24 +02:00
DebaucheryLibrarian 0e4fd12d70 Added more loggers to http module. 2021-10-20 01:46:56 +02:00
DebaucheryLibrarian 522584711b Added Teen Mega World scraper. 2021-09-17 03:30:49 +02:00
DebaucheryLibrarian 7bd858f96b Generalized Adult Empire subsite scraper, added West Coast Productions. 2021-08-30 01:13:32 +02:00
DebaucheryLibrarian bed329cd8c Using bulk insert utility for alert notifications to prevent duplicate errors. 2021-06-04 03:10:41 +02:00
DebaucheryLibrarian c979173422 Rescraping upcoming scenes. Fixed language and scene deep scraping for Dorcel scraper. 2021-06-02 03:27:32 +02:00
DebaucheryLibrarian c386a9098f Generalized Team Skeet scraper, added MYLF network and various Team Skeet partner channels. 2021-03-29 22:22:56 +02:00
DebaucheryLibrarian 193af9bab5 Fixed session options in http module. 2021-03-23 15:25:21 +01:00
DebaucheryLibrarian 336b91c872 Refactored http timeout handling. 2021-03-17 02:09:34 +01:00
DebaucheryLibrarian 1703e9a541 Moved http timeout cancelation before pipeline to prevent large files from getting canceled. 2021-03-16 03:59:36 +01:00
DebaucheryLibrarian a0be8f0aa3 Accounting for missing options in http utility timeout function. 2021-03-15 04:13:09 +01:00
DebaucheryLibrarian cb447da7d0 Added harder timeouts to http utility. Split owner and query database users. 2021-03-15 04:11:14 +01:00
DebaucheryLibrarian a275d0c855 Smaller tag logo on small displays. Centering tag tile lazy image to prevent load jumps. Including SASS breakpoint variable file automatically. 2021-03-07 16:39:54 +01:00
DebaucheryLibrarian bb20659934 Refactored clips into chapters. 2021-02-27 00:37:22 +01:00
DebaucheryLibrarian f310fec869 Added S3 experiment. 2021-02-20 01:12:44 +01:00
DebaucheryLibrarian 7ff222ce25 Passing recursive parameters to all scraper methods. Using throttle parameters in MindGeek scraper, fixed missing slug breaking scene and actor URLs. 2021-02-10 03:23:48 +01:00
DebaucheryLibrarian 79b51eca67 Added BaDoink profile scraper. Improved convert wrapper. 2021-02-03 21:03:35 +01:00
DebaucheryLibrarian 6d93083581 Removed superfluous MindGeek scrapers. 2021-02-03 00:46:59 +01:00
DebaucheryLibrarian e3ef0a0d69 Added Top Web Models profile scraper. 2021-01-16 04:10:43 +01:00
DebaucheryLibrarian 12c3f0d7b9 Added Top Web Models framework. 2021-01-14 01:37:50 +01:00
DebaucheryLibrarian db4e74fb99 Fixed Nubiles base poster query, handling trailing commas in qu source set. Added profile scene scraper to Dogfart. Added tag photo. 2021-01-11 16:20:01 +01:00
DebaucheryLibrarian 8739ec08cf Fixed qu init selector. Fixed Aziani scene page scope. 2021-01-02 03:20:39 +01:00