boiii
/
traxxx
forked from DebaucheryLibrarian/traxxx
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,993 Commits 7 Branches 0 Tags 3.5 GiB
Commit Graph

815 Commits

Author SHA1 Message Date
DebaucheryLibrarian cde760c1ea Preventing Bang! scraper from redundant deep scraping when base release is available. 2022-02-12 16:22:00 +01:00
DebaucheryLibrarian c7710afdbf Fixed Bang! scraped failing when scene has no photos. 2022-02-12 03:26:40 +01:00
DebaucheryLibrarian 496c29e569 Configured Reality Kings to fetch session from RK scene overview. 2022-02-11 22:14:44 +01:00
DebaucheryLibrarian c4f0b48932 Fixed cover dimensions on movie tile to show lazy image, showing missing cover icon. 2022-02-09 00:19:54 +01:00
DebaucheryLibrarian f7bbab08bd Added Porn Pros main channel and working URLs for Flexible Positions, Public Violations and Squirt Disgrace. Added bisexual tag photos. 2022-02-08 23:59:44 +01:00
DebaucheryLibrarian 9b7879bff7 Removed channel ID from Gamma update query to prevent duplicate scenes. Added sexuality tags with covers to overview. 2022-02-08 00:19:34 +01:00
DebaucheryLibrarian 1dfa034332 Handling 'page not found' with 200 OK in Gamma scraper. 2022-02-07 22:29:09 +01:00
DebaucheryLibrarian 3d349c413f Fixed Adult Time scraper. 2022-02-07 22:16:43 +01:00
DebaucheryLibrarian fffe9080f3 Added stand-alone Spizoo sites. 2022-02-07 21:58:27 +01:00
DebaucheryLibrarian 4bcf7ef45b Storing associations if existing media are found. 2022-02-07 01:57:09 +01:00
DebaucheryLibrarian ceaf5a3217 Not storing shallow movie data when argument is disabled. 2022-02-05 17:14:34 +01:00
DebaucheryLibrarian 3d80e9d77b Added movie URL file support. Fixed Gamma movie API scraper movie URL. 2022-02-03 00:04:48 +01:00
DebaucheryLibrarian f1a7f2f905 Added Gamma movie API scraper. 2022-02-02 23:51:15 +01:00
DebaucheryLibrarian 10b8abf706 Updated Zero Tolerance channel to Zero Tolerance Films. 2022-02-02 22:25:04 +01:00
DebaucheryLibrarian b9f3eb85f9 Ignoring Gamma master categories in update scrape as these seem to contain irrelevant tags. 2022-02-01 21:18:54 +01:00
DebaucheryLibrarian 815d56d334 Preventing media associations when use-reference is unavailable. 2022-01-30 22:15:07 +01:00
DebaucheryLibrarian 22864105ac Writing media associations to disk for debugging. 2022-01-30 17:05:18 +01:00
DebaucheryLibrarian 52218c30f8 Not fetching mobile Gamma page when photos are not expected. 2022-01-29 16:38:36 +01:00
DebaucheryLibrarian 5c5a11fdca Added Bang Bros members page scraper (not for full scenes). 2022-01-29 01:21:41 +01:00
DebaucheryLibrarian 90bc2f3724 Changed release media association existence check, added log for details. 2022-01-28 04:05:29 +01:00
DebaucheryLibrarian 67d0a9e0e0 Fixed scene entity tag association. 2022-01-26 01:47:34 +01:00
DebaucheryLibrarian 452f725332 Switched poster URL preference in Gamma scraper to prevent cropped image on XEmpire. 2022-01-25 15:34:15 +01:00
DebaucheryLibrarian 7ceec1c739 Added indexes to scene tag and movie scene tables improve performance. Removed scene count from channel overview, too expensive. 2022-01-24 00:15:11 +01:00
DebaucheryLibrarian cd8f690ac6 Preferring network over channel in deep URL channel matching. 2022-01-23 02:46:42 +01:00
DebaucheryLibrarian c557f0f1f0 Improved Gamma image sources to prevent cropped images. 2022-01-22 23:20:08 +01:00
DebaucheryLibrarian 20da2d1cf6 Reusing batch ID for movies to preserve new-flag. 2022-01-20 00:54:10 +01:00
DebaucheryLibrarian 372db86927 Disabled MindGeek session bundling to analyze Too Many Requests errors. 2022-01-16 22:24:47 +01:00
DebaucheryLibrarian 57d46f5842 Fixed latest query in Gamma scraper. 2022-01-16 17:17:45 +01:00
DebaucheryLibrarian bbd788699e Fixed GraphQL query for scenes REST API. Added entry ID to scene search document. 2022-01-12 17:50:23 +01:00
DebaucheryLibrarian 506971b44b Chunked duplicate check to prevent postgres stack depth errors. 2022-01-10 02:17:17 +01:00
DebaucheryLibrarian baebdbc0bb Adding comment listing sister sites for non-exclusive Gamma API scenes. 2022-01-07 01:07:41 +01:00
DebaucheryLibrarian 6760c11881 Disabled why-is-node-running to assess OOM. 2021-12-31 01:45:03 +01:00
DebaucheryLibrarian 3b958a02ef Added memory profiler configuration options. 2021-12-27 22:36:01 +01:00
DebaucheryLibrarian 299f257a01 Logging memory usage on media fetch. 2021-12-27 17:29:23 +01:00
DebaucheryLibrarian 3bfe1e8fdb Storing scenes before fetching movies. 2021-12-20 03:29:14 +01:00
DebaucheryLibrarian 7d992f4129 Added triggers to memory dump. 2021-12-20 02:28:59 +01:00
DebaucheryLibrarian 280032937f Added centralized movie page fetcher. Added memory heap dump. 2021-12-20 02:22:10 +01:00
DebaucheryLibrarian dc187a9a3a Added execute method to qu, removed runScripts from Gamma's fetchMovie to observe effect on memory usage. 2021-12-19 23:03:44 +01:00
DebaucheryLibrarian 1c0c30be11 Improved and re-enabled why-is-node-running logger. 2021-12-18 23:44:51 +01:00
DebaucheryLibrarian 8e4be06675 Closing JSDOM window after deep scrapes in an attempt to save memory. 2021-12-18 23:22:35 +01:00
DebaucheryLibrarian 65e2b72c6a Fixed Gamma movie scraper failing when data object is not available. 2021-12-13 01:20:02 +01:00
DebaucheryLibrarian 5b5688438c Fixed Gamma movie scene query. Added tag photos. 2021-12-11 22:46:55 +01:00
DebaucheryLibrarian ec4e7bc12a Updated Jules Jordan scraper for Sperm Swallowers and The Ass Factory. 2021-12-05 23:59:17 +01:00
DebaucheryLibrarian 26d15c0a6f Updated Jules Jordan scraper for Manuel Ferrara and Girl Girl. 2021-12-05 23:43:23 +01:00
DebaucheryLibrarian e88554666c Reduced media concurrency to assess effect on memory. Moved qu context removal to deep scrape runner. Updated movie graphql queries. 2021-12-05 02:54:55 +01:00
DebaucheryLibrarian abd063a578 Removed window map from http module to prevent memory pile-up with update scraping. 2021-12-04 00:32:28 +01:00
DebaucheryLibrarian 1918dd4ad5 Removing query methods from XEmpire release. 2021-12-02 00:45:22 +01:00
DebaucheryLibrarian 61daf5f037 Added debug log for jsdom window closing. 2021-12-01 23:44:25 +01:00
DebaucheryLibrarian 60c151ff6a Reduced deep scrape concurrency to 1. 2021-12-01 23:39:09 +01:00
DebaucheryLibrarian e41f9fa937 Added waitImmediate to deep scrape, reduced concurrency. 2021-12-01 23:30:10 +01:00