Commit Graph

912 Commits

Author SHA1 Message Date
DebaucheryLibrarian f7bbab08bd Added Porn Pros main channel and working URLs for Flexible Positions, Public Violations and Squirt Disgrace. Added bisexual tag photos. 2022-02-08 23:59:44 +01:00
DebaucheryLibrarian 9b7879bff7 Removed channel ID from Gamma update query to prevent duplicate scenes. Added sexuality tags with covers to overview. 2022-02-08 00:19:34 +01:00
DebaucheryLibrarian 1dfa034332 Handling 'page not found' with 200 OK in Gamma scraper. 2022-02-07 22:29:09 +01:00
DebaucheryLibrarian 3d349c413f Fixed Adult Time scraper. 2022-02-07 22:16:43 +01:00
DebaucheryLibrarian fffe9080f3 Added stand-alone Spizoo sites. 2022-02-07 21:58:27 +01:00
DebaucheryLibrarian 4bcf7ef45b Storing associations if existing media are found. 2022-02-07 01:57:09 +01:00
DebaucheryLibrarian ceaf5a3217 Not storing shallow movie data when argument is disabled. 2022-02-05 17:14:34 +01:00
DebaucheryLibrarian 3d80e9d77b Added movie URL file support. Fixed Gamma movie API scraper movie URL. 2022-02-03 00:04:48 +01:00
DebaucheryLibrarian f1a7f2f905 Added Gamma movie API scraper. 2022-02-02 23:51:15 +01:00
DebaucheryLibrarian 10b8abf706 Updated Zero Tolerance channel to Zero Tolerance Films. 2022-02-02 22:25:04 +01:00
DebaucheryLibrarian b9f3eb85f9 Ignoring Gamma master categories in update scrape as these seem to contain irrelevant tags. 2022-02-01 21:18:54 +01:00
DebaucheryLibrarian 815d56d334 Preventing media associations when use-reference is unavailable. 2022-01-30 22:15:07 +01:00
DebaucheryLibrarian 22864105ac Writing media associations to disk for debugging. 2022-01-30 17:05:18 +01:00
DebaucheryLibrarian 52218c30f8 Not fetching mobile Gamma page when photos are not expected. 2022-01-29 16:38:36 +01:00
DebaucheryLibrarian 5c5a11fdca Added Bang Bros members page scraper (not for full scenes). 2022-01-29 01:21:41 +01:00
DebaucheryLibrarian 90bc2f3724 Changed release media association existence check, added log for details. 2022-01-28 04:05:29 +01:00
DebaucheryLibrarian 67d0a9e0e0 Fixed scene entity tag association. 2022-01-26 01:47:34 +01:00
DebaucheryLibrarian 452f725332 Switched poster URL preference in Gamma scraper to prevent cropped image on XEmpire. 2022-01-25 15:34:15 +01:00
DebaucheryLibrarian 7ceec1c739 Added indexes to scene tag and movie scene tables improve performance. Removed scene count from channel overview, too expensive. 2022-01-24 00:15:11 +01:00
DebaucheryLibrarian cd8f690ac6 Preferring network over channel in deep URL channel matching. 2022-01-23 02:46:42 +01:00
DebaucheryLibrarian c557f0f1f0 Improved Gamma image sources to prevent cropped images. 2022-01-22 23:20:08 +01:00
DebaucheryLibrarian 20da2d1cf6 Reusing batch ID for movies to preserve new-flag. 2022-01-20 00:54:10 +01:00
DebaucheryLibrarian 372db86927 Disabled MindGeek session bundling to analyze Too Many Requests errors. 2022-01-16 22:24:47 +01:00
DebaucheryLibrarian 57d46f5842 Fixed latest query in Gamma scraper. 2022-01-16 17:17:45 +01:00
DebaucheryLibrarian bbd788699e Fixed GraphQL query for scenes REST API. Added entry ID to scene search document. 2022-01-12 17:50:23 +01:00
DebaucheryLibrarian 506971b44b Chunked duplicate check to prevent postgres stack depth errors. 2022-01-10 02:17:17 +01:00
DebaucheryLibrarian baebdbc0bb Adding comment listing sister sites for non-exclusive Gamma API scenes. 2022-01-07 01:07:41 +01:00
DebaucheryLibrarian 6760c11881 Disabled why-is-node-running to assess OOM. 2021-12-31 01:45:03 +01:00
DebaucheryLibrarian 3b958a02ef Added memory profiler configuration options. 2021-12-27 22:36:01 +01:00
DebaucheryLibrarian 299f257a01 Logging memory usage on media fetch. 2021-12-27 17:29:23 +01:00
DebaucheryLibrarian 3bfe1e8fdb Storing scenes before fetching movies. 2021-12-20 03:29:14 +01:00
DebaucheryLibrarian 7d992f4129 Added triggers to memory dump. 2021-12-20 02:28:59 +01:00
DebaucheryLibrarian 280032937f Added centralized movie page fetcher. Added memory heap dump. 2021-12-20 02:22:10 +01:00
DebaucheryLibrarian dc187a9a3a Added execute method to qu, removed runScripts from Gamma's fetchMovie to observe effect on memory usage. 2021-12-19 23:03:44 +01:00
DebaucheryLibrarian 1c0c30be11 Improved and re-enabled why-is-node-running logger. 2021-12-18 23:44:51 +01:00
DebaucheryLibrarian 8e4be06675 Closing JSDOM window after deep scrapes in an attempt to save memory. 2021-12-18 23:22:35 +01:00
DebaucheryLibrarian 65e2b72c6a Fixed Gamma movie scraper failing when data object is not available. 2021-12-13 01:20:02 +01:00
DebaucheryLibrarian 5b5688438c Fixed Gamma movie scene query. Added tag photos. 2021-12-11 22:46:55 +01:00
DebaucheryLibrarian ec4e7bc12a Updated Jules Jordan scraper for Sperm Swallowers and The Ass Factory. 2021-12-05 23:59:17 +01:00
DebaucheryLibrarian 26d15c0a6f Updated Jules Jordan scraper for Manuel Ferrara and Girl Girl. 2021-12-05 23:43:23 +01:00
DebaucheryLibrarian e88554666c Reduced media concurrency to assess effect on memory. Moved qu context removal to deep scrape runner. Updated movie graphql queries. 2021-12-05 02:54:55 +01:00
DebaucheryLibrarian abd063a578 Removed window map from http module to prevent memory pile-up with update scraping. 2021-12-04 00:32:28 +01:00
DebaucheryLibrarian 1918dd4ad5 Removing query methods from XEmpire release. 2021-12-02 00:45:22 +01:00
DebaucheryLibrarian 61daf5f037 Added debug log for jsdom window closing. 2021-12-01 23:44:25 +01:00
DebaucheryLibrarian 60c151ff6a Reduced deep scrape concurrency to 1. 2021-12-01 23:39:09 +01:00
DebaucheryLibrarian e41f9fa937 Added waitImmediate to deep scrape, reduced concurrency. 2021-12-01 23:30:10 +01:00
DebaucheryLibrarian e29cbc9fea Closing JSDOM window after deep scrape in an attempt to save memory. Reduced deep scrape concurrency to 5. 2021-12-01 17:26:13 +01:00
DebaucheryLibrarian b93a5715cb Updated Jules Jordan scraper for new update page layout. 2021-12-01 00:00:24 +01:00
DebaucheryLibrarian 87094a9498 Replaced cheerio with qu in Gamma scraper. 2021-11-29 02:49:07 +01:00
DebaucheryLibrarian f04eb72891 Added upcoming scraper to PurgatoryX. 2021-11-28 01:20:39 +01:00
DebaucheryLibrarian 9d7183ac69 Added PurgatoryX scraper. 2021-11-27 23:55:16 +01:00
DebaucheryLibrarian 20d0d860d3 Fixed MindGeek scraper trying to acquire session from mindgeek.com 2021-11-22 02:51:52 +01:00
DebaucheryLibrarian 6b4aa64d74 Improved MindGeek scraper session check to prevent crash when network session isn't available yet. 2021-11-22 02:44:03 +01:00
DebaucheryLibrarian e81864ebde Removed verbose http queued output. 2021-11-21 00:47:32 +01:00
DebaucheryLibrarian 0305a22a24 Removed stray console log from media module. 2021-11-21 00:43:35 +01:00
DebaucheryLibrarian 2be3ea9bbb Explicitly adding headers to http module result. 2021-11-21 00:41:12 +01:00
DebaucheryLibrarian 98c103c625 Fixed uuid import. Changed fs rmdir to rm. 2021-11-21 00:31:09 +01:00
DebaucheryLibrarian f0b7678444 Fixed upsert failing on empty insert array due breaking Knex API change. 2021-11-21 00:19:10 +01:00
DebaucheryLibrarian 26539b74a5 Updated dependencies. Added periodic memory logger. 2021-11-20 23:59:48 +01:00
DebaucheryLibrarian ccb99e278c Added periodic memory logger. 2021-11-20 23:59:15 +01:00
DebaucheryLibrarian 29b8c5e38e Including unextracted scenes in date determination. 2021-10-28 02:10:30 +02:00
DebaucheryLibrarian 0864154a0e Added unextracted property to keep paginating when extracting scenes. 2021-10-28 01:59:53 +02:00
DebaucheryLibrarian a22c4d5679 Added beforeNetwork hook, used by MindGeek. Added Filthy Kings to Gamma. 2021-10-27 17:19:23 +02:00
DebaucheryLibrarian 100a35b4e8 Added before scene fetch method to prevent e.g. unnecessary session requests, moved scraper assignment to entity lookup. Removed channel URL hostname matching.. 2021-10-26 23:42:32 +02:00
DebaucheryLibrarian 6c5d4389fe Not parsing HTML with jsdom when using http module directly to save memory. Added loading ellipsis to release grid pages. 2021-10-25 02:06:24 +02:00
DebaucheryLibrarian 0e4fd12d70 Added more loggers to http module. 2021-10-20 01:46:56 +02:00
DebaucheryLibrarian 972b15e72d Added memory profiling. 2021-10-17 23:56:49 +02:00
DebaucheryLibrarian 49f891ba44 Ignoring 1-second scene duration from MindGeek API. 2021-10-17 19:59:05 +02:00
DebaucheryLibrarian 167df35d37 Fixed tags module not dealing with empty releases or tags, added origin tag column to chapter tag table. 2021-10-17 00:28:13 +02:00
DebaucheryLibrarian b6ad2903f3 Storing original tags. 2021-10-10 00:04:21 +02:00
DebaucheryLibrarian dbff3e9539 Added Brad Montana. Added teaser link to trailer video. 2021-09-28 20:45:22 +02:00
DebaucheryLibrarian 522584711b Added Teen Mega World scraper. 2021-09-17 03:30:49 +02:00
DebaucheryLibrarian a316da3083 Retrieving Spizoo trailers. 2021-09-14 00:42:51 +02:00
DebaucheryLibrarian b164c5dad7 Added Spizoo. 2021-09-13 01:29:39 +02:00
DebaucheryLibrarian d542889827 Added sections and pagination to stash page. 2021-09-12 00:05:45 +02:00
DebaucheryLibrarian 7bd858f96b Generalized Adult Empire subsite scraper, added West Coast Productions. 2021-08-30 01:13:32 +02:00
DebaucheryLibrarian 69a7e8f13d Fixed PornDoe scraper. 2021-08-26 01:14:54 +02:00
DebaucheryLibrarian 23537e0243 Removed Gamma console logs. 2021-08-24 02:56:59 +02:00
DebaucheryLibrarian 245f69a122 Fixed movie bindings. 2021-08-24 02:54:40 +02:00
DebaucheryLibrarian bb384029ac Disabled eager searching except for entities. Updating movie search document on store. 2021-08-23 01:54:22 +02:00
DebaucheryLibrarian 6a8c9d89cb Using paginated full text search for movies, combined actor search and fetch to allow combining search with filters. 2021-08-22 22:25:20 +02:00
DebaucheryLibrarian f00e37490c Finished Cum Louder scraper, updated Vixen scraper. Added tag posters. 2021-08-17 19:25:10 +02:00
DebaucheryLibrarian b08cb46ae5 Removed stray console log. 2021-08-15 13:21:02 +02:00
DebaucheryLibrarian aacfd1b29d Updated object-merge-advanced API use. 2021-08-15 13:16:48 +02:00
DebaucheryLibrarian a848d6991b Added Diabolic and Cum Louder, added content type expect option to media sources to fix Vixen thumbnails. 2021-08-09 10:31:12 +02:00
DebaucheryLibrarian 6847ef690c Added Arch Angel, updated BAM Visions scraper to accomodate Arch Angel (different network, same unidentified CMS). 2021-07-06 00:01:44 +02:00
DebaucheryLibrarian 23b41fc4f3 Fixed Bang scraper. Added Kink affiliate, tag photos. 2021-07-05 00:06:18 +02:00
DebaucheryLibrarian eb7009832a Added rudimentary affiliate banner setup. Separated login and signup disable. Added various tag photos. 2021-06-28 00:05:24 +02:00
DebaucheryLibrarian d1480da076 Added effective date column. Changed warning page theme. 2021-06-19 18:09:58 +02:00
DebaucheryLibrarian ab1329dd67 Updating entity ID for rescraped scenes with network entry IDs enabled. 2021-06-13 16:49:27 +02:00
DebaucheryLibrarian e9a0700742 Improved upcoming update query. 2021-06-06 01:09:32 +02:00
DebaucheryLibrarian 3f473589ad Using bulk insert utility for alert notifications to prevent duplicate errors (fixed). 2021-06-04 03:22:40 +02:00
DebaucheryLibrarian bed329cd8c Using bulk insert utility for alert notifications to prevent duplicate errors. 2021-06-04 03:10:41 +02:00
DebaucheryLibrarian c979173422 Rescraping upcoming scenes. Fixed language and scene deep scraping for Dorcel scraper. 2021-06-02 03:27:32 +02:00
DebaucheryLibrarian 846b860c06 Hiding scene photos and trailers from guests. 2021-05-15 02:51:52 +02:00
DebaucheryLibrarian 3f55b90ab8 Adding alerted scene to stashes. 2021-04-29 01:45:01 +02:00
DebaucheryLibrarian 8bf9fff7dc Triggering notifications for children of alert entities. Showing icons in alert entity search to distinguish networks and channels. 2021-04-26 00:48:31 +02:00
DebaucheryLibrarian eed563e06f Updating video player when switching scene page. 2021-04-25 04:20:38 +02:00
DebaucheryLibrarian fc1c2fc2f3 Added notification clear, improved notification styling. 2021-04-25 03:08:50 +02:00
DebaucheryLibrarian f8a3bf6a64 Updated scene URLs in Vixen scraper. 2021-04-22 19:49:11 +02:00
DebaucheryLibrarian c5e74c33b7 Improved alert notifications. 2021-04-22 19:44:23 +02:00
DebaucheryLibrarian 95f3b1c03a Added rudimentary notifications for set alerts. 2021-04-17 01:10:45 +02:00
DebaucheryLibrarian 7f25846d55 List alerts in profile 2021-04-05 00:48:03 +02:00
DebaucheryLibrarian d36e52d5d1 Added row level security to alert tables. Added alerts to user query. 2021-04-04 22:52:54 +02:00
DebaucheryLibrarian da0cbced15 Added alert dialog. Fixed image rotation EXIT data being discarded. 2021-04-04 21:52:19 +02:00
DebaucheryLibrarian 010da8954b Fixed MYLF scraper failing when channel is missing, fixed profile measurement matching. Added MYLF Selects channel. 2021-03-29 22:47:43 +02:00
DebaucheryLibrarian c386a9098f Generalized Team Skeet scraper, added MYLF network and various Team Skeet partner channels. 2021-03-29 22:22:56 +02:00
DebaucheryLibrarian 3961f83ef6 Fixed animated poster being overwritten as jpeg. 2021-03-26 01:47:40 +01:00
DebaucheryLibrarian 193af9bab5 Fixed session options in http module. 2021-03-23 15:25:21 +01:00
DebaucheryLibrarian c70f500acc Fixed stashed check breaking on empty stashes in PostGraphile plugins. 2021-03-21 04:34:04 +01:00
DebaucheryLibrarian 9ff70e5578 Separated full heart button into component. 2021-03-21 03:46:59 +01:00
DebaucheryLibrarian 348aa91832 Added stash menu to release page, returning stashes from stash API to avoid reloading or local interpolation. 2021-03-21 03:23:58 +01:00
DebaucheryLibrarian eee47111a6 Added delete stash icons and dialog. 2021-03-20 23:03:13 +01:00
DebaucheryLibrarian 07643870cd Updating stash button locally on actor and scene page. 2021-03-20 18:12:06 +01:00
DebaucheryLibrarian e88cf4e3f4 Separated user page stash component. 2021-03-20 02:49:17 +01:00
DebaucheryLibrarian 4bc6ff846d Added public visibility toggle to user page stashes. 2021-03-20 02:03:30 +01:00
DebaucheryLibrarian e301e2184c Fixed undefined user ID in postgres function. Fixed and improved mobile alignment for new and stash icons on scene tile. 2021-03-20 00:12:12 +01:00
DebaucheryLibrarian 731a2792c5 Added favorite stash heart to scene tiles. 2021-03-19 03:27:48 +01:00
DebaucheryLibrarian 862a29bb6e Using thumbnail size instead of original photo size in image tags. 2021-03-18 04:36:04 +01:00
DebaucheryLibrarian 5a2e93e900 Added various tag photos and descriptions. 2021-03-17 05:11:17 +01:00
DebaucheryLibrarian 336b91c872 Refactored http timeout handling. 2021-03-17 02:09:34 +01:00
DebaucheryLibrarian 0d7a03f3e5 Allowing auth to be disabled in config. 2021-03-16 04:12:05 +01:00
DebaucheryLibrarian 1703e9a541 Moved http timeout cancelation before pipeline to prevent large files from getting canceled. 2021-03-16 03:59:36 +01:00
DebaucheryLibrarian 1fb7d384fb Merge branch 'master' into experimental 2021-03-15 04:16:36 +01:00
DebaucheryLibrarian 7c7b38e869 Inserting user ID to PostGraphile if available. 2021-03-15 04:16:32 +01:00
DebaucheryLibrarian a0be8f0aa3 Accounting for missing options in http utility timeout function. 2021-03-15 04:13:09 +01:00
DebaucheryLibrarian cb447da7d0 Added harder timeouts to http utility. Split owner and query database users. 2021-03-15 04:11:14 +01:00
DebaucheryLibrarian 77b40817f2 Added favorites button to actor page. 2021-03-15 03:30:47 +01:00
DebaucheryLibrarian e371e9725a Added stashes with experimental row security policies. Added tag photos. 2021-03-14 04:54:53 +01:00
DebaucheryLibrarian 816529b0ca Added user sign up and login. 2021-03-13 04:26:24 +01:00
DebaucheryLibrarian 00c06778ef Only scraping profile source matching actor entity, changed avatar entropy cut-off from 6 to 5.5. 2021-03-11 04:16:59 +01:00
DebaucheryLibrarian a71752b18b Associating directors separately from actors. Added tag photos. 2021-03-08 01:06:57 +01:00
DebaucheryLibrarian de460f53b1 Changed album close behavior so album can be closed when visiting URL directly. 2021-03-07 16:54:20 +01:00
DebaucheryLibrarian a275d0c855 Smaller tag logo on small displays. Centering tag tile lazy image to prevent load jumps. Including SASS breakpoint variable file automatically. 2021-03-07 16:39:54 +01:00
DebaucheryLibrarian 44bb9b33d9 Added intermittent process report to debug freezes. Removed original images. 2021-03-07 02:09:37 +01:00
DebaucheryLibrarian 17e6f5a5da Storing directors. 2021-03-07 00:01:02 +01:00
DebaucheryLibrarian 254e933740 Added permanent filter for invalid actor associations. 2021-03-04 00:31:31 +01:00
DebaucheryLibrarian 780993eb63 Added debug log for actor associations. 2021-03-03 13:56:50 +01:00
DebaucheryLibrarian c2ec4c15e3 Added basic filter for actor boob size. 2021-03-01 02:41:53 +01:00
DebaucheryLibrarian eca24a7c65 Added WP boob filter to actors page. 2021-02-28 03:38:54 +01:00
DebaucheryLibrarian 8eebcae85d Using direct parent as release actor entity instead of highest parent. 2021-02-27 21:59:33 +01:00
DebaucheryLibrarian c2a008afbe Added mimetype check to teasers and trailers. Added chapters to MindGeek scraper, fixed scene ID extraction getting stuck on numbers in domain name. Ordering chapters by timestamp. 2021-02-27 18:05:06 +01:00
DebaucheryLibrarian a45c5f8f37 Added tags to chapters in API. 2021-02-27 17:19:07 +01:00
DebaucheryLibrarian e55bc1ef33 Re-added trailer to scene API query. 2021-02-27 04:05:38 +01:00
DebaucheryLibrarian e60153ad01 Curating chapter media. 2021-02-27 03:57:37 +01:00
DebaucheryLibrarian 2deed3a7eb Added internal GraphQL client, using GraphQL for scenes API. 2021-02-27 03:52:27 +01:00
DebaucheryLibrarian bb20659934 Refactored clips into chapters. 2021-02-27 00:37:22 +01:00
DebaucheryLibrarian 0eba0461c9 Added error handler to web server. 2021-02-26 19:39:48 +01:00
DebaucheryLibrarian f018735052 Added relevance filter to search REST API. 2021-02-26 17:29:02 +01:00
DebaucheryLibrarian 4ca6c37cc8 Added relevance to REST release search API, sorting by relevance rank. Improved search result table column naming. 2021-02-26 17:22:54 +01:00