boiii
/
traxxx
forked from DebaucheryLibrarian/traxxx
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,503 Commits 8 Branches 0 Tags 4.7 GiB
Commit Graph

661 Commits

Author SHA1 Message Date
DebaucheryLibrarian af67d733ad Added profile flush. 2020-12-30 02:23:43 +01:00
DebaucheryLibrarian 770e5b75a5 Added profile interpolate command line argument. 2020-12-30 00:16:05 +01:00
DebaucheryLibrarian a8f68f4993 Fixed boolean handling in actor profile curation. 2020-12-29 23:44:38 +01:00
DebaucheryLibrarian 361e8f1bd3 Fixed Kelly Madison title regex. 2020-12-29 18:53:55 +01:00
DebaucheryLibrarian 0f052a0631 Fixed actor boolean logic, addressing missing boob, tattoo and piercing info. Removed scroll background, fixed actor photo load event and padding. 2020-12-29 04:20:33 +01:00
DebaucheryLibrarian bc944c2373 Restored Mike Adriano scraper. 2020-12-29 02:05:22 +01:00
DebaucheryLibrarian 8dd10f7e77 Adjusting tooltip arrow position, added open and close events. Fixed search tooltip layout. 2020-12-29 00:42:02 +01:00
DebaucheryLibrarian 6baa6b0802 Fixed fallback create dates in scene tiles. Fixed Mike Adriano entryIds and trailers for Nympho. 2020-12-28 01:29:34 +01:00
DebaucheryLibrarian 77ec2d3747 Using advanced merge library to maximize scraped data. 2020-12-27 23:43:08 +01:00
DebaucheryLibrarian 9c926a1d81 Fixed Mike Adriano dates for scene pages. 2020-12-27 23:36:15 +01:00
DebaucheryLibrarian ded414577f Fixed default actor avatar allocation. Fixed lazy loading in actor photos component. 2020-12-27 22:45:38 +01:00
DebaucheryLibrarian 27e5583849 Using generic session ID variable for to determine consent warning, rather than dedicated property. 2020-12-20 19:49:57 +01:00
DebaucheryLibrarian 5f4039c5d4 Added sharpness and re-added entropy to avatars, ignoring low-entropy photos as main avatar and in profile photo list. 2020-12-20 04:21:28 +01:00
DebaucheryLibrarian bd77d4347d Tied consent warning to session. 2020-12-19 23:01:17 +01:00
DebaucheryLibrarian be4d025505 Fixed trailer path regex for some Hush sites. 2020-12-19 02:35:41 +01:00
DebaucheryLibrarian 67ed249239 Added poster to API scene overview. 2020-12-19 01:03:15 +01:00
DebaucheryLibrarian 0d41fb48dc Removed outdated movie releases from API. Added API endpoint for scene posters. 2020-12-19 00:40:36 +01:00
DebaucheryLibrarian 4ccd8bf07a Fixed missing initial value in scraper reduce, breaking first network (21 Naturals). 2020-12-19 00:19:38 +01:00
DebaucheryLibrarian d67dca60fc Fixed birthdate interpolation and query. 2020-12-17 03:43:09 +01:00
DebaucheryLibrarian cd8e810c35 Fixed various Kelly Madison scraper issues. 2020-12-17 02:05:01 +01:00
DebaucheryLibrarian cc83b832f1 Attach base actor to scenes scraped from profile. 2020-12-16 00:50:58 +01:00
DebaucheryLibrarian 07f0249717 Upgraded pg, knex and postgraphile. Updated nvmrc version to latest node LTS. Fixed Gamma scraper not passing request headers to new http module. 2020-12-14 03:16:47 +01:00
DebaucheryLibrarian 71196688ae Removed console log. 2020-12-05 02:44:39 +01:00
DebaucheryLibrarian f0bec85ef8 Added Fuck'n'Drive and Jizz On Teens latest layouts to First Anal Quest scraper. 2020-12-05 02:24:31 +01:00
DebaucheryLibrarian 2e0fba3de9 Added First Anal Quest and Double View Casting profile scrapers. 2020-12-04 23:53:20 +01:00
DebaucheryLibrarian be1821b9eb Changed --inspect to --report to avoid conflict with Node's own debug tools. 2020-12-02 21:26:55 +01:00
DebaucheryLibrarian 2656e3adb0 Added First Anal Quest and Double View Casting latest and scene scraper. 2020-12-02 03:17:32 +01:00
DebaucheryLibrarian 6f19a8a642 Using ageFromBirth for age on scene date. Using infinite depth and colors for inspect. 2020-12-01 23:00:14 +01:00
DebaucheryLibrarian 9a61d2305c Added fixed actor age. Added male profiles to Littlr Caprice Dreams scraper. Added various tag photos. 2020-11-29 03:59:47 +01:00
DebaucheryLibrarian 71c884fe48 Improved Little Caprice Dreams scraper. 2020-11-28 00:46:30 +01:00
DebaucheryLibrarian 9a183c7ffb Added channel matching to Little Caprice Dreams. 2020-11-27 03:23:12 +01:00
DebaucheryLibrarian 60485751e2 Added support for upcoming scenes to Bang scraper. 2020-11-26 22:07:54 +01:00
DebaucheryLibrarian 4e559f63e3 Including all children of included networks, separated included children into dedicated property. 2020-11-26 04:26:52 +01:00
DebaucheryLibrarian 1b407254a7 Increased hard limit in release API. 2020-11-26 04:01:01 +01:00
DebaucheryLibrarian 0b86def315 Ignoring non-scene page (possible CF or similar protection) in Vixen scraper. 2020-11-26 03:27:21 +01:00
DebaucheryLibrarian 6633ce78d0 Returning empty array instead of null from empty Vixen page. 2020-11-26 03:14:32 +01:00
DebaucheryLibrarian 54df9d0c78 Fixed empty page breaking Vixen scraper. 2020-11-26 03:13:43 +01:00
DebaucheryLibrarian 980efbc93d Added series as channels with logos and photo album scraping to Little Caprice. Added various tag photos. 2020-11-24 04:29:44 +01:00
DebaucheryLibrarian 711a9441a6 Added Little Caprice Dreams scraped (WIP). 2020-11-23 04:32:56 +01:00
DebaucheryLibrarian 8688c28d0f Replaced queueMethod in media sources for new interval/concurrency options. 2020-11-23 00:38:22 +01:00
DebaucheryLibrarian ca7c8f0afd Updated place resolve module to use new HTTP module. 2020-11-23 00:30:20 +01:00
DebaucheryLibrarian 0633197793 Removed direct bhttp usage from scrapers in favor of local http module. Deleted legacy scrapers, as old code is available via git repo history. 2020-11-23 00:05:02 +01:00
DebaucheryLibrarian 3d427f7e1d Allowing HTTP rate limits to be set by configuration or argument. 2020-11-22 23:50:24 +01:00
DebaucheryLibrarian 6a5063cf32 Fixed PornCZ scene photos attribute. 2020-11-22 04:13:21 +01:00
DebaucheryLibrarian 081a5a1e8c Updated HTTP call in Gamma scraper. 2020-11-22 04:09:44 +01:00
DebaucheryLibrarian b9b777c621 Using new HTTP module with a dynamic rate limiter. 2020-11-22 04:07:09 +01:00
DebaucheryLibrarian 77f9193669 Updated Dorcel scraper, added movie support. 2020-11-19 02:01:13 +01:00
DebaucheryLibrarian d14ef90136 Scraping actor scenes from Hussie Pass. Adding entity to actor base releases. 2020-11-15 23:50:04 +01:00
DebaucheryLibrarian df9a6eac05 Updated Hussie Pass scraper for new site design. Added cock size to profiles. 2020-11-15 04:33:24 +01:00
DebaucheryLibrarian b3a5d7f379 Fixed JayRock status response, 2020-11-13 01:23:06 +01:00
DebaucheryLibrarian 8a22ff07a6 Merged legacy JayRock scraper into new scraper for CosPimps. 2020-11-13 01:16:17 +01:00
DebaucheryLibrarian 2063d66550 Ignoring placeholder avatar in Jay Rock scraper. 2020-11-10 22:44:15 +01:00
DebaucheryLibrarian a3c18ca577 Rewrote Jay Rock scraper for new website. 2020-11-04 04:02:51 +01:00
DebaucheryLibrarian 8586817963 Fixed Insex html table selector. 2020-11-01 03:25:30 +01:00
DebaucheryLibrarian c1fffe5cdb Added alternative layout to Insex for updated Topgrl and Sexually Broken sites. 2020-11-01 02:58:21 +01:00
DebaucheryLibrarian 3855d96135 Removed 'null' from scene tile URL if no slug is available. 2020-10-30 17:51:47 +01:00
DebaucheryLibrarian 39f8c037a5 Replaced bhttp with patched fork. Improved Jesse Loads Monster Facials scraper reliability (WIP). Added various tag photos. 2020-10-30 17:37:10 +01:00
DebaucheryLibrarian c37d4ad01f Filtering invalid actors from releases before storing. 2020-10-29 16:06:20 +01:00
DebaucheryLibrarian b188bc5744 Filtering out empty or unidentified scenes from update scraper, with warning. Improved Jesse Loads Monster Facials reliability. 2020-10-29 15:20:59 +01:00
DebaucheryLibrarian 4469376dd2 Using temporary table instead of WHERE IN to stack depth error when finding duplicate actors. 2020-10-28 03:50:52 +01:00
DebaucheryLibrarian bf9b334b73 Adding scraper config by scraper slug to current 'includes' parameter. 2020-10-28 01:36:13 +01:00
DebaucheryLibrarian 1869877178 Added release covers table to flush routine. 2020-10-27 02:40:30 +01:00
DebaucheryLibrarian 7257776ba8 Mapping avatar to media ID in orphan delete. 2020-10-27 02:34:06 +01:00
DebaucheryLibrarian 1d3ec96e8d Detecting profile avatars in orphaned media flush. 2020-10-27 02:09:52 +01:00
DebaucheryLibrarian 99a7bfeb4c Added traxxx dummy network. 2020-10-26 01:42:38 +01:00
DebaucheryLibrarian 0027be65eb Fixed CzechAV to accomodate teasers instead of photos. Fixed error trying to flush non-existent media file. 2020-10-25 21:43:36 +01:00
DebaucheryLibrarian 57a44d6643 Expanded README. 2020-10-25 01:29:25 +02:00
DebaucheryLibrarian 0bd7fca876 Added orphaned media flush and batch release flush. 2020-10-25 00:52:40 +02:00
DebaucheryLibrarian 6791053c83 Fixed entity alias available through wrong type endpoint. 2020-10-20 21:04:29 +02:00
DebaucheryLibrarian 9f3c686913 Added thumbnail and favicon to entity REST API. 2020-10-20 15:37:42 +02:00
DebaucheryLibrarian ecdd6d8fb0 Added logo path to entity API. 2020-10-20 15:28:58 +02:00
DebaucheryLibrarian 60eb599416 Added alias to entity search query. 2020-10-20 00:25:32 +02:00
DebaucheryLibrarian 6b17f9d1f2 Allowing entity to be fetched by alias. 2020-10-20 00:21:15 +02:00
DebaucheryLibrarian d795266114 Removed no-date-limit as argument due yargs conflict. 2020-10-20 00:08:14 +02:00
DebaucheryLibrarian 2f8fca0327 Added missing-date as config and argument alias. 2020-10-20 00:05:23 +02:00
DebaucheryLibrarian 4a900cbbeb Renamed nullDateLimit to noDateLimit in config. 2020-10-20 00:03:22 +02:00
DebaucheryLibrarian 5bdbb5ec62 Renamed null-date-limit to no-date-limit, added old as alias. 2020-10-20 00:01:29 +02:00
DebaucheryLibrarian 593ce27312 Added rudimentary scene and entity scene remove. 2020-10-19 02:02:21 +02:00
DebaucheryLibrarian 2536405dba Fixed entity API database query. 2020-10-18 00:01:34 +02:00
DebaucheryLibrarian ca22aedaaa Added rudimentary API documentation to README. 2020-10-17 22:54:00 +02:00
DebaucheryLibrarian e6c52002f0 Added tags and entities to REST API.. 2020-10-16 23:00:03 +02:00
DebaucheryLibrarian f38233053e Removed debug log. 2020-10-15 01:45:13 +02:00
DebaucheryLibrarian 99a4751c20 Returning results from new pagination. 2020-10-14 03:17:03 +02:00
DebaucheryLibrarian 7c856c267d Added revised next page determination. 2020-10-12 04:08:22 +02:00
DebaucheryLibrarian 8aefb8eddb Added and updated tag photos. 2020-10-07 03:40:19 +02:00
DebaucheryLibrarian 3f843cc0fc Added version to stats page. 2020-09-25 21:21:26 +02:00
DebaucheryLibrarian 566c20ea7e Added various tag photos. Renamed some toy tags. 2020-09-21 05:11:24 +02:00
DebaucheryLibrarian e78bfe4c22 Upgraded knex and pg versions. 2020-09-19 00:12:15 +02:00
DebaucheryLibrarian c94e1aaea9 Handling missing trailers in Kink scraper. 2020-09-18 23:25:15 +02:00
DebaucheryLibrarian 3789ef51f2 (Temporarily) removed studio filter from entity query for performance reasons. 2020-09-18 22:43:45 +02:00
DebaucheryLibrarian 3c9468b0f1 Fixed wrong MindGeek session acquire URL. 2020-09-18 03:27:00 +02:00
DebaucheryLibrarian a4929819df Using channel URL instead of composed URL for session retrieval, should fix Brazzers. 2020-09-18 02:54:05 +02:00
DebaucheryLibrarian a9fa71e455 Fixed predata parameter in Assylum scraper. 2020-09-17 14:49:45 +02:00
DebaucheryLibrarian 1a8de4fcf6 Added mimetype verification option to media source to ensure server returned a plausible file. Added additional fallbacks to Jules Jordan poster scraper for Amateur Allure. 2020-09-17 02:30:58 +02:00
DebaucheryLibrarian 0190ee9531 Fixed Porn Doe's poster query. Checking style attribute existence in qu before attempting to use it. 2020-09-17 00:35:41 +02:00
DebaucheryLibrarian 6fb15fb591 Added Dorcel Club with scene and actor scraping. Added count method to qu. 2020-09-16 04:55:30 +02:00
DebaucheryLibrarian a8c525f4fc Hard-coded Pascal White as the male actor for Pascal's Sub Sluts. Added female gender to all Sub Sluts. 2020-09-16 02:38:59 +02:00
DebaucheryLibrarian 7c4dd03a8c Ignoring 'Lockdown Submissions' as actor for Pascals Subsluts. 2020-09-16 01:45:58 +02:00
DebaucheryLibrarian 7fd7005776 Added profile scraper to Pascals Subsluts. 2020-09-16 01:42:15 +02:00
DebaucheryLibrarian bc34c6edb4 Added Bang Bros Vault channel. 2020-09-14 16:01:27 +02:00
DebaucheryLibrarian ba7419d3b0 Added basic Pascals Subsluts scraper. 2020-09-14 02:40:27 +02:00
DebaucheryLibrarian beeaebbfb7 Added CzechAV. 2020-09-14 00:53:41 +02:00
DebaucheryLibrarian b57b0a38f5 Fixed infinite pagination when scraping upcoming scenes. 2020-09-13 01:41:52 +02:00
DebaucheryLibrarian eb6337f6fb Fixed upcoming Jules Jordan scene without teaser breaking scraper. 2020-09-11 02:36:36 +02:00
DebaucheryLibrarian 0e8024adf1 Changed next page determination to ensure --after is followed even if there are no unique releases. 2020-09-10 23:49:24 +02:00
DebaucheryLibrarian bec097f14d Returning status codes from Gamma fetch. 2020-09-10 17:54:23 +02:00
DebaucheryLibrarian 7f8704ee2c Using http instead of bhttp for Gamma. 2020-09-10 17:41:29 +02:00
DebaucheryLibrarian 5ee5b270ef Added avatars to Gamma latest API. 2020-09-10 03:56:09 +02:00
DebaucheryLibrarian 88eeab410b Added filter parameter to Gamma scraper to distinguish Evil Angel exclusives from channel scenes. 2020-09-10 03:43:16 +02:00
DebaucheryLibrarian 0d4893b13c Added Evil Angel channels with logos. 2020-09-10 03:17:19 +02:00
DebaucheryLibrarian ac66606135 Including networks in filter count. Preserving query between date ranges. Allowing --latest to be used without --last. 2020-09-09 03:45:00 +02:00
DebaucheryLibrarian 8aaa88770f Accounting for Windows paths in logger. 2020-09-08 15:44:55 +02:00
DebaucheryLibrarian 03ba35d65a Fixed include object. Fixed qu's undefined URL handling. 2020-09-08 03:26:34 +02:00
DebaucheryLibrarian e90bb63a8f Added American Pornstar. Improved Jules Jordan scraper to accomodate for American Pornstar. Changed entity logo mogrify settings to ensure both minimum height and width. 2020-09-05 01:56:54 +02:00
DebaucheryLibrarian 3c84a814a8 Added Zero Tolerance with Addicted 2 Girls and GenderX. 2020-09-04 03:07:28 +02:00
DebaucheryLibrarian c187a27123 Added Hookup Hotshot. 2020-09-03 22:22:12 +02:00
DebaucheryLibrarian f6353ca14c Storing actor profile URL when provided from scene page. 2020-08-31 02:43:41 +02:00
DebaucheryLibrarian 1bfdf4b232 Storing actor profiles from scene pages. 2020-08-30 04:18:47 +02:00
DebaucheryLibrarian 53b0101a12 Added new Naughty America sites. 2020-08-26 23:59:29 +02:00
DebaucheryLibrarian 8611d738b0 Using UTC to query date ranges. Removed stray console log from MindGeek scraper. 2020-08-26 02:01:38 +02:00
DebaucheryLibrarian 52f66e7982 Fixed undefined location in FreeOnes scraper. 2020-08-24 18:24:07 +02:00
DebaucheryLibrarian 7fed5b7138 Moved Brazzers to MindGeek scraper to support new site. 2020-08-24 05:13:34 +02:00
DebaucheryLibrarian d8b8dfa299 Not creating batch ID when no movies are to be stored. 2020-08-23 03:41:29 +02:00
DebaucheryLibrarian c3d771c8fc Hush scraper uses children from entity argument for filter regexp, instead of making its own database request. 2020-08-23 03:31:37 +02:00
DebaucheryLibrarian 278b74e78c Providing duplicate releases in predata. Using duplicates for filtering scenes without channel in Hush scraper. 2020-08-23 02:43:10 +02:00
DebaucheryLibrarian 3a5ea3dd9a Returning duplicate releases from pagination. 2020-08-22 04:22:56 +02:00
DebaucheryLibrarian 5f5c48ea05 Fixed pagination behavior for upcoming scenes. 2020-08-22 01:57:23 +02:00
DebaucheryLibrarian 7c052fedfc Fixed map error. 2020-08-22 00:28:22 +02:00
DebaucheryLibrarian 1c17cd1be5 Improved update scrape pagination and limits. 2020-08-21 03:55:51 +02:00
DebaucheryLibrarian 4ec89e2cc8 Added upcoming, profile and detailed scene actor scraping to InTheCrack. Fixed clip upsert. 2020-08-20 23:35:18 +02:00
DebaucheryLibrarian 501e764c21 Renamed chapters to clips. Fixed Vixen trailers. 2020-08-20 19:52:02 +02:00
DebaucheryLibrarian 2835c66694 Added chapters and shoot location. Added In The Crack. 2020-08-20 04:57:38 +02:00
DebaucheryLibrarian fd4477bc50 Improved 'new' sorting. 2020-08-19 21:48:55 +02:00
DebaucheryLibrarian e896d52968 Added separate task queue for video streams to prevent ffmpeg overstressing the CPU. Fixed entity parent in scene REST API. 2020-08-17 15:53:20 +02:00
DebaucheryLibrarian b3435c97c3 Added footer and basic stats page. 2020-08-15 19:04:33 +02:00
DebaucheryLibrarian d7974f057f Fixed scene tile detail word wrap. Fixed Score posters. 2020-08-15 02:05:35 +02:00
DebaucheryLibrarian 50c5f921f5 Using new bulk insert utility for releases, media and actors. 2020-08-14 23:21:53 +02:00
DebaucheryLibrarian e996a45bf5 Added new bulk upsert utility. 2020-08-14 23:05:25 +02:00
DebaucheryLibrarian b3f784686f Improved entity provision behavior. 2020-08-14 00:32:59 +02:00
DebaucheryLibrarian 77566eae0d Fixed and documented entity configuration and query. 2020-08-13 23:59:54 +02:00
DebaucheryLibrarian 59e2124407 Removed type property from scenes API. 2020-08-13 16:10:58 +02:00
DebaucheryLibrarian f8c9b69f4b Allowing --actors-update to be used without --actors. 2020-08-12 21:04:38 +02:00
DebaucheryLibrarian d14034d38b Defauling --actors-update to 1900-01-01 2020-08-12 21:00:50 +02:00
DebaucheryLibrarian 7413d7db25 Improved and documented actor profile scraping. 2020-08-12 20:51:08 +02:00
DebaucheryLibrarian 5cabeed19d Modularized release component between movie and scene. Added Kink Classics channel. 2020-08-12 03:30:20 +02:00
DebaucheryLibrarian 40aed1086f Showing actors, tags and date on movie tiles. 2020-08-10 21:39:55 +02:00
DebaucheryLibrarian a7d5bef93f Filtering undefined scenes property from movies. Added movie page scraper to Elegant Angel. 2020-08-08 18:10:59 +02:00
DebaucheryLibrarian b4f0501765 Scraping from Cherry Pimps when available. Showing cover in movie tile. 2020-08-02 03:44:14 +02:00
DebaucheryLibrarian 767437d9aa Added movie tile. Fixed actor header. Larger breakpoint for nav menu. 2020-08-01 15:11:07 +02:00
DebaucheryLibrarian 6c5a62353c Moved movies to separate table. 2020-07-25 03:44:19 +02:00