boiii
/
traxxx
forked from DebaucheryLibrarian/traxxx
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,904 Commits 8 Branches 0 Tags 4.7 GiB
Commit Graph

506 Commits

Author SHA1 Message Date
ThePendulum b1b7cd6d50 Fixed Whale Member posters and photos. 2020-05-20 02:23:45 +02:00
ThePendulum c0898b84d6 Showing all unique descriptions on profile with network logo. Fixed Fame Digital scraper. 2020-05-19 04:46:49 +02:00
ThePendulum 0c4628677f Fixed profile location interpolation. Generalizing ethnicity, hair color and eye color. 2020-05-19 01:10:32 +02:00
ThePendulum 4826ae8571 Added oil tag. Removed Fame Digital from profile config. 2020-05-18 04:28:38 +02:00
ThePendulum 885aa4f627 Passing context object with site or network instead of scraper slug and 'site or network' to all profile scrapers. 2020-05-18 03:22:03 +02:00
ThePendulum 8733fdc657 Improved actor scraping and display. 2020-05-18 01:22:56 +02:00
ThePendulum 61965d480c Added birthday to Bang actor scraper. 2020-05-17 05:08:41 +02:00
ThePendulum f42ca7bd52 Added Bang! actor scraper. Fixed date interpolation. Showing date and age of death on profile (only if actor has already died). 2020-05-17 04:59:09 +02:00
ThePendulum 985ab9d2dc Added profile interpolation. 2020-05-17 03:00:44 +02:00
ThePendulum 05ee57378a Storing actor avatars. Using 1 second interval queue for location resolve as per OSM code of conduct. 2020-05-16 04:36:45 +02:00
ThePendulum 11eb66f834 Switched to tabs. Adding missing actor entries when scraping actors, with batch ID. 2020-05-14 04:26:05 +02:00
ThePendulum f1eb29c713 Added thumb and lazy image scripts. Added FreeOnes and Boobpedia as sites. 2020-05-13 23:17:39 +02:00
ThePendulum 6040a3f41f Associating actors without network. 2020-05-13 02:56:20 +02:00
ThePendulum a223f933ce Added 2 minute timeout to media fetch. 2020-04-26 04:21:57 +02:00
ThePendulum 2ac879d276 Fixed Vixen scraper, using new token URL for trailers. 2020-04-26 03:51:59 +02:00
ThePendulum fc58850e56 Added media limit sampling. 2020-04-11 22:49:37 +02:00
ThePendulum cb68319ac0 Added lazy loading to tag photos. Changed tag thumb location. 2020-04-08 14:50:43 +02:00
ThePendulum 4eaacf5697 Expanded new media module. Added network to channel site to fix actor glitch. 2020-03-30 03:01:08 +02:00
ThePendulum a5d859471d Added Jesse Loads Monster Facials. Added various logos. 2020-03-29 23:42:41 +02:00
ThePendulum 93d4f0ff1a Added WIP media module. Returning releases from release search database function. Fixed page loop in update module. 2020-03-29 04:00:46 +02:00
ThePendulum 6d9f96c5d5 Improved duplicate filtering, now also applying to upcoming updates. Updated Gamma fetchLatest method parameters. Added shortcut for SFW-mode. 2020-03-28 04:37:04 +01:00
ThePendulum 95d115b585 Added Aziani. Added URL origin parameter to relevant qu methods. 2020-03-28 01:40:02 +01:00
ThePendulum fdb2b132f6 Improved release storage module. Added new tags module. Added movie scraping. 2020-03-22 03:50:24 +01:00
ThePendulum d765543b30 Improved update runner. Improved HTTP module API, added default user agent. Added PornCZ and Czechav logos. 2020-03-21 02:48:24 +01:00
ThePendulum 0f09fd53eb Refactoring deep scrape. Added tag posters. 2020-03-16 04:10:52 +01:00
ThePendulum 8d484ba728 Added tag photos. Auto-adding BTS tag to BAM Visions scenes. 2020-03-13 21:54:44 +01:00
ThePendulum 01b59f0a9b Changed withReleases boolean to include object in Vixen and Fame Digital scrapers. 2020-03-12 02:21:59 +01:00
ThePendulum 370605554b Added profile scraper with scenes to BAM Visions. Passing 'includes' object instead of withReleases boolean to Gamma. 2020-03-12 02:19:45 +01:00
ThePendulum 25795e2cce Added latest and scene scrapers to BAM Visions. 2020-03-12 00:59:32 +01:00
ThePendulum 37e188a0df Added site aliases. Migrated various scrapers to qu. Added BAM Visions base. 2020-03-12 00:15:25 +01:00
ThePendulum 0d0acb6f3c Fixed release sites for profile scraping. 2020-03-11 03:01:37 +01:00
ThePendulum db63be8f92 Photo plucker will use discarded photos as fallback. Returning high res photo sources from LegalPorno. 2020-03-10 04:42:15 +01:00
ThePendulum 5c55750c0c Fixed qu issues. Fixed media issues. Simplified and expanded date component in search query. 2020-03-10 00:17:57 +01:00
ThePendulum 638757b6e4 Added rudimentary movie relations. 2020-03-09 05:06:37 +01:00
ThePendulum 6cbb7f9c1e Major API change for 'q', renamed to 'qu', refactored modules. Fixed Gamma URL entry ID regex. 2020-03-09 02:02:29 +01:00
ThePendulum acad99bdfe Changed q get and geta APIs to include status, refactored scrapers. Showing front- and back-cover on movie tiles and release page (fix). Removed icons from main navigation. Returning scenes from Jules Jordan movie scraper. 2020-03-08 04:23:10 +01:00
ThePendulum 628c5a2013 Fixed Gamma scene scraper extracting the wrong scene ID from non-standard URLs. 2020-03-07 23:40:38 +01:00
ThePendulum cd12b53071 Added SexyHub. Added favicon. 2020-03-07 04:27:20 +01:00
ThePendulum ff3e956fc7 Added mobile album scraping to Blowpass, improved wrapper. 2020-03-07 02:35:13 +01:00
ThePendulum ea0e37aa49 Fixed Vivid requesting unavailable HTTPS resource, and breaking on non-existing parameters object. 2020-03-07 00:32:04 +01:00
ThePendulum 6820608a09 Gamma won't attempt to fetch photos when 'photos' parameter is false. 2020-03-06 23:50:52 +01:00
ThePendulum aad91fe4ae Added Girls Under Arrest to Adult Time. Added mobile album support to some Girlsway sites. Fixed Moms On Moms link. 2020-03-06 23:40:15 +01:00
ThePendulum a579e3c88f Enabled mobile album scraping for Burning Angel and Fame Digital. 2020-03-06 23:08:55 +01:00
ThePendulum 5d468b58fd Enabled mobile album fetching for Pure Taboo and Fantasy Massage. 2020-03-06 19:44:20 +01:00
ThePendulum 96eef822d6 Allowing mobile parameter to be set on Gamma network. Added mobile to 21Naturals and 21Sextreme, moved 21Sextury mobile to network. 2020-03-06 19:26:23 +01:00
ThePendulum 5c63bd860b Tags module filters out undefined tags. Gamma will return [] instead of undefined when no tags are available. 2020-03-06 18:59:32 +01:00
ThePendulum 90172ea19a Added mobile album support to Gamma scraper. 2020-03-06 04:28:01 +01:00
ThePendulum fa7b2a7a23 Fixed subnetwork spacing in network sidebar. Using derived entry IDs for main Hush scrapers. 2020-03-06 03:15:49 +01:00
ThePendulum db01599569 Added profile scraper with releases to Hush. Added qtexts to q to return text nodes individually. Including network in profile site. 2020-03-06 02:49:55 +01:00
ThePendulum 3889faee26 Added optional sequential scraping and acc release injection. Added Hush Pass and Interracial Pass logos. 2020-03-05 23:01:03 +01:00
ThePendulum fd6e90e74c Added tour layout scraper to Hush, enabling Interracial POVs, POV Pornstars and See Him Fuck. 2020-03-05 20:31:11 +01:00
ThePendulum 1407700511 Added Interracial Pass sites. Fixed Hush removing poster from base release. 2020-03-05 17:07:07 +01:00
ThePendulum 981b121980 Split Hussie Pass into Hush networks Hush Pass, Hussie Pass and Interracial Pass. 2020-03-05 05:07:14 +01:00
ThePendulum f10e4af29b Allowing scrapers to force channel allocation attempt. Added Hush Pass subsite handling to Hussie Pass scraper. 2020-03-05 03:44:27 +01:00
ThePendulum 956afa6ae7 Added Hussie Pass scraper. 2020-03-05 02:47:52 +01:00
ThePendulum 6c3cba1b87 Added actor photos to Brazzers scene scrape. Added no-video poster to Score. Not flattening actor avatar fallbacks. 2020-03-04 17:21:40 +01:00
ThePendulum e79a6b33fb Added 'newly added' filter. Handling paywalled videos in Private scraper. Added shoot ID to search. 2020-03-02 03:41:41 +01:00
ThePendulum e57f440665 Added Amateur Allure. 2020-03-01 05:28:08 +01:00
ThePendulum b03775fa07 Using generic slugify for MindGeek channel. 2020-02-29 05:00:50 +01:00
ThePendulum f1f33080f6 Ignoring undefined video entropy. 2020-02-28 03:56:58 +01:00
ThePendulum 646ff064a7 Updated Score scraper to accept site and with-releases argument. 2020-02-25 22:32:13 +01:00
ThePendulum 6d1f30f703 Passing matching site to profile scrapers. Allowing scrapers to pass avatar metadata. Added scraper and copyright properties to media. Auto-adding copyright from site or scraper to avatars. Separated Porn Pros from Whale Member. 2020-02-24 03:12:58 +01:00
ThePendulum 0ae7d2669a Added profile scraping to Private. 2020-02-24 00:31:36 +01:00
ThePendulum 164681427a Fixed Kelly Madison posters, marking frontpage video as teaser. 2020-02-23 22:37:14 +01:00
ThePendulum 0d719d88ea Removed Boob Pedia as gender source, unreliable. 2020-02-23 22:26:14 +01:00
ThePendulum 8359f78e2e Fixed RK scraper returning dick size as bust size. 2020-02-23 22:01:12 +01:00
ThePendulum 915eb75719 Refactored Vixen scraper, using API endpoint and added actor profile and releases scraper. Release scraper will return base release when present and 'deep' argument is false. 2020-02-22 23:25:10 +01:00
ThePendulum 5e1a1005f1 Refactored Vixen scene scraper. Using better poster source for Vixen. Returning video as teaser instead of trailer. 2020-02-22 05:29:02 +01:00
ThePendulum e5c6ccd252 Scraping upcoming Vixen scenes. Fetching release media groups sequentially to prevent collisions. 2020-02-22 04:37:48 +01:00
ThePendulum 349a5a506e Queueing and batching media HTTP requests for improved reliability. 2020-02-22 03:22:30 +01:00
ThePendulum 13a8221b92 Added New Sensations. Returning null from q's date formatter when date is invalid. 2020-02-21 03:51:57 +01:00
ThePendulum 377970f874 Added parent-child relations to network, showing parent network in sidebar. Added Burning Angel using Gamma API. 2020-02-20 02:35:23 +01:00
ThePendulum 2c490753a4 Removed stray console log from Gamma scraper. 2020-02-19 04:50:10 +01:00
ThePendulum 8889ea5cf3 Using hasTeaser param instead of date comparison to skip upcoming release teasers in Gamma scraper. 2020-02-19 04:49:54 +01:00
ThePendulum 372ff54261 Added teaser source to Gamma scraper. Not adding Gamma teaser for upcoming releases. 2020-02-19 04:47:20 +01:00
ThePendulum 97f5e49187 Refactored media module. Returning 320p and 720p videos from MindGeek as teasers instead of trailers. 2020-02-19 04:41:53 +01:00
ThePendulum 40bf476ea6 Fixed Porn Pros scraper. Added various Score site logos. 2020-02-18 16:00:36 +01:00
ThePendulum d1532e1bec Fixed scene URLs for Full Porn Network. 2020-02-15 02:47:50 +01:00
ThePendulum 30e58fb397 Added actor description to Full Porn Network profile scraper. 2020-02-15 02:06:21 +01:00
ThePendulum b6fe91b016 Added profile avatar and release scraping to Full Porn Network. 2020-02-15 02:04:46 +01:00
ThePendulum 880cc64022 Added Full Porn Network scraper. 2020-02-15 01:50:22 +01:00
ThePendulum 5e461f1f1a Added proper Gamma parameters to 21Sextury sites. 2020-02-13 23:05:28 +01:00
ThePendulum 1ff8d37d89 Added upcoming support to Nubiles. Renamed q's formatDate to extractDate, added actual formatDate. 2020-02-13 04:11:32 +01:00
ThePendulum c9c7a33585 Added Cherry Pimps/Pimp.XXX. 2020-02-13 03:44:04 +01:00
ThePendulum b7abd805e2 Added Girl Girl to Jules Jordan. 2020-02-12 23:49:22 +01:00
ThePendulum cd2ca65903 Added Nubiles network. 2020-02-12 23:00:32 +01:00
ThePendulum 98d29c0af0 Added Insex fallback for Paintoy. 2020-02-12 16:26:08 +01:00
ThePendulum 48b10d0f49 Using brief mode on Insex. 2020-02-12 04:59:15 +01:00
ThePendulum b8074205ef Added Insex. Renamed q's stand-alone date function. Separated q's trim function. Release tile uses cover if available, and poster is not available. 2020-02-12 04:39:57 +01:00
ThePendulum 2f70de8e11 Added preflight method to scrapers. Added Assylum. 2020-02-12 01:54:54 +01:00
ThePendulum df4a2b0bb3 Fetching old scene links from Vivid search API. 2020-02-11 19:05:12 +01:00
ThePendulum dd6a1d9bfd Added Vivid network. Added ASMR Fantasy to Adult Time. Storing deep URL in database. Added href to header links. 2020-02-11 04:58:18 +01:00
ThePendulum ce448da7e0 Added 21 Naturals and 21 Sextreme sites. 2020-02-10 23:11:11 +01:00
ThePendulum 139f0ce7cb Allowing release scrapers to return actor details. Added True Amateurs. 2020-02-09 23:25:54 +01:00
ThePendulum 0f513266a0 Added Black for Wife to JayRock. Switched parameters field to JSON type. 2020-02-09 19:41:39 +01:00
ThePendulum 9d9eda29be Added scene count to actor inspect. Preferring network slug over data brand for scene URLs in MindGeek scraper, since milehighmedia.com's brand is milehigh, resulting in milehigh.com. 2020-02-09 03:09:06 +01:00
ThePendulum e61ed2bb5f Added 21Sextreme and 21Naturals networks. Scraping all actor release pages for Brazzers. 2020-02-09 02:01:39 +01:00
ThePendulum 8ca458a8ff Added Moms on Moms to Girlsway. 2020-02-08 05:16:04 +01:00
ThePendulum d2cb74a252 Added Fantasy Massage sites. Improved Private scraper, added movie link. 2020-02-08 04:52:32 +01:00
ThePendulum 1546e0836c Split Girlsway from Adult Time. Added Fantasy Massage. Using Gamma scraper for Pure Taboo. Added photo path parameter to Gamma scraper. 2020-02-08 02:49:39 +01:00
ThePendulum 5ba308f07a Added Adult Time. Adding context to logger. 2020-02-07 19:53:16 +01:00
ThePendulum 6b56bad21e Added Twistys. 2020-02-07 04:44:01 +01:00
ThePendulum 30963b94dd Added profile releases for classic Fame Digital sites (Silvia Saint and Silverstone DVD). 2020-02-07 03:40:11 +01:00
ThePendulum be5dd4acd8 Added actor release URL resolver for Peter North on Fame Digital. 2020-02-07 01:59:43 +01:00
ThePendulum ea9c2dfe67 Scraping all actor release pages for Gamma. Improved actor matching for Gamma API. 2020-02-07 01:48:21 +01:00
ThePendulum 631ac34573 Blowpass now uses Gamma module for latest and upcoming. 2020-02-07 01:06:39 +01:00
ThePendulum be83505ecf Added channel support to Gamma scene scraper. 2020-02-06 23:59:32 +01:00
ThePendulum 44f0064b42 Cleaned up Gamma profile scraper. 2020-02-06 23:52:00 +01:00
ThePendulum 65f98c6387 Refactored 21Sextury module to use Gamma API. 2020-02-06 23:51:13 +01:00
ThePendulum 6e1de52a40 Added Fame Digital. Added actor release scraping to DDF Network. Improved q and Gamma scraper. 2020-02-06 23:15:28 +01:00
ThePendulum 054dfba6b5 Fixed Score module to scrape releases for actors with just one page. 2020-02-06 00:22:31 +01:00
ThePendulum d4801bb240 Returning window.document instead of element as document from q. Fixed actor collisions when scrapers return same scene multiple times. Scraping all Score actor release pages. Fixed 21Sextury and PureTaboo photo scraping. 2020-02-05 23:57:55 +01:00
ThePendulum 38bab672d4 Better handling of empty profiles. Modified Score scraper for Big Tit Terry Nova. Improved Naughty America and various other logos. 2020-02-05 01:41:30 +01:00
ThePendulum f921bb4ae9 Generating and using URL slugs for releases, improver slugify module. Added 'extract' parameter to MindGeek scraper to get scenes not associate with a channel (see Digital Playground). Added various high res logos. 2020-02-04 03:12:09 +01:00
ThePendulum ef602a3a15 Added basic filename copy. Added HTTP helper to q. Fetching all actor release pages from Naughty America. Added various high res network logos. 2020-02-04 00:18:53 +01:00
ThePendulum 5bae5b6e5f Adapted Score scraper for 18eighteen, 40 Something Mag and 50 Plus Milfs. Updated Score network logo and favicon. 2020-02-03 02:57:53 +01:00
ThePendulum 944f091ca5 Adapted Score scraper for Scoreland2. 2020-02-03 02:12:02 +01:00
ThePendulum a671190fff Adapted Score scraper for Score Classics. 2020-02-03 02:04:47 +01:00
ThePendulum a96680875c Fixed media duplicate fallbacks being ignored. 2020-02-03 00:51:25 +01:00
ThePendulum a45bebddac Adapter Score scraper for Score Videos. 2020-02-03 00:39:43 +01:00
ThePendulum 0ed1b2eff9 Added generic photo page extract method to media module, to allow pre-filtering sources and relief Dogfart scraper. Added 'transsexual' site tag to Trans Angels. 2020-02-02 22:36:33 +01:00
ThePendulum 10d2ba78d0 Fixed Score searching for actors infinitely. 2020-02-02 15:26:24 +01:00
ThePendulum a97c6defca Added teaser support. Added Score network with scraper for Scoreland. Improved q. Added assets. 2020-02-02 05:14:58 +01:00
ThePendulum 87e2d6bbfd Added actor releases to MindGeek module. 2020-02-01 04:42:35 +01:00
ThePendulum cde9aba0cb Redundant actor sources can now be bundled in configuration. Fixed Men network actor path. 2020-02-01 04:14:08 +01:00
ThePendulum abf9893f9d Fixed web server plugins. 2020-02-01 03:02:58 +01:00
ThePendulum 3541a9c402 Integrated Blowpass into generic Gamma scraper. 2020-02-01 02:26:00 +01:00
ThePendulum 94bf207397 Added Wicked network. Merged Evil Angel, XEmpire and Wicked into generic Gamma scraper. 2020-02-01 01:15:40 +01:00
ThePendulum b7f51a8deb Added avatar and actor releases to Bang Bros scraper. 2020-01-31 21:43:16 +01:00
ThePendulum 504bcd02e3 Added Naughty America profile and actor releases scraper. 2020-01-31 19:25:42 +01:00
ThePendulum 2fef4b4314 Fetching and returning releases for Evil Angel actors. 2020-01-31 00:39:49 +01:00
ThePendulum 3f113310e3 Added Trans Angels to MindGeek. Interpreting MindGeek 'other' gender as transsexual. 2020-01-31 00:25:51 +01:00
ThePendulum c300d4d251 Added trailer support to Vogov. Set preferred trailer resolution to 480p. Added 480p and other resolution trailers support to XEmpire. 2020-01-30 03:53:54 +01:00
ThePendulum ff61094b69 Added Men network and Icon Male to MindGeek. Added entropy filter to media module to help filter out generic avatars. Added Pure Taboo. Various logo updates. 2020-01-30 01:14:31 +01:00
ThePendulum 8b840f52d1 Added description property to LegalPorno scraper (rarely used, see GIO337). 2020-01-29 03:19:38 +01:00
ThePendulum 81ede3f511 Improved MindGeek avatar fix. 2020-01-29 02:31:55 +01:00
ThePendulum fc675ae144 Added Metro HD network using MindGeek scraper. Fixed MindGeek profile scraper avatar issue. 2020-01-29 02:24:19 +01:00
ThePendulum 76852daf6d Added VogoV (no trailer yet). Fixed MindGeek profile scraper. 2020-01-28 03:05:53 +01:00
ThePendulum 6d4fd5fd77 Added MindGeek profile scraper for all MG sites. 2020-01-27 22:54:14 +01:00
ThePendulum 32a0188b72 Fixed Boobpedia scraper. Catching non-OK responses for Vixen scraper. 2020-01-27 03:07:06 +01:00
ThePendulum eca65f6b4d Inspecting performance. 2020-01-27 00:41:04 +00:00
ThePendulum bceded3ebd Added gender indicator to actor tiles. Fixed PornHub scraper fetching default avatars. 2020-01-25 01:26:13 +01:00
ThePendulum c93116dee2 Added actor expand button to profile. 2020-01-24 23:36:06 +01:00
ThePendulum 6ef5c82c6a Added URL helper to q. Getting social links from Boobpedia. 2020-01-23 23:32:51 +01:00
ThePendulum 42d07e86f8 Context q methods can accept alternative element as first argument. Added BoobPedia profile scraper. Showing fake boobs icon again. 2020-01-23 22:45:46 +01:00
ThePendulum 9db0e97512 Improved actor profile flow. Added images. 2020-01-23 03:52:12 +01:00
ThePendulum f8175f6054 Added generic Gamma photo and actor scraper for XEmpire, 21Sextury, Blowpass and Evil Angel. 2020-01-22 22:25:58 +01:00
ThePendulum 4e4323704a Getting more and higher quality Vixen photos. 2020-01-18 23:10:31 +01:00
ThePendulum daee426ba6 Removed site and tag matching from Dogfart scraper. Filtering duplicate tag associations before insert attempt. 2020-01-17 01:55:54 +01:00
ThePendulum 4074a5bb64 Added Mile High Media scraper. 2020-01-16 21:56:33 +01:00
ThePendulum 61fee5e4f6 Added q scraping helper. Added Perfect Gonzo scraper. 2020-01-16 21:04:44 +01:00
ThePendulum 5dda81535d Added PornPros satellite sites. 2020-01-15 01:01:16 +01:00
ThePendulum 75c53d338a Added Porn Pros sites and scraper. 2020-01-14 21:45:30 +01:00
ThePendulum 9255467091 Refactored MOFOS scraper to use generic MindGeek scraper. Added Digital Playground and Fake Hub. 2020-01-14 19:56:46 +01:00
ThePendulum ef76909d3c Merged Reality Kings and Babes into MindGeek scraper. Kept classic latest wrapper for Look At Her Now and Tranny Surprise. 2020-01-14 04:50:42 +01:00
ThePendulum f5704451f8 Wrapping up 2020-01-14 01:51:18 +01:00
ThePendulum 8c76568f44 Added Babes network 2020-01-14 01:50:11 +01:00
ThePendulum 859cb7e1f3 Added support for Family Strokes. 2020-01-13 23:45:09 +01:00
ThePendulum 97cb4a061e Added Team Skeet scraper. 2020-01-12 05:30:46 +01:00
ThePendulum 8f72e61928 Added Jay Rock network. Fixed site logo not showing up without URL. 2020-01-11 05:31:04 +01:00
ThePendulum f572f38b5c Added Jay's POV scraper. 2020-01-10 22:10:11 +01:00
ThePendulum 4b36de2f55 Fixed Evil Angel upcoming and actor association issues. Moving from console.log to logger. 2020-01-10 02:43:04 +01:00
ThePendulum a773e0bf54 Added Manuel Ferrara site to Jules Jordan. Refactored Jules Jordan photo scraper for better compatability and quality. 2020-01-08 05:12:14 +01:00
ThePendulum 0a19f2e624 Added Bang! deep scrape. Improved network page layout. Added Bang Bros logos. 2020-01-07 04:23:28 +01:00
ThePendulum bb5b8085f6 Added Bang! scraper. Showing studio on tile. Added favicons to tiles. 2020-01-06 05:19:38 +01:00
ThePendulum 70e27a6cd9 Moved networks to GraphQL. 2020-01-03 00:59:02 +01:00
ThePendulum 5a6bf2b42f Further refactoring. Fixed actor pages and more. 2019-12-31 03:12:52 +01:00
ThePendulum 07a6c77ce2 Improvements, GrapQL experiments. Fixed Babel dependencies. 2019-12-15 22:16:55 +01:00
ThePendulum 7ba716cd6f Experimenting using GraphQL in favor of REST. 2019-12-15 05:42:51 +01:00
ThePendulum fed2b0be8a When poster is not available during scraping, first photo is stored as poster; removed client-side fallback. Added screencap album fallback to Jules Jordan scraper. Simplified JJ page traversal. 2019-12-13 16:59:04 +01:00
ThePendulum c998fcf933 Added photo album support to Blowpass scraper. 2019-12-13 05:04:04 +01:00
ThePendulum e69b7744c8 Showing movie covers as poster when available. 2019-12-13 03:57:01 +01:00
ThePendulum 77307d2d13 Refactored media module to generalize avatar and poster storage into photo storage. 2019-12-13 03:28:52 +01:00
ThePendulum a03a00f5d9 Added Sperm Swallowers and The Ass Factory to Jules Jordan scraper. Added photo source. 2019-12-12 05:18:43 +01:00
ThePendulum 0b819713b5 Fixed off-by-one in photo plucker. Fixed source duplicate photo function not handling fallback sources. 2019-12-12 04:04:35 +01:00
ThePendulum a310f9bb1d Added support for release photo fallbacks. Limited photo fetching concurrency. Modifier XEmpire scraper for AllBlackX support and improved photo scraping. Added movie property to Evil Angel scraper. 2019-12-12 03:12:05 +01:00
ThePendulum b9af7255f6 Fixed release actor origin bug. 2019-12-11 05:15:23 +01:00
ThePendulum 18f37e3fbd Moved all tag photos to public asset directory. 2019-12-11 02:25:25 +01:00
ThePendulum b9bac6d8f9 Added Kelly Madison profile scraper. 2019-12-10 22:35:00 +01:00
ThePendulum 3956994580 Swapped poster and photo for Kelly Madison scraper. 2019-12-10 05:19:08 +01:00
ThePendulum 85e55eebaf Added Kelly Madison Media (Teen Fidelity) release scraper. 2019-12-10 05:10:08 +01:00
ThePendulum b6123b9418 Added source maps. Fixed DDFNetwork avatar and improved photo fetch error handling. 2019-12-10 00:30:55 +01:00
ThePendulum 04a89efa58 Refactored 21sextury scraper. 2019-12-09 05:00:49 +01:00
ThePendulum 30f4a418e3 Allowing scrapers to return multiple trailer qualities, implemented for Brazzers. Removed tag and site matching from Brazzers scraper. 2019-12-07 04:17:14 +01:00
ThePendulum 98b9c885ec Added support for Reality Kings sites 'Look At Her Now' and 'Tranny Surprise'. 2019-12-07 00:42:47 +01:00
ThePendulum 7cb888801f Updated Naughty America scraper, added media support.. 2019-12-05 04:26:54 +01:00
ThePendulum 2697aaa0af Removed release_date selection, not working as desired. Removed site and tag fetching from Kink scraper. 2019-12-05 01:33:57 +01:00
ThePendulum 71cb85c3e1 Allowing scrapers to return raw tags and site URLs or slugs, to gradually remove site and tag fetching from individual scrapers. Added media and deep fetchin support to Perv City scraper. 2019-12-05 01:26:22 +01:00
ThePendulum bead69de49 Added basic actor and network overview. Added DDF Network actor scraper. Various bug fixes and layout improvements. 2019-11-30 05:55:32 +01:00
ThePendulum 0dbe853f39 Resolving actor birth and residence place before storage. Layout improvements. 2019-11-29 05:46:06 +01:00
ThePendulum 4be508b388 Drastic actor page redesign. Storing one avatar per actor, other profile photos as 'photo' role; no longer assuming first photo is avatar. 2019-11-28 05:36:22 +01:00
ThePendulum d113123778 Fixed countries seed file. Updated MOFOS scraper. Improved Reality Kings scraper. Limiting photos for XEmpire scraper. 2019-11-27 04:58:38 +01:00
ThePendulum 31a8f1cac1 Improved profile scrapers. Sorting avatars on actor page by index. 2019-11-21 04:24:55 +01:00
ThePendulum 9224b441e2 Added Brazzers and Jules Jordan as profile sources. Changed profile structure for proper bust-waist-hip properties and improved stability. 2019-11-21 04:05:32 +01:00
ThePendulum 9fcc40dd17 Added avatars. Added PornHub and LegalPorno actor profile scrapers. 2019-11-20 04:53:36 +01:00
ThePendulum 3ec7b15886 Improved scraping and association behavior. 2019-11-19 04:36:15 +01:00
ThePendulum e8130c3634 Scraping actor profiles from FreeOnes. 2019-11-17 03:56:45 +01:00
ThePendulum bd041c528d Fixed ID number/string incompatability in duplicate detection. Expanded Reality Kings scraper to handle older scenes and fix URLs. 2019-11-17 00:45:31 +01:00
ThePendulum 2857387441 Added bash CLI shorthand. Properly setting parameters in site results. Removed obsolete URL restriction from Reality Kings scraper. 2019-11-16 23:37:33 +01:00
ThePendulum b489c8fc33 Major refactor, cleand up site scrape module, fixed and cleaned up release scrape module. Removed old CLI code 2019-11-16 03:33:36 +01:00
ThePendulum 5620dfcb65 Added filter bar to all pages. Added 'upcoming' marker. Improved date tidbit. 2019-11-15 05:10:59 +01:00
ThePendulum bf223adc55 Redesigned release page. Added 'single site' marker, linking directly to network page. Fixed Dogfart scraper duration. 2019-11-14 05:13:38 +01:00
ThePendulum 117d669a69 Added Private Black, improved Private scraper to handle their non-private.com sites. 2019-11-14 00:00:29 +01:00
ThePendulum f9f9cc7977 Added tag filter to releases query, enabled on homepage. 2019-11-13 03:14:24 +01:00
ThePendulum 832e96ced1 Improved site and network pages. Fixed various issues. 2019-11-12 01:22:20 +01:00
ThePendulum 681f1bca85 Added tag page. Added default 'anal' tag to Vixen scraper for Tushy and Tushy Raw. 2019-11-11 05:18:28 +01:00
ThePendulum e9277028e2 Added Mike Adriano scraper (True Anal, All Anal, Swallowed, Nympho). Updated README. 2019-11-09 01:22:50 +01:00
ThePendulum f8f6428e36 Added media support to Private scraper. 2019-11-05 01:41:13 +01:00
ThePendulum c466d791f5 Removed upsert from store releases, observing effects; fixes ? inserted as . Removed query from stored Dogfart URLs. 2019-11-05 00:54:47 +01:00
ThePendulum e4399380d3 Fixed photo plucker to prevent duplicate at end. 2019-11-04 06:04:43 +01:00
ThePendulum 5745cd33d8 Added Dogfart scraper. Added 'date added' property to release page. 2019-11-04 05:47:37 +01:00
ThePendulum 2198c7ceb0 Added media support to Bang Bros scraper. Added untracked files. 2019-10-31 01:53:26 +01:00
ThePendulum 382e40b651 Added studio support. Added studios for LegalPorno. Improved media fetch error handling. Fixed DDFNetwork scraper, added media support. 2019-10-30 04:45:42 +01:00
ThePendulum ffe2564bb3 Added full photo album to Jules Jordan scraper. 2019-10-29 04:49:27 +01:00
ThePendulum eb50af7b81 Added media support to Brazzers scraper. 2019-10-29 03:50:39 +01:00
ThePendulum d5073a73ce Added media support to XEmpire (HardX) scraper. 2019-10-29 03:13:56 +01:00
ThePendulum a4d936523b Added media to LegalPorno scraper. 2019-10-29 01:47:16 +01:00
ThePendulum f7f5e8a306 Deriving transsexual tag from actors in Evil Angel scraper. 2019-10-28 02:12:11 +01:00
ThePendulum 2c32ed6549 Completed Evil Angel scraper. Added trans tags. 2019-09-28 03:26:04 +02:00
ThePendulum 298eabe56e Improved Vixen and XEmpire scrapers. Added media to Blowpass scraper. Improved release fetch code. 2019-09-26 03:27:01 +02:00
ThePendulum 96d8cfbcb6 Improved media handling, added trailer support. Fetching media from Vixen network frontpages. 2019-09-25 04:52:58 +02:00
ThePendulum b12d74cbb3 Fixed Vixen scraper, site restructured. Added Deeper to Vixen network. 2019-09-25 02:44:25 +02:00
ThePendulum c78f7f957f Merge branch 'master' of unknown.name:pendulum/traxxx 2019-09-25 01:38:21 +02:00
ThePendulum 05263a97ea Scrolling content container instead of whole page. Improved title source in XEmpire scraper. Fixed scene page breaking when no photos are available. 2019-09-25 01:38:06 +02:00
boiii a9f64a82a2 Legalporno title scene ID fix 
Old regex would fail on some videos
e.g.
Little Sofi Smile plays with two big cocks AF001
Whitney Wright Gets Punished by Two Big Black Cocks AB015
 2019-09-23 18:32:47 +02:00
ThePendulum cf4987ba48 Using media database for images. 2019-09-10 16:48:04 +02:00
ThePendulum 926c227b85 Refactored database to separate IDs from slugs 2019-09-08 03:53:09 +02:00
ThePendulum f2a6b58370 Added release page with photo viewer. Added photo count to database. Improved styling. 2019-07-06 05:29:12 +02:00
ThePendulum e3558fc0c5 Using grid layout with thumbnails. 2019-05-08 05:50:13 +02:00
ThePendulum 3df5a821b6 Saving results to database. Showing webpage. 2019-05-06 02:01:57 +02:00
ThePendulum b486b6abdd Added Bang Bros scraper. 2019-04-12 03:12:53 +02:00
ThePendulum 82419be402 Changed scene fetch title source for BlowPass scraper. 2019-04-10 04:44:36 +02:00
ThePendulum 7dd2432837 Changed title source for 21Sextury scene fetch, 2019-04-10 03:57:34 +02:00
ThePendulum bc01cbe6dc Added Bang Bros data (no scraper yet). Added 21Sextury scene fetch date fallback. 2019-04-10 03:42:20 +02:00
ThePendulum 763ffb341a Added Naughty America scraper. 2019-04-08 03:07:16 +02:00
ThePendulum d526e4daea Added Reality Kings scraper. Improved site finder. 2019-04-08 01:49:45 +02:00
ThePendulum ec056a177a Added 21Sextury scraper. Various improvements. 2019-04-07 20:51:14 +02:00
ThePendulum d5b9fec0db Added duration property to scraper template. 2019-04-07 05:21:27 +02:00
ThePendulum 6b9d99f495 Added description property to recent scrapers (apparently forgotten). Reconfigured eslint for scrapers. 2019-04-07 05:19:44 +02:00
ThePendulum c8abb302e4 Added DDFNetwork scraper. Modified tag matching query to be case insensitive. 2019-04-07 05:01:06 +02:00
ThePendulum 885a815022 Added tag groups. Added MOFOS scraped. Improved entry ID handling. 2019-04-07 02:15:57 +02:00
ThePendulum 3a90f98d41 Added Blowpass scraper. Split shootId and pageId. 2019-04-06 23:24:26 +02:00
ThePendulum 2b818e379a Scrapers can now iterate through pages. Filtering unique releases before saving to database. Improved scrapers and rendering. 2019-04-05 03:45:40 +02:00
ThePendulum 6190247ac5 Added Brazzers scraper. Removed scene-only sites in database in favor of automatic network fallback. 2019-04-04 20:22:47 +02:00
ThePendulum b3beeef3e4 Added Private scraper. Added Vixen scraper to repository. 2019-04-04 04:00:28 +02:00
ThePendulum 439d3225ec Added Vixen scraper. Added LegalPorno studio IDs. 2019-04-01 02:45:15 +02:00
ThePendulum 8421cd8648 Added parameters column to sites database, fixes Perv City scraper. Getting shoot ID from all existing scrapers. 2019-03-26 01:26:47 +01:00
ThePendulum c60ceeabfa Fixed actor list and title trimming for Jules Jordan scraper. 2019-03-25 04:15:45 +01:00
ThePendulum 4d7f323e86 Replaced network and tag files with SQLite database. 2019-03-25 03:57:33 +01:00
ThePendulum 845f46e9ad Added Kink scraper with elaborate site specification. 2019-03-24 05:28:18 +01:00
ThePendulum 4fcabb4aae Added tags and duration to scraping. Added LegalPorno scraper. 2019-03-24 01:29:22 +01:00
ThePendulum e8d4b76403 Improved module structure. Added individual scene scrapers for Jules Jordan and XEmpire. 2019-03-23 22:48:39 +01:00
ThePendulum 1a170bfbd5 Added Jules Jordan scraper. Added render argument to stop going into ncurses mode during dev. 2019-03-18 04:46:53 +01:00
ThePendulum 02c421b737 Displaying results in terminal table with select, scroll and experimental search. Expanded broken character handling for PervCity scraper. 2019-03-11 04:19:36 +01:00
ThePendulum af9565b296 Completed Perv City scraper. Outputting results as list. 2019-03-04 04:19:03 +01:00
ThePendulum 53ea694894 Added include and exclude config options. Moved network definitions to dedicated file. Scraping all XEmpire sites. Added definitions for Perv City. 2019-03-04 02:46:33 +01:00