boiii
/
traxxx
forked from DebaucheryLibrarian/traxxx
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,465 Commits 8 Branches 0 Tags 4.7 GiB
Commit Graph

647 Commits

Author SHA1 Message Date
ThePendulum 4074a5bb64 Added Mile High Media scraper. 2020-01-16 21:56:33 +01:00
ThePendulum 61fee5e4f6 Added q scraping helper. Added Perfect Gonzo scraper. 2020-01-16 21:04:44 +01:00
ThePendulum 5dda81535d Added PornPros satellite sites. 2020-01-15 01:01:16 +01:00
ThePendulum 75c53d338a Added Porn Pros sites and scraper. 2020-01-14 21:45:30 +01:00
ThePendulum 9255467091 Refactored MOFOS scraper to use generic MindGeek scraper. Added Digital Playground and Fake Hub. 2020-01-14 19:56:46 +01:00
ThePendulum ef76909d3c Merged Reality Kings and Babes into MindGeek scraper. Kept classic latest wrapper for Look At Her Now and Tranny Surprise. 2020-01-14 04:50:42 +01:00
ThePendulum f5704451f8 Wrapping up 2020-01-14 01:51:18 +01:00
ThePendulum 8c76568f44 Added Babes network 2020-01-14 01:50:11 +01:00
ThePendulum 859cb7e1f3 Added support for Family Strokes. 2020-01-13 23:45:09 +01:00
ThePendulum 97cb4a061e Added Team Skeet scraper. 2020-01-12 05:30:46 +01:00
ThePendulum 8f72e61928 Added Jay Rock network. Fixed site logo not showing up without URL. 2020-01-11 05:31:04 +01:00
ThePendulum f572f38b5c Added Jay's POV scraper. 2020-01-10 22:10:11 +01:00
ThePendulum 89afc5e169 Switching to logger. 2020-01-10 04:40:41 +01:00
ThePendulum 4b36de2f55 Fixed Evil Angel upcoming and actor association issues. Moving from console.log to logger. 2020-01-10 02:43:04 +01:00
ThePendulum 863df38b29 Simplified tag media seed file. Renamed tags. Media refactor experiment. 2020-01-08 23:33:24 +01:00
ThePendulum 4c28c8b640 Fixed duplicate query. Updated Jules Jordan logo's. 2020-01-08 19:23:50 +01:00
ThePendulum a773e0bf54 Added Manuel Ferrara site to Jules Jordan. Refactored Jules Jordan photo scraper for better compatability and quality. 2020-01-08 05:12:14 +01:00
ThePendulum 0a19f2e624 Added Bang! deep scrape. Improved network page layout. Added Bang Bros logos. 2020-01-07 04:23:28 +01:00
ThePendulum 89064e9e0c Simplified single-site network overview. Added Jules Jordan favicons. 2020-01-07 00:30:51 +01:00
ThePendulum bb5b8085f6 Added Bang! scraper. Showing studio on tile. Added favicons to tiles. 2020-01-06 05:19:38 +01:00
ThePendulum 72b175e9e2 Removed views in favor of PostGraphile filter and sort plugins. Updated site modules to GraphQL. Added tag posters. 2020-01-04 02:51:58 +01:00
ThePendulum 70e27a6cd9 Moved networks to GraphQL. 2020-01-03 00:59:02 +01:00
ThePendulum e77dbca954 Fixed actor data and avatar fetching and display. 2020-01-02 17:13:57 +01:00
ThePendulum 5a6bf2b42f Further refactoring. Fixed actor pages and more. 2019-12-31 03:12:52 +01:00
ThePendulum 1c43884102 Refactoring client to reflect database changes. 2019-12-19 04:42:50 +01:00
ThePendulum 31aee71edb Normalized database. Updated seed files. Simplified seed upsert. 2019-12-19 02:35:07 +01:00
ThePendulum 6950a76cb5 Abstracted fragments and curation. Using GraphQL for tags. 2019-12-16 05:30:25 +01:00
ThePendulum f4c2e6c08c Replaced default height and weight fields with fields taking units argument. 2019-12-16 02:39:13 +01:00
ThePendulum 577c03f9b7 Fixed queries. 2019-12-15 23:46:42 +01:00
ThePendulum ce92d13327 Fixed site fallback handling. 2019-12-15 23:14:43 +01:00
ThePendulum 07a6c77ce2 Improvements, GrapQL experiments. Fixed Babel dependencies. 2019-12-15 22:16:55 +01:00
ThePendulum 7ba716cd6f Experimenting using GraphQL in favor of REST. 2019-12-15 05:42:51 +01:00
ThePendulum fed2b0be8a When poster is not available during scraping, first photo is stored as poster; removed client-side fallback. Added screencap album fallback to Jules Jordan scraper. Simplified JJ page traversal. 2019-12-13 16:59:04 +01:00
ThePendulum c998fcf933 Added photo album support to Blowpass scraper. 2019-12-13 05:04:04 +01:00
ThePendulum e69b7744c8 Showing movie covers as poster when available. 2019-12-13 03:57:01 +01:00
ThePendulum 77307d2d13 Refactored media module to generalize avatar and poster storage into photo storage. 2019-12-13 03:28:52 +01:00
ThePendulum a03a00f5d9 Added Sperm Swallowers and The Ass Factory to Jules Jordan scraper. Added photo source. 2019-12-12 05:18:43 +01:00
ThePendulum 0b819713b5 Fixed off-by-one in photo plucker. Fixed source duplicate photo function not handling fallback sources. 2019-12-12 04:04:35 +01:00
ThePendulum a310f9bb1d Added support for release photo fallbacks. Limited photo fetching concurrency. Modifier XEmpire scraper for AllBlackX support and improved photo scraping. Added movie property to Evil Angel scraper. 2019-12-12 03:12:05 +01:00
ThePendulum b9af7255f6 Fixed release actor origin bug. 2019-12-11 05:15:23 +01:00
ThePendulum 18f37e3fbd Moved all tag photos to public asset directory. 2019-12-11 02:25:25 +01:00
ThePendulum b9bac6d8f9 Added Kelly Madison profile scraper. 2019-12-10 22:35:00 +01:00
ThePendulum 3956994580 Swapped poster and photo for Kelly Madison scraper. 2019-12-10 05:19:08 +01:00
ThePendulum 85e55eebaf Added Kelly Madison Media (Teen Fidelity) release scraper. 2019-12-10 05:10:08 +01:00
ThePendulum b6123b9418 Added source maps. Fixed DDFNetwork avatar and improved photo fetch error handling. 2019-12-10 00:30:55 +01:00
ThePendulum 04a89efa58 Refactored 21sextury scraper. 2019-12-09 05:00:49 +01:00
ThePendulum 222260d3cf Generalized tags to allow them to be assigned to anything. 2019-12-07 04:41:16 +01:00
ThePendulum 30f4a418e3 Allowing scrapers to return multiple trailer qualities, implemented for Brazzers. Removed tag and site matching from Brazzers scraper. 2019-12-07 04:17:14 +01:00
ThePendulum c4d620c6ff Refactored media module for better duplicate handling, readability and DRY. 2019-12-07 03:59:55 +01:00
ThePendulum 98b9c885ec Added support for Reality Kings sites 'Look At Her Now' and 'Tranny Surprise'. 2019-12-07 00:42:47 +01:00
ThePendulum 7cb888801f Updated Naughty America scraper, added media support.. 2019-12-05 04:26:54 +01:00
ThePendulum 2697aaa0af Removed release_date selection, not working as desired. Removed site and tag fetching from Kink scraper. 2019-12-05 01:33:57 +01:00
ThePendulum 71cb85c3e1 Allowing scrapers to return raw tags and site URLs or slugs, to gradually remove site and tag fetching from individual scrapers. Added media and deep fetchin support to Perv City scraper. 2019-12-05 01:26:22 +01:00
ThePendulum 55e3130062 Move tag posters and photos to media database. 2019-12-04 21:58:08 +01:00
ThePendulum cf81aa99e0 Added rudimentary tags page. Improved social match behavior. 2019-12-01 05:32:47 +01:00
ThePendulum bead69de49 Added basic actor and network overview. Added DDF Network actor scraper. Various bug fixes and layout improvements. 2019-11-30 05:55:32 +01:00
ThePendulum 0dbe853f39 Resolving actor birth and residence place before storage. Layout improvements. 2019-11-29 05:46:06 +01:00
ThePendulum 4be508b388 Drastic actor page redesign. Storing one avatar per actor, other profile photos as 'photo' role; no longer assuming first photo is avatar. 2019-11-28 05:36:22 +01:00
ThePendulum 884ef248e4 Fixed actor birthdate and social links scraping. 2019-11-27 05:12:51 +01:00
ThePendulum d113123778 Fixed countries seed file. Updated MOFOS scraper. Improved Reality Kings scraper. Limiting photos for XEmpire scraper. 2019-11-27 04:58:38 +01:00
ThePendulum de36ed97e4 Added profile scraping in try/catch block. Re-added boobs enhanced info to actor page. 2019-11-21 11:43:07 +01:00
ThePendulum 31a8f1cac1 Improved profile scrapers. Sorting avatars on actor page by index. 2019-11-21 04:24:55 +01:00
ThePendulum 9224b441e2 Added Brazzers and Jules Jordan as profile sources. Changed profile structure for proper bust-waist-hip properties and improved stability. 2019-11-21 04:05:32 +01:00
ThePendulum 9fcc40dd17 Added avatars. Added PornHub and LegalPorno actor profile scrapers. 2019-11-20 04:53:36 +01:00
ThePendulum a13d92b84e Fixed date bug. 2019-11-19 04:41:56 +01:00
ThePendulum 3ec7b15886 Improved scraping and association behavior. 2019-11-19 04:36:15 +01:00
ThePendulum e8130c3634 Scraping actor profiles from FreeOnes. 2019-11-17 03:56:45 +01:00
ThePendulum bd041c528d Fixed ID number/string incompatability in duplicate detection. Expanded Reality Kings scraper to handle older scenes and fix URLs. 2019-11-17 00:45:31 +01:00
ThePendulum 2857387441 Added bash CLI shorthand. Properly setting parameters in site results. Removed obsolete URL restriction from Reality Kings scraper. 2019-11-16 23:37:33 +01:00
ThePendulum b489c8fc33 Major refactor, cleand up site scrape module, fixed and cleaned up release scrape module. Removed old CLI code 2019-11-16 03:33:36 +01:00
ThePendulum 5620dfcb65 Added filter bar to all pages. Added 'upcoming' marker. Improved date tidbit. 2019-11-15 05:10:59 +01:00
ThePendulum 0575dbc7e4 Generalized filters bar, added to network page. 2019-11-15 02:37:17 +01:00
ThePendulum 1c3f17ec82 Added timerange filters. Refactored releases module for more efficient queries. 2019-11-15 01:27:58 +01:00
ThePendulum bf223adc55 Redesigned release page. Added 'single site' marker, linking directly to network page. Fixed Dogfart scraper duration. 2019-11-14 05:13:38 +01:00
ThePendulum 117d669a69 Added Private Black, improved Private scraper to handle their non-private.com sites. 2019-11-14 00:00:29 +01:00
ThePendulum f9f9cc7977 Added tag filter to releases query, enabled on homepage. 2019-11-13 03:14:24 +01:00
ThePendulum 832e96ced1 Improved site and network pages. Fixed various issues. 2019-11-12 01:22:20 +01:00
ThePendulum 681f1bca85 Added tag page. Added default 'anal' tag to Vixen scraper for Tushy and Tushy Raw. 2019-11-11 05:18:28 +01:00
ThePendulum 50816d7b28 Generating thumbnails. Added site overview page. 2019-11-11 03:20:00 +01:00
ThePendulum 6ed484af76 Moved icons into network directories. Improved network logo handling. 2019-11-10 04:47:41 +01:00
ThePendulum a9b91ff76c Added actors page. Added site logos with overview on network page. 2019-11-10 04:20:22 +01:00
ThePendulum 82c9bc664a Added network page and sites back-end. Split release tiles into own component. 2019-11-09 04:43:58 +01:00
ThePendulum e9277028e2 Added Mike Adriano scraper (True Anal, All Anal, Swallowed, Nympho). Updated README. 2019-11-09 01:22:50 +01:00
ThePendulum 0567b1b2ff Refactored seed files to use programmatic upsert instead of SQL duplicate ignore. 2019-11-06 02:47:10 +01:00
ThePendulum f8f6428e36 Added media support to Private scraper. 2019-11-05 01:41:13 +01:00
ThePendulum c466d791f5 Removed upsert from store releases, observing effects; fixes ? inserted as . Removed query from stored Dogfart URLs. 2019-11-05 00:54:47 +01:00
ThePendulum e4399380d3 Fixed photo plucker to prevent duplicate at end. 2019-11-04 06:04:43 +01:00
ThePendulum 5745cd33d8 Added Dogfart scraper. Added 'date added' property to release page. 2019-11-04 05:47:37 +01:00
ThePendulum 2198c7ceb0 Added media support to Bang Bros scraper. Added untracked files. 2019-10-31 01:53:26 +01:00
ThePendulum 382e40b651 Added studio support. Added studios for LegalPorno. Improved media fetch error handling. Fixed DDFNetwork scraper, added media support. 2019-10-30 04:45:42 +01:00
ThePendulum ffe2564bb3 Added full photo album to Jules Jordan scraper. 2019-10-29 04:49:27 +01:00
ThePendulum eb50af7b81 Added media support to Brazzers scraper. 2019-10-29 03:50:39 +01:00
ThePendulum d5073a73ce Added media support to XEmpire (HardX) scraper. 2019-10-29 03:13:56 +01:00
ThePendulum a4d936523b Added media to LegalPorno scraper. 2019-10-29 01:47:16 +01:00
ThePendulum 5f853792b8 Added icon component. Stylized filter bar. 2019-10-28 02:54:37 +01:00
ThePendulum f7f5e8a306 Deriving transsexual tag from actors in Evil Angel scraper. 2019-10-28 02:12:11 +01:00
ThePendulum 0e784a274f Reworked tag matching. Added tag filter groundwork. 2019-10-28 00:58:54 +01:00
ThePendulum 2c32ed6549 Completed Evil Angel scraper. Added trans tags. 2019-09-28 03:26:04 +02:00
ThePendulum 298eabe56e Improved Vixen and XEmpire scrapers. Added media to Blowpass scraper. Improved release fetch code. 2019-09-26 03:27:01 +02:00
ThePendulum 96d8cfbcb6 Improved media handling, added trailer support. Fetching media from Vixen network frontpages. 2019-09-25 04:52:58 +02:00
ThePendulum b12d74cbb3 Fixed Vixen scraper, site restructured. Added Deeper to Vixen network. 2019-09-25 02:44:25 +02:00
ThePendulum c78f7f957f Merge branch 'master' of unknown.name:pendulum/traxxx 2019-09-25 01:38:21 +02:00
ThePendulum 05263a97ea Scrolling content container instead of whole page. Improved title source in XEmpire scraper. Fixed scene page breaking when no photos are available. 2019-09-25 01:38:06 +02:00
boiii a9f64a82a2 Legalporno title scene ID fix 
Old regex would fail on some videos
e.g.
Little Sofi Smile plays with two big cocks AF001
Whitney Wright Gets Punished by Two Big Black Cocks AB015
 2019-09-23 18:32:47 +02:00
ThePendulum cf4987ba48 Using media database for images. 2019-09-10 16:48:04 +02:00
ThePendulum 02e81a8be9 Refactored site accumulation to work with new database structure 2019-09-08 04:06:49 +02:00
ThePendulum 926c227b85 Refactored database to separate IDs from slugs 2019-09-08 03:53:09 +02:00
ThePendulum f2a6b58370 Added release page with photo viewer. Added photo count to database. Improved styling. 2019-07-06 05:29:12 +02:00
ThePendulum b8aa81b3f1 Transitioning to Vue. Installed environment and restored homepage scene overview. 2019-06-03 05:31:38 +02:00
ThePendulum fda6d1e475 Added release page. 2019-05-19 01:34:08 +02:00
ThePendulum e3558fc0c5 Using grid layout with thumbnails. 2019-05-08 05:50:13 +02:00
ThePendulum 8eb2dcfd89 Storing and fetching tags. 2019-05-06 23:41:38 +02:00
ThePendulum 3df5a821b6 Saving results to database. Showing webpage. 2019-05-06 02:01:57 +02:00
ThePendulum d044aff9dd Destroying knex when done to ensure process exit. 2019-04-12 03:54:34 +02:00
ThePendulum b486b6abdd Added Bang Bros scraper. 2019-04-12 03:12:53 +02:00
ThePendulum 82419be402 Changed scene fetch title source for BlowPass scraper. 2019-04-10 04:44:36 +02:00
ThePendulum 7dd2432837 Changed title source for 21Sextury scene fetch, 2019-04-10 03:57:34 +02:00
ThePendulum bc01cbe6dc Added Bang Bros data (no scraper yet). Added 21Sextury scene fetch date fallback. 2019-04-10 03:42:20 +02:00
ThePendulum be74f84781 Expanded README. 2019-04-08 04:07:39 +02:00
ThePendulum db05bcf666 Added site stats to README. 2019-04-08 03:09:37 +02:00
ThePendulum 763ffb341a Added Naughty America scraper. 2019-04-08 03:07:16 +02:00
ThePendulum d526e4daea Added Reality Kings scraper. Improved site finder. 2019-04-08 01:49:45 +02:00
ThePendulum ec056a177a Added 21Sextury scraper. Various improvements. 2019-04-07 20:51:14 +02:00
ThePendulum d5b9fec0db Added duration property to scraper template. 2019-04-07 05:21:27 +02:00
ThePendulum 6b9d99f495 Added description property to recent scrapers (apparently forgotten). Reconfigured eslint for scrapers. 2019-04-07 05:19:44 +02:00
ThePendulum c8abb302e4 Added DDFNetwork scraper. Modified tag matching query to be case insensitive. 2019-04-07 05:01:06 +02:00
ThePendulum 885a815022 Added tag groups. Added MOFOS scraped. Improved entry ID handling. 2019-04-07 02:15:57 +02:00
ThePendulum 3a90f98d41 Added Blowpass scraper. Split shootId and pageId. 2019-04-06 23:24:26 +02:00
ThePendulum 2b818e379a Scrapers can now iterate through pages. Filtering unique releases before saving to database. Improved scrapers and rendering. 2019-04-05 03:45:40 +02:00
ThePendulum cbb4fdc919 Added utils directory to repository. 2019-04-04 21:10:44 +02:00
ThePendulum 6190247ac5 Added Brazzers scraper. Removed scene-only sites in database in favor of automatic network fallback. 2019-04-04 20:22:47 +02:00
ThePendulum 784542253b Expanded tag database. 2019-04-04 04:44:37 +02:00
ThePendulum b3beeef3e4 Added Private scraper. Added Vixen scraper to repository. 2019-04-04 04:00:28 +02:00
ThePendulum 439d3225ec Added Vixen scraper. Added LegalPorno studio IDs. 2019-04-01 02:45:15 +02:00
ThePendulum 8421cd8648 Added parameters column to sites database, fixes Perv City scraper. Getting shoot ID from all existing scrapers. 2019-03-26 01:26:47 +01:00
ThePendulum c60ceeabfa Fixed actor list and title trimming for Jules Jordan scraper. 2019-03-25 04:15:45 +01:00
ThePendulum 4d7f323e86 Replaced network and tag files with SQLite database. 2019-03-25 03:57:33 +01:00
ThePendulum 845f46e9ad Added Kink scraper with elaborate site specification. 2019-03-24 05:28:18 +01:00
ThePendulum e85a0f45f9 Improved copy & quit options. 2019-03-24 03:08:51 +01:00
ThePendulum 4fcabb4aae Added tags and duration to scraping. Added LegalPorno scraper. 2019-03-24 01:29:22 +01:00
ThePendulum e8d4b76403 Improved module structure. Added individual scene scrapers for Jules Jordan and XEmpire. 2019-03-23 22:48:39 +01:00
ThePendulum 1a170bfbd5 Added Jules Jordan scraper. Added render argument to stop going into ncurses mode during dev. 2019-03-18 04:46:53 +01:00
ThePendulum ddbed3c142 Added colors to ratings. TUI renderer accounts for style encodings in formatted values. 2019-03-11 04:51:28 +01:00
ThePendulum 02c421b737 Displaying results in terminal table with select, scroll and experimental search. Expanded broken character handling for PervCity scraper. 2019-03-11 04:19:36 +01:00
ThePendulum af9565b296 Completed Perv City scraper. Outputting results as list. 2019-03-04 04:19:03 +01:00
ThePendulum 53ea694894 Added include and exclude config options. Moved network definitions to dedicated file. Scraping all XEmpire sites. Added definitions for Perv City. 2019-03-04 02:46:33 +01:00
ThePendulum 71aa31dda6 Experimentally scraping data from HardX. 2019-03-03 04:18:33 +01:00