DebaucheryLibrarian
|
5576fed590
|
Fixed existing stash selection in transfer tool.
|
2023-06-08 01:37:27 +02:00 |
DebaucheryLibrarian
|
56534800d8
|
Added stash transfer tool.
|
2023-06-08 01:16:44 +02:00 |
DebaucheryLibrarian
|
e36ba59d27
|
Auto-recognizing m3u8 sources as stream. Fixed Bang Bros poster and trailer.
|
2023-06-05 04:49:44 +02:00 |
DebaucheryLibrarian
|
d3da2359de
|
Refactored Bang! scraper, added My Stepdaughters Friend.
|
2023-06-05 03:32:24 +02:00 |
DebaucheryLibrarian
|
adda78f0c6
|
Refactored New Sensations scraper.
|
2023-06-05 02:13:36 +02:00 |
DebaucheryLibrarian
|
164757ee26
|
Matching URLs to entity using hostname rather than slug to minimize collisions. Fixed missing Cum Louder POV logo.
|
2023-06-04 21:50:59 +02:00 |
DebaucheryLibrarian
|
caf37ba9fb
|
Transferring release media types separately to prevent race conditions.
|
2023-06-04 04:17:01 +02:00 |
DebaucheryLibrarian
|
18e91d54f1
|
Transfer tool finds existing media by source.
|
2023-06-04 01:24:18 +02:00 |
DebaucheryLibrarian
|
84c59bd05a
|
Removed redundant description replace in transfer tool.
|
2023-06-04 01:20:38 +02:00 |
DebaucheryLibrarian
|
e0f7db8187
|
Improved skipped scene reporting in transfer tool.
|
2023-06-04 01:13:23 +02:00 |
DebaucheryLibrarian
|
13e38c487f
|
Ignoring skipped scenes in final tally.
|
2023-06-04 01:05:59 +02:00 |
DebaucheryLibrarian
|
33cab26d3b
|
Fixed transfer status, moved media logging to debug level.
|
2023-06-04 01:04:21 +02:00 |
DebaucheryLibrarian
|
c9201430ea
|
Logging index in transfer.
|
2023-06-04 00:58:35 +02:00 |
DebaucheryLibrarian
|
a4c82a377b
|
Returning existing release ID in case new scene for existing movie was encountered.
|
2023-06-04 00:49:21 +02:00 |
DebaucheryLibrarian
|
66f4244779
|
Logging skipped releases.
|
2023-06-04 00:42:29 +02:00 |
DebaucheryLibrarian
|
54798f87da
|
Improved duplicate media handling in transfer tool.
|
2023-06-04 00:23:45 +02:00 |
DebaucheryLibrarian
|
f356135722
|
Using line-separated JSON to save memory in transfer tool.
|
2023-06-03 22:41:15 +02:00 |
DebaucheryLibrarian
|
5103a07e5f
|
Accumulating boolean instead of full scene.
|
2023-06-03 21:51:09 +02:00 |
DebaucheryLibrarian
|
4b9a0e6bab
|
Extended and improved transfer tool. Moved scenes up on movie page.
|
2023-06-03 02:51:42 +02:00 |
DebaucheryLibrarian
|
1b4d973e7b
|
Added transfer tool, WIP. Added Savage Gangbang to Kink.
|
2023-05-31 00:29:54 +02:00 |
DebaucheryLibrarian
|
54b1c03a8d
|
Added Fabulous Cash sites.
|
2023-04-06 01:28:59 +02:00 |
DebaucheryLibrarian
|
1a24d39761
|
Updated MG session acquisition.
|
2023-03-27 00:01:38 +02:00 |
DebaucheryLibrarian
|
a7d094f0b6
|
Updatead AnalVids studios.
|
2023-03-19 17:31:47 +01:00 |
DebaucheryLibrarian
|
b0692d78ac
|
Improved showcase view.
|
2023-01-14 23:44:32 +01:00 |
DebaucheryLibrarian
|
306744f415
|
Restored upcoming in Bang! scraper, added Bang! Adventures site.
|
2023-01-03 23:32:52 +01:00 |
DebaucheryLibrarian
|
5cda0f741d
|
Added Kink Men to scrapers module.
|
2023-01-03 23:15:56 +01:00 |
DebaucheryLibrarian
|
45715773f1
|
Updated Kink profile scraper.
|
2023-01-02 00:54:10 +01:00 |
DebaucheryLibrarian
|
eb759a0dec
|
Fixed Kink Men Test Shoots slug.
|
2023-01-01 02:03:18 +01:00 |
DebaucheryLibrarian
|
0d846c8871
|
Updated Kink channel inventory, separated Kink Men.
|
2022-12-31 17:50:39 +01:00 |
DebaucheryLibrarian
|
cd98b9d147
|
Changed deep merge so base release photos are only used when no deep photos are available.
|
2022-12-29 23:07:08 +01:00 |
DebaucheryLibrarian
|
16de5e2a2c
|
Removed stray console log.
|
2022-12-29 00:17:20 +01:00 |
DebaucheryLibrarian
|
9dd3348535
|
Removed headless option from kink scraper.
|
2022-12-28 23:06:08 +01:00 |
DebaucheryLibrarian
|
74214bc060
|
Added puppeteer to http module, refactored Kink scraper to utilize it.
|
2022-12-28 01:34:12 +01:00 |
DebaucheryLibrarian
|
78b1bd2e48
|
Improved Love Her Films profile scraping.
|
2022-12-15 23:22:52 +01:00 |
DebaucheryLibrarian
|
b5e308562e
|
Merged filters into new settings dialog, added experimental summary field.
|
2022-11-28 03:33:46 +01:00 |
DebaucheryLibrarian
|
114d7bdff4
|
Added scene avatars and improved HTML fallback to Bang! scraper.
|
2022-11-27 04:43:27 +01:00 |
DebaucheryLibrarian
|
6edd62c337
|
Refactored Bang! scraper to match new website, first use of unprint.
|
2022-11-27 04:22:58 +01:00 |
DebaucheryLibrarian
|
2943aea4d8
|
Added showcased migration. Added Love Her Films scraper.
|
2022-10-25 23:13:24 +02:00 |
DebaucheryLibrarian
|
a6905e4664
|
Added showcased column, hiding most LegalPorno/AnalVids studios. Refined Adult Time network structure.
|
2022-10-05 22:01:27 +02:00 |
DebaucheryLibrarian
|
463e0e1fe7
|
Added and restructured various Adult Time series and studios.
|
2022-10-02 22:49:38 +02:00 |
DebaucheryLibrarian
|
0fc37e46d2
|
Fixed media module trying to fetch invalid source URLs. Added Accidental Gangbang to Adult Time.
|
2022-09-27 20:09:46 +02:00 |
DebaucheryLibrarian
|
3db8b80164
|
Added CF resolver to http module. Using priority lookup in tags seed.
|
2022-08-15 23:51:51 +02:00 |
DebaucheryLibrarian
|
cd187fac16
|
Added campaign retrieval function, added banner to homepage.
|
2022-07-18 02:42:30 +02:00 |
DebaucheryLibrarian
|
01b37f087f
|
Added native site support to Perv City scraper for DP Diva.
|
2022-07-10 01:57:24 +02:00 |
DebaucheryLibrarian
|
85c73bad77
|
Improved MindGeek actor scraper.
|
2022-05-30 00:05:33 +02:00 |
DebaucheryLibrarian
|
43d239a6ae
|
Added Ricky's Room.
|
2022-05-29 21:10:44 +02:00 |
DebaucheryLibrarian
|
0fa36b17bf
|
Refactored upcoming scenes in Vixen scraper.
|
2022-05-24 00:22:33 +02:00 |
DebaucheryLibrarian
|
527112d5da
|
Refactored Vixen scraper, moved XEmpire to generic Gamma scraper after site update, added BiPhoria.
|
2022-05-15 23:28:56 +02:00 |
DebaucheryLibrarian
|
b9556c9c86
|
Set Kink rate limits, added method parameter for arbitrary requests.
|
2022-04-10 21:51:22 +02:00 |
DebaucheryLibrarian
|
cc63cc652a
|
Using network page for Dogfart sites without complete index.
|
2022-04-07 23:22:40 +02:00 |
DebaucheryLibrarian
|
1d84830423
|
Supporting JSON in scenes file. Added Dogfart link finding utility.
|
2022-04-07 16:06:38 +02:00 |
DebaucheryLibrarian
|
b00b8f4a96
|
Using channel URL for Dogfart photos and network URL as fallback.
|
2022-04-04 23:56:26 +02:00 |
DebaucheryLibrarian
|
0d95746689
|
Added update url parameter to Dogfart scraper.
|
2022-04-04 18:21:51 +02:00 |
DebaucheryLibrarian
|
153f28c494
|
Added last login column to users table.
|
2022-04-04 00:23:37 +02:00 |
DebaucheryLibrarian
|
25e0575c2b
|
Fixed description query in Dogfart scraper.
|
2022-04-03 23:31:36 +02:00 |
DebaucheryLibrarian
|
5cbf122d6f
|
Scraping Dogfart scenes from native sites.
|
2022-04-03 23:00:05 +02:00 |
DebaucheryLibrarian
|
762b3984a3
|
Ignore join links for trailers in Dogfart scraper.
|
2022-04-03 01:29:16 +02:00 |
DebaucheryLibrarian
|
9be80e2be9
|
Returning unextracted scenes from Kelly Madison / Teen Fidelity scraper. Fixed Dogfart profile scraper to use extract scenes.
|
2022-04-03 00:53:27 +02:00 |
DebaucheryLibrarian
|
574c117ab0
|
Refactored Dogfart scraper to use qu and return unextracted scenes.
|
2022-04-03 00:49:39 +02:00 |
DebaucheryLibrarian
|
5e499c3685
|
Added chunking to media duplicate queries to prevent overloading parameters. Added DP Diva to Perv City (coming soon).
|
2022-04-02 00:32:23 +02:00 |
DebaucheryLibrarian
|
5352186319
|
Insex not fetching video when not required.
|
2022-03-31 23:01:54 +02:00 |
DebaucheryLibrarian
|
39813d4461
|
Updated Insex scraper.
|
2022-03-31 22:46:54 +02:00 |
DebaucheryLibrarian
|
a19a77e165
|
Optionalized qualities.
|
2022-03-31 14:34:10 +02:00 |
DebaucheryLibrarian
|
18b219850e
|
Storing scene qualities. Updated Perv City scraper.
|
2022-03-31 14:11:13 +02:00 |
DebaucheryLibrarian
|
33a327a04b
|
Merge branch 'master' into experimental
|
2022-03-30 23:00:29 +02:00 |
DebaucheryLibrarian
|
61123fdb6a
|
Added Accept-Language header to MindGeek requests, seems to help with acquiring sessions.
|
2022-03-30 01:17:54 +02:00 |
DebaucheryLibrarian
|
15c9af8057
|
Added dedicated movie photo table, renamed scene photo function.
|
2022-03-27 23:42:03 +02:00 |
DebaucheryLibrarian
|
e93e8ace5c
|
Added deep scene force parameter to MindGeek scraper.
|
2022-03-27 00:27:26 +01:00 |
DebaucheryLibrarian
|
0dad5b0d68
|
Added series removal utils.
|
2022-03-26 17:56:22 +01:00 |
DebaucheryLibrarian
|
fd8170f223
|
Added series.
|
2022-03-26 17:40:20 +01:00 |
DebaucheryLibrarian
|
5ff076cac3
|
Added DP Star Sex Challenges to Digital Playground.
|
2022-03-09 23:26:48 +01:00 |
DebaucheryLibrarian
|
c6e977f842
|
Added movie support to MindGeek scraper.
|
2022-03-04 23:32:09 +01:00 |
DebaucheryLibrarian
|
5deba6b90f
|
Passing resolved parameters into before-scrape methods.
|
2022-02-25 22:13:41 +01:00 |
DebaucheryLibrarian
|
17dfeac1af
|
Renamed LegalPorno to AnalVids, added missing studios.
|
2022-02-24 23:28:24 +01:00 |
DebaucheryLibrarian
|
582269cfaa
|
Added Radical metadata layout scraper and the Got Filled and Inserted channels.
|
2022-02-24 22:33:00 +01:00 |
DebaucheryLibrarian
|
690d2bb3ed
|
Added MOFOS to default rate limit config, added light logos.
|
2022-02-20 16:52:29 +01:00 |
DebaucheryLibrarian
|
8386230f33
|
Added missing Team Skeet sites.
|
2022-02-14 01:27:56 +01:00 |
DebaucheryLibrarian
|
5918364cf5
|
Preventing unnecessary deep scrapes in Team Skeet scraper.
|
2022-02-13 01:34:39 +01:00 |
DebaucheryLibrarian
|
d9f0db6e3c
|
Fixed reading screen ID from non-existent item in Bang scraper.
|
2022-02-12 17:19:47 +01:00 |
DebaucheryLibrarian
|
5da1acc38d
|
Added media resize.
|
2022-02-12 17:16:20 +01:00 |
DebaucheryLibrarian
|
c7c62e220d
|
Removed console log from Bang scraper.
|
2022-02-12 16:22:41 +01:00 |
DebaucheryLibrarian
|
cde760c1ea
|
Preventing Bang! scraper from redundant deep scraping when base release is available.
|
2022-02-12 16:22:00 +01:00 |
DebaucheryLibrarian
|
c7710afdbf
|
Fixed Bang! scraped failing when scene has no photos.
|
2022-02-12 03:26:40 +01:00 |
DebaucheryLibrarian
|
496c29e569
|
Configured Reality Kings to fetch session from RK scene overview.
|
2022-02-11 22:14:44 +01:00 |
DebaucheryLibrarian
|
c4f0b48932
|
Fixed cover dimensions on movie tile to show lazy image, showing missing cover icon.
|
2022-02-09 00:19:54 +01:00 |
DebaucheryLibrarian
|
f7bbab08bd
|
Added Porn Pros main channel and working URLs for Flexible Positions, Public Violations and Squirt Disgrace. Added bisexual tag photos.
|
2022-02-08 23:59:44 +01:00 |
DebaucheryLibrarian
|
9b7879bff7
|
Removed channel ID from Gamma update query to prevent duplicate scenes. Added sexuality tags with covers to overview.
|
2022-02-08 00:19:34 +01:00 |
DebaucheryLibrarian
|
1dfa034332
|
Handling 'page not found' with 200 OK in Gamma scraper.
|
2022-02-07 22:29:09 +01:00 |
DebaucheryLibrarian
|
3d349c413f
|
Fixed Adult Time scraper.
|
2022-02-07 22:16:43 +01:00 |
DebaucheryLibrarian
|
fffe9080f3
|
Added stand-alone Spizoo sites.
|
2022-02-07 21:58:27 +01:00 |
DebaucheryLibrarian
|
4bcf7ef45b
|
Storing associations if existing media are found.
|
2022-02-07 01:57:09 +01:00 |
DebaucheryLibrarian
|
ceaf5a3217
|
Not storing shallow movie data when argument is disabled.
|
2022-02-05 17:14:34 +01:00 |
DebaucheryLibrarian
|
3d80e9d77b
|
Added movie URL file support. Fixed Gamma movie API scraper movie URL.
|
2022-02-03 00:04:48 +01:00 |
DebaucheryLibrarian
|
f1a7f2f905
|
Added Gamma movie API scraper.
|
2022-02-02 23:51:15 +01:00 |
DebaucheryLibrarian
|
10b8abf706
|
Updated Zero Tolerance channel to Zero Tolerance Films.
|
2022-02-02 22:25:04 +01:00 |
DebaucheryLibrarian
|
b9f3eb85f9
|
Ignoring Gamma master categories in update scrape as these seem to contain irrelevant tags.
|
2022-02-01 21:18:54 +01:00 |
DebaucheryLibrarian
|
815d56d334
|
Preventing media associations when use-reference is unavailable.
|
2022-01-30 22:15:07 +01:00 |
DebaucheryLibrarian
|
22864105ac
|
Writing media associations to disk for debugging.
|
2022-01-30 17:05:18 +01:00 |
DebaucheryLibrarian
|
52218c30f8
|
Not fetching mobile Gamma page when photos are not expected.
|
2022-01-29 16:38:36 +01:00 |
DebaucheryLibrarian
|
5c5a11fdca
|
Added Bang Bros members page scraper (not for full scenes).
|
2022-01-29 01:21:41 +01:00 |
DebaucheryLibrarian
|
90bc2f3724
|
Changed release media association existence check, added log for details.
|
2022-01-28 04:05:29 +01:00 |
DebaucheryLibrarian
|
67d0a9e0e0
|
Fixed scene entity tag association.
|
2022-01-26 01:47:34 +01:00 |
DebaucheryLibrarian
|
452f725332
|
Switched poster URL preference in Gamma scraper to prevent cropped image on XEmpire.
|
2022-01-25 15:34:15 +01:00 |
DebaucheryLibrarian
|
7ceec1c739
|
Added indexes to scene tag and movie scene tables improve performance. Removed scene count from channel overview, too expensive.
|
2022-01-24 00:15:11 +01:00 |
DebaucheryLibrarian
|
cd8f690ac6
|
Preferring network over channel in deep URL channel matching.
|
2022-01-23 02:46:42 +01:00 |
DebaucheryLibrarian
|
c557f0f1f0
|
Improved Gamma image sources to prevent cropped images.
|
2022-01-22 23:20:08 +01:00 |
DebaucheryLibrarian
|
20da2d1cf6
|
Reusing batch ID for movies to preserve new-flag.
|
2022-01-20 00:54:10 +01:00 |
DebaucheryLibrarian
|
372db86927
|
Disabled MindGeek session bundling to analyze Too Many Requests errors.
|
2022-01-16 22:24:47 +01:00 |
DebaucheryLibrarian
|
57d46f5842
|
Fixed latest query in Gamma scraper.
|
2022-01-16 17:17:45 +01:00 |
DebaucheryLibrarian
|
bbd788699e
|
Fixed GraphQL query for scenes REST API. Added entry ID to scene search document.
|
2022-01-12 17:50:23 +01:00 |
DebaucheryLibrarian
|
506971b44b
|
Chunked duplicate check to prevent postgres stack depth errors.
|
2022-01-10 02:17:17 +01:00 |
DebaucheryLibrarian
|
baebdbc0bb
|
Adding comment listing sister sites for non-exclusive Gamma API scenes.
|
2022-01-07 01:07:41 +01:00 |
DebaucheryLibrarian
|
6760c11881
|
Disabled why-is-node-running to assess OOM.
|
2021-12-31 01:45:03 +01:00 |
DebaucheryLibrarian
|
3b958a02ef
|
Added memory profiler configuration options.
|
2021-12-27 22:36:01 +01:00 |
DebaucheryLibrarian
|
299f257a01
|
Logging memory usage on media fetch.
|
2021-12-27 17:29:23 +01:00 |
DebaucheryLibrarian
|
3bfe1e8fdb
|
Storing scenes before fetching movies.
|
2021-12-20 03:29:14 +01:00 |
DebaucheryLibrarian
|
7d992f4129
|
Added triggers to memory dump.
|
2021-12-20 02:28:59 +01:00 |
DebaucheryLibrarian
|
280032937f
|
Added centralized movie page fetcher. Added memory heap dump.
|
2021-12-20 02:22:10 +01:00 |
DebaucheryLibrarian
|
dc187a9a3a
|
Added execute method to qu, removed runScripts from Gamma's fetchMovie to observe effect on memory usage.
|
2021-12-19 23:03:44 +01:00 |
DebaucheryLibrarian
|
1c0c30be11
|
Improved and re-enabled why-is-node-running logger.
|
2021-12-18 23:44:51 +01:00 |
DebaucheryLibrarian
|
8e4be06675
|
Closing JSDOM window after deep scrapes in an attempt to save memory.
|
2021-12-18 23:22:35 +01:00 |
DebaucheryLibrarian
|
65e2b72c6a
|
Fixed Gamma movie scraper failing when data object is not available.
|
2021-12-13 01:20:02 +01:00 |
DebaucheryLibrarian
|
5b5688438c
|
Fixed Gamma movie scene query. Added tag photos.
|
2021-12-11 22:46:55 +01:00 |
DebaucheryLibrarian
|
ec4e7bc12a
|
Updated Jules Jordan scraper for Sperm Swallowers and The Ass Factory.
|
2021-12-05 23:59:17 +01:00 |
DebaucheryLibrarian
|
26d15c0a6f
|
Updated Jules Jordan scraper for Manuel Ferrara and Girl Girl.
|
2021-12-05 23:43:23 +01:00 |
DebaucheryLibrarian
|
e88554666c
|
Reduced media concurrency to assess effect on memory. Moved qu context removal to deep scrape runner. Updated movie graphql queries.
|
2021-12-05 02:54:55 +01:00 |
DebaucheryLibrarian
|
abd063a578
|
Removed window map from http module to prevent memory pile-up with update scraping.
|
2021-12-04 00:32:28 +01:00 |
DebaucheryLibrarian
|
1918dd4ad5
|
Removing query methods from XEmpire release.
|
2021-12-02 00:45:22 +01:00 |
DebaucheryLibrarian
|
61daf5f037
|
Added debug log for jsdom window closing.
|
2021-12-01 23:44:25 +01:00 |
DebaucheryLibrarian
|
60c151ff6a
|
Reduced deep scrape concurrency to 1.
|
2021-12-01 23:39:09 +01:00 |
DebaucheryLibrarian
|
e41f9fa937
|
Added waitImmediate to deep scrape, reduced concurrency.
|
2021-12-01 23:30:10 +01:00 |
DebaucheryLibrarian
|
e29cbc9fea
|
Closing JSDOM window after deep scrape in an attempt to save memory. Reduced deep scrape concurrency to 5.
|
2021-12-01 17:26:13 +01:00 |
DebaucheryLibrarian
|
b93a5715cb
|
Updated Jules Jordan scraper for new update page layout.
|
2021-12-01 00:00:24 +01:00 |
DebaucheryLibrarian
|
87094a9498
|
Replaced cheerio with qu in Gamma scraper.
|
2021-11-29 02:49:07 +01:00 |
DebaucheryLibrarian
|
f04eb72891
|
Added upcoming scraper to PurgatoryX.
|
2021-11-28 01:20:39 +01:00 |
DebaucheryLibrarian
|
9d7183ac69
|
Added PurgatoryX scraper.
|
2021-11-27 23:55:16 +01:00 |
DebaucheryLibrarian
|
20d0d860d3
|
Fixed MindGeek scraper trying to acquire session from mindgeek.com
|
2021-11-22 02:51:52 +01:00 |
DebaucheryLibrarian
|
6b4aa64d74
|
Improved MindGeek scraper session check to prevent crash when network session isn't available yet.
|
2021-11-22 02:44:03 +01:00 |
DebaucheryLibrarian
|
e81864ebde
|
Removed verbose http queued output.
|
2021-11-21 00:47:32 +01:00 |
DebaucheryLibrarian
|
0305a22a24
|
Removed stray console log from media module.
|
2021-11-21 00:43:35 +01:00 |
DebaucheryLibrarian
|
2be3ea9bbb
|
Explicitly adding headers to http module result.
|
2021-11-21 00:41:12 +01:00 |
DebaucheryLibrarian
|
98c103c625
|
Fixed uuid import. Changed fs rmdir to rm.
|
2021-11-21 00:31:09 +01:00 |
DebaucheryLibrarian
|
f0b7678444
|
Fixed upsert failing on empty insert array due breaking Knex API change.
|
2021-11-21 00:19:10 +01:00 |
DebaucheryLibrarian
|
26539b74a5
|
Updated dependencies. Added periodic memory logger.
|
2021-11-20 23:59:48 +01:00 |
DebaucheryLibrarian
|
ccb99e278c
|
Added periodic memory logger.
|
2021-11-20 23:59:15 +01:00 |
DebaucheryLibrarian
|
29b8c5e38e
|
Including unextracted scenes in date determination.
|
2021-10-28 02:10:30 +02:00 |
DebaucheryLibrarian
|
0864154a0e
|
Added unextracted property to keep paginating when extracting scenes.
|
2021-10-28 01:59:53 +02:00 |
DebaucheryLibrarian
|
a22c4d5679
|
Added beforeNetwork hook, used by MindGeek. Added Filthy Kings to Gamma.
|
2021-10-27 17:19:23 +02:00 |
DebaucheryLibrarian
|
100a35b4e8
|
Added before scene fetch method to prevent e.g. unnecessary session requests, moved scraper assignment to entity lookup. Removed channel URL hostname matching..
|
2021-10-26 23:42:32 +02:00 |
DebaucheryLibrarian
|
6c5d4389fe
|
Not parsing HTML with jsdom when using http module directly to save memory. Added loading ellipsis to release grid pages.
|
2021-10-25 02:06:24 +02:00 |