Separated actor expand buttons. Refactored Brazzers scraper. Fixed actor releases not included in shallow scrape. Added number query and data-src default to qu img. Updated README. Removed post-install migrate and seed.

This commit is contained in:
2020-07-09 02:00:54 +02:00
parent 17d46e804e
commit 44a8ced30c
10 changed files with 289 additions and 326 deletions

View File

@@ -7,50 +7,45 @@ Use [nvm](https://github.com/creationix/nvm) to install a recent version of Node
`npm install`
### Set up database
Install PostgreSQL, make sure password authentication is enabled (scram-sha-256) and create a database with a privileged user.
Install PostgreSQL, make sure password authentication is enabled (scram-sha-256) and create a database with a privileged user. For optimal search engine performance, copy `traxxx.stop` to your PostgresQL text search directory, usually `/usr/share/postgresql/tsearch_data/ or `/usr/local/share/postgresql/tsearch_data/`.
For optimal search engine performance, copy `traxxx.stop` to your PostgresQL text search directory, usually `/usr/share/postgresql/tsearch_data/ or `/usr/local/share/postgresql/tsearch_data/`.
### Configuration
Do not modify `config/default.js`, but instead create a copy at `config/local.js` containing the properties you wish to change. If you have set `NODE_ENV`, copy `assets/js/config/default.js` to `assets/js/config/[environment].js`. After setting up PostgreSQL and configuring the details, run the following commands to create and populate the tables, and build the project:
`npm run migrate`
`npm run seed`
`npm start`
You can also use `npm run flush` to run both steps at once, and wipe the database completely later.
### Configuration
Do not modify `config/default.js`, but instead create a copy at `config/local.js` containing the properties you wish to change. If you have set `NODE_ENV`, copy `assets/js/config/default.js` to `assets/js/config/[environment].js`.
### Building
To build traxxx, run the following command:
### Options
`npm start -- --option value`
`npm run build`
To generate the thumbnails for logos and tag photos, run:
`npm run logos-thumbs`
`npm run tags-thumbs`
### Run
`./traxxx --option value` or `npm start -- --option value`
* `--server`: Run the web server
* `--fetch`: Fetch updates instead of running the webserver. Without further arguments, it will use the networks and sites defined in the configuration file.
* `--site [site ID]`: Fetch updates from a specific site. The site ID is typically the site name in lowercase and without cases or special characters. For example, Teens Like It Big is teenslikeitbig.
* `--network [network ID]`: Fetch updates from all sites of a specific network. The network ID is composed similarly to the site ID.
* `--after "[time]"`: Do not fetch scenes older than this. Example values are: `"1 month"`, `"2 weeks"`, `"3 years"`.
* `--scene [URL]`: Try to retrieve scene details from its official site or network URL.
* `--deep`: Follow each release link found running `--site` or `--network` and scrape it for more details. Enabled by default at the moment of writing; use `--no-deep` to only save information found on the overview pages.
* `--copy`: Try to copy relevant results to the clipboard. When used with `--scene`, it will copy the filename as defined in the config with all the details filled in.
* `--all`: Fetch updates from the channels and networks in the configuration file.
* `--channel [slug] [slug]`: Fetch updates from specific channels. The slug is the channel's name in lowercase and without cases or special characters. For example, Teens Like It Big is teenslikeitbig.
* `--network [slug] [slug]`: Fetch updates from all sites of a specific network. The network slug is composed similarly to the channel slug.
* `--after "[time]"`: Do not fetch scenes older than this period or date. Example values are: `"1 month"`, `"3 years"`, `"2019-01-01"`.
* `--scene [URL]`: Try to retrieve scene details from its official channel or network URL.
* `--deep`: Follow each release link found running `--channel` or `--network` and scrape it for more details. Enabled by default ; use `--no-deep` to only save information found on the overview pages.
## Developers
### Options
#### Developers
* `--no-save`: Do not store retrieved information in local database, forcing re-fetch.
* `--debug`: Show full error stack trace.
### Generating thumbnails
Ensure each tag or sfw category directory has a `thumbs` and `lazy` directory: `for dir in \*; do mkdir "$dir/thumbs $dir/lazy"; done`
Using ImageMagick's bulk tool `mogrify` to generate 240px thumbnails and 90px lazy pre-loading images:
* Generate thumbnails within tag or sfw directory: `mogrify -path lazy -resize x240 -quality 90% \*.jpeg`
* Generate lazy loading images within tag or sfw directory: `mogrify -path lazy -resize x90 -quality 90% \*.jpeg`
* Generate thumbnails for all tags or categories in `tags` or `sfw` directory: `for dir in \*; do mogrify -path "$dir/thumbs" -resize x240 -quality 90% "$dir/\*.jpeg"; done`
* Generate lazy loading images for all tags categories in `tags` or `sfw` directory: `for dir in \*; do mogrify -path "$dir/lazy" -resize x90 -quality 90% "$dir/\*.jpeg"; done`
* `--level`: Change log level to `silly`, `verbose`, `info`, `warn` or `error`.
## Supported networks & sites
768 sites on 62 networks, continuously expanding!
896 channels on 64 networks, continuously expanding!
* 21Naturals
* 21Sextreme
@@ -60,7 +55,6 @@ Using ImageMagick's bulk tool `mogrify` to generate 240px thumbnails and 90px la
* Assylum
* Aziani
* Babes
* BAM Visions
* Bang!
* Bang Bros
* Blowpass
@@ -75,24 +69,20 @@ Using ImageMagick's bulk tool `mogrify` to generate 240px thumbnails and 90px la
* Fame Digital
* Fantasy Massage
* Full Porn Network
* Gamma Entertainment
* Girlsway
* Hush
* Hush Pass
* Hussie Pass
* Insex
* Interracial Pass
* JayRock Productions
* Jesse Loads Monster Facials
* Jules Jordan
* Kelly Madison Media
* Kelly Madison Media (Teen Fidelity)
* Kink
* LegalPorno
* Men
* Metro HD
* Mike Adriano
* Mile High Media
* Mind Geek
* MOFOS
* Naughty America
* New Sensations
@@ -107,10 +97,11 @@ Using ImageMagick's bulk tool `mogrify` to generate 240px thumbnails and 90px la
* SCORE
* Sexy Hub
* Team Skeet
* Teen Core Club
* Twistys
* Vivid
* Vixen
* VogoV
* Whale Member
* Whale Member (Holed, POVD)
* Wicked
* XEmpire