reddit-post-dump

A powerful utility to fetch almost all of a reddit user's content. It supports many image and video hosts and has offers extensive filenaming options.

Features

Most features are optional and can easily be disabled!

• Freely configure target paths with variables like username, formatted date, post title, subreddit, image ID, index in album and many more. All variables can be used to create directories
• Fetch any (reasonable) amount of users in one go, with their profile image and description
• Search various archives for posts no longer listed on reddit, but for which the source or preview is still available
• Save image descriptions or other variables as metadata (EXIF for JPEG only, broader support coming soon!)
• Avoid duplicates
• Extract single images from albums

Supported hosts

• Reddit text/self, images and videos*
• Imgur
• Gfycat
• Eroshare archive

Plans and ideas

• Avoid redownloading unless specified otherwise
• Support for more image hosts (e.g. vidble, erome)
• Watch-mode (keep the process running and automatically save new posts for specified users)
• Templates for text/self posts (use any variable inside text files)
• Support for subreddits
• Expand metadata support to PNGs, GIFs and videos
• Save additional details to an index file
• Only download non-default profile images (avoid standard avatars)

Installation

reddit-post-dump requires a arbitrarily recent version of Node.js. Before use, dependencies must be installed as follows:

npm install

Usage

npm start -- (--user <username> | --post <post-id>)

Optional arguments

• --users <username> [<username>...]: You may fetch posts from multiple users by supplying a space-separated list of usernames to --users.
• --posts <post-id> [<post-id>...]: Fetch multiple posts by supplying a space-separated list of post IDs to --posts.
• --limit <number>: Maximum amount posts per user to fetch content from. Limit is applied after filtering out ignored, cross- and reposts. Posts requested directly by ID may be discarded as duplicates, but are not otherwise affected by the limit.
• --sort <method>: How posts should be sorted while fetched. This affects the $postIndex variable, and in combination with a --limit decides what posts will be included.
• --ignore <prop> [<prop>...]: Ignore posts with any of the following properties: pinned, stickied, hidden, over_18, spoiler.
• --exclude <source> [<source>...]: Do not include posts from these sources (e.g. self, reddit, imgur, gfycat, ...). Should not be used in combination with --include.
• --include <source> [<source>...]: Only include posts from these sources (e.g. self, reddit, imgur, gfycat, ...). Should not be used in combination with --exclude.

Examples

• npm start -- --user AWildSketchAppeared
• npm start -- --users ShittyWatercolour AwildSketchAppeared --limit 10 --sort top
• npm start -- --user GallowBoob --limit 50 --ignore pinned stickied

Reddit videos

The audio stream for videos with sound uploaded to reddit directly (v.redd.it) is hosted separately, and will be saved as such alongside the video (typically {filename}-0 and {filename}-1). For them to be muxed into a single file automatically, ffmpeg must be available on the system, and the separate source files will be deleted when muxing has succeeded. If ffmpeg is not available, the separate files will remain as is.

Configuration

The default configuration aims to be sensible. However, a multitude of options make this utility particularly powerful.

To change the configuration, please refer to config/default.js. I recommend not editing this file directly, but instead making a copy config/local.js, as the default configuration might be overwritten in updates and can be a useful reference for restoring any detrimental configuration errors. The structure of config/local.js must match the structure of the default configuration, but does not necessarily need to contain any properties you do not wish to override. If preferred, you may instead use JSON in config/local.json.

API keys

Unfortunately, it is necessary to register for the reddit and imgur APIs for this application to work reliably. Example details have been provided in config/default.js, but must be overwritten (preferably in config/local.js). More information on registering APIs will become available in this section soon.

Library

Path patterns dictate where and how a file will be saved. Various variables and options are available, and you may use subdirectories divided by /.

Variables

Base

$base is an optional variable intended to set the beginning most paths have in common. The variable must be added to each path manually and is not prefixed automatically as to allow for exceptions.

User

• $user or $username: The nickname of the reddit user that submitted the post
• $userId: The ID of the reddit user that submitted the post
• $userCreated: The creation date or birthday of the reddit user, formatted according to dateFormat described below
• $userVerified (boolean): Whether the reddit user is verified
• $userVerifiedEmail (boolean): Whether the reddit user has verified their e-mail address
• $userGold (boolean): Whether the reddit user is a gold member of reddit

Profile

Many reddit users have a 'subreddit' of their own in the form of a profile (not to be confused with users that have created an actual subreddit for themselves). These variables are only available for users that have enabled this.

• $profileTitle: The title of the reddit user's profile
• $profileId: The ID of the reddit user's profile
• $profileDescription: The description of the reddit user's profile
• $profileOver18 (boolean): Whether the profile contains adult content and requires an 'over 18' age confirmation

Post

• $postId: The ID of the reddit post
• $postTitle: The title of the reddit post
• $postUser: The user that submitted the post, almost always equivalent to the --user command line argument
• $postDate: The submission date of the reddit post, formatted by the dateFormat configuration described below
• $postIndex: The index of the post according to the sort method
• $host: Name of the source the content was hosted on

Album

• $albumId: The ID of the media host album
• $albumTitle: The title of the media host album
• $albumDescription: The description of the media host album
• $albumDate: The submission date of the media host album, formatted by the dateFormat configuration described below

Item (individual image, video or text)

• $itemId: The ID of the individual image or video
• $itemTitle: The title of the individual image or video
• $itemDescription: The description of the individual image or video
• $itemDate: The submission date of the individual image or video, formatted by the dateFormat configuration described below
• $itemIndex: The index of the individual image or video in an album, offset by the indexOffset configuration described below
• $extracted (boolean): Whether the item has been extracted as the only item in an album
• $preview (boolean): Whether the image is a reddit preview because it was unavailable on the original host
• $ext: The extension of the medium. Must typically be included, but may be omitted for self (text) posts on Unix systems

booleans

Some variables are booleans and indicate whether or not a property applies. When you use a boolean variable, you must configure a string of text that is only inserted in place of a boolean variable when the variable is true.

booleans

Some variables are booleans and indicate whether or not a property applies. When you use a boolean variable, you must configure a string of text that is only inserted in place of a boolean variable when the variable is true.

dateFormat

Affects the representation of $postDate, $albumDate and $itemDate and defaults to YYYYMMDD. See this documentation for an overview of all available tokens.

titleLength

Titles can sometimes be longer than you prefer your filenames to be, or even overflow the operating system's limit (255 bytes for Linux). This property cuts off titles at a fixed number of characters.

indexOffset

Arrays start at 0, but as to not tire myself out debating the matter, you may offset it my any numerical value you like. Affects the $itemIndex variable for album items.

slashSubstitute

The patterns represent Unix file paths, and a / therefore indicates a new directory. You may freely use directories in your paths, but titles or descriptions may contain a / that is not supposed to create a new directory. All instances of / in a variable value will be replaced with the configured slash substitute.

album.extractSingleItem

Some albums contain only one image or video. By setting album.extractSingleItem to true (default), the item will be saved in accordance to the individual item patterns rather than the album patterns. An extracted item will inherit the title and description of the album if it has none of its own. Extracted items will have a truthy $extracted boolean variable.