Rescraping an actor does not update the updated_at #38

New Issue

Ghost · 2021-02-13T01:14:04Z

Ghost commented

2021-02-13 01:14:04 +00:00

Running the following I would expect that it would update any actors that do not have information from any scraper that is newer than the 1st of Feb 2021.

npm start -- --force --actors-update "2021-02-01"

The SQL was incorrect as it should be >= and not <=, also exclude any actors with an alias for or it will fail as it tries to create a duplicate name.

 SELECT actors.name
	FROM actors
	WHERE NOT EXISTS (
		SELECT *
		FROM actors_profiles
		WHERE actors_profiles.actor_id = actors.id
		AND actors_profiles.updated_at >= (?)
	) AND alias_for IS NULL

The following shoudl be anything with that does not have a profile new that today so everything?

npm start -- --force --actors-update

but it uses new Date(0, 0, 0) set in interpretAfter in argv.js, and the following was not being used to set it to the current date.


argv.actorsUpdate || new Date()

curateProfileEntry should also set this value if it is an update, so that the profile knows that this is the correct data at the current date.

if (profile.update) curatedProfileEntry.updated_at = new Date().toDateString();

Running the following I would expect that it would update any actors that do not have information from any scraper that is newer than the 1st of Feb 2021. npm start -- --force --actors-update "2021-02-01" The SQL was incorrect as it should be >= and not <=, also exclude any actors with an alias for or it will fail as it tries to create a duplicate name. ``` SELECT actors.name FROM actors WHERE NOT EXISTS ( SELECT * FROM actors_profiles WHERE actors_profiles.actor_id = actors.id AND actors_profiles.updated_at >= (?) ) AND alias_for IS NULL ``` The following shoudl be anything with that does not have a profile new that today so everything? npm start -- --force --actors-update but it uses new Date(0, 0, 0) set in interpretAfter in argv.js, and the following was not being used to set it to the current date. ``` argv.actorsUpdate || new Date() ``` curateProfileEntry should also set this value if it is an update, so that the profile knows that this is the correct data at the current date. `if (profile.update) curatedProfileEntry.updated_at = new Date().toDateString();`

pendulum changed title from ~~a rescrape of an actor does not update the updated_at value~~ to Rescraping an actor does not update the updated_at value

2021-02-25 01:54:44 +00:00

pendulum changed title from ~~Rescraping an actor does not update the updated_at value~~ to Rescraping an actor does not update the updated_at

2021-02-25 01:54:50 +00:00

Sign in to join this conversation.