The Big Programming Thread - Page 1015

On January 07 2020 22:56 Nesserev wrote:

Doing everything with selenium sounds like a massive pain and limiting your options; just use Google's Youtube Data API to check for new videos, download the videos, and then process the videos locally.

1. uses database to keep track of relevant channels and their videos
   
2. periodically scan channel for new videos
   
3. download videos
   
4. use visual/audio recognition library to determine timestamps
   

Btw, wouldn't it be better to search for the countdown signal? It's very distinct, and should be easy to recognize.

On January 08 2020 13:49 tofucake wrote:
Parsing websites is fucking awful. Ignoring the part where bad devs make everything 43 nested tables and good devs make things properly but every time a site is deployed every id and class name changes, it's fucking hard.

Parsing most sites isn't even usually worth it, as there are typically APIs to get the raw and specific data you want from the source, and if not, someone else has already written a scraper and made an API that'll give you that anyway.

Unless you want to get into something like adblock tech or the like, scraping is a path to the dark side of HTML.

On January 10 2020 07:34 WarSame wrote:
Say you have a table with 3 important columns for some event, Id, CreationDate, and EventDate. Let's say that there are multiple entries for one person at one event, and a new entry gets created every time their information changes. How can you use SQL to grab the most recent Id and EventDate from them for each EventDate?

Say their event is from 2019-05-03 to 2019-05-08 they would have 1 entry for each of those dates. I am trying to get the Id, and EventDate with a query without tons of subqueries/joins. Is there some efficient way of doing this that I'm missing?

SELECT Id, CustomerId, EventDate FROM events
WHERE CustomerId = someId
ORDER BY EventDate DESC
LIMIT 1

On January 11 2020 01:23 WarSame wrote:
It's not my DB and I can't redesign it but I have the same questions as y'all :D

I can tie the Event Snapshot(1 day of an event) to a customer, so that part I'm not worried about. What I'm trying to do is find the most recent updates to the event before a certain date cutoff. Say I'm looking before the end of March, I want to find the last Event Snapshot at the end of March. So I need to pull out the most recent Event Snapshot for that Event, but I need to do so for each date of the Event(so I can sum up revenues of the event, for example).

@Acrofales, thanks! Our DB on the backend is SQL Server but I'm trying to do this in SalesForce Developer Console(bleh) so it doesn't translate directly. I asked about SQL because I figured it was the most familiar to everyone but now I see the flaw in my plan. Do you know of a way without using any tailored SQL solution? i.e. system agnostic, using just joins and the like.

On January 10 2020 07:34 WarSame wrote:
Say you have a table with 3 important columns for some event, Id, CreationDate, and EventDate. Let's say that there are multiple entries for one person at one event, and a new entry gets created every time their information changes. How can you use SQL to grab the most recent Id and EventDate from them for each EventDate?

Say their event is from 2019-05-03 to 2019-05-08 they would have 1 entry for each of those dates. I am trying to get the Id, and EventDate with a query without tons of subqueries/joins. Is there some efficient way of doing this that I'm missing?

latest_creations = defaultdict(int)
latest_ids = defaultdict(int)
for e_id, creation_date, event_date in entries:
  latest = latest_creations[event_date]
  entry_time = convert_to_timestamp(creation_date)
  if latest < entry_time:
    latest_creations[event_date] = entry_time
    latest_ids[event_date] = e_id