Why do recommendation algorithms suck?

It's difficult to talk about a good or better algorithm without quantifying what that means. Youtube aren't trying to build a good algorithm. They're optimizing for a metric like time spent on the website. If your algorithm could beat their current solution in an A/B test, I'm sure that they would be up for it.

I’ve given up on content-based recommendation. The issue is that people’s interests in their topics and subtopics is primarily dictated by their individual circumstance and environmental real-world context. That is, if you are able to perfectly know the interests of an individual today, perfectly label the topic of all content, you’ll still give them incorrect content after a few days—-and with news article recommendations, by the next session of platform engagement (since they’ve sufficiently extracted the maximum value from the present topics by the nature of the session).

I remember when the Netflix prize was underway, there were certain movies—Napoleon Dynamite was one of them specifically—where it was incredibly difficult to determine if someone would enjoy it based on their other ratings. And yet it was also enormously popular, so lots of people ended up reviewing it.

Honestly, read up on the articles discussing the Netflix prize, they tended to mention a lot of unexpected gotchas. For instance, it turns out, ratings aren’t really independent. If you rate something good, it will affect how you rate the next thing. Stuff like that.

Definitely not an ML person perspective:

One type of recommendation system is user-user recommendation, so it is taking stuff you've (A) looked at, comparing it to what other's have seen (B), then trying to find what from A that B might like from weighting features of each possible item to find the item that person might like most (and therefore, do the action you want them to: view, buy, etc)

So if you view Video 1 (action), Video 3 (action) and Video 4 (comedy).

And someone else views Video 4 (comedy), Video 2 (comedy) and Video 3 (action).

If we are only using genre as a feature, the system would want to recommend you Video 2 (a comedy you haven't seen) and it would recommend the other user Video 1 (an action video they haven't seen).

Your system is introducing new features that could be useful.

In my opinion, the reason why recommendation systems suck is either people learning how to game it and therefore have their stuff recommended when their item shouldn't be, the system not having enough data or the system just not being great at figuring out what you like due to bad weights.

EDIT: Fully re-reading this post, it seems my post is a bit off-topic.

With that said, ML is not a dirty word or has to be some magical thing. It is applying stats to a large dataset. It's not anything fancy nor does it have to use the new shiny thing.

ML can be as simple as: go through this list of users and videos, given these weights, what videos would each user likely like most?

TikTok as an interesting take on recommendations compared to other platforms.

Instead of presenting a list of recommendations (like YouTube, Netflix or Instagram does), TikTok infers what it thinks you will like and directly shows it to you. Then based on how you react: if you like the video, follow the creator, watch it until the end, or swipe away, they know with much more accuracy the type of content you enjoy. This allows for more testing on their part too.

On the other hand, on platforms like YouTube, there are a lot of factors beyond the video itself that influences if a person even clicks on it (like the thumbnail, the title, the number of views, who the creator is), so if someone doesn't click on a video, that doesn't tell you if they would have liked it or not. Because of that their data to make recommendations isn't as accurate.

YouTube recommendations could be made enormously more useful with two trivial (really!) fixes:

SELECT * FROM videos WHERE NOT EXISTS (SELECT 1 FROM votes WHERE video_id = videos.id AND user_id = $user_id) AND NOT EXISTS (SELECT 1 FROM not_interested WHERE video_id = videos.id AND user_id = $user_id);

Aka don't recommend anything I've already voted on or already explicitly told you I'm not interested in. That would remove at least 98% of my recommendations.

Even without that improvement, though, YouTube is an amazing service, and the reliability is first class. I can't remember the last time it was actually down. Personally I suspect the YouTube team is just tasked with keeping things ticking over, and that they're too busy putting out fires and coping with tech debt to actually develop useful features.

It's more complicated to the point that Netflix offered a $1,000,000 prize for someone who could build a better one.

https://en.wikipedia.org/wiki/Netflix_Prize

I think there can be some additions:

1) for user to be able to manage those percentages directly or using various pre-made profiles and maybe allow some randomness mixed in;

and

2) "exploraton mode" where topics with content that is "disliked but not conflicting with the core values", "disliked but they may try to convince me here" and "initially neutral, unknown or not entirely irrelevant to them" are suggested.

I also think that attempting to extract the core values as hidden variables and using them to predict, or guess from oher users and external news, the future direction of their evolution can also be useful.

On a surface level I would like this better. It's not a black box, and seems like a decent spread.

My question would be how do you get all of your content classified in a reasonable way?

Another reason you didn't list is that it is surprisingly difficult to generate recommendation algorithms that don't empower hate speech. As an example we had to update the filters downstream of the main recommendation model dozens of times over the course of a week to stop Epstein and Pizzagate "researchers" from connecting with each other during the last election