I recently read some articles about designing feed system, the reason I am interested in it is because a feed system design involves a lot of knowledge in system design(NoSQL or relational, affinity ranking and scalability). This is also a popular question for technical interview.
Traditional design is pretty simple with one app sever and one database, while this is not scalable at all, maybe it works for small site, but what if a site grows large like Twitter and Facebook?
How to store feed for user
A feed system is special, because usually you will need to show a feed for a user, then you need to fetch all feeds a user liked, commented, mentioned and so on. A good way is to implement a message box for each user with redis, if the data growing vastly, we can further consider clustering with Cassandra
Push or Pull
When a user generated a post, do we push it to all of the user’s follower right away? or we only render it when one of the follower request to view the feed. This is a problem since celebrities’ posts always have a large number of followers, which pushing takes a lot of time.
For users who have a large number(it’s always need a way to figure out how large is large) of followers, we should apply pull model, and apply push model for others.
How to display the feed
A general way is to display the feed by timestamp, but for a system that cares about user experiences, we also need to take the affinities between user and followers, and the type of post into consideration.
Basically, after we get the feed, we need an algorithm to rank the feeds, there are 3 features used here to ranking:
- Affinity score: For each feed, affinity score evaluates how close you are with this user. For example, you are more likely to care about feed from your close friends instead of someone you just met once.
- Edge weight. Edge weight basically reflects importance of each edge. For instance, comments are worth more than likes.
- Time decay. The older the story, the less likely users find it interesting.
References:
http://blog.gainlo.co/index.php/2016/04/05/design-news-feed-system-part-2/
https://thenewstack.io/building-scalable-news-feed-applications-using-redis-and-cassandra/
https://www.slideshare.net/danmckinley/etsy-activity-feeds-architecture/