How to Measure Employee Productivity Without Micromanaging
Jul 15, 2025 in Guide: How-to
Learn how to measure employee productivity with methods that build trust. Discover modern metrics, tools, and strategies to foster growth, not fear.
Not a member? Sign up now
Ensuring all users and objects get exposure
Paulo Maia on May 25, 2022
In this blog post, you’ll learn about some examples of decision processes you can use in recommender systems: do you see any usage for recommending less popular products as a way to improve your business? You will see it now!
Let’s imagine a use case where you are building a MOOC platform (like Udemy/Coursera). Your CEO means to increase student engagement and recurrence in the platform, and wants to know the preferred courses for each student.
Your team of Data Scientists, after brainstorming, decides you should build a model which predicts the affinity each user has for each course, and build a decision process on top of it. Your marketing team is able to send a fixed amount of newsletters per week with limited slots, of the best courses for each user, at no associated cost.
MOOCs websites, being online services, can easily generate a lot of user data with no major acquisition effort. For this example, the most relevant characteristics are:
The focus of this blog post is not to explore data sources, so let’s assume we have a model that takes in this data.
There are three major groups of models for recommendation systems: content-based, collaborative filtering, and hybrid. At NILG.AI, we have built content-based algorithms for a lot of our clients, as they allow you to compare products by their characteristics, reducing cold start in new products.
For more details when comparing these types of models, we recommend an in-depth article such as this one.
Now towards the main topic of the blog post: how do you go from a list of affinities per user to an actual recommendation in the newsletters? There are multiple options for creating the final newsletter:
There is no actual best rule for this: one approach we have followed in the past was to run a hybrid approach, where you combine sections of the newsletter using different strategies. You need to test out these proportions in different AB Tests.
So… it does make sense that you can end up recommending less popular products to improve your business: you are giving exposure to less popular courses, preventing them from ending up in a feedback loop. If the courses are rarely recommended, they don’t get a lot of sales/clicks. If they don’t get a lot of sales/clicks, they are rarely recommended!
To measure if the model is doing well, besides the technical metrics (Lift, ROC AUC, Average Precision, …) there are some business KPIs you can measure. These KPIs, for these types of businesses, are typically based on positive feedback signals, such as click rates and sales. Some are more short-term but lead to a less relevant intent signal (such as clicks).
For instance, you can focus on:
Take into consideration that, while some KPIs might increase, others might decrease: for instance, you can dramatically increase course exposure rate but decrease the overall experience: this would happen if you randomly recommend a course, for instance.
Using strategies such as Student to Course recommendations, you should be improving your business-centric KPIs and decreasing your user-centric KPIs, when compared to a baseline. On the other hand, Course to Student is an approach that is more user-centric. There’s always a trade-off you need to consider and discuss with the business teams.
Regarding AB Testing Strategies: you will need to experiment with the proportion type of each strategy if you want to use the hybrid strategy as mentioned above. The number of groups should be dependent on the number of active users you have and the time you want to run these experiments for.
Note that in many cases, there’s a budget associated with course recommendation (for instance, you can only recommend N courses per week, due to a maximum number of sales, caused by digital license issues). In this case, AB Testing can lead to unreliable conclusions due to cannibalization bias: the recommendations in one strategy are influencing the others, breaking the rules of standard AB Testing. We’ll talk more about this later in another blog post!
There’s a lot to be done after having a prediction of how much a student likes a certain product. It’s up to the Data Scientists to help define these constraints, together with the rest of the team. With a huge diversity of approaches to follow, there’s really not a “best solution”. You should quickly iterate to generate value, while continuously running R&D activities and monitoring the performance of the current approach.
Remember that before jumping into the complexities of building personalized recommendations, it’s equally crucial to have a solid eCommerce. We recommend our partner Pipecodes in case you’re looking to improve your eCommerce.
Is this the solution you need? If so, contact us. Otherwise, contact us anyway, and let’s find a better solution together!
Like this story?
Special offers, latest news and quality content in your inbox.
Jul 15, 2025 in Guide: How-to
Learn how to measure employee productivity with methods that build trust. Discover modern metrics, tools, and strategies to foster growth, not fear.
Jul 14, 2025 in Use Case
Learn how to automate repetitive tasks efficiently with practical tips and tools. Boost productivity and free up time for what truly matters.
Jul 14, 2025 in Guide: How-to
Discover how to develop a data integration strategy that unifies your data, boosts efficiency, and drives growth. Learn practical steps today!
Cookie | Duration | Description |
---|---|---|
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |