Saving INR 2Cr annually with Internal Tools

19 min readApr 13, 2024

Automating workflows and with Internal Tools

Comprehensive Impact

Scaler’s Course Pause feature started generating INR 2Cr in business revenue each month.
We ended up saving the engineering team’s 8 Story Points each month that used to go in manually changing batches, pausing, or resetting courses.
A staggering 98% reduction in the number of support tickets in this bucket.
Activated 20% of churned users with Course Reset and Roadmap Plan.

🥅 Context & Objectives:

Our support team was getting 700 support tickets per day — quite a high amount. So, our goal with these initiatives was to reduce the number and frequency of support requests by adding new features to the product (specifically the dashboard) to make the overall experience more user-friendly.

We had two objectives:

Reducing the number of support requests per day by automating certain things that currently require the intervention of the OPS team so that users can carry out these tasks themselves without long wait times.
Reducing the turnaround time on support requests, that would eventually lead to better customer satisfaction and a reduction in the detractor percentage during NPS feedback, as a majority of detractors had stated that they were heavily dissatisfied with the customer support.

📈 Data & Analysis

We were provided raw data on all support ticket concerns and the categories they fell into so we could figure out which features were most problematic.

Distribution of Support Tickets (Paid & Free Product)

Through this data, we identified the categories that most support tickets fell into, specifically:

Most commonly requested support queries

Apart from this, customers who cited bad customer support as the reason for their negative feedback made up the largest percentage (21.46%) of NPS (Net Promoter Score) detractors.

The major issue here was the customer support team having to handle each query manually and separately, resulting in high turnaround times.

🎧 Handling of support queries

Process of resolution of support tickets

👀 Narrowing down the problem:

After studying and discussing the distribution of support tickets for a week, we decided to focus on the following features that formed most support queries.

Phase 1: Batch Shift/SAT Reset (Quick Fix) — Allowing students who want to shift batches post/pre-SAT or re-attempt SAT to do this themselves rather than reaching out to support.
Phase 2: Revamped Course Pause + Reset — Reworking the Course Resume feature to incorporate various support requests that can be productized.
Namely, introducing ‘Back-Dated Course Pause’ and ‘Course Reset’ features which let mentees restart or resume their course from a past lecture if they want to.
Phase 3: Batch Change Tool — Building a single-point solution for most batch change-related issues. The sub-buckets could consist of the time change, day change, language change, level change, pause & reset.
Phase 4: Self-Serve Tool — Automating responses from our end to give the most relevant solutions to a user with a common query to reduce the load on live chat (preferably with ChatGPT).

Out of these, I was fully responsible for Phase 1 and Phase 3.

🔂 Phase 1️: Batch Shift/SAT Reset (Quick Fix)

🏆🚨⏳ Impact created (as of 02/04/2023):

No. of people who reset their SAT: 225
Time reduced for the process: 12–18 hours to 1 minute.
Reduction in support tickets: 98% in the first iteration (100% in the second iteration)

🧪 What is SAT?

SAT stands for Scaler Assessment Test — a mandatory test to check the skill level of a learner and allot them a batch according to their skill set.

This segregation of mentees into batches is done as follows:

Beginner — People who are not familiar with coding
Intermediate — People with a basic knowledge of coding
Advanced — People who are confident in their coding proficiency

Sometimes, due to miscellaneous issues on the learners’ end (loss of electricity or having to quit the test midway), students on an Advanced level would get allotted the Beginner batch.

Post this, they would connect with support to request that they be allowed to retake the exam as they were not satisfied with their current batch. In most cases, they were given a reset link.

The Project Manager and I brainstormed and decided to let users are-attempt the SAT on their own but limit the use of the new ‘SAT Reset’ or ‘SAT Re-Attempts’ features to 2 times. Learners were now able to attempt the assessment total 3 times without having to contact support.

This basic feature addition was only a quick fix, but its introduction led to a greater than 98% reduction in support tickets and helped 162 people reset their SAT within a minute. This simple process would previously take 12–18 hours, hence saving over 3,600 resolution hours.

⏸️ Phase 2: Introduction of paid pauses and designing the course ‘reset’ feature.

🏆💰 🚨 Impact created (as of 25/05/2023–4 months):

Revenue generated: INR 90,62,100
Months of paid pauses taken: 3,270
Total paid course pause users: 1519
Total course reset taken: 1651
Total course reset duration: 3,673

⏸️ Course Pause & Paid Pause Extension

What is Course Pause?

Course pause is a feature where we allow learners to pause and take a break from their course for up to 3 months. The free version had been implemented before I joined Scaler.

These pauses work only in 1 month intervals. So, when a learner resumes their course, they will do so in the next month’s batch with the same syllabus.

Why did we introduce Paid Extension?

We receive ~100 queries per month wherein learners have exhausted their course pause and request an extension.

While we allow 1 more month in case of emergencies, ~50% requests have to be denied. Even in case of emergencies where more than 1 month is required, the case has to be escalated to leadership, which is not feasible in the long run.

Analyzing the current data

Data suggested at least 15% of people paused their course within 3 months and 36% within 3–6 months of starting it.
Of the 4,263 individual pauses analyzed, only 2,631 were unique (61.71%), meaning ~1200 learners paused their course more than once.
~100 people used 2 course pauses within a 3-month time frame.
An analysis of Problem Solving Percentage (PSP) also indicated that learners’ PSP increased upon resuming their course after a pause.

Solutions & Brainstorming:

Considering the data, we decided to introduce longer pauses in case a learner needs them after exhausting their 3 free months.
We priced them as follows:

This addition allows users to take an extended pause with a nominal fee compared to the actual course fees.

When introducing learners to this feature, we laid out also our reason for charging to pause a course for an extended time:

Learning is the only way for mentees to get desired outcomes. We at Scaler want to make sure it happens. To make sure that the pause feature is used more responsibly and sincerely, we introduced paid extensions to pauses/resets at a nominal price. To know more — Read FAQs
For cases where people are unable to finish their course due to immediate emergencies or want to redo a set of lectures, we introduced a ‘back- dated’ course pause feature and called it ‘Course Reset’

🔁 Course Reset:

Why Course Reset?

A significant number of learners become inactive in the middle of their course due to unforeseen circumstances like increases in workload, medical emergencies, etc.

Once the backlog of lectures and coursework starts piling up, it becomes difficult for these learners to understand future lectures and solve assignments.

This cycle continues till the learner gets too demotivated to get back on track and eventually gives up on their learning journey. The data analysed for the Reset feature was the same as that for Course Pause as the ability to Reset did not exist previously.

Analyzing the current data

Around 4,376 learners who hadn’t finished their course accessed the dashboard between July 1st and October 25th, but hadn’t used it for more than 15 days. This group represented about 20% of ~20,000 learners currently taking classes.
We inferred that 20% of learners were thus inactive — an alarmingly high amount.

Validation from users

The above learners were sent an email to express their interest in resuming live classes from where they left off.
Assuming an industry-standard 25% email opening rate, we can guess that 1,094 learners read the email. Of them, 385 responded positively to the feature, suggesting a 35.19% interest from the target audience.

How would the reset feature work?

We allow learners to reset their course to an earlier point in their journey via the dashboard.
Learners can pick a favorable point to resume their journey by paying a nominal fee in comparison to the course fees.
We were to enable the feature for both Academy and DSML after refund period.
We display CTAs on the learner’s dashboard when they have been inactive for >= 1 month. [Inactive: Having < 10% attendance in live classes]
Learners can reset to any previous point in the course within the past 6 months.
We limit the maximum reset duration to 6 months. Our analysis suggested that the edge case for reset requests was after 5 months of inactivity while the mode was 3 months. Hence, 6 months would cater to virtually all scenarios.
We limit the minimum reset duration to 1 month so the learner can be placed in that month’s batch.

With policies and ideas in place, we were quick with UI deliverables. I actively participated in the discussion and planning phase of this feature but was only involved in coming up with the base idea and elements for its UI since I was also responsible for the Batch Change Tool, which would be scalable in addition to being consistent with this feature’s UI.

Once built, the Batch Change Tool would house the Course Pause and Reset features.

I will talk about these UI changes in Phase 3 — Batch Change Tool.

First Rework of Pause/Reset UI

Phase 3: Batch Management Tool with Updated Course Pause/Reset UI

🏆💡⏳🚨 Impact created (as of 25/05/23– 1.5 months):

No. of batch shifts done: 937 batch shifts
Time reduction for batch shift: 3–4 Days to 2 minutes (~100%)
Support time reduction: 10 Minutes to 2 minutes (Templated responses sent informing about BCT)
Reduction of support tickets: 54% with just organic discovery (without feature release announcements etc)

What were our goals?

With this tool, we aimed to reduce the number of support requests (about 700 per day) related to batch shifts, pauses and resets by over 90%.

How did we start?

We started by analyzing more than 400 support tickets receives over 30 days tagged with ‘batch change’. Once we were done with our analysis, we grouped these requests under 6 sub-buckets representing the majority of users’ concerns.

Analysis of Support tickets

We mapped out how repeated requests under the previously divided sub-buckets were resolved (accepted or rejected, along with exceptional cases).

Then, we correlated this data with the points in a learner’s journey where students generally raised each category of tickets by mapping them as follows.

Mapping Data

With data mapping in place, we scheduled a call with the OPS team to understand their perspective and give them some context on what we were building.

The summary of the 90-minute call is as follows (thanks ChatGPT):

- Learners must take the SAT to be allocated a batch
- Those who don’t attempt the SAT are automatically assigned to a Beginner batch
- Learners can select their day and time preferences within the first week after giving the SAT, and those who don’t are automatically assigned to a random batch on the level assessed by the SAT
- Each batch requires a time commitment of 10/12 hours per week
- Learners can switch batches only through a request to the OPS team
- Learners can pause their course at any time, but they can only resume under certain conditions (such as being assigned the next month’s batch)
- AMAs are conducted to allocate specializations
- A LinkTree link to FAQs and information is shared with people who need information
- Learners are moved to analytics if they don’t take the specialization test
- If learners don’t like their teacher, they can pause the course if it is an individual issue or the batch will be sent out feedback forms if it is a collective issue
- Learners who fail a module are assigned to a refresher batch for just revision
- Once learners pass the test, they are added to a new batch starting with the next module

Equipped with a far better understanding of how the overall learner journey functions and at what point requests are raised, we were ready to brainstorm ideas.

Brainstorming & Ideation

We brainstormed solutions to some specific questions:

How do we deal with each category of tickets?
How much of an effort would it be to manually resolve this category compared to the time it would take to code a solution?
What are the probable loopholes and edge cases of each feature?
At what points could the feature’s flow break?
Where on the dashboard can we include entry points for BCT?

Proposed Flows

After intense brainstorming, we came up with a list of rough ideas for dealing with and automating each category of support tickets.

Writing down the proposed flows

In the end, we narrowed down the following flows:

Reaching the BCT
Pausing a course
Time/day or time+day change
Handling situations where the requested time/day batch is unavailable
Language shift — within 7 days of starting the course
Language shift — post the 7-day period
Specialization shift — within 14 days post specialization test (needs further understanding, awaiting response)
High-level to low-level & low-level to high-level batch change — for Advanced learners
High-level to low level & low-level to high-level batch change — for Intermediate learners
High-level to low level & low-level to high-level batch change — for Beginner learners
High-level to low level & low-level to high-level batch change — for Beginner learners who have run out of SAT resets

We considered separating the day shift and time shift flows, but it made more sense to club them together for increased user-friendliness and also saved real estate.

Wireframing Different Approaches

Approach 1: Clubbing options into discrete categories listed within the ‘Batch Change Tool’ so users can directly choose their desired action themselves.

Approach 2: Asking users about their desired change with a dropdown menu of predetermined issues identified by us and giving probable solutions using expandable and scrollable cards

Initially, we began work in Approach 2, where the user could choose their problem and relevant solutions would come up. However, it ended up being too complicated for an MVP as we didn’t have the time to devise conditions that would cover the many diverse problems users could face.

So, to save time, we ultimately settled on Approach 1, which would clearly list the actions the user could take along with helpful headings, icons, and descriptions.

User Testing With UI Iteration V1 — Validation testing

When requested batches are available vs unavailable

We tested UI flows based on Approach 1 with 8–10 users consisting of 2 personas —

Users who had previously requested a batch change and could help us understand how the Batch Change Tool compared to manually requesting a change
Users who had not previously requested a batch change.

Interview Questions and Insights

How would you typically go about managing your course on Scaler Academy, such as taking a course pause or changing your schedule?
We recently added a batch changing feature to Scaler Academy. Can you walk me through how you would use this feature and what you would expect to happen?
If the name of your batch was removed from the dashboard, do you think it would cause any confusion or make it harder for you to find the information you need?
Can you tell me what your understanding is of the options available on the dashboard, such as schedule, level, pause, specialization, and language? How do you think these options would be helpful to you?
We asked users to try out the schedule and level flows on Scaler Academy. Did you find these flows easy to understand and navigate, or did you encounter any confusion or difficulties?
Do you have any specific feedback or suggestions for improving the Scaler Academy dashboard or user experience?
Are there any features or functionalities that you would like to see added to Scaler Academy in the future to make it more helpful or useful for you?

Interview Insights

All 10 users representing both personas tested our wireframes to help us gauge if they could navigate and understand the new feature and if they would use it when they needed to.

To summarize our insights:

The primary action for shifting/managing batches was raising a support ticket, but some users felt that it would be within the Manage Course feature.
Most users prefer having batch names on the dashboard but won’t be bothered if it is removed.
Users showed varying levels of understanding about the five options on the dashboard, with some confusion about batch level and course level.
Change Schedule Flow — Overall, users read the warnings and restrictions and understood that they may need to watch lecture recordings, but found the feature a bit difficult to grasp at first.
Change Batch Type Flow — Users were initially confused about the feature’s purpose, with some users being unaware of multiple course levels and unsure if they were moving up or down.
Some users clicked CTAs to proceed further without reading the details.
Other feedback included suggestions for additional FAQs, difficulty understanding the notes feature, poor visibility of the search option, and a desire for real-time support.
The users liked that the feature will be coming soon and would have loved it if the feature was already present, while we had earlier assumed that batch changes would not happen very frequently

From from these insights, we inferred that the users were not able to intuitively figure out the flow of the Batch Change Tool because of so much information being presented all at once.

Armed with productive feedback, we proceeded to make changes to the UI.

Final UI

Our goals with this third iteration were to:

Reduce visual clutter.
Improve communication of info for every step and card.
Ensure users are aware of the implications of their actions by introducing checks at a couple of stages.
Build it with components and auto layouts as it would be (close to) the final version.
Ensure that it flows with and resonates with the new mentee dashboard designs (most recently revamped section) by incorporating those values (stroke, colors) were incorporated here as well.
Show unavailable/unselected sections with the introduction of the Panda.
Keep the upcoming incorporation of Course Pause and Course Reset features into the BCT in terms of UI.

New improvements from the previous iteration

Pause & Reset features

Quick fix feature introduction — Iteration 1

The Course Pause UI I worked on — Iteration 2 — Unexpanded

The Course Pause UI I worked on — Expanded

Changes compared to 1st Iteration

Since I was working on the overall Batch Change tool, I was also closely involved with the Pause and Reset flows so I could understand how they functioned and fix any issues they might have. After all, my designs were Revamp V2!

I actively continuously identified flow problems and logic problems and kept fixing them in my own designs. As a result, when the feedback for Revamp V1 was in, I was easily able to provide solutions for the same :D

Things I improved were:

Improving the feature descriptions

Better reasoning (click to expand)

Adding a table to explain the pricing and credit availability on the feature landing page

Pause/Reset Pricing Policy

Laying out how the pause/reset feature works and what steps the user needs to follow to use it successfully

How pause/reset works

Explaining why we charge a fee for an extended course pause

Reason for charging a small fee

Changing the sidebar approach to input details to a modal approach with discrete steps and clear messaging

Insights post 2nd Usability test:

We tested the new prototypes with 6 users — 3 who had seen the feature earlier and 3 who had shown interest in Course Reset but hadn’t seen it earlier. As such, we gained some useful insights:

Users liked the flow was going and remarked that it was easier to navigate.
Users really liked the introduction of the Reset feature as they appreciated being able to go back and reset their course and lectures to a previous stage.
Users were curious about the pricing — some thought it was reasonable while others thought that they should have to pay more after already paying so much for the course.
Overall, the UI felt intuitive to the new set of users and they were not confused about the messaging and information anymore.

Final Final 2 UI

After a discussion with the tech team, we learned that introducing the ability to choose a lecture was feasible but would take some effort. So, for V0, we decided to show just the current schedule of the user’s current batch and that of the batch the mentee might be interested in.

We implemented only the Change Schedule and Pause/Reset functionalities for this initial release.

We also added a text input box to the popup so we could the reasons why people are requesting batch changes and study how the feature performs post-release.

Lessons Learned

There are all kinds of users out there, and some might surprise you!
Users may often frantically click everything that looks clickable, so you should do user testing with a fully clickable prototype.
Keeping every stakeholder in the loop about the project and proper documentation is incredibly important. It was very easy for me to write this case study as everything was already documented.
While you should be proud of your designs, you shouldn’t get attached to them. Product design is all about embracing change!
If a feature is working and stable, you don’t always need to mess with it or improve it right then.

Future Scope

Allowing learners to choose the lecture at which they will shift to the new batch.
Making a Pause/Reset flow where users don’t have to choose if they want to pause or reset. They can simply choose a reason from a predetermined drop-down menu (which we have been collecting since release) and will be given an option accordingly.