Automating the advising process using a chatbot

Student Helper Bot is a closed domain retrieval based chatbot built using Rasa framework. Rasa is an open source machine learning framework for building contextual AI assistants and chatbots. I have divided the whole process in 5 sections.

Note: This is my Master’s thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Information Technology.

1. Current problems with the advising process
2. Solution – “Use a Chatbot”
3. Data collection and storage
4. Rasa Framework
   • Natural Language Understanding (NLU)
   • Dialogue Management
5. Testing and improving the chatbot using Rasa X
6. Deployment

---

Current problems with the advising process

Usually, the objective of the advising function is to improve students’ understanding of their academic career as they undertake their chosen programs in the university or college. Often, the ratio of students to academic course adviser in any institution could grow as high in extreme cases requiring more human resources.

• Face-to-face advising is getting harder for students as the number of advisors shrinks and enrollment increases.
• The current advising process requires you to pick a time from office hours of your advisor or schedule an appointment with your advisor. The time slot might not be flexible to your schedule which causes a change in priorities.
• Students must wait for appointments during office hours and it can stretch for weeks.

---

Solution – “Use a Chatbot”

I designed a chatbot to solve the current problems with the advising process.

• Students do not need to schedule an appointment with their advisor.
• Students can access the chatbot from anywhere and at any time.
• Students can get all the information about courses and faculties in one place.
• During advising period, students can select classes and get pin.
• Advisors can use the chatbot to check if their students have selected classes or not.
• During registration period, students can use the pin to register for classes.

---

Data collection and storage

1. Faculty data - It consists of the name, title, department, location, phone number, email, and website which is collected from CS department website through web scraping. I used requests library for making HTTP requests and beautifulsoup4 library for pulling data out of HTML, navigating, searching, and extracting useful information. Then, I created a new SQLite database and inserted all the collected information into a table using the sqlite3 library. SQLite is a lightweight disk-based database that does not require a separate server. A table is created for advisors which consists of name and email. A JSON file is created for each advisor which stores information of their students like student id, name, status, course no, course name, and pin. The bot will fetch this information when the advisor asks for it.

2. Student data - I created a database in SQLite for students. Student’s names and emails are created using a dataset found on Kaggle. Information like student id, advisor, status, hold, and the pin is randomly generated and assigned and all the information is then inserted into a table. For each student, an individual table is created which consists of data about courses that they might have taken in the past i.e. course no, course name, and grade. These tables are used when students select classes and the bot can check if the student has taken the required prerequisite or not.

3. Courses data - It consists of course, title, days & time, dates, room, credits, instructor, cross-listed courses, prerequisites, and link to the course which is collected from the following website using Selenium and Web Scraping. Selenium is a browser automation tool. Using selenium, I was able to collect data about courses for the required semester i.e. fall, spring, summer. All the collected data is then stored in a JSON file using the json library in Python. There is a reason why I used dictionaries to store courses data. Looking up a value in a list takes O(n) time because the whole list needs to be iterated through until the value is found. Looking up a key in a dictionary takes O(1) time because it is a hash table. This makes a huge difference in the response time of the chatbot.

---

Rasa Framework

Rasa open source consists of two main components:

• NLU - an open-source natural language processing tool for intent classification and entity extraction.
• Core - a framework for machine learning-based, contextual decision making and dialogue management.

Natural Language Understanding (NLU)

NLU helps assistants to handle unstructured inputs that are governed by poorly defined and flexible rules and convert them into structured form that an assistant can understand and act upon. To build assistants that understand natural language, we must distill speech into a structured ontology using a combination of rules, statistical modeling, or other techniques. Entities must be extracted, identified, and resolved, and semantic meaning must be derived within context, and be used for identifying intents.

Creating NLU training data

The development of an AI assistant should start with a conversation design. Conversation design is a process of asking yourself who your users are, understanding what the users will use the assistant for, and crafting the most typical conversations they are likely to have with the assistant. It is an important step for understanding the purpose and usability of the assistant and will make the process of creating the training data easier and more efficient. In my case, I was familiar with the advising process. The advising process is where a student selects classes of their choice, book an appointment with their advisor, and then meet their respective advisor to get a pin. Then students use the pin to register. I have created following intents:

Note: For simplicity, I have only provided examples of few intents. The actual NLU files contain more intents and a lot of conversations/utterances for each intent.

• contact_request - Students can ask for the phone number, email, website, title, location, and contact information of a professor.

## intent:contact_request
- I need [phone number](phone) of Professor [Yusuf](pname)
- I want [phone no](phone)
- What is [email](email) of Professor [Chad](pname)
- I need [email address](email) of Professor [Sixia](pname)
- I need [website](website) of Professor [Williams](pname:Chad)
- What is [website](website) of Professor [Fatemeh](pname)
- What is the [title](p_title) of Professor [Chad](pname)?
- [position](p_title) of Professor [Irena](pname)?
- Where can I [find](p_location) Professor [Neli](pname) in CCSU?
- Where is the [office](p_location) of Professor [Abdollahzadeh](pname:Fatemeh)?
- I want to [reach out](contact) to Professor [Stan Kurkovsky](pname:Stan)
- I need to [contact](contact) Professor [Irena](pname)

      Here () represents an entity and [] represents entity value.

- I want to [reach out](contact) to Professor [Stan Kurkovsky](pname:Stan)

      A user might call a professor by the first name, last name or full name. To handle such cases, I       have created synonyms for each professor. For example,

## synonym:Stan
- Kurkovsky
- Stan Kurkovsky
- Kurkovsky Stan

• course_information - For a course, students can ask for the following information: room, days and time, instructor, prerequisite, title, credit, info, dates, cross-listed, courses, and all_courses.

## intent:course_information
- What is the [room no](room) for [CS 550](course_no)
- [Room number](room) for [CS 460](course_no)
- What [days](dandt) is [CS 505](course_no) taught?
- [Days and time](dandt) of [CS 460](course_no)
- [Instructor](instructor) name of [CS 113](course_no), [CS 500](course_no)
- How many [instructor](instructor) teach [CS 501](course_no)?
- What are the [prerequisite](prerequisite) of [CS 550](course_no)?
- [prereq](prerequisite) [CS 501](course_no)
- [Title](title) of [CS 502](course_no)
- Tell me the [Title](title) of [CS 500](course_no) and [CS 550](course_no)
- [Credits](credit) of [CS 530](course_no)
- How many [credit](credit) is [CS 550](course_no)?
- Give me [information](info) about [CS 407-1](course_no) and [CS 407-2](course_no)
- What is this [CS 354](course_no) [about](info)?
- What is the [date](dates) for [CS 550](course_no)?
- [Dates](dates) [CS 460](course_no)
- What are the [cross-listed](cross-listed) courses for [CS 460](course_no)?
- [CS 500](course_no) [cross listed](cross-listed)
- Which [courses](courses) are taught by Professor [Neli](pname)?
- [courses](courses) taught by [Chad](pname)
- List of [all courses](all_courses) offered in this semester
- [List of courses](all_courses) offered in Spring

• selection - Selection is the process where a student tells the chatbot their chosen courses and the bot will check for the prerequisites for each course and reply to the user their pin.

## intent:selection
- I need to select courses
- Select classes [CS 253](course_no), [CS 254](course_no) and [CS 291](course_no)
- Select courses
- I need pin for registration
- Help me get my pin
- Can you give me my pin?
- I need to pick courses
- Can I pick [CS 410](course_no), [CS 500](course_no) and [CS 501](course_no)
- My selected courses are [CS 550](course_no), [CS 580](course_no)
- Can I choose [CS 407-1](course_no) and [CS 407-2](course_no)

• registration - Registration is the process where a student tells the chatbot their pin. The chatbot will validate the pin and then add the student courses to the student’s database.

## intent:registration
- How can I [register](register) for classes
- How can I [register](register)
- [Register](register)
- Can you help me [register](register)
- [registration](register)
- What should I do to [register](register)?

• advisor - The advisors can check if their students have selected classes or not.

## intent:advisor
- I want to check my students
- [advisor](advisor)
- [faculty](advisor)
- [advisor](advisor) login
- I'm an [advisor](advisor) and I need to login
- I'm a [faculty](advisor)
- I am a [faculty](advisor) and I need to login
- i want to [check](advisor) if students have selected classes or not
- I need to [override](advisor) my student's request

Similary, I have created training data for other intents.

NLU Pipeline

language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
    ranking_length: 5
  - name: DucklingHTTPExtractor
    url: http://localhost:8000
    dimensions:
    - email
    - number
  - name: EntitySynonymMapper
  - name: ResponseSelector
    retrieval_intent: out_of_scope
    scale_loss: false
    epochs: 100
  - name: ResponseSelector
    retrieval_intent: faq
    scale_loss: false
    epochs: 100
  - name: ResponseSelector
    retrieval_intent: chitchat
    scale_loss: false
    epochs: 100

Explanation of each component mentioned in the pipeline can be found on Rasa documentation.

Dialogue Management

The component responsible for dialogue management in Rasa is called Core. Rather than a bunch of if/else statements, it uses a machine learning model trained on example conversations to decide what to do next. When developing conversational AI assistants with Rasa everything starts with training data. In dialogue management, this data is called stories - an example of conversations between the user and an assistant written in a specific format. In stories, user inputs are expressed as corresponding intents and entities while the responses of an assistant are expressed as action names. There are two types of actions used in Rasa - utterances and custom actions. Utterances are hardcoded messages that a bot can respond with. Custom actions are where you can create your custom code that can be executed i.e. calling an API, fetching data from a database, etc.

Creating training stories

I have created stories for the intent contact_request. A story starts with double hashes. It is important to provide a story name because it is easier to debug. Messages sent by the user are shown as lines starting with the asterisk symbol followed by intent and entities. Entities are surrounded by {}, the same way I defined it in the NLU training data. If the user says, “Tell me the website” and does not specify which professor’s website the user wants, the assistant should ask for the professor’s name, and for that, I used Forms to collect such information. One of the most common conversation patterns is to collect a few pieces of information from a user to do something (book a restaurant, call an API, search a database, etc.). This is also called slot filling. FormAction is used to collect multiple pieces of information in a row. This is a single action that contains the logic to loop over the required slots and ask the user for this information. Using a FormAction, all of the happy paths can be described in a single story. By “happy path”, I mean that whenever you ask a user for some information, they respond with the information you asked for.

## happy path contact request
* ask_howdoing
    - utter_ask_howdoing
* contact_request{"p_location":"find"} OR contact_request{"phone":"phone number"} 
OR contact_request{"p_title":"title"} OR contact_request{"email":"email"} OR 
contact_request{"contact":"contact"} OR contact_request{"website":"website"}
    - professor_form
    - form{"name":"professor_form"}
    - form{"name":null}
    - action_fetch_req_contact

In the above story, the user intent is contact_request, which is followed by the form action professor_form. The form is activated with form{“name”: “professor_form”} and deactivated with form{“name”: null}. The FormAction will only request slots that have not already been set. If a user starts a conversation with “I want the email of Professor Neli”, then the bot will not ask for the professor’s name (pname slot). In the story above, professor_form is the name of the form action. I have created a custom class for an action named action_fetch_req_contact where I defined custom logic for fetching requested entities (i.e. phone no, email, website, title, location, contact) of the professor from the database.

I also created stories for intent contact_request to handle the unhappy paths. Sometimes the users will not always respond with the information you ask of them. In some situations, users may change their mind in the middle of form action and decide not to go forward with their initial request. Such cases are called unhappy paths.

## unhappy path contact request
* ask_howdoing
    - utter_ask_howdoing
* contact_request{"p_location":"find"} OR contact_request{"phone":"phone number"} 
OR contact_request{"p_title":"title"} OR contact_request{"email":"email"} OR 
contact_request{"contact":"contact"} OR contact_request{"website":"website"}
    - professor_form
    - form{"name":"professor_form"}
* deny
    - utter_ask_continue
* affirm
    - utter_great
    - professor_form
    - form{"name":null}
    - action_fetch_req_contact

## very unhappy path contact request
* ask_howdoing
    - utter_ask_howdoing
* contact_request{"p_location":"find"} OR contact_request{"phone":"phone number"} 
OR contact_request{"p_title":"title"} OR contact_request{"email":"email"} OR 
contact_request{"contact":"contact"} OR contact_request{"website":"website"}
    - professor_form
    - form{"name":"professor_form"}
* deny
    - utter_ask_continue
* deny
    - utter_thumbsup
    - action_deactivate_form
    - form{"name": null}
    - utter_anything_else

I have created stories for the intent course_information. For validating the courses entered by the user, I created a course_form. I have created a custom class for an action named action_fetch_req_course_info where I defined custom logic for fetching requested entities (i.e. room, days and time, instructor, prerequisite, title, credit, info, dates, cross-listed, courses, all_courses) of the courses from the JSON file.

## happy path course information
* greet
    - utter_greet
* course_information{"room": "room no"} OR course_information{"dandt": "day"}
OR course_information{"instructor": "instructor"} OR course_information{"prerequisite": "prerequisite"}
OR course_information{"title": "title"} OR course_information{"credit": "credit"}
OR course_information{"info": "information"} OR course_information{"dates": "date"}
OR course_information{"cross-listed": "cross listed"} OR course_information{"all_courses": "all courses"}
    - course_form
    - form{"name": "course_form"}
    - form{"name": null}
    - action_fetch_req_course_info
* thankyou
    - utter_noworries

## happy path courses taught by professor
* greet
    - utter_greet
* course_information{"courses":"courses"}
    - professor_form
    - form{"name":"professor_form"}
    - form{"name":null}
    - action_fetch_prof_courses

I have created stories for intent course_information to handle the unhappy paths.

I have created stories for the intent selection. There are two types of stories for intent selection. One of the stories is when a student has not selected classes before, and another is when the student has already selected classes.

## happy path not selected classes
* greet
  - utter_greet
* selection{"select":"select"}
  - login_form
  - form{"name": "login_form"}
  - form{"name": null}
  - action_get_pin
  - slot{"selected": false}
  - course_form
  - form{"name": "course_form"}
  - form{"name": null}
  - action_start_sel_process
* goodbye
  - utter_goodbye

The bot remembers students’ pins, so the students do not have to remember their pin. During the registration period, the student can ask the bot for their pin. For authentication, students or advisors will be first asked for their CCSU email and then an OTP (One Time Password) is sent to their email. They need to enter this OTP to verify themselves. I created a login_form to validate users’ email and code.

After verifying the user, I created an action named action_get_pin to check if they have selected classes before or not and it will set the slot selected to True or False. In the story above, the slot selected is set as False. After that, I validate the courses by calling the course_form. I created an action named action_start_sel_process which will check for the following things:

• Check if the user has selected duplicate courses or not

  

• Check if the user has any hold or not

  

• Check if the user is part-time or full time and have taken courses accordingly or not

  

• Check if they have taken the required prerequisites for the selected courses or not

  I created a table named courses which have columns cno and prereq. I stored all the information on courses in Course_Information.json. For all the courses, I took their prerequisites and extracted necessary information using regular expression and stored them in the database. The reason why I had to do this because the bot will not understand the meaning from the prerequisite, so I converted and stored them in a special format. Below is an example of a prerequisite before and after conversion.

  

  

  For checking the prerequisites, I divided the prerequisites into three groups:
  1. Prerequisites with length 1

  

  

  2. Prerequisites with length 2

  

  

  3. Prerequisites with length > 2

  

  

Below is the story for intent selection when a student has already selected their classes.

## happy path selected classes
* greet
  - utter_greet
* selection{"select":"select"}
  - login_form
  - form{"name": "login_form"}
  - form{"name": null}
  - action_get_pin
  - slot{"selected": true}
  - action_slots_reset
* thankyou
  - utter_noworries

I created stories to handle the unhappy paths.

I created stories for the intent advisor. When the students select their classes, I add those classes to the JSON file of their advisor.

## happy path advisor
* greet
  - utter_greet
* advisor{"advisor":"advisor"}
  - action_slots_reset
  - ad_login_form
  - form{"name": "ad_login_form"}
  - form{"name": null}
  - action_fetch_advisor_info
* thankyou
  - utter_noworries

I created a form named ad_login_form to validate the advisor. I created an action named action_fetch_advisor_info which will fetch information from a JSON file and send it as a response.

I created a story for the intent registration.

## happy path registration
* registration
 - login_form
 - form{"name": "login_form"}
 - form{"name": null}
 - pin_form
 - form{"name": "pin_form"}
 - form{"name": null}
 - action_registration_confirmation

I created a pin_form to validate pins. I created an action named action_registration_confirmation which will fetch courses from JSON file of their advisor and add the courses to the students individual table in the database.

I created stories to handle unhappy paths.

I created a story for intent assist.

## ask for assistance
* greet
  - utter_greet
* assist
  - utter_provide_goal_info
  - utter_faculty_info
  - utter_course_info
  - utter_provide_advisor_info

Similarly, I created stories for other intents.

Domain

A domain defines a universe in which an assistant operates –

1. what intents and entities it can understand
2. which custom actions and utterances an assistant should be ready to respond with
3. how those utterances look like
4. what information an assistant should remember throughout the conversation and used in a certain context. Intents, entities, slots, responses, actions, and forms are specified in a file called “domain.yml”.

Policies

policies:
  - name: TEDPolicy
    max_history: 10
    epochs: 20
    hidden_layers_sizes: 
     dialogue: [256,128]
     label: []
    number_of_transformer_layers: 2
    batch_size:
    - 32
    - 64
  - name: MemoizationPolicy
  - name: FallbackPolicy
  - name: FormPolicy
  - name: MappingPolicy

Explanation of each component mentioned in the policies can be found on Rasa documentation.

---

Testing and improving the chatbot using Rasa X

I shared my chatbot with a small group of 6 people. I monitored their conversations and chatbot actions using Rasa X. Rasa X is a free toolset that layers on top of Rasa Open Source and helps you leverage conversations to improve your assistant. Some of the improvements that I made:

• I created more complex stories to handle unhappy paths.
• I summarised important intents of the chatbot to Telegram buttons, which I found was helpful in terms of user experience.
• I found out that users would click on different buttons without finishing a conversation/story. So, I had to create unhappy paths to handle such cases.

---

Deployment

I deployed the bot on Telegram. Telegram is a cloud-based instant messaging and voice over IP service. Telegram provides Bot API. The Bot API allows you to easily create programs that use Telegram messages for an interface. Bots are third-party applications that run inside Telegram. Users can interact with bots by sending them messages, commands, and inline requests. You control your bots using HTTPS requests to our Bot API. To create a new bot, you need to have a conversation with BotFather. BotFather helps you create new bot accounts and manage your existing bots. BotFather will ask you to choose a name and username for your bot. Once you are done with that it will provide you with an API token. This token is used to control the bot.

To make your Bot answer to requests from your Telegram users you need to manually request for updates to the Bot API or you can register a Webhook to automatically being called once updates are available. A Webhook is a URL you transmit to Telegram once. Whenever a new update for your bot arrives, Telegram sends that update to the specified URL. The quickest and easiest way to set a Webhook for your Bot is to issue a GET request to the Bot API. All you must do is to call the setWebHook method in the Bot API via the following url:

where

• My_bot_token is the token you got from BotFather when you created your Bot
• url_to_send_updates_to is the URL of the piece of code you wrote to implement your Bot behavior (must be HTTPS)

I ran the bot locally. Rasa Core server listens on port 5005 by default. I used ngrok for Webhook. Ngrok is multiplatform tunneling, reverse proxy software that establishes secure tunnels from a public endpoint such as the internet to a locally running network service while capturing all traffic for detailed inspection and replay. I ran ngrok on port 5005 on the terminal using the command ./ngrok http 5005.

It will give us a forwarding URL and this URL is used to call the setWebHook method in the Bot API. Now when the user sends a message to our bot, the telegram will forward the message to the specified ngrok URL and then ngrok will forward that message to our local server.