2020-12-27 00:00:00 +0000

Writing a Chatbot in Django: Part 1

While brainstorming Django app ideas, I thought that it would be cool to design a personal assistant chatbot that can schedule appointments based on my availability and meeting time preferences. This would function as a simple chatbot version of Calendly. Ideally this application can even be integrated with a service like Twilio so that users could text message the bot. Over the next series of articles, we will explore writing such a chatbot in Python Django.

Rule Based Chatbots

One of my favorite parts of working with Django is the availability of Python’s amazing AI and ML libraries. After doing a little bit of research, I decided that a rule based chatbot would be sufficient for the purpose of scheduling appointments. There are no shortage of rule based chatbot guides. In their simplest form, a rule based bot will look for keywords to determine the ‘intent’ of the user’s request and pick an appropriate response. For a simple yet capable rule based bot, I recommend checking out Building a Rule Based Chatbot in Python.

Unfortunately, there are far too many ways to request an appointment without having to enforce an awkward limited set of commands that the user can use. These types of bots will often appear as button based bot. At that point the bot starts to resemble a webform embedded in a text message client. This is convenient but it’s not very conversational. My goal is to strike the balance between a rule based bot and a bot with a little more conversational ability.

Chatterbot

Python has many high level chatbot frameworks. I decided to go with Chatterbot for several reasons:

Trains in real time based on user input
Supports Django ORM integration
Modular and extendable data preprocessors and logic adapters

The Project

For this tutorial I will start small and create a simple terminal application without Django. The project will be organized into a chatbot directory with the main chatbot.py program, a preprocessors.py module for custom data preprocessing functions, and a logic directory containing schedule_adapter.py that will be be used to interpret schedule requests.

├── chatbot
│   ├── __init__.py
│   ├── chatbot.py
│   ├── logic
│   │   ├── __init__.py
│   │   └── schedule_adapter.py
│   └── preprocessors.py
├── database.db
├── db.sqlite3
├── requirements.txt
└── tests
    ├── __init__.py
    ├── test_chatbot.py
    └── test_logic_adapters.pyc

As is the case with most python projects, I recommend building a virtual environment. This makes installing and managing dependencies easier by localizing everything within your project. Don’t forget to add myenv (or whatever you call your virtual env) to .gitignore!

$ python3 -m venv myenv

For this project I have chosen the following dependencies. If you like to freeze your dependencies into a requirements.txt file then I recommend using pip-chill > requirements.txt which will only output the dependencies of the packages that you installed and omit the child dependencies.

chatterbot==1.1.0
chatterbot-corpus==1.2.0
delorean==1.0.0
en-core-web-sm==2.1.0
freezegun==1.0.0
ipdb==0.13.4
pip-chill==1.0.0
pylint==2.6.0

Writing Custom Logic Adapters

As mentioned previously, Chatterbot supports modular logic adapters that can be used to add behaviors to your chatbot. Logic adapters are loaded into the chatbot when they are instantiated.

# chatbot.py
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer

# Create a new instance of a ChatBot
bot = ChatBot(
    'Terminal',
    storage_adapter='chatterbot.storage.SQLStorageAdapter',
    logic_adapters=[
        'logic.schedule_adapter.Schedule',
         {
            'import_path': 'chatterbot.logic.BestMatch',
            'default_response': "I'm sorry I don't understand. I schedule appointments. Is there a date and time you have in mind?",
            'maximum_similarity_threshold': 0.90
        }
    ],
    preprocessors=[
        'preprocessors.format_dates'
    ],
    database_uri='sqlite:///database.db'
)

We are using our custom logic adapter called logic.schedule_adapter.Schedule along with a base adapter called chatterbot.logic.BestMatch. The included BestMatch adapter is set to return a default response when none of the conditions for the other logic adapters are satisfied. The primary role of this bot will be to detect, interpret, and parse schedule requests. For this iteration we will keep things fairly simple and ensure that we can correspond with our bot like below.

$ I would to schedule an appointment for january 5th at 5 PM
=> scheduling appointment for 1/05/21 at 17:00:00

Corresponding in this manner requires that our Logic Adapter can detect the user’s intent to schedule an appointment. All LogicAdapter classes need to respond to a can_process method that is used to invoke the adapter if the method returns True. This can be done with a simple regex condition. If can_process returns True then the class will process the request with the process method.

from chatterbot.logic import LogicAdapter
from delorean import parse
from re import search
import spacy

class Schedule(LogicAdapter):

    def __init__(self, chatbot, **kwargs):
        super().__init__(chatbot, **kwargs)

    def can_process(self, statement):
        if search('(make|schedule).*appointment', statement.text):
            return True
        else: 
            return False

    def process(self, input_statement, additional_response_selection_parameters):
        from chatterbot.conversation import Statement

        nlp = spacy.load('en_core_web_sm')
        doc = nlp(input_statement.text)

        entities = {}
        for ent in doc.ents:
            entities[ent.label_] = ent.text

        date = entities.get('DATE')
        time = entities.get('TIME')

        if date and time:
            appointment = parse(date+" "+time, dayfirst=False).datetime.strftime("%m/%d/%y at %H:%M:%S")
        elif date:
            appointment = parse(date, dayfirst=False).date.strftime("%m/%d/%y")
        elif time:
            appointment = parse(time).datetime.strftime("%m/%d/%y at %H:%M:%S")
        else:
            return Statement(text="please provide a preferred date and time for your appointment", confidence=1)

        response = "scheduling appointment for {}".format(appointment)
        return Statement(text=response, confidence=1)

The process method assumes that data has been sanitized by the data preprocessor functions (more on this in the next section). We can use the natural language processing library spacy to identify the ‘entities’ (date and times) in the user’s request. At this stage, we are not concerned about making the appointments. We only want to prove that we can interpret the datetimes from the natural language inputs.

The ‘en_core_web_sm’ model loaded by spacy does a very good job at extracting the date and time entities. To further parse the extracted date and time we turn to a library that deals with time transformations. This library is appropriately called Delorean. It’s parse function is used to parse any date format or natural language representation to a standard m/d/y h:m:s format.

Once this is hooked up to a calendar with available times, the bot will be able to schedule the appointment if it’s available by confirming the next available time with an app that we will create in the next tutorial. In cases where a date and time is provided the bot will try to fill that time. If only the date is provided then the bot will pick the next available time for that day. If only the time is provided, then the bot will schedule the appointment for the day it was requested. In all cases, the bot will confirm the time.

For this tutorial we will just have the bot return a standard message notifying the user that the appointment is being scheduled for the proposed time.

Preprocessor Functions

Spacy does a great job at detecting entities. However, there were a few gaps in it’s ability to detect lowercase months and dates entered in the american mm/dd/yyyy format. I could have added examples to the model directly but a much easier approach is to format these likely inputs into the format that spacy can interpret.

Chatterbot preprocessors are simply functions that can be added to the Chatbot’s preprocessor kwarg. These functions will be evaluated before passing the input to any logic adapters. In our case this is perfect because we can use a little bit of regex sorcery to sanitize our inputs to a date format and month format that Spacy can interpret.

# preprocessors.py
from re import search, sub, split

def format_dates(statement):
    pattern = r'(\d{2}|\d{1})(-|\/)(\d{2}|\d{1})(-|\/)\d{4}'

    bad_date_format = search(pattern, statement.text)
    if bad_date_format:
        date = bad_date_format.group()
        m, d, y = split(r"\/|-", date)
        date = "-".join([y,m,d])
        statement.text =  sub(pattern, date, bad_date_format.string)

    return statement

Formatting dates involves detecting any dates that are delimited with ‘/’ or ‘-‘ and are assembled in mm/dd/yyyy or m/d/yyyy format. These dates are then split by their delimiter and reordered in ISO format (the preferred date format of Spacy).

def capitalize_months(statement):
    jan_apr = r'\b(jan|january)\b|\b(feb|february)\b|\b(mar|march)\b|\b(apr|april)\b'
    may_aug = r'\b(may)\b|\b(jun|june)\b|\b(jul|july)\b|\b(aug|august)\b'
    sept_dec = r'\b(sept|september)\b|\b(oct|october)\b|\b(nov|november)\b|\b(dec|december)\b'
    pattern = jan_apr+may_aug+sept_dec

    month = search(pattern, statement.text)

    if month:
        capitalized_month = month.group().capitalize()
        statement.text = sub(pattern, capitalized_month, month.string)

    return statement

We want to look out for both lowercase abbreviations and months. Neither format is detected as a date by Spacy. After detecting the pattern, the matching portion of the statement is capitalized and spliced back into the original statement.

Putting It All Together

In the end we have a chatbot that is conversational and can handle a pretty wide variety of appointment requests.

bot = ChatBot(
    'Terminal',
    storage_adapter='chatterbot.storage.SQLStorageAdapter',
    logic_adapters=[
        'logic.schedule_adapter.Schedule',
         {
            'import_path': 'chatterbot.logic.BestMatch',
            'default_response': "I'm sorry I don't understand. I schedule appointments. Is there a date and time you have in mind?",
            'maximum_similarity_threshold': 0.90
        }
    ],
    preprocessors=[
        'preprocessors.format_dates',
        'chatbot.preprocessors.capitalize_months'
    ],
    database_uri='sqlite:///database.db'
)

trainer = ChatterBotCorpusTrainer(bot)
trainer.train('chatterbot.corpus.english.greetings')

print('Type something to begin...')

while True:
    try:
        user_input = input()

        bot_response = bot.get_response(user_input)

        print(bot_response)

    # Press ctrl-c or ctrl-d on the keyboard to exit
    except (KeyboardInterrupt, EOFError, SystemExit):
        break

The bot is conversational (can perform basic greetings) with the addition of the english.greetings corpus included in the ChatterBotCorpusTrainer library.

Conclusion

I hope you are as excited as I am about the progress we made. At this point we have a conversational bot that can do a pretty good job at detecting a user’s intent to schedule an appointment, extract a datetime, and respond with a confirmation of the proposed meeting. Next time we will drop this application into a Django application and actually have our chatbot schedule appointments based on preferred times and availability set in the Django admin.