6 Best Practices for NLP Implementation

Recent advances in artificial intelligence and machine learning have made natural language processing so powerful that the latest models have outpaced human performance in current standard data sets.

In education, we’ve seen the use of NLP in many effective ways, from machine translation and helping students improve their writing skills, to enhancing learning experiences. For example, Google Translate helps make educational content useful to more students around the world. Duolingo uses artificial intelligence to determine how difficult language learning content is. Grammarly helps students with error-free writing, and TurnItIn helps teachers detect plagiarism. At Quizlet, we leverage ML and NLP to categorize written answers, generate questions, and make sense of our content, among other things.

Having spent most of my career advancing (or leading teams to advance) ML and NLP problem-solving for users and businesses, here are some guidelines I recommend keeping in mind when approaching NLP projects.

  1. I know your problem:
    For beginners who are starting with a machine learning problem, it is easy to get lost in theory and code. Ensure that you understand the problem and the hypotheses well by writing them down and performing an exploratory analysis of the data.
  2. Collect your data: The data you use to train and validate your NLP models is critical to their success and deserves you to take this step seriously, and think through creative solutions. For example, for our Subject Classifier training data, we used existing user-generated content that contains the subject names in the titles. (For example, we could indicate that the content titled “Photosynthesis Chapter 3” was about photosynthesis.) For other issues, we collected training data through human annotations or asking users. Some models like OpenAI’s GPT-3 only need a few data points to learn a task, but these come with tradeoffs.
  3. Sharing output examples: One of the best ways for others to understand exactly what you’re working on is to share examples of results. When we created Advanced Questions, the examples helped illustrate the value this new feature can bring to everyone and were critical to prioritizing the project in the product roadmap. Researching the results yourself also helps you come up with ideas on how to improve the algorithm.
  4. Agree on success metrics: In addition to sharing examples, measure and share overall performance. To estimate the quality of the algorithm, we often named a sample of hundreds of outputs. Agree on important metrics (eg false positives, coverage) and acceptable thresholds. For example, we built a semantic (“smart”) matrix to classify free-form text answers. We decided that we should aim to increase the coverage of correct correct answers while keeping “false corrections” below 3%.
  5. Start simple (if you can): Some problems don’t need a fancy algorithm. For example, our Definition Suggestion are just the most common definitions of a given word, which use a simple arithmetic function.
  6. Be vigilant: If creating content, be aware of biased and offensive/inaccurate content. All sophisticated NLP paradigms are trained on internet scripts, i.e. human behavior that can be problematic. We used OpenAI to create example sentences for language learning and had to use their content filter (and our own filter on top of that) to filter out potentially offensive content. It is also important to have firewalls and opportunities for users to provide feedback.

Neuro Linguistic Programming (NLP) has the potential to help improve the user experience and create new features that were not previously possible. There are many courses and technical resources to help you learn technology and tools, and these steps will help you take advantage of them in real-world settings.


Leave a Comment