What does "Arrival" have to do with learning NLP?

Plus, OpenAI's new product, and Poker-driven decision-making


This week the Altdeep Causal Modeling in Machine Learning Workshop opens enrollment for its Summer cohort. Enrollment closes in 4 days.


Ear to the Ground

To learn NLP, start with P, but don’t forget the NL

There is a new natural language processing (NLP) specialization at Coursera. The curriculum looks like a comprehensive and strong introduction. I’m excited.

Courses such as this teach more about the P, and less about the NL.

I am a fan of the Sapir–Whorf hypothesis, which is the core idea behind the film Arrival.

Arrival Movie (2016) | Reviews, Cast & Release Date in ...

The idea is that language shapes the fundamental brain cognition of those who speak it. One of the consequences is that you can’t get a deep understanding of a problem unless you have abstractions that you can use to describe and parse the problem.

Linguistic semantics provides those abstractions for the NL. To solve the hard problems of NLP, one needs to have a vernacular for describing the semantic meanings implicitly captured by those cutting-edge models, and ideally make models that make that meaning more explicit. So I think it is wise to take the above course, then follow up with (such as this one). Otherwise, you risk making more algorithmic versions of Clever Hans — Clever Hans was a horse that had allegedly learned arithmetic but had really just learned cues given by its owner. I argue that without a linguistic vernacular, you are more make the Clever Hans mistake articulated in this The Gradient article.


OpenAI releases first product

OpenAI unveiled an API product that exposes endpoints for some of its language models, including the controversial GPT-3. The product is currently focused on text-generation, but we can expect it to expand to other applications.

I’ve played around with it, looking for ways to incorporate it into the AI-powered chatbot I work on. The verdict is still out — the trick is finding a task that is useful to solve while not so complex or open-ended that the text generation would behave in unpredictable ways.

Even if I identified tasks to I could solve with this API, we'd still be far from deciding whether or not it made business sense to incorporate this into our product. But I think AI teams like mine are much more likely to make use of such a product. The NLP problems we want to solve (and the business value in solving them) are much different than that of OpenAI, tech like this might help us focus more on the problems that matter.

I might be alone in my suggestion that AI-teams working on a problem within a specific domain and set of customers might be the primary users of this API. The posts about this release suggest that general engineering teams can use this an AI-microservice they can connect to their app. Maybe, but I have my doubts. Best practices do not exist to incorporating stochastic artifacts from a third party services into an app.

Go Deeper

Decision Science for Fun and Profit

I’m looking forward to the release of The Biggest Bluff by Maria Konnikova, who writes and lectures on how learning to play poker lead to better decision-making.

The core of AI and data science is decision-making under uncertainty. So I pay attention to structured finite games like Poker, and how humans like Maria Konnikova and Annie Duke, as well as AI like Liberatus, try to solve them.

I think people who work on sequential decision algorithms, such as bandits or reinforcement learning, need to read books like these. The language is the same. Take, for example, this excerpt from her book:

Most real-world environments are ... "wicked": there's a mismatch between action and feedback because of external noise. Activities with elements of surprise, uncertainty, the unknown: suddenly, you're not sure whether what you've learned is accurate or not, accurately executed or not.

BREAKING: Author and journalist Maria Konnikova becomes ...

Liked this post? Chances are your friends might like it too. Please share.