(转) Reinforcement Learning for Profit

Reinforcement Learning for Profit

July 17, 2016

Is RL being used in revenue generating systems today?

 

Recently, one of my facebook friends, and alumni of the University of Alberta (with a PhD in Computing Science), Cosmin Paduraru posed a question:

Where is Reinforcement Learning used in revenue generating systems today?

I have been thinking about this lots over the last month as I attended two international conferences on Artificial Intelligence and Machine Learning (ICML and IJCAI) in NYC, USA. It is important to explore future prospects both inside and outside academia — In case you need a catch up, I am currently at the University of Alberta working on a PhD in Computing Science with a focus on Reinforcement Learning and Artificial Intelligence.

With the success of modern AI systems — out of the winter and into the spring — many companies have invested and continue to invested heavily into modern AI systems, backed by teams of leading researchers in the field (e.g. Facebook, Google, Microsoft, IBM, Twitter, etc.).

With that said, maybe Cosmin is right, Reinforcement Learning (Sutton and Barto 1998, and this killer-intro by the fantastically talented Andrej Karpathy) is seemingly publicly underrepresented in currently deployed systems making money in the real world, or is it?

Adapted from Sutton and Barto 1998 and WALL-E

Luckily I was at the International Joint Conference on Artificial Intelligence where I was attending a panel discussion on The Business of AI, the panel was composed of all speakers from the industry day. A desirable venue to solicit a wide variety of opinions from thought leaders in the field.

So I posed the question to them, their responses went as follows:

Peter Norvig (Director of Research at Google): “well… AlphaGo made a million bucks and then gave it away”… a recent tweet from Demis Hassabis (Google DeepMind) confirms:

Pleased to confirm the recipients of the #AlphaGo $1m prize! @UNICEF_uk, @CodeClub, and the American, European and Korean Go associations

— Demis Hassabis (@demishassabis) June 6, 2016

Peter Stone (Founder and President, Cogitai. Professor UoT (Austin)) gave lots of great examples of recent applications:

He said,“We are on the cusp of moving from the academic lab to the industry for RL, adaptation, and lifelong learning…We are at the cusp, and that is the main motivation from Cogitai”

He also referenced work by Thomas G. Dietterich on invasive species management, wildfire suppression, by Joelle Pineau on applying RL in healthcare, and by Andrew Ng and Drew Bagnall on helicopter control. All of these could be as a practical demonstrations of specific, developing industrial applications.

Hiroaki Kitano (President & CEO SONY Computer Science Laboratories) said that this is a current research area for Sony and to expect profitability using these and advancing RL algorithms in 2-5 years. Almost 10 years after Sony’s last robotic venture, the Aibo, Sony CEO Kazuo Hirai has just recently (late June 2016) said “the robots we are developing can have emotional bonds with customers, giving them joy and becoming the objects of love”.

Guruduth Banavar (Chief Science Officer, Cognitive Computing, IBM Research) predicted that this is going to happen, sooner rather than later, and his prediction was that it will happen in the domain of conversational systems, dialog systems, and understanding the larger context of conversations. He also mentioned that the illustrious Gerald Tesauro (the man behind TD-Gammon) is working on these problems. Interesting that he did not mention Watson…

Some interesting answers from industry leaders. But I was surprised that no one mentioned:recommender systems (like those on Amazon, Netflix, Yelp, and nicely formalized as an RL problem in 2005 by Shani et al.), are these systems all collaborative filtering? Surely not.

No one mentioned that Google Reinforcement Learning Architecture (here is a quick summary), which I can only imagine could be behind some of the personal recommendations and rankings that Google does behind-the-scenes on Search, YouTube, and maybe … Maps?

No one mentioned contextual bandits, sometimes called associative RL (as discussed by Li et al. 2010 for news recommendation), for serving ads and news stories. These systems are surely deployed on large-scale news sites by the publishers to maximize click-through-ratios and create a personalized experience. Microsoft recently announced Multiworld Testing Decision Service, for making context based decisions… I guess there were no Microsoft representatives on the panel to toot this horn (thanks for the catch Pardis)

With so much potentially out there, why was there no mention of these use cases for reinforcement learning? Where else could RL be hiding in the money-making wild? RL seems like an ideal candidate for systems of personalization on large-scale, sequential decision-making problems… so what am I missing?