What Wall Street didn't see coming
Generative models, reinforcement learning, model efficiency, DeepMind and NVIDIA's impact weren't top of mind on Wall Street back in 2016...
In November 2016, Goldman Sachs published its first major report on artificial intelligence. I read it on a flight from London to San Francisco and jotted down my thoughts for the authors whom I’d met. Back then, Wall Street was mostly indifferent to AI. Today, it’s obsessed. Every analyst on TV sings NVIDIA’s praises and AI progress as the engine for economic growth.
Revisiting my 2017 notes in response to the GS report, it’s striking how many of the missing themes have since gone mainstream. I got a few things wrong too. Turns out that we did finally get a 10x better chatbot that everyone now uses and developers are happy to call intelligence from third party APIs.
Hope you enjoy the read!
Overall, I think the report is great and the use cases are interesting. However:
- There's far more than 150 companies [in AI today].
- The report, however, does not touch on generative machine learning models. This class of models is distinct from discriminative models, which are tasked with separating, labelling, or categorising data. Generative models instead learn representations of data and can then output new data altogether. You can see examples where Facebook showed they can erase the middle of an image and then ask a generative model to fill in the blank on the basis of it's understanding of the image's content.
- It also talks a bunch about efficiency that is gained from ML when applied to optimisation problems, but it doesn't talk about the importance of creating ML models and algorithms that are less and less data "hungry" and thus less computationally and energy intensive (because the models are smaller at training and runtime). This is crucial.
- It also doesn't talk about reinforcement learning at all, which is a big oversight in my view, especially when considering problems that involve exploring (through trial and error) an environment (e.g. a robotic control task) to learn the optimal way of reaching a certain goal (e.g. Atari games). This hasn't really made it into real world production yet, but is coming.
- It also doesn't talk about transfer learning, which is a strategy to train an ML model on one data modality such as text and then take that model and apply it to images. These pre-trained networks have been shown to reduce the amount of new training data they need to solve tasks on a new data modality.
- The piece is very light on DeepMind, which I think is by far the most compelling organisation in the world right now. It also says that "In 2013, Google paid more than $400mn to acqui-hire DeepMind Technologies; according to press reports, a team of roughly a dozen.", which is false. The team was much bigger than 12 people!
- NVIDIA is also doing a hell of a lot more in AI than just making GPUs - that company is smashing it right now. If you want to get a good view, I suggest you come along to their GTC in May 2017 (if I got the date correctly). You'll be blown away. Markets clearly didn't appreciate the extent to which it is a market leader, as evidenced by the 30% pop. Should have trusted my own knowledge instead of reading market reports on that one :)
- Sentences like this in the description of Baidu brain just don't mean much tbh "1) An AI algorithm that mimics the human neural network, with a large number of parameters trained over hundreds of billions of samples." I think you gotta be careful with layman descriptions the verge on biological references or too handwaving reference to "lots of data". What data are you talking about.
- "Neural networks become more effective the more data that they have, meaning that as the amount of data increases the number of problems that machine learning can solve using that data increases." That's not technically true. It's not the volume of data that matters per se, it's the volume of quality data that specifically describes the task you want to solve that matters. The more this increases, the more problems we can consider solving, if we have the right approaches of course. This relies on both building models of problems and training algorithms.
- I find the description of deep learning rather confusing and they also don't actually explain exhibit 4 to the reader. It also misses an important point that unlike non-deep learning ML models, deep learning does not require the developer to engineer their own features (i.e. the variables in the data that have to be tweaked in order for the ML model to output a classification). That's super powerful.
- They should explain why over fitting is bad. Of course the problem is that your model will not generalise well to data beyond the training examples it has seen.
- I'm not so sure on the claim that horizontal APIs for ML as a service will be a good business. From research I've done with several hundred developers, basically no one wants to consume a paid for API for an ML task if it's going to be an important feature of their application. Developers hate 3rd party dependencies for functionality and especially in the AI world. Maybe larger corporates will who can't get hold of ML engineers would consume those APIs from Google, but very unlikely if it's a core part of the offering the corporate sells to its customers.
- "Industry consortiums in areas such as retail or advertising could pool data to better compete against larger competitors (e.g., retailers could pool data to better compete against Amazon’s recommendation engine)." This would be neat, but I'm not convinced it'll happen. Maybe amongst small players, but not the big boys. In fact, open standards for training data is an area the US government is pushing.
- I'm bearish on bots. I do not really buy that they're the future user interface. I challenge the authors to find an example product or service that is best procured/consumed with via a bot - I'm talking 10x better. I'm still looking. I think this area is overhyped.
- The piece is light on context awareness for digital assistants. I think this is the big area - if you can identify the intent in language or user behaviour, you can pre-empt their next actions. That's a holy grail and makes a big impact on user experiences.
Read more:
Incisive & prescient, especially from 2016!