SAM RANSBOTHAM: In AI projects, perfection is impossible, so
when inevitable errors happen, how do you manage them?
Find out how Nasdaq does it when we
talk with Douglas Hamilton, the company's head of AI research.
Welcome to Me, Myself, and AI, a podcast
on artificial intelligence in business.
Each episode, we introduce you to someone innovating with AI.
I'm Sam Ransbotham, professor of information systems
at Boston College.
I'm also the guest editor for the AI and Business Strategy
Big Ideas program at MIT Sloan Management Review.
SHERVIN KHODABANDEH: And I'm Shervin Khodabandeh,
senior partner with BCG, and I colead BCG's AI practice
in North America.
Together, MIT SMR and BCG have been
researching AI for five years, interviewing hundreds
of practitioners and surveying thousands
of companies on what it takes to build and to deploy and scale
AI capabilities across the organization
and really transform the way organizations operate.
SAM RANSBOTHAM: Today we're talking with Douglas Hamilton.
He's the associate vice president and head
of AI research at Nasdaq.
Doug, thanks for joining us.
Welcome.
DOUG HAMILTON: Thanks, Sam and Shervin.
Great to be here today.
SAM RANSBOTHAM: So our podcast is Me, Myself, and AI,
so let's start with can you tell us
a little bit about your current role at Nasdaq?
DOUG HAMILTON: In my current role,
I head up AI research for Nasdaq at our Machine Intelligence
Lab.
The role itself here is a little bit unique,
since many, many roles within Global Technology, which
is our engineering organization, are very much
so business-unit-aligned, so they'll
work with one of our four core business units,
whereas this role really services every single area
of the business.
That means that we're servicing market technology, which
is the area of Nasdaq's business that produces software
that powers 2,300 different companies in 50
different countries, powers 47 different markets
around the world, as well as bank and broker
operations, compliance, and [regulatory] tech
for making sure that they are compliant with their local
authorities.
We service, of course, our investor intelligence line
of business, which is how we get data from the market
into the hands of the buy and sell side,
so they can build products and trading strategies
on top of those.
We service, of course, the big one
that people think about mostly, which is market services, which
is the markets themselves; that's our core equities
markets and a handful of options and derivatives
markets as well.
And then finally, corporate services --
that actually deals with the companies that are listed
on our markets and their investor relationship
departments.
So really, we get to work across all
of these different lines of business, which
means that we get to work on a huge number of very
interesting and very diverse problems in AI.
Really, the goal of the group is to leverage
all aspects of cutting-edge artificial intelligence,
machine learning, and statistical computing
in order to find value in these lines of business,
whether it's through productivity
plays, differentiating capabilities, or just continued
incremental innovation that keeps Nasdaq's products leading
edge and keeps our markets at the forefront of the industry.
In this role, I have a team of data scientists
that are doing the work, writing the code, building the models,
munging the data, wrapping it all up in optimizers,
and creating automated decision systems.
So my role, really, I think, day to day,
is working with our business partners
to find opportunities for AI.
SHERVIN KHODABANDEH: Doug, maybe to bring this to life a bit,
can you contextualize this in the context of a use case?
DOUG HAMILTON: I'll talk about one of our favorite use cases,
which is a minimum volatility index that we run.
So the minimum volatility index is an AI-powered index that we
partnered with an external [exchange-traded funds]
provider, Victory Capital, on.
The goal of this index is to basically mimic
Nasdaq's version of the Russell 2000 --
it's a large and mid-cap index --
and then essentially play with the weights of that index,
which are normally market-cap-weighted,
in such a way that it minimizes the volatility exposure of that
portfolio.
What made that project really difficult
is that minimizing volatility is actually
a fairly easy and straightforward problem if you
want to treat it linearly.
That is, you look at a bunch of stocks,
you look at their historical volatility performance,
you pick a bunch of low-volatility shares,
you slap them together, boom -- you get a pretty low-volatility
portfolio.
And that's actually fairly straightforward to solve,
from using linear methods to solve
it, numerical programming, etc., and you
can wrap linear constraints around it
to make sure that you're not deviating too much
from the underlying portfolio.
You're still capturing the general themes of it.
You're not overexposing yourself to different industries.
That's actually fairly easy to do.
However, when this becomes really interesting
is, wouldn't it be cool if you found two stocks that
worked against each other, so they could actually
be quite volatile, but the portfolio, when mixed together,
actually becomes less volatile than even two
low-volatility shares, because they're constantly
working against each other?
That is, they have this nice contravarying action
that cancels each other out so you
can capture the median growth without the volatility
exposure.
That'd be great.
Now, that becomes a nonlinear problem.
And it becomes a very noisy, almost nonconvex problem
at that point too.
But you still have all these constraints
you need to wrap around it.
These are simulated annealing, genetic algorithms,
[Markov Chain Monte Carlo-style] optimizers.
And those also behave pretty well
when we have soft constraints that
generally guide the solutions back into the feasibility zone.
The problem they have is when you give them hard constraints.
They don't like hard constraints; they break a lot.
So, what we had to do is rearchitect
a lot of these algorithms to be able to handle
these hard constraints as well.
SHERVIN KHODABANDEH: What would be a hard constraint?
DOUG HAMILTON: I'll give you an example of a soft constraint
and a hard constraint.
It would be really nice if you have a portfolio, when
you go to rebalance it, if its total turnover was
less than 30%, let's say, because it
gets really expensive to rebalance it otherwise.
A hard constraint might be that no holding can
vary by more than 2% between the optimized portfolio
and the parent portfolio.
So if the parent portfolio is 10% Microsoft,
let's say, then the optimized portfolio
has to be between 8% and 12%, right?
So that's an example of a hard constraint.
If it's 7.9%, we're in violation of the governing
documents of the index, and everybody
gets into a lot of trouble.
SHERVIN KHODABANDEH: Got it.
That's a good one.
OK.
So you're saying hard and soft constraints together
form a tougher problem.
DOUG HAMILTON: A considerably tougher problem,
because these algorithms deal well with nonlinearity.
Particularly, these Monte Carlo Markov Chain-style algos
do not deal well with those hard constraints, where
they must meet these criteria.
And when you have --
I think in that one, we had 4,000 constraints,
something like that -- almost nothing meets them.
So if you take this hard culling approach,
then you're left with no viable solutions
to gain density around.
So we had to spend a lot of time working with the team to figure
out what the appropriate solution architecture should be
-- algorithmically, etc. -- to overcome that challenge,
how we set up those experiments, what sort of experiments we
need to set up, how we test it, and, of course,
how we actually communicate to the client that the solution is
better than what they currently have.
SHERVIN KHODABANDEH: Doug, this example that you talked about
on volatility -- is [it] one of hundreds of use cases that
your team does, or one of tens of use cases?
[I'm] just trying to get a sense of the scale of the operations
here.
DOUG HAMILTON: Within Nasdaq, what we
represent is the center of excellence
for artificial intelligence.
So this is one of I'd say it's in the dozens of use cases
that are either live or that we're
exploring at any point in time.
On top of that, obviously, we have robust relationships
across the business with third-party vendors that help
us with all sorts of internal use cases --
where maybe it's not something we're looking to sell
to the outside world, or something where we can leverage
existing technology in a better way than building it in-house
-- that also certainly are part of our AI story as well.
SAM RANSBOTHAM: I was thinking about your example of finding
the matching [stocks].
We think about digital twins; it's
almost a digital un-twin stock that you're
trying to match with.
That has to change, though, at some point.
How often are you revisiting these?
How are you keeping them up to date
so that you don't end up with things suddenly moving together
when you thought they were moving the opposite [way]?
DOUG HAMILTON: The nice thing about the world of indexing
is that it's almost statutory how
you do that, in that when we look at other models
that we have in production, we usually
do this in one of two ways.
We usually do it either in an ad hoc way,
through telemetry, looking at model performance
and looking for some sort of persistent degradation
in performance, as well as, of course,
having some sort of regularly scheduled maintenance
for many of our products.
For indexes, we're basically told,
"Here's how often you rebalance, and here's
how often you're allowed to make the change."
So in this case, we rebalance twice a year,
so every six months is when we go back and take a look.
SAM RANSBOTHAM: Let's switch a little bit
to say, how did you end up doing this?
What in your background led you to be
able to do all those things?
DOUG HAMILTON: I'm fortunate in that I got my first data
science job in 2015.
I'll tell you how I ended up there.
My very first job was in the Air Force.
I was enlisted in the Air Force in an operational position
as an electronics technician; I spent a lot of time
shocking myself.
It was not the most fun thing in the world,
but I was 22, so it was hard not to have fun.
And what I realized ...
I have this exposure to an operational world
and was able to gain some leadership experience
early on through that as well.
I used the GI Bill to go to school --
the University of Illinois -- [where] I finished
an undergraduate degree in math.
I was very convinced I wanted to go
become a professional mathematician, a professor.
I had some great professors there that I was working with
and was on the theoretical math track: real analysis, topology,
etc.
And that was great until the summer
before I graduated: I had this wonderful internship
in an astronomy lab, where we were studying
a star in the last phase of its life,
and it was going to have no earthly application whatsoever,
and I was just bored and realized I
didn't want to be in academia.
As many people do who are in quant fields
and faced with such an existential crisis,
I decided I was going to go become a software developer.
And what being a software developer
mainly helped me figure out was that I
didn't want to be a software developer, so I
went to MIT to study systems engineering and management
and really focused a lot of my effort in operations research
while I was there.
I had a colleague in the class at Boeing,
who was looking to start up a data science group,
so he suggested my name, and that's
how I got started working at Boeing in manufacturing quality
and standing up an advanced analytics and data science
group there.
I worked there for a couple of years
and then, like many people who go and try
to operate in the real world, became a little disillusioned
by the real world and decided to retreat
into the world of finance, where I found Nasdaq.
I worked as a data scientist here for a few years
before moving into a management position.
I think that's the story in a nutshell.
SHERVIN KHODABANDEH: So Doug, from airplanes
to financial markets, it seems like all
of the examples you gave are where the stakes are
quite high, right?
DOUG HAMILTON: Yes.
SHERVIN KHODABANDEH: I mean, the cost of being wrong or an error
or a failure -- maybe not a catastrophic failure,
but even that, I mean -- any kind of error is quite high.
So how do you manage that in the projects
and in the formulization of the projects?
DOUG HAMILTON: I'm really glad you asked that,
because this is my opportunity to talk smack
about academic AI a little while,
so I'm going to start off doing that.
SAM RANSBOTHAM: Be careful.
There's a professor here, so --
SHERVIN KHODABANDEH: Keep going.
Sam would love that.
Keep going.
DOUG HAMILTON: Really, I think it all
starts with being more concerned about your error rather
than your accuracy.
One of the things I've been really disappointed about
in academic AI over the last couple of years is that --
really, it's related to this AI ethics talk that we have these
days, where people were shocked to find out that when you build
a model to, let's say, classify some things,
and you look at some minority cohort within the data,
that the model doesn't classify that all that well.
And it's like, "Yeah" -- because that's oftentimes,
if you're not careful about it, what models learn.
And you're absolutely right; the stakes here are quite high,
so what we want to be very conscious of is not just trying
to get the high score -- which, when I read a lot of papers,
it seems like we're in high-score land rather than
in utility land.
Even when I talk to many entry-level candidates,
a lot of them talk about trying to get the high score through
juicing the data rather than being really careful about how
they think about the modeling process --
so they're very focused on the score: "What's the accuracy?
What's the accuracy?
How do we get the accuracy higher?
Let's get rid of the outliers; that'll
make the accuracy higher."
Well, it turns out the outliers are
the only thing that matters.
So, what we are very concerned about,
of course, is making sure our accuracy is very high,
making sure our square scores, whatever, are very high;
making sure that the metrics that
are associated with business value are incredibly high.
However, in order to make sure we're hedging our risks, what
is as important, if not more important,
is being keenly aware of the distribution of the error
associated with your model.
No matter what project we're working on, whether
it's in our index space, whether it's in our corporate services
space, whether it's in productivity and automation,
or if it's in new capabilities, we want to make sure that
our error is distributed very uniformly,
or at least reasonably uniformly,
across all the constituent groups that we might be
unleashing this model on -- making sure that if there are
areas where it doesn't perform well,
we have a good understanding of the calibrated interval
of our models and systems, so that when we're outside of that
calibrated interval, frankly, at the very least,
we can give somebody a warning to let
them know that they're in the Wild West now
and they should do this at their own risk.
And maybe it's a little caveat emptor
at that point, but at least you know.
Really, I think those are the two most important things
to help manage those risks: being eminently concerned about
the distribution of your error, and being really,
really well aware about where your model works and where it
doesn't.
There's a number of other things that everybody does these days
around [personally identifiable information] protection
and making sure that there's a robust review process involved.
More recently, we've been able to make sure
that every single project we're working on
has at least one other person on it,
so that two people have to agree that this is the best
path forward and that these are the right numbers that
are coming out.
SHERVIN KHODABANDEH: So you gave a very good series
of examples about algorithmically and technically
and mindset-wise some of the steps that folks
need to take to manage and understand the errors
and be ahead of them rather than being surprised by them.
I mean, on one hand ... so you have to have an eye toward
the riskiness of it and how that could be managed.
And on the other hand, you talked
about being the center of excellence
and the place within Nasdaq where
the state of the art in this space is being defined.
How do you balance the need to watch out
for all those pitfalls and errors and conservatism,
with pushing the art forward?
In terms of a managerial orientation,
how do you do that?
DOUG HAMILTON: I think preaching that conservatism internally
to your own team.
When I first started, I had this great manager at Boeing.
On the one hand, when she was reviewing our work,
it was always very, very critical of what we were doing
-- very careful about making sure we're being very careful
and cautious.
And then, as soon as we went to a business partner or a client,
"Oh, this is the greatest thing ever.
You're not going to believe it."
And I think that's a very important part of this;
those two angles of internal conservatism
and external optimism are really very necessary to making sure
that you don't just build high-performing, risk-averse AI
systems, but also that you see rapid and robust maturation
and adoption of the technology.
SAM RANSBOTHAM: Well, it ties back to your talking
about understanding the error distribution.
You can't really get a hold of that
unless you do understand that error distribution well.
Shervin and I have been talking recently about --
it's come up a few times; he'll remember better than I have --
about just this whole idea of noninferiority.
That the goal of perfection is just unattainable,
and if we set that out for any of these AI systems,
then we're never going to adopt any of them.
And the question is, it's like you
say, it's a balancing thing of "How much off
of that perfection do we accept?"
We certainly want improvements over humans,
but we also want improvements over humans eventually.
It doesn't have to be improvement right out
of the gate, if you think that there's
some potential for that.
SHERVIN KHODABANDEH: Let me use that as a segue to ask
my next question.
So you've been in the AI business for some time.
How do you think the state of the art
is evolving, or has evolved, or is going to evolve in the years
to come?
Obviously, technically it has been [evolving], and it will.
But I'm more interested in [the] nontechnical aspects
of that evolution.
How do you see that?
DOUG HAMILTON: When I first got started,
the big papers that came out were probably [on]
the [generative adversarial network] and [residual neural
network]; both came out actually about the same time.
[In a ] lot of ways, to me that represented the pinnacle
of technical achievement in AI.
Obviously, there's been more since then,
obviously we've done a lot, obviously a lot of things
have been solved.
But at that point, we figured a lot of things out.
And it opened the door to a lot of really good AI and machine
learning solutions.
When I look at the way the technology has progressed
since then, I see it as a maturing ecosystem
that enables business use.
So whether this is things like transfer learning,
to make sure that when we solve one problem,
we can solve another problem, which
is incredibly important for achieving economies of scale
with AI groups, or it's things like AutoML
that help to make everybody at least this kind of idea
of a citizen data scientist, where software engineers
and analysts can do enough machine learning research
or machine learning work that they can prove something out
before they bring it to a team like ours
or their software engineering team.
I think these are the sorts of maturing technologies
that we've seen come along that make machine learning much more
usable in business cases.
I think beyond that, historically what
we've seen is the traditional business
case for artificial intelligence have been all-scale plays.
I think these maturing technologies
are these technologies that are allowing
us to mature models, reuse them, and achieve economies of scale
around the AI development cycle.
As these get better and better, we're
going to see more use cases open up
for "Computers are good at it."
And we've certainly seen it when we
look at how hedge funds and high-frequency traders operate.
They're all using machine learning all over the place,
because it's better for research purposes than ad hoc trial
and error and ad hoc rules.
By the same token, we've seen it in game-playing machines
for years.
So the idea that we'll have more and more of these situations
where [the] computer is just better at it,
I think we're going to see that more and more.
Certainly, this is, I think, the thesis
behind self-driving cars, right?
Driving is the thing that people do
worst, that we do most often, and, provided that you can work
out the edge cases, which is really hard,
there's no reason why computers shouldn't be better
at driving than people are.
SHERVIN KHODABANDEH: I was going to ask, what
about those problems where computers alone or humans alone
can't be as good, but the two of them
together are far better than each of them on their own?
DOUG HAMILTON: When there is a computer-aided process
or an AI-aided process, we can then usually break that down
into two things -- at least two processes.
One is a process that the person is good at doing,
and the other is a thing that the computer is doing.
But if you can imagine computer-aided design,
there's many things that a computer
is good at in computer-aided design
that it is helping the person with.
One of them is not coming up with creative solutions
and creative ways to draw out the part that they're
trying to design, but it's very good at things
like keeping track of which pixels are populated
and which aren't, the 3D spatial geometry of it, etc.
And that's what it's good at -- and then,
the actual creative part is what the person's good at.
Maybe a person is not so good at generating new and novel
designs for, let's say, furniture.
Maybe you're Ikea and you want to design new furniture.
So maybe people aren't particularly
good at generating these things out of the blue,
but they're pretty good at looking at it and saying,
"Well, hang on a second.
If you design the chair that way,
it's got a giant spike in the back,
and it's going to be very uncomfortable,
so let's get rid of that, and then let's try again."
So there's this process of generating and fixing,
or generating and editing, that we can break it down to.
And the computer might be better at generating
and the person is better at editing
for these real-world or these latent requirements that
are very difficult to encode.
SAM RANSBOTHAM: All right.
Well, thanks for taking the time to talk with us
and to learn about all that you, and in particular Nasdaq,
are doing.
We've heard about, for example, project selection,
balancing risk, and how you pick those projects.
We learned about how important understanding error is
and all the different possible cases that you see
for artificial intelligence.
It's a pretty healthy bit to cover in just one session.
We appreciate your input on all those topics.
DOUG HAMILTON: Thanks, Sam.
Thanks, Shervin.
It's been a pleasure speaking with you.
SAM RANSBOTHAM: Please join us next time.
We'll talk with Paula Goldman, chief ethical and humane use
officer at Salesforce.
ALLISON RYDER: Thanks for listening
to Me, Myself, and AI.
We believe, like you, that the conversation
about AI implementation doesn't start and stop
with this podcast.
That's why we've created a group on LinkedIn, specifically
for leaders like you.
It's called AI for Leaders, and if you join us,
you can chat with show creators and hosts,
ask your own questions, share insights, and gain access
to valuable resources about AI implementation
from MIT SMR and BCG.
You can access it by visiting mitsmr.com/AIforLeaders.
We'll put that link in the show notes,
and we hope to see you there.