-: How can governance
of artificial intelligence help organizations?
The word governance can come with a lot of baggage
and some negative connotations,
but governance can enable organizations too.
The question is how?
We'll close out the season with a discussion
with Kay Firth-Butterfield.
She's the head of artificial intelligence
and machine learning for the executive committee
of the World Economic Forum.
With Kay, we'll learn not only about her specific background
in the legal profession, but she'll also help us think
about what we've learned overall this season.
Welcome to "Me, Myself, and AI,"
a podcast on artificial intelligence and business.
Each week, we introduce you to someone innovating with AI.
I'm Sam Ransbotham, professor of information systems
at Boston College.
I'm also the guest editor for the AI
and Business Strategy Big Idea Program
at MIT Sloan Management Review.
-: And I'm Shervin Khodabandeh, senior partner with BCG,
and I co-lead BCG's AI practice in North America.
And together, BCG and MIT SMR have been researching AI
for four years, interviewing hundreds of practitioners,
and surveying thousands of companies on what it takes
to build and deploy and scale AI capabilities
and really transform the way organizations operate.
-: Well first, Kay, let me just officially say we're thrilled
to have you talk with us today.
Thanks for taking the time, welcome.
-: Thank you.
-: Of course, so Kay you've got a fascinating job or actually
really jobs that's you've got so many things going on.
So for our listeners, can you introduce yourself
and describe your current roles?
-: Yes, certainly.
I'm Kay Firth-Butterfield, and I am head of AI
and machine learning at the World Economic Forum.
So what that essentially means is that we work
with multi-stakeholders, so companies, academics,
governments, international organizations, and civil society
to really think through the governance of AI.
So when I say governance, I say it very much with a small g,
we're thinking about everything from norms
through to regulation but AI, we feel, is less susceptible
to regulation.
-: Can you tell us how you got there?
Give us a little bit of background about your career to date
and how did you end up in this role?
-: I am by background a human rights lawyer.
I am a barrister, that's the type of trial lawyer
that wears the wig and gown.
And I got to a point in my career where I was being
considered for a judicial appointment.
In the UK they kindly sort of try out whether
you want to be a judge and whether they think you're really
good at it.
I don't know what their view was, but my view was that
it wasn't the culmination of a career in the law
that I really wanted.
And I had been very interested in the impact of technology
on humans and human rights,
and so it gave me this wonderful opportunity to rethink
where my career would go.
So I was fortunate to be able to come to Austin and teach
AI, law, international relations to pursue my own studies
around law and AI and international relations and AI
and the geopolitical implications of this developing
technology.
And then purely by luck, met a person on a plane
from Heathrow to Austin, it's 10 hours.
He was this chair and CEO of an AI company who was thinking
about AI ethics.
And this was back in 2014 when hardly anybody apart from me
and the dog and some other people were thinking about it.
And so he asked me as we got off the plane if I would like
to be his chief AI ethics officer.
And so that's really how I moved into AI, but obviously
with the social justice, with the ideas of what benefits
AI can bring to society and also cognizant of what we might
have to be worrying about.
And so I have been vice chair of the IEEE's initiative
on ethically aligned design since 2015.
I was part of the Asilomar Conference thinking about
ethical principles for AI back again in 2015.
And so my career ended up with me taking this job
at the forum in 2017.
I say ended up but maybe not, who knows?
-: Yeah, we won't call it an end just yet.
So what does artificial intelligence mean?
-: Well, part of the problem and part of its complexity
is that AI means different things to different people.
So AI means one thing to an engineer and another thing
to a person who's using it as a member of the public
or through their phone.
-: So we're shifting our definition as we go
and we'll continue to as well.
-: Yeah, there's that old adage that it's not in artificial
intelligence once it's done.
-: And how much of that do you think is education
and is sort of stemming from lack of understanding and lack
of education versus a technical or process complexity
inherent in putting all that governance in place, right?
I mean, I guess part of it is you can't really manage
or govern that which you don't really quite understand.
Is that most of the battle and once everybody understands it
because it's common sense, then they begin to say well,
now how we could the govern this like anything else we would
govern because now we understand it?
-: Yes, well, I think that it's organizational change,
it's education and training for employees, but it's also
thinking very carefully about product design so that
if you are actually developing algorithmic product,
what's the path of that from the moment that you dream up
the idea to the moment that you release it, either to other
businesses or into customers and maybe even beyond that.
-: I couldn't help but pick up on one of the things you said
about governance as being negative.
But one of our studies a few years ago found that healthcare
shared data more than other industries.
And that seems counterintuitive, but when we dug into it
what we found is they knew what they could share.
They had structure about it and so that structure then
enabled them to know what they could do,
know what they couldn't do.
Whereas other places when they talked about data sharing,
they were well, let's have to check with our compliance
department and have to check and see what we can do.
And you know, there's much less checking because it's
explicit and the more explicit we can be.
And that's an enabling factor of governance versus
this sort of oppressive factor of governance.
-: Yes, I think just governance has got itself a bad name
because, you know, regulation impedes innovation,
and that's not necessarily so.
I think that at the moment, we're exploring
all these different soft governance ideas.
Largely because to begin with, yes, we will probably see
regulation, the EU said we will see regulation out of Europe
around things like facial recognition and use of AI
and human resources because they're classified as high risk
cases, but a lot are not necessarily high risk cases.
What they are are things that businesses want to use,
but they want to use wisely.
So what we have done as well as create a lot of toolkits
for example, and guidelines and workbooks.
Say that companies or governments can say, "Oh yes,
this can guide me through this process" of, for example,
procurement of artificial intelligence.
Just to give you an example, we surveyed a number
of our members of boards on their understanding
of artificial intelligence.
They didn't really understand artificial intelligence
terribly well.
And so what we did was developed an online tool for them
to understand artificial intelligence but also then to say,
"Okay, my company is going to be deploying artificial
intelligence, what are my oversight responsibilities?"
and long questionnaires of things that you might want to ask
your board if you're on the audit committee or the risk
committee or you're thinking about strategy.
So really digging into the way that boards should be
thinking across the enterprise about the deployment of AI.
-: Yeah because I'm guessing most people need that guidance.
-: Yeah, most people for sure need that guidance,
and I think this is a very well-placed point you're making.
What we don't want to happen is to be so far behind
in understanding and education and governance
of any technology, where then it becomes such a black box
that it's a huge activation energy for anybody to get there.
And we heard that also from Slawek Kierner from Humana,
we heard that from Arti at H&M, was the importance of really
big cross-organizational training, not just for the board
and not just for the handful, but for everybody almost like
you know, I think we heard from Porsche that they actually
did training for their entire technology organization.
This is AI, this is what it could do right,
this is what it could do wrong, it's what you need to learn.
And by the way, this is how it can give you all these
new designs that you as a engineer or a designer can explore
to design the next-generation model.
And this is how it could be your friend.
But I think you're pointing out that it's time for us
to really internalize all of these as not nice to haves
but critical even I would say almost first step
before getting too far ahead.
-: Yes, absolutely.
And in fact, there's a company in Finland that requires
everybody to learn something about AI even at the very most
basic level, and they have a course for their employees
which is important.
Obviously not everybody can master the math, but you don't
even have to go that far.
-: Or should, I can't help but build off of your
human rights background.
One of the things that strikes me is there's incredible
advances with artificial intelligence use by organizations,
particularly large organizations, particularly well-funded
large organizations.
How do we as individuals stand a chance here?
Do we each need our own individual AI working for us?
How can we empower people to work in this perhaps
lopsided arrangement?
-: Yes, I think the imbalances of power is something that
we have to address as both individuals and as companies.
You know there are some companies with more AI capabilities
than others as non-profits and also as a world because
at the moment the concentration of AI, talent, skills,
and jobs is very skewed around the world.
And we really have to think globally about how AI
is deployed on behalf of humans.
And what makes us human and where we want to be maybe
in 15 or 20 years when AI can do a lot of the things
that we are doing currently.
So I think that it's systemic and structural conversations
that we have to have in all those different layers as well.
-: Right, the systemic and structural issues are big
because, I have to say, I don't think most companies intend
to start AI with an evil bent.
I mean, they're not cackling and rubbing their hands
together and applauding.
I think these things are more insidious
and systemic than that.
So how do we do that?
-: In my experience of working with a lot of companies,
governments, et cetera, I would say you're absolutely right.
Companies want to go in doing the right thing,
and what we need to be doing is making sure
that we help them do the right thing.
And it's very much that perhaps a lack of understanding
of the technology is going to skew how they use it.
And so those are all areas that we have been trying to focus
on at the forum so that people who go into using AI with
the right mindset actually come out with the right results.
And you know, your company is a little piece of society.
The idea should be that everybody works together because
you're actually going to end up with a better product.
And I think to your point, the better we enable our
customers or the general public to understand AI,
the less scary it will be.
I also fear that there are many companies
that are being told to go out and get AI.
And they actually don't know what it is that they're getting
or really what the benefit is going to be
or what the downsides might be.
So having the board being capable of asking
the right question is absolutely crucial but, you know,
we're currently working on a similar toolkit for different
types of C-suite officer so that they too can be empowered
to understand more.
But I also see the need for thinking carefully about AI
as a top down and bottom up.
You know, that's why I go back to that survey that you did,
where understanding across the organization
is actually so important.
And I think where you're seeing some of the developments
amongst the companies that have been dealing with this like,
Microsoft, they went for an ether committee.
They went for really sort of thinking about strategically
how are we using AI.
And so I think that we have the benefits of what they learnt
early on that we can then begin to bring into the sector,
from board to designer.
-: And the good part about that is that education component
keeps it from just being ethics theater, kind of the thin
veneer to put the stamp on it and check the box that yes,
we've done the ethics thing.
But I guess what's the role for business in trying to
educate people to have a better human-machine collaboration?
Obviously we've heard a lot about the potential for AI
to affect workplace and job security, but people are already
incredibly busy at work.
What potential is there for AI to kind of free us from
some of these mundane things and lead to greater innovation?
When we talk with Gina Chung at DHL, she's in the innovation
department and that's where they're focusing AI efforts.
Is this a pipe dream or is there a potential here?
-: No, I think that it's certainly not a pipe dream
and most people have innovation labs both in the companies
and countries and UNICEF has an innovation lab,
we were talking about children and AI.
So the potential for AI to free us from some of the things
that we see as mundane, the potential for it to help us
to discover new drugs, to work on climate change.
There are all the reason that I stay working in this space,
and you might say, "Well, you work on governance,
doesn't that mean that you just see AI as a bad thing?"
And that's not true.
Just as an example, at the moment we have problems
just using Zoom for education because there are many kids
who don't have access to broadband.
So that brings us against the questions of rural poverty
and the fact that many people move from rural communities
to cities and yet, if we look at the pandemic,
cities tend to be bad for human beings.
So all the conversations that we should be having
I'm thinking about the innovations that AI will create,
which allow that sort of cross-function of rural
to be as wealthy as city.
We should be having really deep structural conversations
about what our future looks like.
Does it look like "Blade Runner" cities?
Or does it look like something else?
-: You were mentioning, I guess, I was suggesting kids
were one extreme and yet you were all ready had been talking
about board level which seems like another extreme.
It seems like there's a lot of other people between
those two extremes that would need to learn how to work
together alongside and I guess just looking
for some practical, how do businesses get people
to be comfortable with their teammate as a machine
versus their teammate as a normal worker?
Actually for example, we've seen people completely
impatient with robots.
You know, if it's not perfect right off the bat then
why am I bothering teaching this machine how to do this?
You'd never be that impatient with another co-worker.
You remember when you were first learning to do a job.
So how do we get that same sort of I guess maybe empathy
for the poor little machine?
-: Yeah well, I think as I say I do think it's an education
and training piece that that the company has to put in
but also it's important because sometimes we over trust
to the technology, so the computer told us to do it.
You know that something that we'd been noticing for example,
in the criminal sentencing problems that we've been having,
where judges have been over reliant upon the fact
that the machine's telling them this.
And so it's that education to not over trust the machine
and also trust the machine is not going to take your job,
is not going to be spying on you,
you know, there is sort of a lot of things that employees
are frightened of and so you've got to make sure
that they have some better understanding of what that robot
or machinery is going to do with them.
And that it's a human-machine interaction as opposed
to one dominating the other.
-: What's your thinking on to bring about large-scale
understanding and change, not just at the board level,
but from the fabric of the organization, how important
is that companies begin to understand the different modes
of interaction between AI and human
and begin to test some of those things?
-: Obviously, that's really important.
We do have a project that's actually led by Salesforce
called the Responsible Use of Technology.
And we see in that, what we're trying to do is to bring
together all the different companies, like BCG,
who are actually thinking about these issues
and come up with some best practices.
So how do you help your employees to really think
about this interaction with AI?
How do you make sure that the company itself is focused
on ethical deployment of technology,
and where your employees are going to be working
specifically with the technology that they don't fear it?
I think there's a lot of fear and that is at the moment
probably not useful at all.
-: You clearly can't be friends with somebody
if you're afraid of them.
-: Yes, and what we are seeing is that, you know,
when I was talking about AI and ethics in 2014,
very few people were talking about it.
Now everybody, not everybody, but every enlightened person
is talking about it and business is talking about it.
And we're talking about business here.
Businesses talking about it, governments talking about it.
Governments are talking about it in the, if there
is something that is unsafe usually we regulate the unsafe.
So I think actually the time is now to be having
these conversations.
Do we regulate?
Do we depend upon more soft law approaches?
Because what we are setting now in place is the future.
And it's not just our terrestrial future, but that if we're
going to go to Mars, we're going to use a lot of AI.
We need to be really having these conversations,
and one of the things that we have been doing is having
a conversation that looks at positive futures.
So you can sort of look across the panoply of sci-fi
and it's almost all dystopian.
And so what we wanted to do is say, "Okay,
we have this potential with AI, what do we want to create?"
And so we brought sci-fi writers and AI scientists
and business and economists and people together
to really, sort of, have that conversation.
So we're having the conversation about AI ethics,
but the next conversation has to be how do we systematically
want to grow and develop AI for the benefit of the world
and not just sectors of it?
-: I could recall the flavor of these kinds of conversations
I would have five years ago,
it was very heavily tech focused.
What does that tell you in terms of a profile of, you know,
future leaders of AI, what is the right sort of traits,
skills, sort of profiles do you think?
-: I think we will see, so I have a humanities background,
I think we'll see more humanities so, you know,
there's the AI piece that the technologists have to work on.
But what we do know is that, there's a Gartner study
that says that by 2022 if we don't deal with the bias,
85% for algorithms will be erroneous because of the bias.
If that's anywhere near true, that's really bad for your R&D
and your company.
So what we know is that we have to create
those multi-stakeholder teams, and also I see the future
of AI, this discussion as part of ESG.
So I see the AI ethics discussion moving into that more
social realm of the way that companies think about
some of the things that they do.
-: And that's something that we heard from for example,
Prakah at Walmart that they're, they're thinking big picture
about how these would connect and remove inefficiencies
from the process and that certainly has ESG implications.
What we've seen with some of the other folks we've discussed
artificial intelligence in business with, is that
they've transferred learning from things that they've done
in one organization to another.
They've moved this education component that you've mentioned
before, does not happened within companies, it's happened
across companies and it's happened across functional areas.
How do we encourage that?
How do we get people to have those adverse experiences?
-: Yes, I think that that's a. right
and b. really important that we do.
So I was actually talking to somebody yesterday who had set
up some really good resources and training around artificial
intelligence in a bank then moved to government
and then moved to a yet another private sector job
and is doing the same thing.
And many of the trainings that we need to be thinking about
with artificial intelligence are cross-sectoral.
So we did an interesting look at all the ethical principles
that are out there, there are over 190 now from The Beijing
Principles through to the Asilomar ones, et cetera.
-: That's different from 2014.
-: It's very different from 2014.
And one of the things that a lot of people sort of have said
to me in the past is well, whose ethics are you talking
about anyway?
And what we found was actually there were about 10 things
that were ubiquitous to all of those 190 different
ethical principles.
So there are 10 things that we care about as human beings
wherever we are in the world.
And those are 10 things that are actually fairly crossed
for sectorial, so there are about safety and robustness.
They're about accountability, transparency,
explainability.
They're about that conversation we had earlier
human-machine interaction.
Then there are about how does AI benefit us as humans.
So I think that ability to be able to take what
you've learned in one sector and move it to another
is important and relatively straightforward.
-: And also it seems very human.
-: Yeah.
-: That's something I think that the machines themselves
are going to struggle with and need
at least our help for a while.
-: Oh, undoubtedly yes.
And it probably doesn't need saying to this audience,
but it's worth saying that these machines
are not really very clever yet.
-: Yeah, there's still time, we're still okay.
-: Thank God for that.
(men and woman laughs)
-: Okay, thank you for taking the time to talk to us,
we've really enjoyed it.
-: Yeah, thank you so much Kay.
It's been a pleasure hearing your views
and your leadership on this topic.
-: Thank you so much to both of you,
it's been a pleasure and a privilege to be with you.
I could have talked on for hours.
-: But we can't because that is the end of our episode,
and that is the end of our first season.
Thank you for joining us on this podcast.
-: Thank you very much.
(calm upbeat music)
-: Thanks for listening to "Me, Myself, and AI",
if you're enjoying the show, take a minute
to write us a review.
If you send us a screenshot, we'll send you a collection
of MIT SMR's best articles on artificial intelligence
free for a limited time.
Send your review screenshot to smrfeedback@mit.edu.
(calm upbeat music)