- You might not often hear terms like empathy
and design thinking
when talking about AI projects,
but on today's episode,
find out how one pharma company's AI center of excellence
takes a holistic approach to technology projects.
(cheerful electronic music)
- I'm Tonia Sideri from Novo Nordisk
and you're listening to "Me, Myself and AI."
- Welcome to "Me, Myself and AI",
a podcast on artificial intelligence and business.
Each episode,
we introduce you to someone innovating with AI.
I'm Sam Ransbotham, Professor of Analytics
at Boston college.
I'm also the AI and business strategy guest editor
at "MIT Sloan Management Review."
- And I'm Shervin Khodabandeh, senior partner with BCG
and I co-lead BCG's AI practice in North America.
Together MIT SMR and BCG have been researching
and publishing on AI for six years.
Interviewing hundreds of practitioners
and surveying thousands of companies on what it takes
to build, and to deploy, and scale AI capabilities,
and really transform the way organizations operate.
- Today. Shervin and I are joined by Tonia Sideri,
head of Novo Nordisk's AI center of excellence.
Tonia, thanks for joining us. Welcome.
Let's get started.
First, maybe, can you tell us what Novo Nordisk does?
- We are a global pharma company.
And we are headquarted here in Denmark,
and we are focusing on producing drugs,
supporting patients with the chronic diseases,
such as diabetes, obesity, hemophilia, and growth disorders.
We are a 100 year-old company, but still growing a lot,
but still very committed to original values of the company
and to our social responsibilities.
There's more than 34 million diabetes patients
using our products,
and we produce more than 50% of the world's insulin supply.
- Currently you lead the AI center of excellence.
So what is an AI center of excellence?
What is your role there? What does that mean?
- AI center of excellence can have different flavors in
different companies, but what we do,
we are a central team located in the company's Global IT.
We are a group of data scientists,
machine learning engineers, and software developers
working via hub-and-spoke model across the company.
So we want to minimize our distance from ourselves
and our experts in the company- our data and domain experts-
by working in cross functional teams, product teams,
across the company.
And we also want to increase the speed
from where we go from a POC of machine learning model
to a production.
And that's why we have analytics partners
working across the company.
And we also have MLOps product team focusing on
creating microservices across the whole machine learning
model lifecycle.
We want to take all the petabytes of data
we consume as a company,
all the way from our molecule identification
to our clinical trials,
to our commercial execution and production
and shipping of the products,
and take them from database, from flat files,
from cloud storage
and convert them to something that is ultimately useful
for the company and ultimately supports patients' lives.
And that's what we are here for.
We want to bring this data to life.
We are around one and a half year old as a team,
and we already have projects across the company.
And we're working with our R and D, for example,
with using knowledge graph
to identify molecules for insulin resistance.
We have deployed different marketing mix model links
and sales uplift recommendations models
across our different commercial regions.
And last but not least,
we have recently deployed a deep learning machine learning
model that use a vision inspection in our inspection lines.
And that's very important
because it's an optimization on an existing process.
However, it gave us a lot of skills
of how to have live machine learning models
in a very regulated setup,
which is a GMP set up,
good manufacturing practices one.
- How does that work? Tell us more about that.
That seems quite interesting.
- We were already using visual inspection the last 20 years
from a rule-based approach that we have optimized.
And now we have used different deep learning models
to improve that.
And of course, with deep learning,
we are increasing the accuracy and the efficiency
of the visual inspection process
and thereby increasing quality
and reducing the amount of good product going to waste
due to being wrongly identified as defect.
So we save product and we optimize our products that way,
in a more efficient way.
And we also produce less waste
of good cartridges going to waste.
But most importantly,
what we get out of this project is the necessary capability
of how to do machine learning in very regulated spaces.
For example, like manufacturing of pharma.
- Tonia, you've been a big advocate of design thinking
in building data products, AI products.
Tell us more about what that means and why it's important.
- Yes, and I think it started first of all by-
I used to be a data scientist myself.
So quite sometimes I found myself working on projects
that could see that they should have been killed earlier.
So my interest in this is how to speed up
our time for failure.
And that's why, when we started the area-
and that was one and a half year ago-
we really committed to actually start our projects
by what we call a data-to-wisdom sprint.
So basically a hackathon, that we work together
with our business colleagues in a period of two weeks,
to really try to see what we can find from the data,
based on specific hypothesis.
And in the end of these two weeks,
then we ask ourselves, "Is there any signal in the noise?
Are the data good enough?
Do we have the necessary technology to scale it further?
And is there any business value out of this?"
And if the answer is yes,
then we go to the next step where we do to a POC,
then to implementation phase and of course operations.
But if the answer is no, then within two weeks,
very quickly, we should be able to kill it.
And these two weeks we really use,
with the help of agile coaches,
also some design thinking techniques.
But for me, it's the outcome of the design thinking,
how do use design thinking as a way
to work cross-functionally and as a way to fail fast.
- That's great.
No wisdom, you're killed.
Sort of like - Exactly.
natural selection, right?
Joking aside, I think this is a great idea
because Sam, like how many times we either see in our data,
when we survey these thousands of companies,
or in our conversations with executives,
where they are doing hundreds of POCs and pilots
but there is just literally no value.
And there is truly what I call, AI fatigue,
across the organization.
'Cause it's like, the whole organization has become
this graduate school of lab, of like,
"Let's try this, let's try that. Let's try that."
So I love the idea of like,
just kill the ones that aren't working,
so you focus on a handful that are valuable.
- Exactly. And for me also, those that are not working,
we also haven't got a lot of learnings.
Because usually the reason that they're not working
is related to data.
So at least we stress test the data for two weeks
based on what we want to achieve.
And then we get some learnings.
If we want to do this model in the future,
what we need to fix in our data to get there.
- Ooh, that's fabulous
because that's actually tying back
and learning from what you-
I mean, it's one thing to just kind of cut a project off
and say, "All right, well,
we're not going to keep dumping money into that
if it's not going to work."
But then there's something else to,
if you keep starting projects just like that
over and over again,
there'd need to be some learning that those are going to fail
or what you can do to improve those in the future.
What kind of numbers are we talking about here?
How much wisdom is there? Is there 2% wisdom, 20% wisdom,
97% wisdom?
- I think it's very dangerous
to try to quantify something like this, right?
(Shervin chuckles)
But one is the data wisdom.
And the other, of course, is the change management wisdom,
because we work together through this hackathon
with our business experts.
So even something fails,
they understand their way of working,
and also we get a glimpse of their reality,
and they get a glimpse of what can be possible.
And I think this wisdom is even more difficult to quantify
because it will have a,
hopefully, a more kind of a wave impact effect
in the future across the company.
- If you look at the total opposite paradigm
for what you're talking about,
is the old-school waterfall way of building
these like, gigantic tech pieces, right?
It was like tech development 20 years ago,
where I remember with the project,
we looked at a hundred companies
building these massive tech products.
And I think it was like 80% of these companies
were building features and functionality
that either nobody needed
or could not be used with the rest of the technology,
but they would only find this out
like 18 months after development had started.
I guess it's totally new way.
But sadly, there's still many organizations
that are operating with that old paradigm
and they spend months in business requirements gathering
and planning and all that.
And I think what you're saying is, let's get a good idea.
Let's start testing.
If it's got something there,
then we double down and we make it big.
But if it doesn't,
then we've learned something.
And if that project, that idea was important,
then we could fix it.
And I really, really like also your point around,
it's not just the technical part,
it's also the change management
and what it takes for it to work.
It's really, really good.
- Exactly. And by saying from that in advance,
then we have no risk of failure because it is how we work.
We have two weeks,
so it's not going to be our reputation on the line
if the project doesn't continue.
And having gated steps also, after even the MVP phase,
also that the ability to kill something there.
And I think that helps,
and also the budget.
The reason that a lot of companies have these long projects
is because they have long paths allocated to this.
But in our case,
we also assess if there's any willingness to pay
from our business sides.
Is what we do useful enough
that our business is willing to invest in it.
- Set the expectations up front.
Sam, imagine you're, you know,
Sam's a college professor, your students come in and say,
professor, I'm warning you ahead of time.
I will fail in two weeks.
- No, no,
Actually it is the opposite.
I go in and say, 90% of you are going to fail.
And I don't think that would go over very well.
Tonia, how do you transfer these learnings back?
You mentioned that you do that.
Is there a process for that?
How do you codify?
How do you make these things explicit and not just lore?
- That's a good question.
And while we grow, we still have to find out
what's the right level of quantification
that is not bureaucratic as well.
But what we do is, first of all, during these two weeks,
we have two demos across the organization,
and especially with the business unit
that we are working on.
So at least that's the change management part
from a broader perspective,
not only from the people are working in the product team.
And then, regarding the data improvements
or technology improvements,
then we bring them back to our data governance
or to the data owners or to our technology organization.
- Okay. That makes sense.
One of the things you talked about,
and something that Shervin and I, I think,
are seeing overall- is that there's a,
let's say, an increase in the-
maybe in the maturity that we're seeing.
I don't know, Shervin,
maybe I'm reading too much into offhand comments
that people are making.
But I'm just seeing much more process getting put in place
around what used to be very ad hoc.
And maybe you're a couple of steps ahead of this, you know,
looking at some of your building block approaches
to making different services consumable.
Can you explain how that works
and how you're developing these building blocks
and how other people are using them?
- Yes. So of course these building blocks
and the idea of providing MLOps services or, in general,
data services comes very much from this yeah,
data mesh approach.
And now it's the new hype,
but especially for the MLOps work,
what I can speak about is,
based on our learning of how long it took
to get the machine learning model validated,
now we are creating microservices-
wrapping existing services -
either open source or from our cloud vendors, all the way
from how we do model versioning, model monitoring,
model validation, ground truth, storage validation,
and then validating these services
as qualified system from a pharma setting.
And in that way,
we reduce the time to market
from when we need to validate a GXP model,
because then we don't expect any data scientists
in the organization to build their own cloud solutions,
to be both a data engineer and software developer
and a validation experts to bring the model into production.
Because by using this pre-qualified validate services,
they can just focus on data science
and use them as components.
And we just building the first service based on a learning
from this visual inspection model.
- This is such a great point.
If you look at a typical data scientist in a company,
there'll be such a wide variation
of how much of their time's actually in,
what you call extracting wisdom or patterns
or building models and testing,
versus all the other stuff that's prep work
and setting up the environmental feature engineering
and things that somebody else has already done,
but in another part of the organization.
I want to ask you, Tonia, about talent.
I mean, you're talking about a way of working
that is driven by design thinking, fail fast,
highly interconnected with the business.
What is the profile of the right skill sets
from a data scientist, engineering perspective
that's going to be successful in that environment?
- That's a good question.
I think the technical skills, of course,
should be given there,
and I can also see the market over time
is getting more and more mature.
So it's easy to find those.
But what is more difficult is these other, softer skills
that make you a good value translator and a collaborator.
And for me, the most important skill of a data scientist
is actually empathy.
Something we don't expect from people from a technical field
usually.
It's the ability to go to the business person mind
and ask themselves if I was a marketeer,
if I was a production operator,
and I had to do the job every day
and I had the problems that they have,
how would I use the data
to something that would be useful for me?
Being able to make this mental leap
needs a lot of understanding of what is the reality
of the other person and ability also to communicate.
So empathy, and of course, curiosity
about the application of your machine learning models
and the other person.
And that's very difficult skills to quantify
or interview for.
It's more a cultural or a character trait.
- It's interesting Shervin.
We're seeing this, maybe this first indication of,
it's getting easier to find these technical skills.
I mean, I think that's an interesting transition.
- Yep. That's like become more of a-
as Tonia, as you're saying-
the table stakes that you need just to get started.
But the real value is the softer skills and empathy,
and it ties well, Sam to what we're seeing as well,
which is when we look at the evolution of companies
that are investing in AI
and we see that technology and data
is only going to get them so far,
but that big leap is all around organizational learning
interactivity with the business process change.
- At least to be fair about data scientist.
There's still a lot of shortage for machine learning
engineers or data engineers or software developers,
but for data science,
because it becomes more mature as a field technically,
it's all the other skills
that can differentiate somebody.
- Tonia, what are you excited about next?
What's coming with artificial intelligence?
I mean, we're focusing on AI and machine learning.
What are you excited about?
What's coming down the pipe?
- I'm actually excited on data now.
It's not so AI related,
but I think it's regarding to the new trend
that now it's data based.
Like in order to fix our digital intelligence and optimize,
let's optimize our data first.
We also are actually going,
investing more in the data mesh concept now.
So for example, treating data as the product.
Meaning that every time we want to make a new,
let's say marketing mix modeling,
we don't have to go through the whole ETL.
- Yeah, no, this is-
I once did a study 10 years ago,
small group, maybe like a couple of hundred people
in one company,
but like 80% of their data scientist time was spent on ETL.
And then yet they had a data engineering group.
- Mm.
- And the irony of it was like-
you're talking about marketing mix optimization,
this was actually for the marketing department.
You've got data scientists next to each other
in two cubicles, working on something,
using exactly the same data pipeline,
but building it from scratch.
Both of them, not even knowing
that they're using the same foundational features.
And yeah, that's a big deal.
- Tonia. I know that you're excited about that.
Because you talk about that in terms of tech indulgence,
it seems very related there.
That Ikea effect perhaps.
- Yes. The tech indulgence.
Yes. For me, that's actually the worst sin
that we make is technical people
because the Ikea effect is the ability, I think,
to give a higher value to something that you build yourself.
And sometimes we tend to stay in a project
because we build it ourselves
or because we think it's so cool
to try the new machine learning algorithm.
And for me,
this tech indulgence is the biggest danger you can have.
And that's why it's important to avoid this risk
by working closer with the business
and actually working these product teams from a hackathon
all the way to an operational product team.
- I love that term, tech indulgence.
- Tonia, we have a segment
where you ask you a series of rapid- fire questions.
So just answer the first thing that comes to your mind.
what's your proudest AI moment?
- I think this visual inspection problem,
we mentioned, not only for the business impact,
but especially for the capability providers,
how to use machine learning in a GxP setting-
and how quickly we work together as a team
with our business experts, with our manufacturing experts
to make this possible.
And how quickly it actually got
to actually get it validated.
- I thought that might be your example
because of how animated you were when you were talking
about that, we can see this in video,
but I think it probably comes across from your voice too.
What worries you about AI?
- As probably everybody on the show says,
how it can be used also,
as a way to replicate our own biases.
But on the other hand,
I think technology also has the ability
to decode these biases,
because maybe it's easier to remove these biases
from technology than with people in the first place.
So it's a double-edged sword,
but it worries me that we can replicate our own biases.
- Bias is a common concern for everyone.
What is your favorite activity that involves no technology.
- Reading books. Definitely.
And I try actually not to use even my Kindle for that,
to be like physical 3D book.
And I can really recommend,
I just finished Ishiguro's book "Klara and the Sun",
about actually an AI robot that lives in a family
and starts getting feelings about this family.
I can really recommend that.
- Well, that sounds great. Actually, I need a new book.
- I love that.
My 12 year old boy grew up in the age of Kindle,
and you know, screens and reading books.
And so the first time he got an old school book
from the library, he was like,
"Dad, these books smell wonderful.
Like, what is this smell?" (laughing)
And I was like, "Yeah."
It's an amazing smell
that even a child of today's day and age can appreciate.
- What was the first career you wanted as a child?
What did you want to be when you grow up?
- It's very weird,
but I want to be a garbage collector
in the surprise of my mother.
- Me too. Me too!
- Really?
(Tonia laughing)
That's a very rare chance to find the fellow...
- Yes. Fellow garbage collector enthusiasts.
- But I tend to think it's somehow related, right?
I mean, you take something
and you convert it to something else
and we collect data and we convert them to something else.
- Yeah, I'm sure there's some garbage analogy in there too
with the data that's perfect.
What's your greatest wish for AI in the future?
- I will say to be really democratized,
but I don't really believe
that it will get democratized anytime soon
because it needs so much conceptual understanding
to really get democratized
that I don't think we're going to get there.
But that's my real wish that everybody has the tools,
but more also know how to use them.
- So by democratize,
you mean everyone has access to those tools?
- Yes, and I think already there's so many platforms there
that can help to have this low-code AI.
But it's more has access to the tool,
but also be able to use them.
So has the right level of necessary knowledge,
enough to be able to use them
and be independent in using them.
And I think for that, it will take a lot of time
because it's not a tool thing.
It's more, again, change management, an educational, thing.
- Tonia, great meeting you.
I think that a lot of what Novo Nordisk has done
with systematizing and developing processes
around machine learning and AI
are things that a lot of organizations could learn from.
We've really enjoyed talking to you. Thank you.
- Yeah, it's been really a pleasure. Thank you.
(electronic music) - Thank you.
- Please join us next time.
When we talk with Jack Berkowitz, chief data officer at ADP
- Thanks for listening to "Me, Myself and AI."
We believe, like you,
that the conversation about AI implementation,
doesn't start and stop with this podcast.
That's why we've created a group on LinkedIn
specifically for listeners like you.
It's called AI for leaders.
And if you join us,
you can chat with show creators and hosts,
ask your own questions,
share your insights
and gain access to valuable resources
about AI implementation from MIT SMR and BCG.
You can access it by visiting mitsmr.com/AIforLeaders.
We'll put that link in the show notes
and we hope to see you there.
(ending tones music)