WEBVTT 1 00:00:02.715 --> 00:00:04.710 - You might not often hear terms like empathy 2 00:00:04.710 --> 00:00:05.820 and design thinking 3 00:00:05.820 --> 00:00:07.800 when talking about AI projects, 4 00:00:07.800 --> 00:00:09.180 but on today's episode, 5 00:00:09.180 --> 00:00:12.030 find out how one pharma company's AI center of excellence 6 00:00:12.030 --> 00:00:14.689 takes a holistic approach to technology projects. 7 00:00:14.689 --> 00:00:16.320 (cheerful electronic music) 8 00:00:16.320 --> 00:00:18.630 - I'm Tonia Sideri from Novo Nordisk 9 00:00:18.630 --> 00:00:21.190 and you're listening to "Me, Myself and AI." 10 00:00:22.283 --> 00:00:24.300 - Welcome to "Me, Myself and AI", 11 00:00:24.300 --> 00:00:27.540 a podcast on artificial intelligence and business. 12 00:00:27.540 --> 00:00:28.373 Each episode, 13 00:00:28.373 --> 00:00:31.410 we introduce you to someone innovating with AI. 14 00:00:31.410 --> 00:00:34.050 I'm Sam Ransbotham, Professor of Analytics 15 00:00:34.050 --> 00:00:35.580 at Boston college. 16 00:00:35.580 --> 00:00:38.850 I'm also the AI and business strategy guest editor 17 00:00:38.850 --> 00:00:40.980 at "MIT Sloan Management Review." 18 00:00:40.980 --> 00:00:44.550 - And I'm Shervin Khodabandeh, senior partner with BCG 19 00:00:44.550 --> 00:00:47.850 and I co-lead BCG's AI practice in North America. 20 00:00:47.850 --> 00:00:51.450 Together MIT SMR and BCG have been researching 21 00:00:51.450 --> 00:00:54.060 and publishing on AI for six years. 22 00:00:54.060 --> 00:00:56.070 Interviewing hundreds of practitioners 23 00:00:56.070 --> 00:00:58.770 and surveying thousands of companies on what it takes 24 00:00:58.770 --> 00:01:02.100 to build, and to deploy, and scale AI capabilities, 25 00:01:02.100 --> 00:01:04.833 and really transform the way organizations operate. 26 00:01:07.560 --> 00:01:10.380 - Today. Shervin and I are joined by Tonia Sideri, 27 00:01:10.380 --> 00:01:12.990 head of Novo Nordisk's AI center of excellence. 28 00:01:12.990 --> 00:01:14.760 Tonia, thanks for joining us. Welcome. 29 00:01:14.760 --> 00:01:16.080 Let's get started. 30 00:01:16.080 --> 00:01:19.410 First, maybe, can you tell us what Novo Nordisk does? 31 00:01:19.410 --> 00:01:21.210 - We are a global pharma company. 32 00:01:21.210 --> 00:01:23.370 And we are headquarted here in Denmark, 33 00:01:23.370 --> 00:01:25.950 and we are focusing on producing drugs, 34 00:01:25.950 --> 00:01:29.430 supporting patients with the chronic diseases, 35 00:01:29.430 --> 00:01:34.430 such as diabetes, obesity, hemophilia, and growth disorders. 36 00:01:35.070 --> 00:01:38.850 We are a 100 year-old company, but still growing a lot, 37 00:01:38.850 --> 00:01:42.150 but still very committed to original values of the company 38 00:01:42.150 --> 00:01:44.070 and to our social responsibilities. 39 00:01:44.070 --> 00:01:47.760 There's more than 34 million diabetes patients 40 00:01:47.760 --> 00:01:49.380 using our products, 41 00:01:49.380 --> 00:01:53.737 and we produce more than 50% of the world's insulin supply. 42 00:01:53.737 --> 00:01:57.450 - Currently you lead the AI center of excellence. 43 00:01:57.450 --> 00:01:59.670 So what is an AI center of excellence? 44 00:01:59.670 --> 00:02:02.460 What is your role there? What does that mean? 45 00:02:02.460 --> 00:02:05.040 - AI center of excellence can have different flavors in 46 00:02:05.040 --> 00:02:06.840 different companies, but what we do, 47 00:02:06.840 --> 00:02:10.473 we are a central team located in the company's Global IT. 48 00:02:11.340 --> 00:02:13.710 We are a group of data scientists, 49 00:02:13.710 --> 00:02:16.920 machine learning engineers, and software developers 50 00:02:16.920 --> 00:02:21.920 working via hub-and-spoke model across the company. 51 00:02:21.960 --> 00:02:26.040 So we want to minimize our distance from ourselves 52 00:02:26.040 --> 00:02:28.920 and our experts in the company- our data and domain experts- 53 00:02:28.920 --> 00:02:31.920 by working in cross functional teams, product teams, 54 00:02:31.920 --> 00:02:33.600 across the company. 55 00:02:33.600 --> 00:02:36.600 And we also want to increase the speed 56 00:02:36.600 --> 00:02:39.243 from where we go from a POC of machine learning model 57 00:02:39.243 --> 00:02:40.830 to a production. 58 00:02:40.830 --> 00:02:44.070 And that's why we have analytics partners 59 00:02:44.070 --> 00:02:45.330 working across the company. 60 00:02:45.330 --> 00:02:48.780 And we also have MLOps product team focusing on 61 00:02:48.780 --> 00:02:51.330 creating microservices across the whole machine learning 62 00:02:51.330 --> 00:02:53.190 model lifecycle. 63 00:02:53.190 --> 00:02:55.950 We want to take all the petabytes of data 64 00:02:55.950 --> 00:02:57.660 we consume as a company, 65 00:02:57.660 --> 00:03:00.570 all the way from our molecule identification 66 00:03:00.570 --> 00:03:01.890 to our clinical trials, 67 00:03:01.890 --> 00:03:04.560 to our commercial execution and production 68 00:03:04.560 --> 00:03:06.780 and shipping of the products, 69 00:03:06.780 --> 00:03:09.810 and take them from database, from flat files, 70 00:03:09.810 --> 00:03:11.140 from cloud storage 71 00:03:12.120 --> 00:03:15.210 and convert them to something that is ultimately useful 72 00:03:15.210 --> 00:03:18.810 for the company and ultimately supports patients' lives. 73 00:03:18.810 --> 00:03:20.010 And that's what we are here for. 74 00:03:20.010 --> 00:03:22.680 We want to bring this data to life. 75 00:03:22.680 --> 00:03:25.470 We are around one and a half year old as a team, 76 00:03:25.470 --> 00:03:28.170 and we already have projects across the company. 77 00:03:28.170 --> 00:03:31.230 And we're working with our R and D, for example, 78 00:03:31.230 --> 00:03:32.940 with using knowledge graph 79 00:03:32.940 --> 00:03:36.750 to identify molecules for insulin resistance. 80 00:03:36.750 --> 00:03:39.900 We have deployed different marketing mix model links 81 00:03:39.900 --> 00:03:42.090 and sales uplift recommendations models 82 00:03:42.090 --> 00:03:44.910 across our different commercial regions. 83 00:03:44.910 --> 00:03:46.140 And last but not least, 84 00:03:46.140 --> 00:03:49.440 we have recently deployed a deep learning machine learning 85 00:03:49.440 --> 00:03:54.000 model that use a vision inspection in our inspection lines. 86 00:03:54.000 --> 00:03:55.260 And that's very important 87 00:03:55.260 --> 00:03:57.782 because it's an optimization on an existing process. 88 00:03:57.782 --> 00:04:00.210 However, it gave us a lot of skills 89 00:04:00.210 --> 00:04:03.060 of how to have live machine learning models 90 00:04:03.060 --> 00:04:05.760 in a very regulated setup, 91 00:04:05.760 --> 00:04:07.590 which is a GMP set up, 92 00:04:07.590 --> 00:04:09.633 good manufacturing practices one. 93 00:04:10.590 --> 00:04:11.793 - How does that work? Tell us more about that. 94 00:04:11.793 --> 00:04:13.860 That seems quite interesting. 95 00:04:13.860 --> 00:04:17.280 - We were already using visual inspection the last 20 years 96 00:04:17.280 --> 00:04:20.130 from a rule-based approach that we have optimized. 97 00:04:20.130 --> 00:04:23.212 And now we have used different deep learning models 98 00:04:23.212 --> 00:04:25.380 to improve that. 99 00:04:25.380 --> 00:04:26.550 And of course, with deep learning, 100 00:04:26.550 --> 00:04:29.190 we are increasing the accuracy and the efficiency 101 00:04:29.190 --> 00:04:31.290 of the visual inspection process 102 00:04:31.290 --> 00:04:33.210 and thereby increasing quality 103 00:04:33.210 --> 00:04:36.300 and reducing the amount of good product going to waste 104 00:04:36.300 --> 00:04:40.080 due to being wrongly identified as defect. 105 00:04:40.080 --> 00:04:45.080 So we save product and we optimize our products that way, 106 00:04:45.210 --> 00:04:46.440 in a more efficient way. 107 00:04:46.440 --> 00:04:48.600 And we also produce less waste 108 00:04:48.600 --> 00:04:50.940 of good cartridges going to waste. 109 00:04:50.940 --> 00:04:51.930 But most importantly, 110 00:04:51.930 --> 00:04:55.740 what we get out of this project is the necessary capability 111 00:04:55.740 --> 00:04:59.010 of how to do machine learning in very regulated spaces. 112 00:04:59.010 --> 00:05:01.950 For example, like manufacturing of pharma. 113 00:05:01.950 --> 00:05:05.440 - Tonia, you've been a big advocate of design thinking 114 00:05:05.440 --> 00:05:09.450 in building data products, AI products. 115 00:05:09.450 --> 00:05:13.650 Tell us more about what that means and why it's important. 116 00:05:13.650 --> 00:05:16.140 - Yes, and I think it started first of all by- 117 00:05:16.140 --> 00:05:17.910 I used to be a data scientist myself. 118 00:05:17.910 --> 00:05:21.540 So quite sometimes I found myself working on projects 119 00:05:21.540 --> 00:05:25.980 that could see that they should have been killed earlier. 120 00:05:25.980 --> 00:05:29.520 So my interest in this is how to speed up 121 00:05:29.520 --> 00:05:31.380 our time for failure. 122 00:05:31.380 --> 00:05:34.200 And that's why, when we started the area- 123 00:05:34.200 --> 00:05:37.620 and that was one and a half year ago- 124 00:05:37.620 --> 00:05:40.892 we really committed to actually start our projects 125 00:05:40.892 --> 00:05:43.740 by what we call a data-to-wisdom sprint. 126 00:05:43.740 --> 00:05:45.990 So basically a hackathon, that we work together 127 00:05:45.990 --> 00:05:48.835 with our business colleagues in a period of two weeks, 128 00:05:48.835 --> 00:05:52.800 to really try to see what we can find from the data, 129 00:05:52.800 --> 00:05:55.020 based on specific hypothesis. 130 00:05:55.020 --> 00:05:57.060 And in the end of these two weeks, 131 00:05:57.060 --> 00:06:01.740 then we ask ourselves, "Is there any signal in the noise? 132 00:06:01.740 --> 00:06:03.450 Are the data good enough? 133 00:06:03.450 --> 00:06:06.719 Do we have the necessary technology to scale it further? 134 00:06:06.719 --> 00:06:09.690 And is there any business value out of this?" 135 00:06:09.690 --> 00:06:10.980 And if the answer is yes, 136 00:06:10.980 --> 00:06:13.703 then we go to the next step where we do to a POC, 137 00:06:13.703 --> 00:06:17.700 then to implementation phase and of course operations. 138 00:06:17.700 --> 00:06:19.890 But if the answer is no, then within two weeks, 139 00:06:19.890 --> 00:06:22.410 very quickly, we should be able to kill it. 140 00:06:22.410 --> 00:06:24.270 And these two weeks we really use, 141 00:06:24.270 --> 00:06:25.830 with the help of agile coaches, 142 00:06:25.830 --> 00:06:27.730 also some design thinking techniques. 143 00:06:27.730 --> 00:06:30.510 But for me, it's the outcome of the design thinking, 144 00:06:30.510 --> 00:06:33.450 how do use design thinking as a way 145 00:06:33.450 --> 00:06:36.243 to work cross-functionally and as a way to fail fast. 146 00:06:37.200 --> 00:06:38.033 - That's great. 147 00:06:38.033 --> 00:06:39.660 No wisdom, you're killed. 148 00:06:39.660 --> 00:06:40.493 Sort of like - Exactly. 149 00:06:40.493 --> 00:06:42.330 natural selection, right? 150 00:06:42.330 --> 00:06:44.100 Joking aside, I think this is a great idea 151 00:06:44.100 --> 00:06:48.324 because Sam, like how many times we either see in our data, 152 00:06:48.324 --> 00:06:50.700 when we survey these thousands of companies, 153 00:06:50.700 --> 00:06:53.310 or in our conversations with executives, 154 00:06:53.310 --> 00:06:57.900 where they are doing hundreds of POCs and pilots 155 00:06:57.900 --> 00:07:00.420 but there is just literally no value. 156 00:07:00.420 --> 00:07:03.660 And there is truly what I call, AI fatigue, 157 00:07:03.660 --> 00:07:05.130 across the organization. 158 00:07:05.130 --> 00:07:08.070 'Cause it's like, the whole organization has become 159 00:07:08.070 --> 00:07:11.797 this graduate school of lab, of like, 160 00:07:11.797 --> 00:07:13.860 "Let's try this, let's try that. Let's try that." 161 00:07:13.860 --> 00:07:14.940 So I love the idea of like, 162 00:07:14.940 --> 00:07:17.850 just kill the ones that aren't working, 163 00:07:17.850 --> 00:07:20.940 so you focus on a handful that are valuable. 164 00:07:20.940 --> 00:07:24.090 - Exactly. And for me also, those that are not working, 165 00:07:24.090 --> 00:07:26.100 we also haven't got a lot of learnings. 166 00:07:26.100 --> 00:07:28.260 Because usually the reason that they're not working 167 00:07:28.260 --> 00:07:30.450 is related to data. 168 00:07:30.450 --> 00:07:33.210 So at least we stress test the data for two weeks 169 00:07:33.210 --> 00:07:34.800 based on what we want to achieve. 170 00:07:34.800 --> 00:07:36.450 And then we get some learnings. 171 00:07:36.450 --> 00:07:38.970 If we want to do this model in the future, 172 00:07:38.970 --> 00:07:41.760 what we need to fix in our data to get there. 173 00:07:41.760 --> 00:07:42.720 - Ooh, that's fabulous 174 00:07:42.720 --> 00:07:44.550 because that's actually tying back 175 00:07:44.550 --> 00:07:46.093 and learning from what you- 176 00:07:46.093 --> 00:07:48.120 I mean, it's one thing to just kind of cut a project off 177 00:07:48.120 --> 00:07:48.953 and say, "All right, well, 178 00:07:48.953 --> 00:07:50.580 we're not going to keep dumping money into that 179 00:07:50.580 --> 00:07:51.413 if it's not going to work." 180 00:07:51.413 --> 00:07:53.070 But then there's something else to, 181 00:07:53.070 --> 00:07:54.990 if you keep starting projects just like that 182 00:07:54.990 --> 00:07:56.190 over and over again, 183 00:07:56.190 --> 00:07:59.040 there'd need to be some learning that those are going to fail 184 00:07:59.040 --> 00:08:02.880 or what you can do to improve those in the future. 185 00:08:02.880 --> 00:08:05.040 What kind of numbers are we talking about here? 186 00:08:05.040 --> 00:08:08.880 How much wisdom is there? Is there 2% wisdom, 20% wisdom, 187 00:08:08.880 --> 00:08:10.980 97% wisdom? 188 00:08:10.980 --> 00:08:11.987 - I think it's very dangerous 189 00:08:11.987 --> 00:08:13.888 to try to quantify something like this, right? 190 00:08:13.888 --> 00:08:14.721 (Shervin chuckles) 191 00:08:14.721 --> 00:08:16.440 But one is the data wisdom. 192 00:08:16.440 --> 00:08:20.010 And the other, of course, is the change management wisdom, 193 00:08:20.010 --> 00:08:22.650 because we work together through this hackathon 194 00:08:22.650 --> 00:08:24.870 with our business experts. 195 00:08:24.870 --> 00:08:26.370 So even something fails, 196 00:08:26.370 --> 00:08:28.260 they understand their way of working, 197 00:08:28.260 --> 00:08:30.900 and also we get a glimpse of their reality, 198 00:08:30.900 --> 00:08:34.740 and they get a glimpse of what can be possible. 199 00:08:34.740 --> 00:08:38.160 And I think this wisdom is even more difficult to quantify 200 00:08:38.160 --> 00:08:40.260 because it will have a, 201 00:08:40.260 --> 00:08:44.550 hopefully, a more kind of a wave impact effect 202 00:08:44.550 --> 00:08:46.830 in the future across the company. 203 00:08:46.830 --> 00:08:50.670 - If you look at the total opposite paradigm 204 00:08:50.670 --> 00:08:52.320 for what you're talking about, 205 00:08:52.320 --> 00:08:56.324 is the old-school waterfall way of building 206 00:08:56.324 --> 00:09:00.210 these like, gigantic tech pieces, right? 207 00:09:00.210 --> 00:09:02.520 It was like tech development 20 years ago, 208 00:09:02.520 --> 00:09:04.830 where I remember with the project, 209 00:09:04.830 --> 00:09:07.170 we looked at a hundred companies 210 00:09:07.170 --> 00:09:10.629 building these massive tech products. 211 00:09:10.629 --> 00:09:14.580 And I think it was like 80% of these companies 212 00:09:14.580 --> 00:09:17.190 were building features and functionality 213 00:09:17.190 --> 00:09:19.380 that either nobody needed 214 00:09:19.380 --> 00:09:22.290 or could not be used with the rest of the technology, 215 00:09:22.290 --> 00:09:23.670 but they would only find this out 216 00:09:23.670 --> 00:09:26.640 like 18 months after development had started. 217 00:09:26.640 --> 00:09:27.960 I guess it's totally new way. 218 00:09:27.960 --> 00:09:31.170 But sadly, there's still many organizations 219 00:09:31.170 --> 00:09:33.390 that are operating with that old paradigm 220 00:09:33.390 --> 00:09:36.780 and they spend months in business requirements gathering 221 00:09:36.780 --> 00:09:38.847 and planning and all that. 222 00:09:38.847 --> 00:09:42.480 And I think what you're saying is, let's get a good idea. 223 00:09:42.480 --> 00:09:44.070 Let's start testing. 224 00:09:44.070 --> 00:09:45.840 If it's got something there, 225 00:09:45.840 --> 00:09:47.850 then we double down and we make it big. 226 00:09:47.850 --> 00:09:49.140 But if it doesn't, 227 00:09:49.140 --> 00:09:50.400 then we've learned something. 228 00:09:50.400 --> 00:09:52.380 And if that project, that idea was important, 229 00:09:52.380 --> 00:09:54.030 then we could fix it. 230 00:09:54.030 --> 00:09:56.340 And I really, really like also your point around, 231 00:09:56.340 --> 00:09:57.904 it's not just the technical part, 232 00:09:57.904 --> 00:09:59.127 it's also the change management 233 00:09:59.127 --> 00:10:00.720 and what it takes for it to work. 234 00:10:00.720 --> 00:10:02.130 It's really, really good. 235 00:10:02.130 --> 00:10:05.040 - Exactly. And by saying from that in advance, 236 00:10:05.040 --> 00:10:07.620 then we have no risk of failure because it is how we work. 237 00:10:07.620 --> 00:10:08.790 We have two weeks, 238 00:10:08.790 --> 00:10:10.740 so it's not going to be our reputation on the line 239 00:10:10.740 --> 00:10:12.183 if the project doesn't continue. 240 00:10:12.183 --> 00:10:16.860 And having gated steps also, after even the MVP phase, 241 00:10:16.860 --> 00:10:19.560 also that the ability to kill something there. 242 00:10:19.560 --> 00:10:20.940 And I think that helps, 243 00:10:20.940 --> 00:10:22.350 and also the budget. 244 00:10:22.350 --> 00:10:24.960 The reason that a lot of companies have these long projects 245 00:10:24.960 --> 00:10:27.930 is because they have long paths allocated to this. 246 00:10:27.930 --> 00:10:28.800 But in our case, 247 00:10:28.800 --> 00:10:31.080 we also assess if there's any willingness to pay 248 00:10:31.080 --> 00:10:32.310 from our business sides. 249 00:10:32.310 --> 00:10:34.350 Is what we do useful enough 250 00:10:34.350 --> 00:10:36.900 that our business is willing to invest in it. 251 00:10:36.900 --> 00:10:38.310 - Set the expectations up front. 252 00:10:38.310 --> 00:10:40.680 Sam, imagine you're, you know, 253 00:10:40.680 --> 00:10:43.170 Sam's a college professor, your students come in and say, 254 00:10:43.170 --> 00:10:45.900 professor, I'm warning you ahead of time. 255 00:10:45.900 --> 00:10:47.340 I will fail in two weeks. 256 00:10:47.340 --> 00:10:48.173 - No, no, 257 00:10:48.173 --> 00:10:49.020 Actually it is the opposite. 258 00:10:49.020 --> 00:10:51.900 I go in and say, 90% of you are going to fail. 259 00:10:51.900 --> 00:10:53.940 And I don't think that would go over very well. 260 00:10:53.940 --> 00:10:57.750 Tonia, how do you transfer these learnings back? 261 00:10:57.750 --> 00:10:58.980 You mentioned that you do that. 262 00:10:58.980 --> 00:11:00.412 Is there a process for that? 263 00:11:00.412 --> 00:11:01.590 How do you codify? 264 00:11:01.590 --> 00:11:06.330 How do you make these things explicit and not just lore? 265 00:11:06.330 --> 00:11:07.230 - That's a good question. 266 00:11:07.230 --> 00:11:09.420 And while we grow, we still have to find out 267 00:11:09.420 --> 00:11:11.880 what's the right level of quantification 268 00:11:11.880 --> 00:11:14.040 that is not bureaucratic as well. 269 00:11:14.040 --> 00:11:17.580 But what we do is, first of all, during these two weeks, 270 00:11:17.580 --> 00:11:19.740 we have two demos across the organization, 271 00:11:19.740 --> 00:11:21.390 and especially with the business unit 272 00:11:21.390 --> 00:11:22.860 that we are working on. 273 00:11:22.860 --> 00:11:25.170 So at least that's the change management part 274 00:11:25.170 --> 00:11:26.550 from a broader perspective, 275 00:11:26.550 --> 00:11:30.030 not only from the people are working in the product team. 276 00:11:30.030 --> 00:11:32.880 And then, regarding the data improvements 277 00:11:32.880 --> 00:11:34.140 or technology improvements, 278 00:11:34.140 --> 00:11:38.070 then we bring them back to our data governance 279 00:11:38.070 --> 00:11:42.510 or to the data owners or to our technology organization. 280 00:11:42.510 --> 00:11:43.740 - Okay. That makes sense. 281 00:11:43.740 --> 00:11:45.060 One of the things you talked about, 282 00:11:45.060 --> 00:11:46.590 and something that Shervin and I, I think, 283 00:11:46.590 --> 00:11:49.680 are seeing overall- is that there's a, 284 00:11:49.680 --> 00:11:51.630 let's say, an increase in the- 285 00:11:51.630 --> 00:11:53.220 maybe in the maturity that we're seeing. 286 00:11:53.220 --> 00:11:54.053 I don't know, Shervin, 287 00:11:54.053 --> 00:11:56.280 maybe I'm reading too much into offhand comments 288 00:11:56.280 --> 00:11:57.120 that people are making. 289 00:11:57.120 --> 00:12:01.530 But I'm just seeing much more process getting put in place 290 00:12:01.530 --> 00:12:05.100 around what used to be very ad hoc. 291 00:12:05.100 --> 00:12:07.320 And maybe you're a couple of steps ahead of this, you know, 292 00:12:07.320 --> 00:12:09.570 looking at some of your building block approaches 293 00:12:09.570 --> 00:12:13.350 to making different services consumable. 294 00:12:13.350 --> 00:12:14.880 Can you explain how that works 295 00:12:14.880 --> 00:12:16.560 and how you're developing these building blocks 296 00:12:16.560 --> 00:12:18.450 and how other people are using them? 297 00:12:18.450 --> 00:12:20.277 - Yes. So of course these building blocks 298 00:12:20.277 --> 00:12:23.760 and the idea of providing MLOps services or, in general, 299 00:12:23.760 --> 00:12:26.490 data services comes very much from this yeah, 300 00:12:26.490 --> 00:12:27.387 data mesh approach. 301 00:12:27.387 --> 00:12:29.550 And now it's the new hype, 302 00:12:29.550 --> 00:12:31.230 but especially for the MLOps work, 303 00:12:31.230 --> 00:12:33.480 what I can speak about is, 304 00:12:33.480 --> 00:12:35.580 based on our learning of how long it took 305 00:12:35.580 --> 00:12:37.920 to get the machine learning model validated, 306 00:12:37.920 --> 00:12:41.160 now we are creating microservices- 307 00:12:41.160 --> 00:12:42.750 wrapping existing services - 308 00:12:42.750 --> 00:12:46.440 either open source or from our cloud vendors, all the way 309 00:12:46.440 --> 00:12:49.350 from how we do model versioning, model monitoring, 310 00:12:49.350 --> 00:12:53.880 model validation, ground truth, storage validation, 311 00:12:53.880 --> 00:12:56.610 and then validating these services 312 00:12:56.610 --> 00:13:00.600 as qualified system from a pharma setting. 313 00:13:00.600 --> 00:13:02.100 And in that way, 314 00:13:02.100 --> 00:13:04.470 we reduce the time to market 315 00:13:04.470 --> 00:13:08.220 from when we need to validate a GXP model, 316 00:13:08.220 --> 00:13:11.370 because then we don't expect any data scientists 317 00:13:11.370 --> 00:13:15.030 in the organization to build their own cloud solutions, 318 00:13:15.030 --> 00:13:17.790 to be both a data engineer and software developer 319 00:13:17.790 --> 00:13:21.540 and a validation experts to bring the model into production. 320 00:13:21.540 --> 00:13:25.320 Because by using this pre-qualified validate services, 321 00:13:25.320 --> 00:13:27.060 they can just focus on data science 322 00:13:27.060 --> 00:13:28.950 and use them as components. 323 00:13:28.950 --> 00:13:32.430 And we just building the first service based on a learning 324 00:13:32.430 --> 00:13:34.413 from this visual inspection model. 325 00:13:35.400 --> 00:13:36.630 - This is such a great point. 326 00:13:36.630 --> 00:13:41.550 If you look at a typical data scientist in a company, 327 00:13:41.550 --> 00:13:43.740 there'll be such a wide variation 328 00:13:43.740 --> 00:13:47.400 of how much of their time's actually in, 329 00:13:47.400 --> 00:13:50.460 what you call extracting wisdom or patterns 330 00:13:50.460 --> 00:13:52.080 or building models and testing, 331 00:13:52.080 --> 00:13:55.680 versus all the other stuff that's prep work 332 00:13:55.680 --> 00:13:58.410 and setting up the environmental feature engineering 333 00:13:58.410 --> 00:14:00.870 and things that somebody else has already done, 334 00:14:00.870 --> 00:14:02.910 but in another part of the organization. 335 00:14:02.910 --> 00:14:06.180 I want to ask you, Tonia, about talent. 336 00:14:06.180 --> 00:14:09.090 I mean, you're talking about a way of working 337 00:14:09.090 --> 00:14:13.560 that is driven by design thinking, fail fast, 338 00:14:13.560 --> 00:14:16.137 highly interconnected with the business. 339 00:14:16.137 --> 00:14:21.137 What is the profile of the right skill sets 340 00:14:21.240 --> 00:14:25.380 from a data scientist, engineering perspective 341 00:14:25.380 --> 00:14:28.200 that's going to be successful in that environment? 342 00:14:28.200 --> 00:14:29.070 - That's a good question. 343 00:14:29.070 --> 00:14:30.720 I think the technical skills, of course, 344 00:14:30.720 --> 00:14:31.920 should be given there, 345 00:14:31.920 --> 00:14:33.900 and I can also see the market over time 346 00:14:33.900 --> 00:14:35.100 is getting more and more mature. 347 00:14:35.100 --> 00:14:37.290 So it's easy to find those. 348 00:14:37.290 --> 00:14:39.990 But what is more difficult is these other, softer skills 349 00:14:39.990 --> 00:14:43.770 that make you a good value translator and a collaborator. 350 00:14:43.770 --> 00:14:47.280 And for me, the most important skill of a data scientist 351 00:14:47.280 --> 00:14:48.570 is actually empathy. 352 00:14:48.570 --> 00:14:51.300 Something we don't expect from people from a technical field 353 00:14:51.300 --> 00:14:52.133 usually. 354 00:14:52.133 --> 00:14:56.100 It's the ability to go to the business person mind 355 00:14:56.100 --> 00:14:59.520 and ask themselves if I was a marketeer, 356 00:14:59.520 --> 00:15:01.980 if I was a production operator, 357 00:15:01.980 --> 00:15:04.650 and I had to do the job every day 358 00:15:04.650 --> 00:15:06.270 and I had the problems that they have, 359 00:15:06.270 --> 00:15:08.250 how would I use the data 360 00:15:08.250 --> 00:15:10.450 to something that would be useful for me? 361 00:15:10.450 --> 00:15:13.260 Being able to make this mental leap 362 00:15:13.260 --> 00:15:16.800 needs a lot of understanding of what is the reality 363 00:15:16.800 --> 00:15:19.950 of the other person and ability also to communicate. 364 00:15:19.950 --> 00:15:22.590 So empathy, and of course, curiosity 365 00:15:22.590 --> 00:15:25.980 about the application of your machine learning models 366 00:15:25.980 --> 00:15:26.813 and the other person. 367 00:15:26.813 --> 00:15:28.920 And that's very difficult skills to quantify 368 00:15:28.920 --> 00:15:30.150 or interview for. 369 00:15:30.150 --> 00:15:34.290 It's more a cultural or a character trait. 370 00:15:34.290 --> 00:15:35.130 - It's interesting Shervin. 371 00:15:35.130 --> 00:15:38.220 We're seeing this, maybe this first indication of, 372 00:15:38.220 --> 00:15:40.560 it's getting easier to find these technical skills. 373 00:15:40.560 --> 00:15:43.230 I mean, I think that's an interesting transition. 374 00:15:43.230 --> 00:15:45.390 - Yep. That's like become more of a- 375 00:15:45.390 --> 00:15:46.770 as Tonia, as you're saying- 376 00:15:46.770 --> 00:15:51.510 the table stakes that you need just to get started. 377 00:15:51.510 --> 00:15:55.890 But the real value is the softer skills and empathy, 378 00:15:55.890 --> 00:15:59.520 and it ties well, Sam to what we're seeing as well, 379 00:15:59.520 --> 00:16:03.540 which is when we look at the evolution of companies 380 00:16:03.540 --> 00:16:05.310 that are investing in AI 381 00:16:05.310 --> 00:16:08.130 and we see that technology and data 382 00:16:08.130 --> 00:16:09.900 is only going to get them so far, 383 00:16:09.900 --> 00:16:13.185 but that big leap is all around organizational learning 384 00:16:13.185 --> 00:16:18.185 interactivity with the business process change. 385 00:16:18.309 --> 00:16:20.343 - At least to be fair about data scientist. 386 00:16:20.343 --> 00:16:23.970 There's still a lot of shortage for machine learning 387 00:16:23.970 --> 00:16:26.940 engineers or data engineers or software developers, 388 00:16:26.940 --> 00:16:28.080 but for data science, 389 00:16:28.080 --> 00:16:30.300 because it becomes more mature as a field technically, 390 00:16:30.300 --> 00:16:31.263 it's all the other skills 391 00:16:31.263 --> 00:16:33.660 that can differentiate somebody. 392 00:16:33.660 --> 00:16:35.220 - Tonia, what are you excited about next? 393 00:16:35.220 --> 00:16:37.200 What's coming with artificial intelligence? 394 00:16:37.200 --> 00:16:39.720 I mean, we're focusing on AI and machine learning. 395 00:16:39.720 --> 00:16:40.890 What are you excited about? 396 00:16:40.890 --> 00:16:42.473 What's coming down the pipe? 397 00:16:42.473 --> 00:16:44.700 - I'm actually excited on data now. 398 00:16:44.700 --> 00:16:46.530 It's not so AI related, 399 00:16:46.530 --> 00:16:48.570 but I think it's regarding to the new trend 400 00:16:48.570 --> 00:16:50.220 that now it's data based. 401 00:16:50.220 --> 00:16:53.040 Like in order to fix our digital intelligence and optimize, 402 00:16:53.040 --> 00:16:55.440 let's optimize our data first. 403 00:16:55.440 --> 00:16:57.300 We also are actually going, 404 00:16:57.300 --> 00:17:00.150 investing more in the data mesh concept now. 405 00:17:00.150 --> 00:17:02.550 So for example, treating data as the product. 406 00:17:02.550 --> 00:17:05.220 Meaning that every time we want to make a new, 407 00:17:05.220 --> 00:17:07.290 let's say marketing mix modeling, 408 00:17:07.290 --> 00:17:09.340 we don't have to go through the whole ETL. 409 00:17:09.340 --> 00:17:10.590 - Yeah, no, this is- 410 00:17:10.590 --> 00:17:13.170 I once did a study 10 years ago, 411 00:17:13.170 --> 00:17:15.150 small group, maybe like a couple of hundred people 412 00:17:15.150 --> 00:17:16.410 in one company, 413 00:17:16.410 --> 00:17:20.880 but like 80% of their data scientist time was spent on ETL. 414 00:17:20.880 --> 00:17:23.250 And then yet they had a data engineering group. 415 00:17:23.250 --> 00:17:24.083 - Mm. 416 00:17:24.083 --> 00:17:25.234 - And the irony of it was like- 417 00:17:25.234 --> 00:17:27.420 you're talking about marketing mix optimization, 418 00:17:27.420 --> 00:17:29.550 this was actually for the marketing department. 419 00:17:29.550 --> 00:17:32.310 You've got data scientists next to each other 420 00:17:32.310 --> 00:17:35.220 in two cubicles, working on something, 421 00:17:35.220 --> 00:17:37.290 using exactly the same data pipeline, 422 00:17:37.290 --> 00:17:39.390 but building it from scratch. 423 00:17:39.390 --> 00:17:40.920 Both of them, not even knowing 424 00:17:40.920 --> 00:17:44.340 that they're using the same foundational features. 425 00:17:44.340 --> 00:17:46.078 And yeah, that's a big deal. 426 00:17:46.078 --> 00:17:47.940 - Tonia. I know that you're excited about that. 427 00:17:47.940 --> 00:17:50.730 Because you talk about that in terms of tech indulgence, 428 00:17:50.730 --> 00:17:52.260 it seems very related there. 429 00:17:52.260 --> 00:17:54.123 That Ikea effect perhaps. 430 00:17:55.054 --> 00:17:57.990 - Yes. The tech indulgence. 431 00:17:57.990 --> 00:18:00.510 Yes. For me, that's actually the worst sin 432 00:18:00.510 --> 00:18:02.190 that we make is technical people 433 00:18:02.190 --> 00:18:05.640 because the Ikea effect is the ability, I think, 434 00:18:05.640 --> 00:18:08.733 to give a higher value to something that you build yourself. 435 00:18:09.840 --> 00:18:12.337 And sometimes we tend to stay in a project 436 00:18:12.337 --> 00:18:14.520 because we build it ourselves 437 00:18:14.520 --> 00:18:16.170 or because we think it's so cool 438 00:18:16.170 --> 00:18:19.200 to try the new machine learning algorithm. 439 00:18:19.200 --> 00:18:20.033 And for me, 440 00:18:20.033 --> 00:18:22.860 this tech indulgence is the biggest danger you can have. 441 00:18:22.860 --> 00:18:25.950 And that's why it's important to avoid this risk 442 00:18:25.950 --> 00:18:27.390 by working closer with the business 443 00:18:27.390 --> 00:18:30.330 and actually working these product teams from a hackathon 444 00:18:30.330 --> 00:18:34.470 all the way to an operational product team. 445 00:18:34.470 --> 00:18:36.273 - I love that term, tech indulgence. 446 00:18:37.470 --> 00:18:38.340 - Tonia, we have a segment 447 00:18:38.340 --> 00:18:40.500 where you ask you a series of rapid- fire questions. 448 00:18:40.500 --> 00:18:43.026 So just answer the first thing that comes to your mind. 449 00:18:43.026 --> 00:18:45.960 what's your proudest AI moment? 450 00:18:45.960 --> 00:18:48.720 - I think this visual inspection problem, 451 00:18:48.720 --> 00:18:51.630 we mentioned, not only for the business impact, 452 00:18:51.630 --> 00:18:54.840 but especially for the capability providers, 453 00:18:54.840 --> 00:18:57.450 how to use machine learning in a GxP setting- 454 00:18:57.450 --> 00:18:59.970 and how quickly we work together as a team 455 00:18:59.970 --> 00:19:03.360 with our business experts, with our manufacturing experts 456 00:19:03.360 --> 00:19:04.500 to make this possible. 457 00:19:04.500 --> 00:19:06.300 And how quickly it actually got 458 00:19:06.300 --> 00:19:08.520 to actually get it validated. 459 00:19:08.520 --> 00:19:09.870 - I thought that might be your example 460 00:19:09.870 --> 00:19:12.510 because of how animated you were when you were talking 461 00:19:12.510 --> 00:19:14.190 about that, we can see this in video, 462 00:19:14.190 --> 00:19:17.070 but I think it probably comes across from your voice too. 463 00:19:17.070 --> 00:19:19.290 What worries you about AI? 464 00:19:19.290 --> 00:19:21.600 - As probably everybody on the show says, 465 00:19:21.600 --> 00:19:23.880 how it can be used also, 466 00:19:23.880 --> 00:19:27.720 as a way to replicate our own biases. 467 00:19:27.720 --> 00:19:29.550 But on the other hand, 468 00:19:29.550 --> 00:19:31.350 I think technology also has the ability 469 00:19:31.350 --> 00:19:32.880 to decode these biases, 470 00:19:32.880 --> 00:19:35.520 because maybe it's easier to remove these biases 471 00:19:35.520 --> 00:19:38.250 from technology than with people in the first place. 472 00:19:38.250 --> 00:19:39.690 So it's a double-edged sword, 473 00:19:39.690 --> 00:19:43.890 but it worries me that we can replicate our own biases. 474 00:19:43.890 --> 00:19:46.410 - Bias is a common concern for everyone. 475 00:19:46.410 --> 00:19:49.422 What is your favorite activity that involves no technology. 476 00:19:49.422 --> 00:19:51.960 - Reading books. Definitely. 477 00:19:51.960 --> 00:19:54.360 And I try actually not to use even my Kindle for that, 478 00:19:54.360 --> 00:19:57.057 to be like physical 3D book. 479 00:19:57.057 --> 00:19:58.650 And I can really recommend, 480 00:19:58.650 --> 00:20:02.160 I just finished Ishiguro's book "Klara and the Sun", 481 00:20:02.160 --> 00:20:05.820 about actually an AI robot that lives in a family 482 00:20:05.820 --> 00:20:07.860 and starts getting feelings about this family. 483 00:20:07.860 --> 00:20:09.690 I can really recommend that. 484 00:20:09.690 --> 00:20:12.240 - Well, that sounds great. Actually, I need a new book. 485 00:20:12.240 --> 00:20:13.073 - I love that. 486 00:20:13.073 --> 00:20:16.680 My 12 year old boy grew up in the age of Kindle, 487 00:20:16.680 --> 00:20:19.020 and you know, screens and reading books. 488 00:20:19.020 --> 00:20:22.290 And so the first time he got an old school book 489 00:20:22.290 --> 00:20:23.347 from the library, he was like, 490 00:20:23.347 --> 00:20:26.183 "Dad, these books smell wonderful. 491 00:20:26.183 --> 00:20:27.270 Like, what is this smell?" (laughing) 492 00:20:27.270 --> 00:20:28.710 And I was like, "Yeah." 493 00:20:28.710 --> 00:20:29.790 It's an amazing smell 494 00:20:29.790 --> 00:20:34.203 that even a child of today's day and age can appreciate. 495 00:20:35.130 --> 00:20:37.470 - What was the first career you wanted as a child? 496 00:20:37.470 --> 00:20:39.630 What did you want to be when you grow up? 497 00:20:39.630 --> 00:20:40.953 - It's very weird, 498 00:20:40.953 --> 00:20:42.380 but I want to be a garbage collector 499 00:20:42.380 --> 00:20:43.343 in the surprise of my mother. 500 00:20:43.343 --> 00:20:44.460 - Me too. Me too! 501 00:20:44.460 --> 00:20:45.594 - Really? 502 00:20:45.594 --> 00:20:47.100 (Tonia laughing) 503 00:20:47.100 --> 00:20:50.280 That's a very rare chance to find the fellow... 504 00:20:50.280 --> 00:20:54.270 - Yes. Fellow garbage collector enthusiasts. 505 00:20:54.270 --> 00:20:56.490 - But I tend to think it's somehow related, right? 506 00:20:56.490 --> 00:20:57.840 I mean, you take something 507 00:20:57.840 --> 00:20:59.400 and you convert it to something else 508 00:20:59.400 --> 00:21:02.970 and we collect data and we convert them to something else. 509 00:21:02.970 --> 00:21:05.460 - Yeah, I'm sure there's some garbage analogy in there too 510 00:21:05.460 --> 00:21:08.100 with the data that's perfect. 511 00:21:08.100 --> 00:21:11.041 What's your greatest wish for AI in the future? 512 00:21:11.041 --> 00:21:13.500 - I will say to be really democratized, 513 00:21:13.500 --> 00:21:14.940 but I don't really believe 514 00:21:14.940 --> 00:21:17.040 that it will get democratized anytime soon 515 00:21:17.040 --> 00:21:20.280 because it needs so much conceptual understanding 516 00:21:20.280 --> 00:21:21.900 to really get democratized 517 00:21:21.900 --> 00:21:23.610 that I don't think we're going to get there. 518 00:21:23.610 --> 00:21:27.150 But that's my real wish that everybody has the tools, 519 00:21:27.150 --> 00:21:29.940 but more also know how to use them. 520 00:21:29.940 --> 00:21:30.960 - So by democratize, 521 00:21:30.960 --> 00:21:33.930 you mean everyone has access to those tools? 522 00:21:33.930 --> 00:21:37.290 - Yes, and I think already there's so many platforms there 523 00:21:37.290 --> 00:21:40.980 that can help to have this low-code AI. 524 00:21:40.980 --> 00:21:43.470 But it's more has access to the tool, 525 00:21:43.470 --> 00:21:45.900 but also be able to use them. 526 00:21:45.900 --> 00:21:48.750 So has the right level of necessary knowledge, 527 00:21:48.750 --> 00:21:50.250 enough to be able to use them 528 00:21:50.250 --> 00:21:52.200 and be independent in using them. 529 00:21:52.200 --> 00:21:54.540 And I think for that, it will take a lot of time 530 00:21:54.540 --> 00:21:55.650 because it's not a tool thing. 531 00:21:55.650 --> 00:21:58.893 It's more, again, change management, an educational, thing. 532 00:21:59.850 --> 00:22:01.170 - Tonia, great meeting you. 533 00:22:01.170 --> 00:22:03.780 I think that a lot of what Novo Nordisk has done 534 00:22:03.780 --> 00:22:07.470 with systematizing and developing processes 535 00:22:07.470 --> 00:22:09.510 around machine learning and AI 536 00:22:09.510 --> 00:22:11.550 are things that a lot of organizations could learn from. 537 00:22:11.550 --> 00:22:13.410 We've really enjoyed talking to you. Thank you. 538 00:22:13.410 --> 00:22:15.463 - Yeah, it's been really a pleasure. Thank you. 539 00:22:15.463 --> 00:22:16.463 (electronic music) - Thank you. 540 00:22:17.790 --> 00:22:18.900 - Please join us next time. 541 00:22:18.900 --> 00:22:22.480 When we talk with Jack Berkowitz, chief data officer at ADP 542 00:22:25.200 --> 00:22:27.690 - Thanks for listening to "Me, Myself and AI." 543 00:22:27.690 --> 00:22:28.920 We believe, like you, 544 00:22:28.920 --> 00:22:31.200 that the conversation about AI implementation, 545 00:22:31.200 --> 00:22:33.390 doesn't start and stop with this podcast. 546 00:22:33.390 --> 00:22:35.190 That's why we've created a group on LinkedIn 547 00:22:35.190 --> 00:22:36.990 specifically for listeners like you. 548 00:22:36.990 --> 00:22:38.760 It's called AI for leaders. 549 00:22:38.760 --> 00:22:39.630 And if you join us, 550 00:22:39.630 --> 00:22:41.475 you can chat with show creators and hosts, 551 00:22:41.475 --> 00:22:43.200 ask your own questions, 552 00:22:43.200 --> 00:22:44.520 share your insights 553 00:22:44.520 --> 00:22:46.500 and gain access to valuable resources 554 00:22:46.500 --> 00:22:50.010 about AI implementation from MIT SMR and BCG. 555 00:22:50.010 --> 00:22:54.933 You can access it by visiting mitsmr.com/AIforLeaders. 556 00:22:56.400 --> 00:22:57.780 We'll put that link in the show notes 557 00:22:57.780 --> 00:22:59.728 and we hope to see you there. 558 00:22:59.728 --> 00:23:02.811 (ending tones music)