WEBVTT

7467afcd-82a9-4c06-a51b-62a6efa416b1-0
00:00:00.960 --> 00:00:02.760
Daniel, thank you so much for
being here.

fece92f2-14cb-4930-bb5c-e56ca47db6ee-0
00:00:03.000 --> 00:00:05.857
What did you recently discover
through your experiment on

fece92f2-14cb-4930-bb5c-e56ca47db6ee-1
00:00:05.857 --> 00:00:07.680
GenAI's impact on knowledge
workers?

0ea91e21-1d1c-40a0-b238-28327727b834-0
00:00:08.000 --> 00:00:08.800
Thanks for having me.

5a0c2fd7-8826-4ee7-b66c-51c89b2e9d03-0
00:00:09.280 --> 00:00:12.308
So we ran a huge controlled
study looking at our own

5a0c2fd7-8826-4ee7-b66c-51c89b2e9d03-1
00:00:12.308 --> 00:00:15.965
consultants and comparing them
to a benchmark set by our expert

5a0c2fd7-8826-4ee7-b66c-51c89b2e9d03-2
00:00:15.965 --> 00:00:16.880
data scientists.

d903bfa0-3867-45b3-902c-5afe3a8d6576-0
00:00:17.240 --> 00:00:20.377
Some of those consultants we
gave generative AI tools to and

d903bfa0-3867-45b3-902c-5afe3a8d6576-1
00:00:20.377 --> 00:00:21.560
some of them we didn't.

d3ddadcf-4c71-4475-85fb-b0785ef1d524-0
00:00:21.720 --> 00:00:24.531
And we gave them a series of
tasks that were actually outside

d3ddadcf-4c71-4475-85fb-b0785ef1d524-1
00:00:24.531 --> 00:00:27.343
of their skill set and outside
of the skill set of generative

d3ddadcf-4c71-4475-85fb-b0785ef1d524-2
00:00:27.343 --> 00:00:27.480
AI.

789f0d9b-5bae-480d-912f-8fe3b27e08b0-0
00:00:27.640 --> 00:00:30.739
And what we found was actually
those consultants that were

789f0d9b-5bae-480d-912f-8fe3b27e08b0-1
00:00:30.739 --> 00:00:33.995
given these tools were able to
reach 86% of the benchmark set

789f0d9b-5bae-480d-912f-8fe3b27e08b0-2
00:00:33.995 --> 00:00:36.885
by the data scientists, which is
a 49 percentage point

789f0d9b-5bae-480d-912f-8fe3b27e08b0-3
00:00:36.885 --> 00:00:39.931
improvement over those data
scientists who were not given

789f0d9b-5bae-480d-912f-8fe3b27e08b0-4
00:00:39.931 --> 00:00:43.398
those tools, which shows us that
we can actually expand the skill

789f0d9b-5bae-480d-912f-8fe3b27e08b0-5
00:00:43.398 --> 00:00:45.920
set of our knowledge workers,
which is amazing.

a39454a9-6523-429a-9d9b-caae49869e0f-0
00:00:45.920 --> 00:00:48.700
And they were also able to
accomplish it 10% faster than

a39454a9-6523-429a-9d9b-caae49869e0f-1
00:00:48.700 --> 00:00:51.675
our data scientists, which is
just really interesting to see

a39454a9-6523-429a-9d9b-caae49869e0f-2
00:00:51.675 --> 00:00:54.650
that they're able to actually
get so much done in this realm

a39454a9-6523-429a-9d9b-caae49869e0f-3
00:00:54.650 --> 00:00:57.479
that is completely beyond what
we would expect from them.

cc9fdd13-7865-40e9-984f-99ffee4a811c-0
00:00:58.240 --> 00:01:00.610
And so what is the difference
between using GenAI to complete

cc9fdd13-7865-40e9-984f-99ffee4a811c-1
00:01:00.610 --> 00:01:00.840
tasks 

731f2650-4550-41df-8572-28aa5addbd72-0
00:01:00.920 --> 00:01:04.000
versus learning new skills?

6163323f-dfb6-4930-aa7e-53d188770e12-0
00:01:04.000 --> 00:01:06.520
And then also, how does that
impact the workforce?

e4e98e02-d1d1-409e-8554-b24dabe7d203-0
00:01:06.720 --> 00:01:07.640
Another great question.

05320d43-6e8a-4653-9624-df3e65999e9e-0
00:01:08.080 --> 00:01:10.973
So we have seen in this
experiment that even though

05320d43-6e8a-4653-9624-df3e65999e9e-1
00:01:10.973 --> 00:01:14.423
these consultants were able to
do so much more, to accomplish

05320d43-6e8a-4653-9624-df3e65999e9e-2
00:01:14.423 --> 00:01:17.539
more than we would usually
expect of them, when we gave

05320d43-6e8a-4653-9624-df3e65999e9e-3
00:01:17.539 --> 00:01:20.822
them the questions about those
tasks immediately after the

05320d43-6e8a-4653-9624-df3e65999e9e-4
00:01:20.822 --> 00:01:23.883
experiment, it was clear that
they did not retain that

05320d43-6e8a-4653-9624-df3e65999e9e-5
00:01:23.883 --> 00:01:24.440
knowledge.

eda3d1b9-d32b-4a7e-88bc-8e2715d07f24-0
00:01:24.920 --> 00:01:27.690
And while this was not designed
to teach them those things, and

eda3d1b9-d32b-4a7e-88bc-8e2715d07f24-1
00:01:27.690 --> 00:01:29.854
they were given a very
constrained time period to

eda3d1b9-d32b-4a7e-88bc-8e2715d07f24-2
00:01:29.854 --> 00:01:32.495
accomplish these tasks, what we
see is an effect that's much

eda3d1b9-d32b-4a7e-88bc-8e2715d07f24-3
00:01:32.495 --> 00:01:34.400
more like putting on an
exoskeleton, right?

423d7914-ea93-47ee-8888-daafd453416d-0
00:01:34.400 --> 00:01:37.337
So these consultants were able
to really power through tasks

423d7914-ea93-47ee-8888-daafd453416d-1
00:01:37.337 --> 00:01:39.600
beyond what they would normally
be able to do.

7d4b7547-49f2-46b4-a6f1-bd846d3732a0-0
00:01:39.840 --> 00:01:43.106
But again, putting those tools
down meant that they did not

7d4b7547-49f2-46b4-a6f1-bd846d3732a0-1
00:01:43.106 --> 00:01:43.760
retain that.

5c9d74f1-2e38-4d31-be39-94f3fbe5052f-0
00:01:44.080 --> 00:01:47.343
So it is important for us to
understand that while incredibly

5c9d74f1-2e38-4d31-be39-94f3fbe5052f-1
00:01:47.343 --> 00:01:50.132
impactful, if we're trying to
upscale our teams more

5c9d74f1-2e38-4d31-be39-94f3fbe5052f-2
00:01:50.132 --> 00:01:53.606
permanently, then we really need
to be looking at how we arm them

5c9d74f1-2e38-4d31-be39-94f3fbe5052f-3
00:01:53.606 --> 00:01:56.080
with tools that are designed for
that purpose.

8b6ca9a8-0dcc-4b67-ad5e-768a23feedb0-0
00:01:56.720 --> 00:01:58.960
What are the risks when
professionals use GenAI

4180ee81-95fe-447c-a2bc-57fbe30426ea-0
00:01:59.000 --> 00:02:01.680
 for tasks outside their usual
skill set?

e64639a9-560c-4604-a6d4-399d994b43c9-0
00:02:02.240 --> 00:02:02.800
Great question.

846c7c0d-37b5-40f6-814c-2bbd787a7e08-0
00:02:02.800 --> 00:02:05.718
So before we actually started
this experiment, we asked our

846c7c0d-37b5-40f6-814c-2bbd787a7e08-1
00:02:05.718 --> 00:02:08.588
consultants what they thought
generative AI was capable of

846c7c0d-37b5-40f6-814c-2bbd787a7e08-2
00:02:08.588 --> 00:02:08.880
doing.

1e12233f-0ceb-45ce-a673-fae0859bda19-0
00:02:08.880 --> 00:02:10.320
You know, could it do A, B, or
C?

fc3b5a28-e2c0-486e-944b-f1f572f9245a-0
00:02:10.840 --> 00:02:14.120
And then we asked the same
question after the experiment.

b0ab0b32-e1aa-43ba-b950-806a4a6aef7e-0
00:02:14.360 --> 00:02:17.054
And what we found is that
regardless of whether they

b0ab0b32-e1aa-43ba-b950-806a4a6aef7e-1
00:02:17.054 --> 00:02:20.155
worked with generative AI tools
during that experiment, they

b0ab0b32-e1aa-43ba-b950-806a4a6aef7e-2
00:02:20.155 --> 00:02:23.002
couldn't discern whether
generative AI could accomplish

b0ab0b32-e1aa-43ba-b950-806a4a6aef7e-3
00:02:23.002 --> 00:02:26.204
A, B, or C, which tells us that
there's a risk of overreliance

b0ab0b32-e1aa-43ba-b950-806a4a6aef7e-4
00:02:26.204 --> 00:02:27.120
on the technology.

5c55f77b-6cc5-472c-b045-cfba08b1feb3-0
00:02:27.280 --> 00:02:29.160
People don't really understand
what it can do.

a6339bdf-e8a7-4f54-90fd-dd6253616ca5-0
00:02:29.480 --> 00:02:32.496
And so it is absolutely critical
that we have experts like those

a6339bdf-e8a7-4f54-90fd-dd6253616ca5-1
00:02:32.496 --> 00:02:35.373
data scientists embedded in our
teams to make sure that we're

a6339bdf-e8a7-4f54-90fd-dd6253616ca5-2
00:02:35.373 --> 00:02:38.158
coming out with the the right
insights, the right findings,

a6339bdf-e8a7-4f54-90fd-dd6253616ca5-3
00:02:38.158 --> 00:02:39.040
the right outcomes.

1590f261-48b4-40ca-9b04-eadc05709ec4-0
00:02:39.720 --> 00:02:42.461
And based on your findings, what
should business leaders be doing

1590f261-48b4-40ca-9b04-eadc05709ec4-1
00:02:42.461 --> 00:02:42.960
differently?

6a4dae6b-101b-47db-884b-0c67227dd728-0
00:02:43.280 --> 00:02:44.200
I think there's a few things.

04a260b3-4cf1-4b4e-890a-f554046a26bb-0
00:02:44.200 --> 00:02:47.280
So first off, this really does
open up the talent pool.

b13ec565-92be-4717-89ac-93b558999281-0
00:02:47.400 --> 00:02:49.707
So before, especially in
something like data science,

b13ec565-92be-4717-89ac-93b558999281-1
00:02:49.707 --> 00:02:51.887
where we have a really
constrained talent pool, we

b13ec565-92be-4717-89ac-93b558999281-2
00:02:51.887 --> 00:02:53.640
should be looking further
afield, right?

b3965156-41c4-4ff8-882d-f74679229c1c-0
00:02:53.640 --> 00:02:56.847
Maybe not looking based solely
on resume, but interviewing with

b3965156-41c4-4ff8-882d-f74679229c1c-1
00:02:56.847 --> 00:02:59.904
generative AI, trying to get a
sense of what people could be

b3965156-41c4-4ff8-882d-f74679229c1c-2
00:02:59.904 --> 00:03:02.360
capable of doing when augmented
with technology.

e7277003-cc9d-4f90-ab12-cae7010bd48e-0
00:03:03.000 --> 00:03:05.452
And the second thing is that
even for workers, we should

e7277003-cc9d-4f90-ab12-cae7010bd48e-1
00:03:05.452 --> 00:03:08.162
really maybe encourage them to
expand or explore opportunities

e7277003-cc9d-4f90-ab12-cae7010bd48e-2
00:03:08.162 --> 00:03:10.615
outside where they normally
would have looked, right? If

e7277003-cc9d-4f90-ab12-cae7010bd48e-3
00:03:10.615 --> 00:03:13.110
that analytical task they maybe
thought was out of reach,

e7277003-cc9d-4f90-ab12-cae7010bd48e-4
00:03:13.110 --> 00:03:15.520
perhaps it's actually something
they could take on now.

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-0
00:03:16.040 --> 00:03:19.314
And then finally, something that
we were really surprised to see

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-1
00:03:19.314 --> 00:03:22.437
is that when we looked at the
performance of our consultants,

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-2
00:03:22.437 --> 00:03:25.560
those that knew how to code,
that learned how to code at some

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-3
00:03:25.560 --> 00:03:28.682
point, were actually much more
effective at accomplishing the

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-4
00:03:28.682 --> 00:03:31.604
tasks, regardless of whether
those tasks required coding,

adab24d8-0bcc-436d-8b8e-9e44e0b0f524-5
00:03:31.604 --> 00:03:32.360
which is crazy.

b4d8643f-eea3-44b9-9ee0-107318e393a2-0
00:03:32.720 --> 00:03:35.472
And to us, it says maybe there's
something in the engineering

b4d8643f-eea3-44b9-9ee0-107318e393a2-1
00:03:35.472 --> 00:03:38.313
mindset that they had to learn
to learn how to code in terms of

b4d8643f-eea3-44b9-9ee0-107318e393a2-2
00:03:38.313 --> 00:03:40.800
breaking problems down into the
logical building blocks

12b1514f-3a23-4e1a-956f-fd776b229b2b-0
00:03:41.080 --> 00:03:43.793
and then thinking through those
that helps them work with this

12b1514f-3a23-4e1a-956f-fd776b229b2b-1
00:03:43.793 --> 00:03:45.560
technology to get to the right
outcomes.

2b3e2818-bdae-49ed-9fed-08ec3d09e44a-0
00:03:45.920 --> 00:03:48.644
And so perhaps there's something
for us as leaders in terms of

2b3e2818-bdae-49ed-9fed-08ec3d09e44a-1
00:03:48.644 --> 00:03:51.240
ensuring that our teams have the
right foundational skills.

f046bc5e-3375-4a33-a5cf-dff970c71bf4-0
00:03:52.080 --> 00:03:55.224
There's a lot of talk about
writing code as being a useless

f046bc5e-3375-4a33-a5cf-dff970c71bf4-1
00:03:55.224 --> 00:03:56.640
skill or coding being dead.

99c538da-22af-468c-b8f3-5d6377e5359e-0
00:03:57.000 --> 00:03:59.331
I'd argue, you know, from this
experiment, we can see that

99c538da-22af-468c-b8f3-5d6377e5359e-1
00:03:59.331 --> 00:04:00.240
maybe that's not right.

6da78f1e-5d5b-4cc9-8af8-3fdf46033eb8-0
00:04:00.360 --> 00:04:03.303
Maybe there is some real value,
even if the syntax changes in

6da78f1e-5d5b-4cc9-8af8-3fdf46033eb8-1
00:04:03.303 --> 00:04:05.440
investing into that real
foundational skill.

eb54df61-9db4-4a3b-82da-aa1b4794c3c0-0
00:04:05.760 --> 00:04:08.000
And if people ask you, what
should your kids study?

b0774341-f656-40c7-8706-1ee12f642878-0
00:04:08.360 --> 00:04:10.280
Maybe coding is still a really
good answer for that.

ac5fc384-f597-409a-b1bf-0d2cb557553c-0
00:04:10.880 --> 00:04:11.760
Thank you so much.

5c7f56cc-f145-484f-8cbe-d23ddb04791e-0
00:04:11.840 --> 00:04:12.200
Thank you.