Exploring the future of Prompt Engineering, Hybrid roles

We spoke to Angelina Yang, Head of Data at Underdog Fantasy, to get her thoughts on the scope of AI to deliver business outcomes and the future of prompt engineering roles.

We delved into the potential commoditization of AI platforms, the FOMO businesses might be feeling in relation to AI, the opacity of pure prompt engineering roles, and the potential for hybrid roles with data science and prompt engineering skill sets.

Quotations have been lightly edited for concision and readability.

Give us your thoughts on this issue:

What do you think about the potential of AI tools like ChatGPT to help businesses?

“[There are] similar types of models that are already a pretty good foundation to build applications. You're probably already seeing a lot of applications coming out right now. It's very, very crowded. It's just amazing how many different use cases people are thinking of building. I do think it will accelerate things and [that] change is going to happen big time in this year, or the next two years.”

Right now, it feels like a lot of businesses have to find their own way into using AI. Do you think anyone can adopt it, or do they have to build specific capabilities for themselves?

“I don't think they [necessarily] need to build specific capabilities. It really depends on what stage the company is at. If you’re at the early stage, you just want something quick and ready to use, and there are a lot of open source options already. Because of open-source and all this advancement in this area. I'm sure there will be more vendors. The AI space is going to be more commoditized and more available to you.”

She touches on the FOMO that businesses may be feeling around the rapid advances in AI:

“So if you are thinking about hiring, I'm not sure we're going to get to the stage where every company needs a team of data scientists to [take advantage of AI]. You might just hire enough people to be able to understand how to use it. I think that's where we're going and that companies should focus on their core competencies. Right now there's a level of FOMO that people feel, that maybe we shouldn't miss something. It's possible that companies will ramp up on hiring because of this fear of missing out.”

On your point about hiring, prompt engineering seems to be this buzzword around LLMs, and businesses are paying a lot of money for people that can create prompts. Do you think that it’s a sustainable type of role?

“It’s a very interesting role. There is not enough understanding [right now] of the role so you don't see it as often as, say data scientists. [Even] data science roles are not well understood yet because it's only come up in the past 20 years. And prompt engineering is as new as six months.”

This newness makes it hard to predict the future of prompt engineering. The role seems to have plenty of potential, but there are also concerns:

“The majority of the companies don't understand it and they don't know how to use it. I think those kinds of roles are at an early adoption phase. It's hard to say whether it will stick or not, but, it might be a paradigm change of how to use AI models. and it's probably a change in how we train AI models as well. [It] feels like it's becoming more black box as well.”

How is it (prompt engineering) black box?

“Let's say I'm the stakeholder and I have a business use case. I hired a prompt engineer to help me make something happen. I'm not sure that my prompt engineer would really be able to explain to me what's happening, even if the thing works.”

In that instance, a traditional data scientist may be better equipped to satisfy stakeholder expectations:

“…versus if I hire a data scientist, who is actually going to build a model and then they understand [and are able to communicate] from a traditional stats perspective what your dependent variable is, what your independent variable is, [etc.]. The numbers for your result. As a stakeholder, oftentimes I want to know why this thing works. I'm not sure if the prompt engineers actually know how to do that yet. That's a gap that might surface over time.”

On the two major needs for stakeholders:

“Turn around time and explainability. But prompt engineers probably satisfy the first one and not the second.”

Maybe a sort of hybrid type role (would help achieve both)?

“That's a good way to look at it. I think this trend is also at an early adoption phase. Probably a small number of data scientists are looking into this and are actually learning about it, playing around with it, and seeing what can happen.”

She gives her impression of the data science community at the moment, with, for example, data scientists saying that they only care about traditional regression type models, and not about large language models.

“[It’s likely that] the majority of use cases are not large language models just yet. It has a lot of use cases but across the industry, all the problems we have may not be the large language model type. So that's a fair point as well, but I agree [that] if you're hybrid and you're learning the new things, you will have more opportunities in the future.”

Your thoughts on the gap between the skills needed to be a data scientist and the skills needed to be a prompt engineer?

“I think they are similar people, just a different way of thinking. You can be a data scientist and then become a prompt engineer, and you can do that fairly quickly. I think it takes longer to train a data scientist rather than a prompt engineer, especially if you have the foundations. I [do] feel prompt engineering is more of a normal, scientific way of thinking in a sense. A normal human, how we learn is through experiments. Prompt engineering makes it more natural for data science.”

She gives an example of the relatively slower and more complicated iterative process (through Jupyter notebooks) that data scientists go through and how prompt engineering may speed it up.

“…prompt engineering [may] make for faster iteration than the traditional data science way. You can experiment faster and it's just more efficient. Maybe the role is changing. Maybe data scientists are going to be prompt engineers in the future.”

Our conversation with Angelina Yang further highlights the potential of AI, and prompt engineering, especially in relation to data science. While there is still a lot of uncertainty, with prompt engineering being in its fledgling stages, Data scientist/Prompt Engineer hybrid roles may cover for the weaknesses associated with each of these roles individually.

Do consider checking out her insightful newsletter covering careers in Machine Learning and AI, interview tips, and industry use cases here.

See you next week.

Do you have a unique perspective on developing and managing data science and AI talent? We want to hear from you! Reach out to us by replying to this email.