Managing the Promise and Perils of Chat GPT Today: A Framework for HR and Business Leaders
A quick demystifying of Chat GPT, what it means for HR & Leaders and what it means for job design today.
Back in 2006, I started using an open source software to identify HR professionals in Surrey, Sussex and Kent for roles for my job in recruitment. It was a game-changer for me. This new platform was an open source platform that made my rolodex of contacts, built up over the years, public knowledge. Of course, this did not just replicate my rolodex (admittedly this was a digital rolodex of contacts), but it augmented it. I used to manually track all the movements of HR professionals in the region, but now this open source software did it for me, and gave me a direct channel to reach out to them. No longer did I need to go undercover and discover their email address or go via the switchboard, I could just use this platform to introduce myself.
Of course, this platform was LinkedIn. I was one of the early adopters to purchase a recruiter license. This gave me unrivalled access, but only until everyone else caught up, which after a couple of years they did, and we all had to find a new edge. Within 10 years 93% of recruiters were on LinkedIn.
History is dotted with early adopters of open source tools making short term competitive advantage. This free innovation cannot be protected against by companies, but only leveraged for short term competitive advantage before its widespread adoption. Of course AI comes with its dangers and warning labels are added long after any benefits are gained by early adoption.
As we seek to navigate the hybrid working era, Open AI quietly released in November 2022 its LLM, Chat GPT. These two are to be seen hand in hand. Working from home affords lots of tech and the ability to focus on many of the tasks Chat GPT excels at.
What is an LLM, Chat GPT 3 vs.4?
Open AI is a business which Microsoft invested $1 billion in. It released Chat GPT (Chat generative pre-trained transformer) which is a form of AI called a Large Language Model (LLM), based on neural networks (a computer model of how a brain works). ChatGPT-3 is available via Open AI for free, is an LLM trained on data prior to June 2020 and its training data base is not connected to the internet. Chat GPT-4 is available as a paid subscription on OpenAI and is what Bing Chat uses, and is connected to the internet. This connection to the internet, gives it about a 40% improvement over GPT-3.
What benefits does it provide??
LLMs like used in Chat GPT-4 provide 3 advantages over prior AI (Noy et al., 2023)
1. Surprising general abilities - LLMs demonstrate unexpected capabilities across many domains despite being trained as general purpose models. Their ability to perform specialist tasks like legal analysis or medical diagnosis without dedicated training expands their potential applications.
2. Direct performance boost - Studies show LLMs can directly improve individual worker performance at tasks like programming, writing, and content creation without extensive technical infrastructure or training. This differs from earlier AI that required organisational integration via internal IT departments.
3. Opacity and unclear failure modes - The strengths and weaknesses of LLMs are often unclear even to their creators. They excel at some tasks inexplicably yet fail at others unexpectedly. This uncertainty makes their impacts hard to predict and integrate without extensive user experience and sharing of best practices.
In combination, these factors create conditions for LLMs to rapidly diffuse into many knowledge work settings in ad hoc ways driven by individual users, before systematic organisational integration. It is likely these are being used right now by employees and suppliers, we just don’t know about it.
How in HR and Leadership does it impact job design?
In Reinventing Jobs, Jesuthansan & Boudreau (2018), look tasks through a 2x2 grid. With Repetitive vs Variable on one axis, Independent vs Interactive on another. Independent processes, like those that are most productive done away from the open plan office, are valuable to either replace with Chat GPT if the task is independent and repetitive, or augment, if independent and variable. With much of this tech available on personal computers, this is the area that employees are most likely to adopt before companies incorporate it, via their slower moving corporate IT. This is a spectrum, and the more variable and interactive tasks become, the less value AI offers. This is where humans excel and is where the exponential value comes from having humans. The more time we can free up from the repetitive processes, the more time people can spend in high value work, which is best done in the office.
Overall, the goal should be to use AI to automate tasks where performance has limited upside, freeing up human time for high-judgment and creative work with exponential returns to enhanced performance. This requires analysing workflows & tasks, not whole jobs to identify components suitable for AI augmentation versus those requiring specialised human skills.
What effect does it really have, on the ground?
Study 1: AI Consultants (Dell’ et al., 2023)
The study by Dell’Acqua et al. examined 758 management consultants from Boston Consulting Group (BCG) completing realistic consulting tasks with and without AI.
With AI access, consultants were 12% more productive, completing tasks 25% faster. AI increased output quality over 40% based on human evaluations.
These improvements occurred for creative ideation, writing, analysis, and problem-solving tasks - areas considered human strengths.
Bottom-performing consultants improved the most with AI, by 43%. But all users benefited, suggesting AI both enhances average performance and levels differences. This shows that AI not only raises the performance ceiling, but importantly raises the floor of performance higher.
However, for a task requiring nuanced qualitative data analysis, consultants with AI performed 19 percentage points worse. Overreliance on AI can thus backfire outside its capabilities.
Consultants successful with AI behaved as “centaurs”, strategically dividing labour, or “cyborgs”, tightly integrating with AI across sub-tasks. This integration will require new workflows. These two categories of roles can inform new job categorisations.
Study 2: AI and Human Creativity(Koivisto & Grassini, 2023)
The study by Koivisto and Grassini compared 256 humans and 3 AI systems on a creative ideation task.
·On average, AI systems produced more creative ideas than humans. However, top human performers matched or exceeded AI. Again, showing how this is a greater at raising the floor than the ceiling of performance.
Humans generated more common ideas and a wider range of quality, while AI was consistently novel.
AI seems to excel at accessing distant concepts but combining them in nuanced, surprising ways. Humans bring context and intuition.
The results suggest both promise and limitations of AI for creative work. While beneficial on average, human ingenuity remains competitive.
What are the limitations of large language models like ChatGPT?
Knowledge breadth but not depth - LLMs have a wide knowledge base but lack deeper expertise in most domains. Their capabilities are superficial outside of language tasks.
Factual inaccuracies - LLMs hallucinate false information or provide incorrect citations about 30% of the time according to analysis by Anthropic. Accuracy is not a primary design goal. They need knowledgeable humans to check their work.
Logical inconsistencies - LLMs often make basic logical errors in multi-step reasoning or math. Their reasoning capabilities are limited.
Opaque failure modes - It is hard to predict when LLMs will make mistakes or identify the root cause, due to their black box nature. Failures can seem arbitrary.
Out-of-distribution inputs - Performance drops dramatically if inputs differ from the training data. LLMs have limited adaptability to new domains or data.
Lack of reasoning about users - LLMs have no explicit model of user knowledge or goals. Their outputs do not adapt to users.
Poor judgment on risks or ethics - LLMs have limited capabilities to weigh legal, ethical, or social risks associated with potential actions or recommendations.
This means, these devices need human oversite. Think of them as Interns for those position holders who spend a lot of time doing creative ideation, writing, analysis, and problem-solving tasks. This does not replace them, but becomes an assistant augmenting their knowledge and carrying out the labour-intensive work. Imagine it as a super calculator but for language, not just numbers.
In summary, whilst new, these findings suggest that with careful implementation, like the calculator, computer, or internet, Chat GPT will provide a temporary competitive advantage to some, but primarily by enhancing human skills, not replacing them. Sustainable advantage will come from effectively leveraging Chat GPT’s strengths while retaining uniquely human skills and perspectives. HR will play a primary role, like that of COVID, in getting employees ready to leverage Chat GPT, reconstructing roles around AI and much more.
If you have got this far, thanks. This is a free blog as its purpose is circulating ideas. In exchange, please consider sharing.
Dell’, F., Saran, A., Mcfowland, R. E., Krayer, L., Mollick, E., Candelon, F., Lifshitz-Assaf, H., Lakhani, K. R., & Kellogg, K. C. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. https://ssrn.com/abstract=4573321
Jesuthasan, R., & Boudreau, J. (2018). Reinventing Jobs: A 4 Step Approach for Applying Automaton to Work. HBR Press.
Koivisto, M., & Grassini, S. (2023). Best humans still outperform artificial intelligence in a creative divergent thinking task. Scientific Reports, 13(1). https://doi.org/10.1038/s41598-023-40858-3
Noy, S., Zhang, M. W., Agarwal, N., Autor, D., Barros, L., Benheim, T., Finkelstein, A., Horton, J., Jäger, S., Leslie, A., Mejia, J., Noy, I., Noy, L., Partridge, E., Rafkin, C., Rao, A., Roussille, N., Roth, C., Schilbach, F., … Wu, S. (2023). Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.