What is the Value of an AI-Generated Cover Letter?

Source: https://chatmaxima.com/

Cover letters used to provide insight to hiring managers and helped them identify which candidates to select for an interview. A well-written cover letter signaled something about the quality of a candidate. Moreover, a well-tailored letter also could signal that a candidate was serious about the particular job opening. Do cover letters still have signaling value in the age of AI?

Several months ago, Jingyi Cui, Gabriel Dias, and Justin Ye published a working paper titled "Signaling in the Age of AI: Evidence from Cover Letters." They studied over 5 million cover letters submitted to 100,000 jobs on freelancer.com platform.  The examined the impact of a new feature on the platform that uses AI to generate cover letters for job candidates.  Perhaps not surprisingly, "Access to the tool increased textual alignment between cover letters and job posts and raised callback rates."  

However, that is not the end of the story.  The key finding pertained to a substantial drop in the correlation between cover-letter tailoring and invitations to interview, as well as a significant drop in the correlation with job offers.  On the other hand, workers' review scores (a metric developed by the platform to evaluate past work experiences) became more meaningful.  The authors conclude "These patterns suggest that as AI adoption increases, employers substitute away from easily manipulated signals like cover letters toward harder-to-fake indicators of quality."  

Finally, the scholars examined whether people spent time revising or editing the AI-generated cover letter.  Many people did not.  Yet, those people who did edit the letters increased their probability of landing the job!   

Interestingly, another study by Galdin and Silbert also studied job candidates on the freelancer.com platform.  They found that the length of applications increased after the introduction of AI tools to help candidates.  At the same time, "employers had a high willingness to pay for workers with more customized applications in the period before LLMs were introduced, but not after."  In short, they discovered a drop in the value of the well-crafted application as a signal of quality.  That drop had important implications.  They write, "Without costly signaling, employers are less able to identify high-ability workers, causing the market to become significantly less meritocratic: compared to the pre-LLM equilibrium, workers in the top quintile of the ability distribution are hired 19% less often, workers in the bottom quintile are hired 14% more often." 

Overestimating What We Know: The Trap of Effortless Search

You would like to learn about a particular topic.  You have two options.  You can search online for information, which AI tools and search engines summarize quickly for you.  Alternatively, you could dig deeper, read an article or two, listen to a podcast, and... maybe even read a book on the subject!  How much do we rely on the easy route to knowledge, and how confident are we that we have become highly knowledgeable through our rapid search for information?

In her amazing book, Uncertain: The Wisdom and Wonder of Being Unsure, author and journalist Maggie Jackson explores this topic.  She writes:

After even a brief online search, information seekers tend to think they know more than they actually do, according to a decade of studies. In one set of five experiments, people were asked to study weighty topics, such as autism or inflation, before taking a quiz on the subject. Half the participants were told to find an online article on the topic, while others were simply given the same information without having to search for it.  People who searched online were far more overconfident going into the quiz.  In one round, they predicted that, on average, they would get two-thirds of the questions right, although they scored less than 50%. 

In contrast, people who had been given the information studied longer, absorbed more, and got about 60% of the questions right - about what they had expected.  Rarely if ever in life are just handed information. Searching and seeking are the human condition. But how we do so matters.  In the virtual realm, we seem to lose the ability to sense that we don't know, the starting point of discernment.  This false confidence blossoms even when people learn nothing from an online search, further studies show. By assuming we can know effortlessly, we close our eyes to our failings and so to chances to explore.  We run from the work of fully attuning to the here and now, finding in hubris a retreat from the challenges of facing up to reality as potent as that of outcome-oriented fear. 

This research suggests that we need to proceed with caution when we jump to conclusions based on a breezy online search or quick prompt on an AI tool such as ChatGPT, Gemini, or Claude.   Ask yourself: What do I actually know?  How deep and accurate is my knowledge?  Should I be making critical decisions based on this superficial knowledge?

Using AI to Create a "Fantasy Board of Directors" for Yourself?

What would it be like if you could have Steve Jobs and Warren Buffett serving as advisers as you lead your team and make difficult decisions? Matt Blumberg, CEO of Markup AI, decided to use AI tools to create what he calls a "fantasy board of directors." Preston Fore reported on Blumberg's invention in Fortune this week. He writes,

"To build the fantasy board, Blumberg used a mix of ChatGPT, Gemini, and Claude to create 5,000-word profiles for each person. The profiles were trained on items in the public domain—with the goal of enabling the AI to respond to problems the way real board members might—grounded in how those business leaders viewed leadership, governance, and performance."

Blumberg describes how it only took a few hours to create this AI-generated fantasy board of directors. He uses the chatbot to prepare for board meetings, garner feedback on proposals and plans, and build better presentations to his company and his actual board. Blumberg said, "I’ll say things like: Hey, I’m doing a presentation for our kickoff meeting next week. What do you think are the top three themes I should hit?" He even used the AI tool to provide him with an annual review of his performance as CEO. Blumberg notes that the feedback was right on the mark.

What Happens When We Compare Ourselves to Generative AI?

Source: https://itsoli.ai/

What happens when we compare ourselves to generative AI?  What conclusions do we reach about our own capabilities? Taly Reich and Jacob Teeny have examined these questions in a new paper titled, "Does Artificial Intelligence Cause Artificial Confidence? Generative AI as an Emerging Social Referent."  The scholars conducted several experiments in which they exposed people to precisely the same work, but told some that it was performed by AI and others that it was completed by human beings.  Interestingly, the  researchers found that people exhibit greater self-confidence (with regard to completing a creative task) when they believe that the work they observed was completed by a generative AI model rather than fellow humans. Teeny offered some explanation in this feature from Kellogg Insight:

"From a practical standpoint, self-confidence is such an important driver of innovation, and past research shows so much of our behavior is driven by the simple perception that we are capable of doing that behavior, whether it’s to undertake a piece of creative work or apply for a dream job... Much of our self-perceptions are based on how we compare ourselves to others. If we’re exposed to people we believe are really good at something, we may think, ‘Oh, I’m not as good at that [task] as I thought I was.’ But if we’re exposed to people who do something poorly or who we believe are less skilled, we think, ‘I’m actually pretty good at that.’”

Of course, simply having more self-confidence does not mean people actually will perform well on a subsequent task.   In one of their studies, Reich and Teeny show that those who had compared themselves to an AI model did no better at a creative task than the individuals who compared themselves to other humans.  

Gender Divide on AI Adoption

Are men more likely than women to adopt generative artificial intelligence tools in their work?  Indeed, that is what scholars Nicholas Otis, Solène Delecourt, Katelynn Cranney, and Rembrand Koning have discovered. They recently published a working paper titled "Global Evidence on Gender Gaps and Generative AI."  Here is an excerpt from their paper:

The findings above document that gender gaps in generative AI are nearly universal. We find women use generative AI less than men in data from 18 studies covering 143,008 people from across the world as well as data on who uses top generative AI websites and apps. Moreover, equalizing access does not appear to fully close the gap, even when presented with the chance to use generative AI, women are less likely to use this new technology than men. Akin to efforts to equalize female labor market participation and pay (England, Levine, and Mishel, 2020), it appears that social, cultural, and institutional frictions have led to a gendered gap in generative AI adoption.

Why does this gender gap exist?  HBS Working Knowledge recently interviewed Rembrand Koning about the research.  The article, by Michael Blanding, reports that, "the research suggests women are concerned about the ethics of using the tools and may fear they will be judged harshly in the workplace for relying on them."  

Why is the gender gap finding so consequential?  Two important ramifications must be considered.  First, will the differential usage lead to the exacerbation of gender biases?  In their paper, the scholars write:

This disparity has the potential to be significant. As generative AI systems are still in their formative stages, the under-representation of women may result in early biases in the user data these tools learn from, resulting in self-reinforcing gender disparities (Cao, Koning, and Nanda, 2023). Such biases in user data—similar to those that have previously led to racial disparities in generative AI performance—could result in generative AI systems that reinforce gendered stereotypes and in tools that are less effective at the tasks more often performed by women (Koenecke et al., 2020; Guilbeault et al., 2024).

Secondly, the lower adoption rates by women may affect their career opportunities and future compensation, according to these scholars.  Thus, further work will be needed to understand what's driving the gender gap, and how it may be affecting employees and their careers.  

What Happens When Your Team Adds an AI Teammate?

Source: Getty Images

Bruce Kogut, Fabrizio Dell’Acqua, and Patryk Perkowski have conducted a study to examine how team performance changes when we replace a human member with an AI agent.  In their research project, more than 100 two-person teams played 12 rounds of a video game. For the first six rounds, only humans played the game. For the next six rounds, the researchers replaced one human on each team with an AI agent. Interestingly, they found that performance in the game initially declined when an AI agent replaced a human team member, though performance ultimately bounced back after several rounds of game play. This effect occurred even though the AI agents were actually superior to humans when playing the game individually.  Kogut explained why team performance declined at first:

Despite the AI’s superior individual performance and the fact that bonuses were paid to the entire team if it performed well, 84% of respondents preferred to play with their human teammates. From surveys conducted at the midpoint and end of the experiment, we learned that AI causes team sociability to fall, and that lessens members’ motivation, effort, and trust.

Perhaps most surprisingly, the scholars found that all-human teams adjacent to a team with an AI agent also experienced a decline in performance.  The scholars described this phenomenon as a spillover effect.  What's going on there?  Kogut explained that the AI agent disrupted the environment, perhaps affecting the routines and processes within the all-human teams.  He likened to the impact that losing an employee, or hiring an inexperienced one, can sometimes have on adjacent teams in an organization because of the disruption of usual work routines.  

Risks of Using AI in Human Resources

Source: https://inc42.com/

Artificial intelligence has the potential to transform the way much work is done in the human resources divisions of companies. For example, Nickle LaMoreaux, IBM's chief human resources officer, told Fortune, “We’ve got over 280 different A.I. automations running inside HR right now. That’s what is different here. It’s making HR more human because we’re spending time on things that matter.” Fortune reports that IBM saved 12,000 hours in 18 months by applying artificial intelligence to a series of human resource tasks.  However, Paige McGlaufin has written an excellent article for Fortune highlighting several critical risks that may emerge as artificial intelligence transforms the way human resources departments do their work.

1.  The potential for bias:  Several companies have learned that artificial intelligence tools exhibit a bias against certain groups of employees or job candidates.  

2. The potential for data leaks:  Firms will have to be highly vigilant to be sure private information about employees and job candidates does not get leaked and misused by others.  

3.  The potential for relationship breakdowns:  Efficiency clearly can be enhanced using artificial intelligence.   However, one has to ask:  Will that efficiency have a detrimental impact on the social connections that are crucial to getting work done and retaining employees?  McGlaufin writes, 

"But that efficiency could come at the cost of interpersonal connections. Imagine a scenario where A.I. tools fully administer the hiring and onboarding process: “If I’m a new employee and A.I. is getting my materials and my laptop, onboarding, and online tutorials, I don’t feel connected to the organization,” says Dustin York, a communications professor at Maryville University. That could spell trouble for retention. “I can easily leave and go somewhere else.”

4. The potential for employee  pushback regarding AI tools:  Many employees exhibit an aversion to the use of artificial intelligence tools for certain tasks.  Julia Dhar, director and managing partner at Boston Consulting Group, told Fortune:   “Change doesn’t come super comfortably to human beings. And if executives and leaders are consistently out there only saying that this change is exciting and energizing, you’re unlikely to bring people with you."  

Discovering Analogous Inspiration: Can Crowdsourcing and Artificial Intelligence Help?

Source: Pixabay

When trying to develop a creative breakthrough, analogous inspiration can be incredibly productive.  I have written about this type of fuel for the creative process many times, including in a recent blog post about hospitals and Formula One race teams.  I also describe analogous inspiration in the Unlocking Creativity book, with an example about the Reebok Pump sneakers.  

One question you may have is:  How do I come up with the perfect analogy? How do I find great experiences or situations outside my industry from which I can draw inspiration? New research suggests that crowdsourcing and artificial intelligence can help. NYU's Stern School of Business recently posted a description of this research conducted by Professor Hila Lifshitz-Assaf, assistant professor of information, operations and management sciences, and her colleagues at other universities around the globe.  Here's a brief description:

Wilbur Wright, for instance, famously got his idea for using wing warping to steer an airplane while twisting a cardboard box. Using similar methods to solve disparate problems is a common theme in the history of innovation. But as problems become more complex and the amount of scientific information explodes, finding helpful analogies can be difficult, said Niki Kittur, a professor in Carnegie Mellon University’s Human-Computer Interaction Institute.

As described in a new report to be published online this week by the Proceedings of the National Academy of Sciences, researchers are addressing this problem by breaking down the process of identifying analogies, using crowd workers to solve individual steps in the process and training AIs to do part of the work automatically.

“We’re developing new tools that could unlock a whole set of interesting possibilities,” said Kittur, the lead author. “We’re just beginning to see how people might use them.”