For 9 years, I’ve been training an AI to do my job. Sadly I couldn’t hit up the neighborhood Jawas for an off-the-shelf droid, so we have had to build our own AI.

So far, I’ve broken up my job into 2,408 tasks, of which we’ve implemented 1,410. These tasks have ranged from the incremental (e.g., “use Section Headings in PowerPoint”) to the transformative. For instance, we recently added a new Highlights feature, which selects and summarizes the top five or so questions from a survey. Like many of our story-driven features, the rules system uses the questionnaire itself for cues about what is important.

Our AI is an expert system, which requires extensive time for a subject-matter expert training it: I’ve invested 545 hours so far.

In contrast, the new generation of AIs are powered by learning models trained on the contents of the World Wide Web. While we have experimented with ChatGPT, as a language learning model, it far too happily disconnects itself from reality. Like that time it invented a career for me as an award-winning Wisconsin-based writer:

Or when it invented citations to articles and studies that weren’t real. Or when it provided recipes using poisonous plants.

As Samantha Quiñones put it:

Intent and ability to conceptualize? These machines can do neither. They produce strings of tokens that statistically appear like they were produced by a human. To an LLM the statements “Neil Armstrong was the first man to walk on the moon” and “Neil Armstrong was the first man to walk on Mars” are differentiated only in that more people have written the former than the latter.

Wolfram Alpha has more on how AIs like ChatGPT actually work. Fun analogy: Emily Bender calls them “stochastic parrots”.

Which doesn’t mean there are times when parroting something back to you can’t be useful. We ourselves have had success using ChatGPT to write code for ResearchStory Enterprise, improving the productivity of our development team. So, yes, we’re using an AI to improve our AI. (Skynet?!)

Now keep in mind that ChatGPT is just one of over eight AI “unicorns” (firms with $1B+ valuations) building large-language learning models, according to The Hustle

Unlike ChatGPT, our AI uses an inference engine, acting on the data and metadata of our surveys via a rules-based approach to decide how to best summarize that data. What’s been the impact? The second year, the AI cut our time to analyze surveys in half: 28 minutes per every hour before. The third year it cut time by three-quarters: 13 minutes per every hour spent before.

The pace of improvement slowed from 2017 to 2019, but then a funny thing happened, the time savings stopped. But quality was increasing.

The more we automated, the more time we had to add other value. The kind only a human can provide.

Goldman Sachs warns that 300 million people are going to lose their jobs to AI. But Brian Livingstone argues “You’re fired if you don’t know how to use GPT-4.” I think Brian’s closer to the truth. Professionals who embrace AI to improve their productivity and the quality of their work are going to outcompete other professionals.

I’ve been trying to replace myself with an AI for almost a decade. Instead, I’ve found it to be less like the Terminator and more like C3PO: an excellent, constantly-learning coworker, enabling me to do my job better than ever. 

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.