Buying generative AI after the first convincing demo

GPT-4's release made new applications plausible. Turning them into enterprise products required evidence of productivity gains, sustainable costs and reliable controls.

Enterprise AI4 min read

Summit supercomputer, Oak Ridge National Laboratory, 2018.Carlos Jones / Oak Ridge National Laboratory · CC BY 2.0

OpenAI released GPT-4 on 14 March 2023, offering text access through ChatGPT and a waitlisted API. The launch made a wider range of document, coding and knowledge tasks appear commercially feasible. The accompanying technical report described strong benchmark results while acknowledging limitations in practical use. An impressive response still needed to become a dependable part of a customer's work.

For enterprise software investors, the gap between a convincing demonstration and a reliable commercial product demanded attention. Product teams could build useful prototypes with an external model. Buyers still needed to establish whether those products improved a specific task, protected their information and worked at an acceptable cost. Subsequent workplace research and enterprise launches helped make those questions more precise.

Useful evidence came from a defined task

The following month, Erik Brynjolfsson, Danielle Li and Lindsey Raymond published the first version of Generative AI at Work. Their study covered 5,179 customer support agents at one company. The original paper reported that access to an AI assistant increased issues resolved per hour by 14 percent on average, with larger benefits among less experienced and lower skilled workers. The pattern suggested that assistance could transmit useful working practices as well as accelerate writing.

The working paper studied a single firm's customer support operation using a system deployed before GPT-4's release. Its findings could not establish GPT-4's effects or productivity gains across the economy. For a buyer, it offered a useful method: identify the task, compare outcomes and examine who benefits. A single average can conceal important differences between experienced users and people learning the job.

Metered model access introduced a new cost line

At launch, GPT-4's API with an 8,192 token context cost $0.03 per thousand input tokens and $0.06 per thousand output tokens. The cost of model access therefore grew with use. Longer inputs, longer answers and repeated attempts could increase the bill without a proportionate improvement in the customer's result.

Illustration of server racks connected to liquid cooling manifolds in a data centre aisle. — Liquid cooling hardware in a data center. Illustration.

The full cost extended beyond the model. Retrieving documents, checking permissions, maintaining data connectors and reviewing uncertain answers all required resources. A low cost response could still be expensive if a skilled employee had to verify it from the beginning. For a software business, the useful unit of economics was a completed, acceptable customer task, including the cost of exceptions and review.

This changed how a pilot should be priced and measured. Average usage alone could hide customers whose long documents or high review needs made service disproportionately costly. Teams needed to examine the distribution of task sizes and failure cases, then connect those observations to packaging and gross margin. An appealing subscription price had to support the workload the product actually created.

Procurement extended beyond the quality of the answer

NIST's AI Risk Management Framework preceded GPT-4's launch. Its Govern, Map, Measure and Manage functions provided a way to organise responsibility and assessment across the life of an AI system. As voluntary guidance rather than a certification, it helped buyers identify the evidence needed for an application's purpose, users and consequences.

Different evidence answered different buying questions

Scroll to view all columns

Comparison	Evidence	What still needed to be established
Model evaluation	Performance on a defined benchmark	Reliability with the customer's documents and exceptions
Workplace study	Observed effects in a particular task and workforce	Benefits for another company's users and operating conditions
Enterprise controls	Available data, identity and administration features	The buyer's configuration, responsibilities and acceptance criteria

A procurement distinction drawn from the model report, workplace study and risk management guidance.

An internal drafting assistant and a system producing information used in a financial decision present different consequences when wrong. Buyers needed to identify which data could enter the service, who could inspect the output and when a person had to approve an action. Evaluation therefore had to cover the surrounding process as well as the generated answer. A product that made uncertainty invisible could create more review work than it removed.

OpenAI's subsequent ChatGPT Enterprise launch addressed business data use, encryption, administration and single sign on. These features reflected the procurement requirements emerging around generative AI. Buyers still had to assess the actual contract, configuration and proposed use to establish whether a deployment was appropriate.

Commercial value depended on the surrounding product

The original GPT-4 announcement warned that the model could hallucinate facts and make reasoning errors. Those limitations shaped product choices. Showing a source passage, preserving a review trail and allowing a user to correct a result could be more valuable than producing a longer answer. In consequential workflows, clear boundaries on what the system could do were part of the product's usefulness.

For investors, access to a capable model was only the beginning of diligence. The harder questions concerned customer distribution, permissioned data, integration effort and evidence that the software saved useful time after review. A durable business needed a reason for customers to keep using and paying for the application as models changed. Measuring the benefit in customers' work offered a firmer basis for investment than a compelling demonstration alone.