Building products with AI: decisions across the life cycle

1. AI product development is a continuing decision cycle

AI product development is a life cycle organized around verifiable decisions. The CAPDS guide does not present that cycle as a project methodology or a prescribed set of ceremonies. Each stage produces artifacts used by the next stage. Each transition has a criterion that someone outside the team can verify. Progress therefore depends on evidence, not only on completing planned work.

The guide distinguishes this cycle from a conventional progression into maintenance. An AI product passes through conception, construction and launch, but it does not reach the same final maintenance state. Product decisions continue through evaluation, monitoring and governance. The practical implication is to plan these activities as part of the product, rather than treating them as temporary work before release.

The scope is broader than choosing an AI approach. CAPDS covers discovery and framing, feasibility, data and privacy, quality evaluation, experience under uncertainty, inference economics, launch and responsible use. These areas are connected. Input quality can limit feasibility. Review effort can change the economic case. The consequence of an error can change the conditions for release.

The Certified AI Product Development Specialist page provides the CAPDS entry point. Use the CAPDS guide for the full treatment and the PMCert glossary for terminology. The organizing question throughout is direct: what evidence authorizes the next product decision?

2. Frame the problem before committing to AI

Framing establishes that the problem exists and that AI is an appropriate response. CAPDS separates refusing a badly formulated scope from refusing the opportunity. A team can preserve the underlying opportunity while changing the proposed solution. The decision is not simply whether to build the requested feature. It is whether the problem and the proposed role for AI withstand examination.

The guide calls for decomposing the problem into its deterministic part and its residual. It also requires calculating net work, including input preparation and review. This changes the product discussion from apparent automation to the work that remains. A proposed capability must be assessed with those activities included, not only with attention to the task performed by the model.

The clause-extraction case illustrates evidence-based framing. Before construction, extraction is performed manually for real clients, and lawyers act on the results. The evidence concerns actual use rather than declared intent. That distinction matters to the product decision: a useful result is demonstrated through what people do with it, rather than through agreement that the proposed feature sounds valuable.

Feasibility then narrows the promise. CAPDS calls for distinguishing volatile knowledge from stable behavior, removing rules requiring guarantees from the prompt, and calculating compound reliability before promising an autonomous flow. It also asks teams to identify constraints that are not model capability. In the contract case, input quality and reliability across the chain change the design. Human verification becomes part of the approach, not an exception added after launch.

3. Treat data and context as product decisions

Data and context decisions concern the inputs the product actually uses, their coverage and the conditions under which they can be used. CAPDS requires verification of coverage in the real input base. It also directs teams to investigate the input before changing the model. This places the source of a quality problem ahead of a preferred technical remedy.

In the clause-extraction case, low-resolution digitized contracts form a relevant part of the collection. Their quality sets a ceiling on the result. The corrective decision is made at capture, rather than through a model change. The lesson is specific: when the input constrains performance, changing the approach without investigating that input does not address the identified constraint.

Context also has operating conditions. CAPDS calls for monitoring index lag and applying permissions during retrieval. These are explicit checks in the product decision framework. The presence of retrieved material is not the end of the inquiry. The team must also examine whether retrieval respects permissions and whether the index has fallen behind.

Privacy decisions extend beyond the original input. The guide requires measuring reidentification through combinations and treating derived use as a decision about legal basis. The recruitment case adds another concern: descriptions of experience vary by educational background and sector, and that variation enters the system as signal. The input source record documents the risk. Data review therefore includes how information is expressed, not only whether a field is present.

4. Evaluate quality in the user's unit of work

Quality evaluation measures whether the product performs in the user's unit of work. CAPDS makes this an explicit requirement. An isolated output can be the wrong unit for a product promise. The relevant question is whether the complete result the user needs is usable, with the required checks and corrections taken into account.

The contract case makes the distinction concrete. Evaluation concerns the fully reviewed contract, not an isolated clause. With ninety-five percent performance per clause and fifteen clauses per contract, fewer than half the contracts would be entirely correct. The guide uses this result to redefine the product promise. A strong component-level figure does not, in that case, justify a promise about the whole contract.

The evaluation method must also make differences visible. CAPDS calls for stratification with sample sizes in critical strata, item-by-item comparisons across versions, an adversarial set and calibration of the automated judge. It also requires a minimum sample size before declaring a difference. These requirements constrain when a team can claim improvement, rather than merely providing more figures to report.

Aggregate performance is not sufficient for every release decision. In the recruitment case, an aggregate result of eighty-seven percent conceals consistently lower performance for a set of profiles. Release is suspended. The correction reduces the weight of attributes correlated with differences in how experience is described. Evaluation by group changes the decision even though an aggregate result is available. The release condition must therefore reflect the product's consequences, not just its overall score.

5. Design the experience and economics around usable results

Experience under uncertainty is the design of interaction around likely errors, review and recovery. CAPDS directs teams to start with the most likely error. It also requires keeping the cost of review below the cost of the task. Review is therefore part of the product's value calculation, not an unlimited safeguard that can be added without examining its burden.

The guide links interaction choices to consequences. It calls for choosing between prior confirmation and rollback according to reversibility, anchoring evidence to the statement and graduating autonomy by confidence band. In the contract case, the extracted clause is linked to its original passage and highlighted in the document. The lawyer confirms or corrects it clause by clause. Corrections then enter the regression set.

Economics uses a related unit: cost per usable result. CAPDS requires projection by distribution, control of context growth, inclusion of evaluation in the budget and pricing aligned with cost behavior. These requirements move the economic assessment beyond the cost of an individual call. They include the work and consumption needed to obtain a result the product can use.

One case records four discarded items for every usable item. The cost per published item is therefore five times the apparent cost per call. The response is to improve input and reduce discards, rather than change approach. In the contract case, evaluation consumes more than production during the first quarter. Together, these examples show why review, rejected results and evaluation must remain visible when deciding whether the product is economically viable.

6. Make launch a controlled exposure decision

Launch is a controlled exposure decision supported by monitoring and containment. CAPDS calls for running in shadow before exposing users, writing rollback thresholds before exposure and instrumenting before the first version. These conditions put operational evidence and response capability ahead of release. The decision is not limited to whether the feature can be made available.

In the contract case, shadow execution uses contracts that have already been analyzed. The output is compared with the recorded human result. This creates a comparison grounded in the task the product is intended to support. It also keeps the release discussion connected to the evaluated unit of work, rather than relying only on examples selected to demonstrate the capability.

Exposure and rollback can be shaped by risk. One case releases by product category, beginning with the category of lowest regulatory risk. The recruitment case uses simultaneous comparison against the previous process. Its rollback threshold is defined by disparity, not only accuracy. These are different release choices, but both connect exposure to a stated condition rather than to general confidence.

CAPDS also requires granular containment and directs teams to contain before diagnosing. The product must support a response while the cause is still being investigated. This is part of the continuing life cycle described by the guide. Monitoring is not a closing report after launch. It supplies evidence for decisions about continued exposure, rollback and containment, using thresholds written before users are exposed.

7. Govern by consequence across the life cycle

Governance is implemented through product controls proportional to risk. CAPDS calls for classification by consequence, measurement of outcomes by group before release, effective human oversight and an appeal path. These are observable product requirements. They connect responsibility to what the product permits, what it records and what affected people can do after a decision.

The recruitment case shows why consequence takes precedence over technical sophistication. Initial ranking determines who receives an interview. The cost of error is high, and a candidate may not know that the system passed them over. The framing record treats that combination as a design constraint. The approach extracts verifiable attributes and avoids an unauditable judgment about cultural fit, even though this reduces the apparent capability.

Oversight is also designed into the interaction. Rankings are suggestions with justification by attribute. Recruiters must record a reason when discarding candidates from the top, creating deliberate friction. Governance includes measurement by group as a release condition, appeal with a stated deadline at the point of decision, full traceability and oversight with authority to disagree. Human involvement is therefore tested through its effective authority.

Across the life cycle, the decision pattern remains consistent. Establish evidence of use. Examine input constraints. Evaluate the complete task and critical groups. Include review and evaluation in the economics. Define exposure and rollback conditions. Match controls to consequences. The CAPDS guide brings these decisions together. Its focus is not memorizing statements, but demonstrating the conduct needed to make, qualify or refuse a product promise.