The NUP Lifecycle for Probabilistic Software
Traditional software development lifecycles (such as Waterfall or Agile Scrum) rely on binary phase gates: write code, pass 100% unit tests, sign off UAT, and deploy. In probabilistic systems, where outputs vary validly and models drift in production, applying deterministic milestones leads to project stalls. The NUP Probabilistic Lifecycle establishes a 6-stage engineering framework—from Discovery and Knowledge Prep to AI Development, Statistical Validation, Staged Release, and Continuous Drift Monitoring—with concrete audit deliverables mapped to ISO 42001, EU AI Act, and NIST AI RMF.
Architectural Orientation: Why AI Demands a New Lifecycle
Engineering leaders cannot manage AI initiatives using the assumption that software artifacts remain frozen post-release. In AI-native architectures, enterprise intelligence evolves continuously: training data shifts, vector indexes expand, model providers update foundation weights, and users prompt models in unforeseen ways.
The Netspective Unified Process (NUP) for Probabilistic Software defines a six-stage lifecycle that replaces binary sign-offs with continuous statistical bounds, feedback loops, and tamper-evident audit evidence.
Interactive Phase Explorer
Click through each stage to explore its primary activities, risks, exit criteria, and compliance artifacts:
NUP 6-Stage Probabilistic Lifecycle Explorer
Discovery & Problem Framing
Defines whether the problem requires probabilistic AI or traditional deterministic algorithms. Assesses data quality, regulatory exposure (FDA SaMD, EU AI Act risk tiers), and establishes statistical success metrics.
Primary Engineering Activities
- Evaluate feasibility: Probabilistic vs. Deterministic problem categorization
- Conduct initial AI risk classification (EU AI Act Prohibited / High-Risk / Minimal)
- Assess training/context data availability, copyright provenance, and representation bias
- Define non-deterministic success criteria (acceptable error rates, latency budgets, cost per token)
- Plan knowledge transformation and data extraction requirements from source documents
Probabilistic Failure Risks
- Applying LLMs to problems requiring 100% exact math or deterministic business rules
- Underestimating latency and token consumption at enterprise scale
- Failing to identify regulatory high-risk classification before architecture freeze
- Approved Problem Framing Brief with explicit probabilistic justification
- Initial AI Risk Classification Matrix signed off by compliance officer
- Target statistical metrics defined (Target Accuracy ≥ 95%, Hallucination Rate ≤ 1%)
Discovery & Problem Framing
Determining applicability, bias risks, and non-deterministic success criteria
Defines whether the problem requires probabilistic AI or traditional deterministic algorithms. Assesses data quality, regulatory exposure (FDA SaMD, EU AI Act risk tiers), and establishes statistical success metrics.
Key Engineering Activities
- Evaluate feasibility: Probabilistic vs. Deterministic problem categorization
- Conduct initial AI risk classification (EU AI Act Prohibited / High-Risk / Minimal)
- Assess training/context data availability, copyright provenance, and representation bias
- Define non-deterministic success criteria (acceptable error rates, latency budgets, cost per token)
- Plan knowledge transformation and data extraction requirements from source documents
- Applying LLMs to problems requiring 100% exact math or deterministic business rules
- Underestimating latency and token consumption at enterprise scale
- Failing to identify regulatory high-risk classification before architecture freeze
Stage Exit Criteria
Knowledge & Data Preparation
Transforming enterprise assets into provenance-tracked, AI-consumable context
Converts unstructured enterprise documents (PDF, Word, Excel) into structured, metadata-rich formats (Markdown, HTML) for retrieval-augmented generation. Analyzes data distributions and establishes strict lineage.
Key Engineering Activities
- Transform complex legacy documents into clean, chunkable Markdown and HTML trust layers
- Establish cryptographic data provenance and chunk-to-source traceability
- Analyze datasets for demographic imbalances, sampling bias, and toxic content
- Design context engineering strategies (chunk sizing, metadata tagging, hybrid sparse/dense indexing)
- Version and snapshot knowledge bases and vector indexes in tamper-evident registries
- Garbage-in, garbage-out: Poorly OCR’d or unformatted tables corrupting LLM context
- Stale vector embeddings producing hallucinated answers based on deprecated policies
- Data leakage: Embedding proprietary or PII data without role-based access control filters
Stage Exit Criteria
AI-Augmented Development
Orchestrating prompt, context, harness, and loop engineering with human oversight
Executes implementation using the 4 Layers of LLM Engineering (L1 Prompt, L2 Context, L3 Harness, L4 Loop). Employs AI coding assistants with strict code provenance and human-in-the-loop review.
Key Engineering Activities
- Implement prompt templates with explicit instructions, constraints, and few-shot exemplars
- Build context injection middleware (surveilr SQL-native middleware, dynamic RAG pipelines)
- Implement harness engineering: validation guards, output schema validators (JSON Schema/Zod)
- Construct agentic loop engineering with bounded recursion, step budgets, and human interruption points
- Track AI-generated code provenance and conduct peer reviews on all synthesized logic
- Prompt injection vulnerabilities allowing malicious user input to hijack system instructions
- Unbounded agentic loops resulting in infinite tool execution and API cost spikes
- Accumulation of AI-generated tech debt without adequate human architectural understanding
Stage Exit Criteria
Validation & Testing
Continuous statistical evaluation, distribution analysis, and adversarial red teaming
Replaces binary unit testing with statistical validation across sample populations. Conducts hallucination measurement, semantic drift tracking, bias audits, and red-team adversarial attacks.
Unlike deterministic testing which enforces binary exact matches, Stage 4 validation requires statistical confidence evaluation across sample populations (e.g. 95% confidence bounds on hallucination rates < 0.8% and retrieval recall ≥ 90%).
Deep Dive into Safety & Hallucination Mitigation (Trust & Retrieval Engineering)Key Engineering Activities
- Execute statistical evaluation suites across test populations with 95% confidence intervals
- Measure hallucination rates using automated LLM-as-judge and ground-truth verification
- Perform adversarial red teaming to probe for prompt extraction, jailbreaks, and harmful content
- Conduct bias and fairness audits across protected demographic attributes
- Verify citation correctness: ensure every generated claim links to an intact source chunk
- Sampling bias in evaluation benchmarks leading to false confidence in edge cases
- Over-reliance on LLM-as-a-judge without human verification of judge reliability
- Failing to test multi-turn conversation escalation and adversarial memory poisoning
Stage Exit Criteria
Deployment & Release
Staged canary rollouts, human escalation gates, and transparent user disclosures
Implements progressive delivery mechanisms (shadow pipelines, canary rollouts) for probabilistic features. Enforces human-in-the-loop escalation gates for high-stakes actions and delivers clear AI disclosures.
Key Engineering Activities
- Deploy AI features in shadow mode to compare model outputs against legacy human workflows
- Execute canary rollouts (5% ➡️ 25% ➡️ 100%) with real-time anomaly tripwires
- Establish human-in-the-loop escalation workflows for outputs with confidence scores < 0.85
- Deploy transparent user-facing AI disclosures ("Generated by AI, verified by human")
- Configure AI gateway routing, rate limiting, token quotas, and prompt caching layers
- Deploying AI systems directly to 100% of users without shadow comparison baseline
- Lack of clear user disclosure violating EU AI Act transparency mandates
- Sudden upstream model API deprecation or unannounced behavior modifications
Stage Exit Criteria
Monitoring & Continuous Improvement
Real-time drift detection, feedback aggregation, and scheduled retraining cycles
Monitors output distributions, retrieval latency, and semantic drift in production. Closes the feedback loop by ingesting user corrections, updating knowledge bases, and executing scheduled evaluation sweeps.
Key Engineering Activities
- Continuously stream and analyze production inference telemetry for semantic drift
- Collect implicit and explicit user feedback (thumbs up/down, user prompt edits, copy actions)
- Track model latency, token costs, cache hit rates, and error classifications in real time
- Execute automated weekly evaluation sweeps against frozen golden benchmark datasets
- Maintain an incident response playbook for prompt injection attacks and harmful outputs
- Silent semantic drift: Model outputs gradually degrading over time without error alerts
- User feedback poisoning: Malicious users submitting false feedback to skew retraining data
- Unmonitored token cost inflation during high-traffic enterprise usage spikes
Stage Exit Criteria
Use this prompt in your AI coding assistant to audit your current project state against NUP Probabilistic exit criteria.
Topic 5: Evidence Requirements for Probabilistic Systems
Community Discussion & Feedback
Attributed peer feedback and official Netspective architecture notes.