01 / INGEST
Prepare approved sources.
Documents, policies, templates, records, and other approved knowledge are ingested, normalized, and tagged with metadata.
Managed AI Architecture
The layers behind connected AI.
Managed AI combines the AI experience employees use with the models, custom skills, knowledge retrieval, approved connections, security controls, and business systems required to make that experience useful.
From user request to business system
AI experience
GPTs · Agents · Automations
Model
ChatGPT · Claude · Copilot · Private/Open Models
Orchestration
Instructions · Skills · Workflow logic · Approvals
Business systems
CRM · ERP · Email · Files · HR · Analytics
Knowledge
RAG · Documents · Databases
Connections
MCP · APIs · Native integrations · Webhooks
Security + governance
Reference Architecture
How the pieces fit together.
The architecture separates the user experience, model layer, business logic, knowledge retrieval, and system connections. That separation makes access easier to govern and lets each component change without rebuilding the entire environment.
RAG and MCP do different jobs. RAG retrieves approved knowledge for context. MCP and APIs connect AI to live systems, data, and actions. A workflow can use both at the same time.
Knowledge Layer
Where the RAG server fits.
A RAG server gives AI controlled access to approved company knowledge. It is the retrieval layer for information the model should reference, not a replacement for the source system and not a general-purpose connection to every application.
02 / INDEX
Store retrieval data.
Embeddings and metadata are stored in a vector database or equivalent retrieval index, with partitions based on access needs.
03 / RETRIEVE
Find only relevant context.
The system searches approved content at request time and returns the most relevant passages instead of sending the entire knowledge base to the model.
04 / GROUND
Support the AI response.
The model uses retrieved content as context for the answer, analysis, draft, or next workflow step.
RAG is best for reusable knowledge that needs controlled retrieval.
Live operational data often belongs behind MCP or an API call instead of being copied into a shared knowledge index.
Connections Layer
MCP, APIs, and credentials.
The connection layer lets AI reach live business data or perform approved actions. MCP can standardize those tool connections, while direct APIs, native integrations, and webhooks remain valid when they are the better fit.
MCP
A governed tool gateway.
An MCP server exposes specific tools and resources to an AI application. The AI does not receive blanket access to the underlying business system. It receives the functions that were intentionally exposed.
- Reusable connection layer for one or more AI experiences
- Defined tools such as search, read, create, update, or query
- Central place for authentication, validation, routing, and logging
- Useful when multiple agents or models need the same governed connection
APIs, Native Integrations, and Webhooks
Direct paths where appropriate.
Not every integration needs MCP. A workflow can use a vendor API, a platform-native connector, or a webhook when that connection is simpler, more reliable, or better supported.
- APIs for structured data access and actions
- Native integrations for supported platform-to-platform connections
- Webhooks for event-driven triggers and notifications
- Per-connection permissions and action limits
API Keys and Secrets
An API key is a credential, not the risk by itself.
The risk depends on what the credential unlocks, what permissions it has, how it is stored, and how it is monitored. A read-only key to a low-sensitivity reporting endpoint is materially different from an administrative key that can modify production data.
AI workflowRequests an approved tool or action
→
MCP / API connectorValidates scope and prepares the request
→
Secrets manager + business systemCredential is used by the connector, not exposed to the model
- Store credentials in a secrets manager or approved credential vault.
- Keep keys out of prompts, source documents, client-side code, and model context.
- Use service accounts and least-privilege scopes instead of personal credentials where possible.
- Separate development, testing, and production credentials.
- Rotate credentials, monitor usage, and revoke access when the connection is retired.
- Prefer distinct credentials per connection or workload when the vendor supports it.
Orchestration Layer
Where custom skills fit.
Custom skills belong in the orchestration and logic layer. They are reusable business capabilities that tell AI how to perform a defined type of work, what tools to use, what output to produce, and which rules or approvals apply.
A Skill Can Contain
Reusable business logic.
- Step-by-step instructions for a specific task
- Required inputs and source rules
- Tool calls to RAG, MCP, APIs, or native actions
- Output structure, templates, and formatting requirements
- Validation checks and exception handling
- Human approval requirements before sensitive actions
Examples
Company-specific capabilities.
- Document creation using approved company templates
- Proposal assembly using approved services, terms, and formatting
- Spreadsheet analysis using company KPI definitions
- Client report creation using approved reporting standards
- Research synthesis using a defined source and citation method
- Content review against company brand, legal, or compliance rules
1. RequestUser, schedule, or system trigger
2. Select SkillChoose the business capability
3. Retrieve KnowledgeUse RAG when context is needed
4. Call ToolsUse MCP, API, native connector, or webhook
5. Apply RulesFormat, validate, and enforce workflow logic
6. ReviewRequire human approval where defined
7. Output / ActionCreate the deliverable or perform the approved action
The skill does not replace RAG or MCP. It coordinates when and how those capabilities are used inside the workflow.
Security Boundaries
Access is defined before the workflow runs.
Security should wrap the full architecture instead of being added at the end. Each layer needs clear identity, permission, data, credential, logging, and approval boundaries.
| Control Area | What It Governs | Architecture Impact |
|---|---|---|
| Identity and access | Who can use the AI experience and which data or tools they can reach. | Permissions should propagate into RAG filters, MCP tools, APIs, and actions. |
| Secrets management | API keys, OAuth tokens, service credentials, certificates, and other connection secrets. | Credentials stay outside prompts and are retrieved only by the connection service that needs them. |
| Data boundaries | Which knowledge, records, clients, departments, or projects may be accessed together. | RAG indexes and live tool calls should enforce the same separation rules. |
| Action permissions | What the AI may read, draft, create, update, send, or delete. | Start read-only where possible, then add narrowly scoped write actions with approval gates. |
| Logging and review | Tool use, retrieval, failures, approvals, model activity, and sensitive actions. | Logs support troubleshooting, auditing, governance, and ongoing Managed AI operations. |
Managed AI Architecture & Roadmap
Architecture should follow the business need.
ITS maps the workflows, data, knowledge, connections, security boundaries, custom skills, and implementation dependencies required for each approved Managed AI solution.