# Core Engine System - Architecture Design ## Overview The Core Engine System is a scalable 3-layer architecture designed to support multiple client platforms (NOW, Saathi, and future platforms) while maintaining clean separation of concerns, testability, and extensibility. --- ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────────┐ │ CLIENT PLATFORMS │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ NOW Platform│ │ Saathi Platform│ │ Future Platform│ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ └─────────┼──────────────────┼──────────────────┼─────────────────┘ │ │ │ │ HTTP/HTTPS │ HTTP/HTTPS │ HTTP/HTTPS │ │ │ ┌─────────▼──────────────────▼──────────────────▼─────────────────┐ │ APIM LAYER (API Gateway) │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ • Request Validation (Pydantic/JSONSchema) │ │ │ │ • Authentication (JWT Token: x-access-token) │ │ │ │ • Authorization (Role-based access control) │ │ │ │ • Rate Limiting Hooks │ │ │ │ • Request/Response Normalization │ │ │ │ • Logging + Tracing (trace_id propagation) │ │ │ │ • Platform Detection (NOW/Saathi) │ │ │ │ • Version Routing (v1, v2) │ │ │ └──────────────────────────────────────────────────────────┘ │ │ NO BUSINESS LOGIC │ └──────────────────────────────┬────────────────────────────────────┘ │ │ Canonical Request │ (with trace_id, platform, version) │ ┌──────────────────────────────▼────────────────────────────────────┐ │ LOGIC LAYER (Orchestration) │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ • Platform-aware Mapping │ │ │ │ - NOW Request → Canonical Model │ │ │ │ - Saathi Request → Canonical Model │ │ │ │ • Canonical Data Model Generation │ │ │ │ • Orchestration across Service Modules │ │ │ │ • Decision Making & Rule Engine Hooks │ │ │ │ • Response Mapping (Canonical → Platform Format) │ │ │ │ • Retry Logic & Circuit Breaker Hooks │ │ │ │ • Async Workflow Support (Events/Outbox) │ │ │ │ • Idempotency Key Handling │ │ │ └──────────────────────────────────────────────────────────┘ │ │ DOMAIN LOGIC + MAPPING LOGIC │ │ (NO PERSISTENCE LOGIC) │ └──────────────────────────────┬────────────────────────────────────┘ │ │ Canonical Request │ (via Service Client/Adapter) │ ┌──────────────────────────────▼────────────────────────────────────┐ │ SERVICE LAYER (Backend Modules) │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ │ │ Loan Service │ │ User Service │ │ Score Service │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ │ │ KYC Service │ │ Auth Service │ │ ... │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ └──────────────────────────────────────────────────────────┘ │ │ • DB Operations (MongoDB via PyMongo) │ │ • External API Integrations │ │ • Internal Business Rules │ │ • Returns Canonical Response │ └───────────────────────────────────────────────────────────────────┘ ``` --- ## Layer Responsibilities ### 1. APIM Layer (API Gateway / Facade Layer) **Purpose**: Thin facade layer that handles all cross-cutting concerns. **Responsibilities**: - ✅ **Request Validation**: Schema validation using Pydantic models - ✅ **Authentication**: JWT token validation from `x-access-token` header - ✅ **Authorization**: Role-based access control checks - ✅ **Platform Detection**: Identify client platform (NOW/Saathi) from headers or path - ✅ **Version Routing**: Route to appropriate version handler (v1/v2) - ✅ **Rate Limiting**: Hook for rate limiting middleware - ✅ **Request Normalization**: Extract common fields (trace_id, user_id, platform) - ✅ **Response Normalization**: Format responses consistently - ✅ **Logging & Tracing**: Structured logging with trace_id propagation - ✅ **Error Handling**: Catch exceptions and format error responses **What it does NOT do**: - ❌ Business logic - ❌ Data transformation - ❌ Direct database access - ❌ Direct service layer calls **Key Components**: - `apim/middleware/` - Authentication, validation, rate limiting - `apim/routes/` - Route handlers (thin, delegate to logic layer) - `apim/validators/` - Request schema validators - `apim/errors/` - Error response formatters --- ### 2. Logic Layer (Orchestration + Bridge Layer) **Purpose**: Intelligence hub that orchestrates business workflows and maps between platform-specific formats and canonical models. **Responsibilities**: - ✅ **Platform Mapping**: Convert NOW/Saathi requests to canonical format - ✅ **Canonical Model Generation**: Create standardized internal data models - ✅ **Orchestration**: Coordinate multiple service layer calls - ✅ **Decision Making**: Business rules and workflow logic - ✅ **Response Mapping**: Convert canonical responses to platform-specific formats - ✅ **Retry Logic**: Implement retry policies for service calls - ✅ **Circuit Breaker**: Prevent cascading failures - ✅ **Idempotency**: Handle idempotency keys for write operations - ✅ **Async Workflows**: Support for event-driven workflows (outbox pattern) **What it does NOT do**: - ❌ Direct database operations - ❌ External API calls (delegates to service layer) - ❌ HTTP request handling (delegates to APIM layer) **Key Components**: - `logic/orchestrators/` - Business workflow orchestrators - `logic/mappers/` - Platform-to-canonical and canonical-to-platform mappers - `logic/models/` - Canonical domain models (shared with service layer) - `logic/clients/` - Service layer client adapters (dependency injection) - `logic/rules/` - Business rule engine hooks --- ### 3. Service Layer (Backend Modules) **Purpose**: Implements actual business operations and data persistence. **Responsibilities**: - ✅ **Business Operations**: Implement module-specific business logic - ✅ **Database Operations**: MongoDB CRUD operations - ✅ **External Integrations**: Call external APIs (FPL, N-Score, etc.) - ✅ **Data Validation**: Validate data before persistence - ✅ **Internal Rules**: Module-specific business rules - ✅ **Canonical Responses**: Return standardized response format **What it does NOT do**: - ❌ Platform-specific formatting - ❌ HTTP request handling - ❌ Authentication/Authorization (handled by APIM) **Key Components**: - `services//` - Individual service modules (loans, users, scores, etc.) - `services//models.py` - Service-specific models - `services//repository.py` - Database access layer - `services//client.py` - External API clients - `services//rules.py` - Business rules --- ## Data Flow Example: Loan Application ### Request Flow ``` 1. NOW Platform → POST /api/v1/loan/apply { "loan_amount": 5000, "purpose": "Business", "user_id": "123" } 2. APIM Layer: - Validates request schema - Authenticates JWT token - Detects platform: "NOW" - Generates trace_id: "abc-123" - Normalizes request with trace_id, platform, user_id - Routes to Logic Layer 3. Logic Layer: - Maps NOW request to canonical: { "amount": 5000, "purpose": "Business", "user_id": "123", "platform": "NOW", "trace_id": "abc-123" } - Orchestrates workflow: a. Call ScoreService.get_n_score(user_id) b. Call LoanService.validate_eligibility(canonical_request) c. Call LoanService.apply(canonical_request) - Maps canonical response to NOW format 4. Service Layer (LoanService): - Validates loan eligibility - Checks N-Score from ScoreService - Creates loan record in MongoDB - Returns canonical response 5. Logic Layer: - Maps canonical response to NOW format 6. APIM Layer: - Formats response with trace_id - Returns to NOW Platform ``` --- ## Canonical Data Model Strategy ### Principles 1. **Single Source of Truth**: Canonical models live in `domain/models/` and are shared between Logic and Service layers 2. **Platform Agnostic**: Canonical models have no platform-specific fields 3. **Versioned**: Canonical models support versioning (v1, v2) 4. **Extensible**: New fields can be added without breaking existing mappings ### Structure ``` domain/ ├── models/ │ ├── __init__.py │ ├── base.py # Base model classes │ ├── loan.py # Loan canonical models │ ├── user.py # User canonical models │ ├── score.py # Score canonical models │ └── common.py # Common models (trace_id, metadata) ├── exceptions.py # Domain exceptions └── constants.py # Domain constants ``` ### Example: Loan Application Canonical Model ```python # domain/models/loan.py from pydantic import BaseModel from typing import Optional from datetime import datetime class LoanApplicationRequest(BaseModel): """Canonical loan application request""" user_id: str amount: float purpose: str term_days: Optional[int] = None platform: str # "NOW" or "Saathi" trace_id: str idempotency_key: Optional[str] = None metadata: Optional[dict] = None class LoanApplicationResponse(BaseModel): """Canonical loan application response""" loan_id: str status: str amount: float approved_amount: Optional[float] = None message: str trace_id: str created_at: datetime ``` --- ## Request/Response Flow ### NOW Platform Request Format ```json { "loan_amount": 5000, "purpose": "Business", "user_id": "123" } ``` ### Saathi Platform Request Format ```json { "amount": 5000, "loan_purpose": "Business Expansion", "customer_id": "123" } ``` ### Canonical Request (Internal) ```json { "user_id": "123", "amount": 5000, "purpose": "Business", "platform": "NOW", "trace_id": "abc-123", "idempotency_key": "req-xyz" } ``` --- ## Error Handling Strategy ### Error Response Format ```json { "success": false, "error": { "code": "LOAN_ELIGIBILITY_FAILED", "message": "User N-Score is below minimum threshold", "details": { "n_score": 35, "minimum_required": 40 } }, "trace_id": "abc-123", "timestamp": "2025-01-15T10:30:00Z" } ``` ### Error Code Categories - `VALIDATION_ERROR` (400): Request validation failed - `AUTHENTICATION_ERROR` (401): Authentication failed - `AUTHORIZATION_ERROR` (403): Authorization failed - `NOT_FOUND` (404): Resource not found - `BUSINESS_RULE_ERROR` (400): Business rule violation - `EXTERNAL_SERVICE_ERROR` (502): External service failure - `INTERNAL_ERROR` (500): Internal server error --- ## Logging & Tracing Strategy ### Structured Log Format ```json { "timestamp": "2025-01-15T10:30:00Z", "level": "INFO", "trace_id": "abc-123", "user_id": "123", "platform": "NOW", "layer": "APIM", "endpoint": "/api/v1/loan/apply", "message": "Request received", "metadata": {} } ``` ### Trace ID Propagation - Generated in APIM layer (UUID) - Passed to Logic layer via canonical request - Passed to Service layer via service client - Included in all logs across layers - Returned in response headers: `X-Trace-Id` --- ## Versioning Strategy ### URL Versioning - `/api/v1/loan/apply` - Version 1 - `/api/v2/loan/apply` - Version 2 ### Implementation - Separate route handlers per version - Shared canonical models (with version support) - Backward compatibility maintained where possible - Deprecation warnings in v1 when v2 is available --- ## Idempotency Strategy ### Idempotency Key - Client sends `Idempotency-Key` header - APIM layer extracts and validates format - Logic layer checks for existing request with same key - Service layer stores idempotency key with request - Returns cached response if key exists ### Storage - MongoDB collection: `idempotency_keys` - TTL index: 24 hours - Key format: `{user_id}:{operation}:{client_key}` --- ## Retry & Circuit Breaker Strategy ### Retry Policy - **Max Retries**: 3 - **Backoff Strategy**: Exponential (1s, 2s, 4s) - **Retryable Errors**: 5xx, timeout, connection errors - **Non-Retryable**: 4xx (except 429), validation errors ### Circuit Breaker - **Failure Threshold**: 5 failures in 60 seconds - **Open Duration**: 30 seconds - **Half-Open**: Allow 1 request to test recovery - **Per Service**: Separate circuit breaker per service module --- ## Caching Strategy ### Cache Layers 1. **APIM Layer**: Response caching for GET requests (Redis) 2. **Logic Layer**: Orchestration result caching 3. **Service Layer**: Database query result caching ### Cache Keys - Format: `{layer}:{operation}:{params_hash}` - Example: `service:loan:get:{loan_id}` - TTL: Configurable per operation (default: 5 minutes) --- ## Testing Strategy ### Unit Tests - **APIM Layer**: Test validation, auth, routing - **Logic Layer**: Test mapping, orchestration, rules - **Service Layer**: Test business logic, DB operations ### Integration Tests - Test full request flow (APIM → Logic → Service) - Test platform-specific mappings - Test error handling across layers ### Test Structure ``` tests/ ├── unit/ │ ├── apim/ │ ├── logic/ │ └── services/ ├── integration/ │ ├── flows/ │ └── mappings/ └── fixtures/ ``` --- ## Security Considerations 1. **Authentication**: JWT token validation in APIM layer 2. **Authorization**: Role-based access control 3. **Input Validation**: Strict schema validation 4. **Rate Limiting**: Prevent abuse 5. **SQL Injection**: Use parameterized queries (PyMongo handles this) 6. **XSS**: Sanitize user inputs 7. **CORS**: Configured for allowed origins 8. **Secrets**: Environment variables, never hardcoded --- ## Scalability Considerations 1. **Horizontal Scaling**: Stateless design allows multiple instances 2. **Database**: MongoDB sharding for large datasets 3. **Caching**: Redis for frequently accessed data 4. **Async Processing**: Background jobs for heavy operations 5. **Load Balancing**: Nginx/ALB in front of APIM layer 6. **Monitoring**: APM tools for performance tracking --- ## Deployment Architecture ``` ┌─────────────┐ │ Nginx │ (Load Balancer) └──────┬──────┘ │ ┌──────▼──────────────────────────┐ │ APIM Layer Instances (Flask) │ (Multiple instances) └──────┬──────────────────────────┘ │ ┌──────▼──────────────────────────┐ │ Logic Layer (Same process) │ └──────┬──────────────────────────┘ │ ┌──────▼──────────────────────────┐ │ Service Layer (Same process) │ └──────┬──────────────────────────┘ │ ┌──────▼──────────────────────────┐ │ MongoDB Cluster │ └──────────────────────────────────┘ ``` --- ## Next Steps 1. ✅ Architecture design (this document) 2. ⏭️ Folder structure implementation 3. ⏭️ Code scaffolding with demo flow 4. ⏭️ Integration with existing codebase 5. ⏭️ Documentation and examples --- **Last Updated**: January 2025