Memory System Deployment Guide¶

Last Updated: April 14, 2025
Status: Complete

This document provides comprehensive guidance for deploying the Neuroca memory system in various environments. It covers initial setup, configuration, performance tuning, and maintenance procedures.

Table of Contents¶

1. Introduction
2. Prerequisites
3. Environment Setup
4. Development Environment
5. Testing Environment
6. Production Environment
7. Backend-Specific Deployment
8. In-Memory Backend
9. SQLite Backend
10. Redis Backend
11. SQL Backend
12. Vector Backend
13. Configuration Management
14. Performance Tuning
15. Monitoring and Maintenance
16. Backup and Recovery
17. Troubleshooting
18. Upgrading and Migration

Introduction¶

The Neuroca memory system is a modular, tiered memory architecture designed for cognitive AI applications. It consists of multiple memory tiers (STM, MTM, LTM) that can be configured to use different storage backends. This guide provides instructions for deploying and maintaining the memory system in various environments.

Prerequisites¶

Before deploying the memory system, ensure you have the following:

• Python 3.10 or higher
• Pip package manager
• Git for version control
• Docker and Docker Compose (optional, for containerized deployment)
• Access to required backend services (Redis, PostgreSQL, etc., if applicable)
• Sufficient system resources (memory, disk space, CPU) based on expected load

Environment Setup¶

Development Environment¶

1. Clone the repository:

git clone https://github.com/organization/neuroca.git
cd neuroca

1. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

1. Install development dependencies:

pip install -e ".[dev]"

1. Set up configuration:

mkdir -p config/dev/backends
cp config/backends/*.yaml config/dev/backends/

1. Modify development configuration files as needed:

Edit files in config/dev/backends/ to adjust settings for development.

1. Set environment variables:

export NEUROCA_ENV=development
export NEUROCA_CONFIG_DIR=config/dev/backends

1. Run tests to verify setup:

pytest tests/unit/memory

Testing Environment¶

1. Set up a clean test environment:

mkdir -p config/test/backends
cp config/backends/*.yaml config/test/backends/

1. Modify test configuration files:

Edit files in config/test/backends/ to use appropriate test settings: - Use in-memory databases where possible - Use isolated test instances for persistent backends - Configure shorter timeouts and smaller cache sizes

1. Set up CI/CD configuration:

Create a .github/workflows/memory-tests.yml file (if using GitHub Actions) with:

name: Memory System Tests

on:
  push:
    branches: [ main, develop ]
    paths:
      - 'src/neuroca/memory/**'
      - 'tests/unit/memory/**'
      - 'tests/integration/memory/**'
  pull_request:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7
        ports:
          - 6379:6379
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
        ports:
          - 5432:5432

    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -e ".[dev]"
      - name: Run tests
        env:
          NEUROCA_ENV: testing
          NEUROCA_CONFIG_DIR: config/test/backends
        run: |
          pytest tests/unit/memory tests/integration/memory

1. Run integration tests:

export NEUROCA_ENV=testing
export NEUROCA_CONFIG_DIR=config/test/backends
pytest tests/integration/memory

Production Environment¶

1. Prepare configuration files:

mkdir -p config/prod/backends
cp config/backends/*.yaml config/prod/backends/

1. Modify production configuration files:

Edit files in config/prod/backends/ to optimize for production: - Increase cache sizes - Optimize performance settings - Configure proper connection pooling - Set up logging level to WARNING/ERROR - Configure security settings

1. Set up environment variables:

export NEUROCA_ENV=production
export NEUROCA_CONFIG_DIR=/path/to/config/prod/backends

1. Use a process manager:

For production deployments, use a process manager like systemd, Supervisor, or PM2.

Example systemd service file (/etc/systemd/system/neuroca.service):

[Unit]
Description=Neuroca AI Service
After=network.target

[Service]
User=neuroca
Group=neuroca
WorkingDirectory=/path/to/neuroca
Environment="NEUROCA_ENV=production"
Environment="NEUROCA_CONFIG_DIR=/path/to/config/prod/backends"
ExecStart=/path/to/neuroca/venv/bin/python -m neuroca.server
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

1. Enable and start the service:

sudo systemctl enable neuroca
sudo systemctl start neuroca

Backend-Specific Deployment¶

In-Memory Backend¶

The in-memory backend is the simplest to deploy but has limitations in terms of persistence and scale.

1. Configuration:

Modify in_memory_config.yaml to set appropriate limits:

in_memory:
  memory:
    initial_capacity: 10000  # Start with enough capacity
    auto_expand: true
    max_capacity: 1000000  # Set based on available system memory

  persistence:
    enabled: true  # Enable persistence for production
    file_path: "/path/to/data/memory_dump.json"
    auto_save_interval_seconds: 300
    save_on_shutdown: true

1. System Requirements:

2. Ensure sufficient RAM for both the application and the memory backend

3. Configure swap space as a backup

4. Monitor memory usage to prevent OOM errors

5. Scaling Considerations:

6. The in-memory backend runs in the application process and doesn't support clustering

7. For higher loads, consider using Redis or SQL backends
8. Shard memory by implementing multiple backend instances for different data types

SQLite Backend¶

SQLite is a lightweight, file-based database suitable for smaller deployments.

1. Configuration:

Modify sqlite_config.yaml to optimize for your environment:

sqlite:
  connection:
    database_path: "/path/to/data/memory_store.db"
    create_if_missing: true

  performance:
    journal_mode: "WAL"  # Write-Ahead Logging for better concurrency
    synchronous: "NORMAL"  # Balance between safety and performance
    cache_size: 10000  # Adjust based on available memory

1. System Requirements:

2. Fast SSD storage for database file

3. Regular filesystem backups

4. File permissions allowing application read/write access

5. Deployment Steps:

# Create data directory
mkdir -p /path/to/data

# Set permissions
chown -R neuroca:neuroca /path/to/data
chmod 750 /path/to/data

# Initialize database (if needed)
python -m neuroca.memory.tools.init_sqlite_db

1. Scaling Considerations:

2. SQLite supports concurrent reads but not concurrent writes

3. For higher concurrency, consider using PostgreSQL or MySQL
4. Monitor file size and implement pruning/archiving for large datasets

Redis Backend¶

Redis provides in-memory storage with persistence and cluster support, suitable for medium to large deployments.

1. Installation:

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install redis-server

# Configure Redis
sudo nano /etc/redis/redis.conf

# Enable Redis to start at boot
sudo systemctl enable redis-server

1. Configuration:

Modify redis_config.yaml:

redis:
  connection:
    host: "redis.example.com"  # Use hostname or IP
    port: 6379
    database: 0
    password: "your_redis_password"  # Ensure Redis is password-protected
    use_ssl: true  # Enable for production

  performance:
    use_connection_pool: true
    max_connections: 20  # Adjust based on concurrency needs

1. Redis Server Configuration (redis.conf):

# Memory management
maxmemory 4gb
maxmemory-policy allkeys-lru

# Persistence
appendonly yes
appendfsync everysec

# Network
bind 127.0.0.1  # Restrict to localhost or internal network
protected-mode yes
requirepass your_redis_password

# Performance
tcp-keepalive 300

1. Security Considerations:

2. Never expose Redis directly to the internet

3. Use strong passwords
4. Consider Redis auth
5. Use SSL/TLS for encryption

6. Configure proper firewalls

7. Scaling Options:

8. Redis Cluster for horizontal scaling

9. Redis Sentinel for high availability
10. Redis Enterprise for managed solutions

SQL Backend¶

SQL backends (PostgreSQL, MySQL) provide robust storage with advanced query capabilities, suitable for large-scale deployments.

1. PostgreSQL Installation:

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

# Create database and user
sudo -u postgres psql
postgres=# CREATE USER neuroca WITH PASSWORD 'your_password';
postgres=# CREATE DATABASE neuroca_memory;
postgres=# GRANT ALL PRIVILEGES ON DATABASE neuroca_memory TO neuroca;

1. Configuration:

Modify sql_config.yaml:

sql:
  connection:
    driver: "postgresql"
    host: "db.example.com"
    port: 5432
    database: "neuroca_memory"
    username: "neuroca"
    password: "your_password"

  pool:
    min_connections: 5
    max_connections: 20

  performance:
    use_batch_inserts: true
    max_batch_size: 1000

1. PostgreSQL Configuration (postgresql.conf):

# Memory configuration
shared_buffers = 1GB
work_mem = 32MB
maintenance_work_mem = 256MB

# Write-ahead log
wal_level = replica

# Query optimization
effective_cache_size = 3GB
random_page_cost = 1.1  # For SSD storage

# Concurrency
max_connections = 100

1. Database Migration:

# Run migrations 
python -m neuroca.memory.tools.run_migrations

1. Scaling Options:

2. Connection pooling with PgBouncer

3. Read replicas for query scaling
4. Table partitioning for large datasets
5. PostgreSQL clustering with tools like Patroni

Vector Backend¶

The vector backend is optimized for semantic search and similarity queries, essential for LTM memory implementation.

1. Configuration:

Modify vector_config.yaml:

vector:
  storage:
    type: "hybrid"  # Use hybrid for both in-memory and file-based
    file_path: "/path/to/data/vector_store.bin"

  vector:
    dimension: 1536  # Match your embedding model
    distance_metric: "cosine"

  index:
    type: "hnsw"  # Hierarchical Navigable Small World graphs
    use_gpu: false  # Set to true if GPU is available

  performance:
    use_multithreading: true
    num_threads: 4  # Adjust based on CPU cores

1. System Requirements:

2. Sufficient RAM for vector index (depends on vector count and dimensions)

3. Fast CPU for vector operations
4. GPU support is optional but recommended for large indexes

5. SSD storage for vector persistence

6. GPU Acceleration (Optional):

Install GPU support packages:

pip install faiss-gpu

Modify configuration to use GPU:

vector:
  index:
    use_gpu: true
    gpu_id: 0  # Use specific GPU if multiple are available

1. Scaling Considerations:

2. Vector search is CPU/GPU intensive

3. Consider load distribution for large vector databases
4. Implement parallel processing for batch operations
5. Use vector compression for large collections

Configuration Management¶

For effective configuration management across environments:

1. Use Environment Variables for Sensitive Data:

export NEUROCA_REDIS_PASSWORD="your_secure_password"
export NEUROCA_DB_PASSWORD="your_database_password"

In configuration files, use placeholder values:

redis:
  connection:
    password: "${NEUROCA_REDIS_PASSWORD}"

1. Version Control for Configuration:

2. Store template configurations in version control

3. Use .gitignore to exclude environment-specific configurations

4. Document required configuration variables

5. Configuration Validation:

# Validate configuration
python -m neuroca.memory.tools.validate_config config/prod/backends/

1. Dynamic Configuration Reloading:

Implement a configuration watcher for runtime updates:

# Check for configuration changes
python -m neuroca.memory.tools.config_watcher

Performance Tuning¶

For optimal memory system performance:

1. Memory Tier Allocation:

2. STM: Use in-memory backend for fastest access

3. MTM: Use Redis or SQLite for balance of speed and persistence

4. LTM: Use Vector backend for semantic search capabilities

5. Cache Configuration:

Adjust cache sizes based on available system memory:

common:
  cache:
    max_size: 10000  # Increase for production

1. Batch Operations:

Use batch operations for bulk data processing:

common:
  batch:
    max_batch_size: 1000  # Increase for better throughput

1. Connection Pooling:

For database backends, configure connection pools:

pool:
  min_connections: 5
  max_connections: 20

1. Indexing Strategy:

Optimize index types for query patterns:

index:
  type: "hnsw"  # For vector search
  ef_search: 100  # Higher for better recall, lower for speed

1. Memory Pruning:

Configure automatic pruning to manage memory growth:

pruning:
  enabled: true
  max_items: 10000
  strategy: "importance"  # Prune by importance/relevance

1. Performance Monitoring:

# Run performance benchmarks
python -m neuroca.memory.performance.benchmark

Monitoring and Maintenance¶

1. Health Checks:

# Check memory system health
python -m neuroca.memory.tools.health_check

1. Metrics Collection:

common:
  metrics:
    enabled: true
    collection_interval_seconds: 60
    export_prometheus: true

1. Log Rotation:

Configure log rotation to manage log growth:

/var/log/neuroca/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 neuroca neuroca
}

1. Regular Maintenance:

Schedule routine maintenance tasks:

# Add to crontab
0 2 * * * /path/to/neuroca/scripts/memory_maintenance.sh

1. Database Vacuuming (PostgreSQL):

-- Run regularly
VACUUM ANALYZE;

Backup and Recovery¶

1. Backup Strategy:

# Backup script
#!/bin/bash

# Stop service or put in maintenance mode
systemctl stop neuroca

# Backup configuration
cp -r /path/to/config/prod /path/to/backup/config-$(date +%Y%m%d)

# Backup data
cp -r /path/to/data /path/to/backup/data-$(date +%Y%m%d)

# For SQL backend, perform database dump
pg_dump -U neuroca neuroca_memory > /path/to/backup/memory-$(date +%Y%m%d).sql

# Restart service
systemctl start neuroca

1. Recovery Procedure:

# Recovery script
#!/bin/bash

# Stop service
systemctl stop neuroca

# Restore configuration
cp -r /path/to/backup/config-20250414 /path/to/config/prod

# Restore data
cp -r /path/to/backup/data-20250414 /path/to/data

# For SQL backend, restore database
psql -U neuroca neuroca_memory < /path/to/backup/memory-20250414.sql

# Restart service
systemctl start neuroca

1. Disaster Recovery Testing:

Regularly test recovery procedures to ensure they work as expected.

Troubleshooting¶

Common issues and solutions:

1. Connection Failures:

2. Check network connectivity

3. Verify credentials and connection parameters
4. Check firewall rules

5. Inspect service logs

6. Performance Degradation:

7. Check system resources (CPU, memory, disk)

8. Review backend-specific metrics
9. Analyze query patterns

10. Check for index fragmentation

11. Memory Leaks:

12. Monitor memory usage over time

13. Check for growing cache sizes
14. Verify proper resource cleanup

15. Implement memory profiling

16. Data Consistency Issues:

17. Verify transaction settings

18. Check for concurrent write conflicts
19. Review error logs

20. Implement data validation

21. Logging:

Enable detailed logging for troubleshooting:

common:
  logging:
    level: "DEBUG"
    log_queries: true

1. Diagnostic Tools:

# Check backend status
python -m neuroca.memory.tools.diagnostic --backend in_memory

# Run consistency check
python -m neuroca.memory.tools.verify_consistency

Upgrading and Migration¶

1. Version Compatibility:

2. Review release notes for breaking changes

3. Check configuration format changes

4. Verify backend compatibility

5. Upgrade Procedure:

# Backup first
./backup_memory_system.sh

# Stop service
systemctl stop neuroca

# Update code
git pull origin main

# Install dependencies
pip install -e ".[prod]"

# Run migrations
python -m neuroca.memory.tools.run_migrations

# Start service
systemctl start neuroca

1. Rollback Plan:

# If upgrade fails, rollback
git checkout v1.2.3  # Previous stable version

# Restore from backup
./restore_memory_system.sh 20250414

# Start service
systemctl start neuroca

1. Backend Migration:

For migrating between backend types:

# Export data from source backend
python -m neuroca.memory.tools.export --backend sqlite --output memory_data.json

# Import data to target backend
python -m neuroca.memory.tools.import --backend redis --input memory_data.json

1. Data Format Migration:

For handling data format changes:

# Transform data format
python -m neuroca.memory.tools.transform --input old_format.json --output new_format.json

This deployment guide covers the essential aspects of deploying and maintaining the Neuroca memory system. For detailed information about specific backend configurations, refer to the Memory System Backend Configuration document.