diff --git a/README.md b/README.md index 4ad8ffd..64e0a8f 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,421 @@ -# Project Overview +# AFA Systems Presence Detection -## Bridge +A comprehensive **Bluetooth Low Energy (BLE) presence detection system** that tracks beacon devices in real-time, calculates locations based on signal strength, and integrates with popular IoT platforms like Home Assistant and Node-RED. -Used for sending messages between MQTT broker and Kafka ... Initial config is done +## 🏗️ System Architecture -## Decoder +The system follows a **microservices architecture** with Apache Kafka as the central message bus: -Decoding BLE beacons -> generating notifications (batery, fall detection) +``` +┌─────────────┐ MQTT ┌─────────────┐ Kafka ┌─────────────┐ +│ MQTT │ ────────► │ Bridge │ ────────► │ Decoder │ +│ Gateway │ │ Service │ │ Service │ +└─────────────┘ └─────────────┘ └─────────────┘ + │ + ▼ +┌─────────────┐ WebSocket ┌─────────────┐ Kafka ┌─────────────┐ +│ Web UI │ ◄────────── │ Server │ ◄───────── │ Location │ +│ Dashboard │ │ Service │ │ Algorithm │ +└─────────────┘ └─────────────┘ └─────────────┘ +``` -Still needs to be reimplemented +### Core Components -## Locations algorithm +#### 🔌 **Bridge Service** (`cmd/bridge/`) +- **Purpose**: MQTT to Kafka message gateway +- **Function**: Subscribes to `publish_out/#` MQTT topics and forwards messages to Kafka `rawbeacons` topic +- **Features**: Configurable MQTT connection, automatic reconnection, error handling -Calculating location -> generating notifications +#### 🔍 **Decoder Service** (`cmd/decoder/`) +- **Purpose**: Processes raw BLE beacon advertisements +- **Supported Formats**: Ingics, Eddystone, Minew B7 +- **Functions**: + - Battery level monitoring + - Fall detection events + - Button press detection + - Device telemetry extraction +- **Output**: Processed events to `alertbeacons` Kafka topic -still needs to be implemented +#### 📍 **Location Algorithm** (`cmd/location/`) +- **Purpose**: Calculates device locations based on RSSI and proximity +- **Features**: + - Weighted location calculation using RSSI values + - Confidence scoring system + - Location change notifications + - Configurable thresholds and parameters +- **Output**: Location events to `locevents` Kafka topic -## Server +#### 🌐 **Server Service** (`cmd/server/`) +- **Purpose**: HTTP API and WebSocket server +- **Features**: + - RESTful API for beacon management (CRUD operations) + - Real-time WebSocket communication + - Settings management interface + - CORS-enabled web interface + - Home Assistant integration endpoints -Publishing to front end and notifications +#### 📊 **Supporting Services** +- **Kafka 3.9.0**: Central message bus with automatic topic creation +- **Kafdrop**: Web-based Kafka monitoring and management UI +- **Node-RED**: IoT workflow automation with pre-configured flows +- **Redis**: Real-time data caching and WebSocket session management +- **BoltDB**: Embedded database for persistent storage -Still needs to be reimplemented \ No newline at end of file +## 🚀 Quick Start + +### Prerequisites + +- Docker and Docker Compose +- Go 1.24+ (for local development) +- MQTT broker (compatible with BLE gateways) + +### Installation + +1. **Clone the repository**: + ```bash + git clone https://github.com/AFASystems/presence.git + cd presence + ``` + +2. **Start the system**: + ```bash + cd build + docker-compose up -d + ``` + +3. **Verify services**: + - **Web Interface**: http://localhost:8080 + - **Kafdrop (Kafka UI)**: http://localhost:9000 + - **Node-RED**: http://localhost:1880 + +### Configuration + +Set the following environment variables: + +```bash +# Web Server +HTTP_HOST_PATH=":8080" +HTTP_WS_HOST_PATH=":8081" + +# MQTT Configuration +MQTT_HOST="tcp://mqtt-broker:1883" +MQTT_USERNAME="your_username" +MQTT_PASSWORD="your_password" + +# Kafka Configuration +KAFKA_URL="kafka:29092" + +# Database +DB_PATH="./volumes/presence.db" +``` + +## 📡 Supported Beacon Types + +### Ingics Beacons +- Battery level monitoring +- Event detection (fall detection, button presses) +- Signal strength tracking + +### Eddystone Beacons +- Eddystone-UID protocol support +- Battery telemetry (Eddystone-TLM) +- Proximity detection + +### Minew B7 Beacons +- Vendor-specific format decoding +- Button counter tracking +- Battery status monitoring +- Multiple button modes + +## 🔧 Configuration Options + +### System Settings (`SettingsVal`) + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `location_confidence` | int64 | 80 | Minimum confidence level for location changes | +| `last_seen_threshold` | int64 | 300 | Time (seconds) before beacon considered offline | +| `beacon_metrics_size` | int | 10 | Number of RSSI measurements to keep for averaging | +| `ha_send_interval` | int64 | 60 | Home Assistant update interval (seconds) | +| `ha_send_changes_only` | bool | true | Send updates only on location changes | +| `rssi_min_threshold` | int64 | -90 | Minimum RSSI value for beacon detection | +| `enforce_rssi_threshold` | bool | true | Filter out weak signals | + +### API Endpoints + +#### Beacon Management +- `GET /api/beacons` - List all registered beacons +- `POST /api/beacons` - Register a new beacon +- `PUT /api/beacons/{id}` - Update beacon information +- `DELETE /api/beacons/{id}` - Remove a beacon + +#### Settings +- `GET /api/settings` - Get current system settings +- `POST /api/settings` - Update system settings + +#### WebSocket +- `/ws/broadcast` - Real-time beacon updates and notifications + +## 🏠 Home Assistant Integration + +### MQTT Auto-Discovery + +The system automatically generates MQTT messages for Home Assistant: + +```yaml +# Example device tracker configuration +device_tracker: + - platform: mqtt + devices: + beacon_001: "Presence/Beacons/beacon_001" +``` + +### Battery Monitoring + +```yaml +# Battery level sensor +sensor: + - platform: mqtt + name: "Beacon Battery" + state_topic: "Presence/Beacons/beacon_001/battery" + unit_of_measurement: "%" + device_class: battery +``` + +### Button Detection + +```yaml +# Button press sensor +binary_sensor: + - platform: mqtt + name: "Beacon Button" + state_topic: "Presence/Beacons/beacon_001/button" + device_class: "button" +``` + +## 📊 Data Models + +### Core Entities + +#### Beacon +```go +type Beacon struct { + Name string `json:"name"` + ID string `json:"beacon_id"` + BeaconType string `json:"beacon_type"` + BeaconLocation string `json:"beacon_location"` + LastSeen int64 `json:"last_seen"` + Distance float64 `json:"distance"` + LocationConfidence int64 `json:"location_confidence"` + HSButtonCounter int64 `json:"hs_button_counter"` + HSBattery int64 `json:"hs_button_battery"` + // ... additional fields +} +``` + +#### BeaconAdvertisement +```go +type BeaconAdvertisement struct { + Hostname string `json:"hostname"` + MAC string `json:"mac"` + RSSI int64 `json:"rssi"` + Data string `json:"data"` + BeaconType string `json:"beacon_type"` + UUID string `json:"uuid"` + Major string `json:"major"` + Minor string `json:"minor"` + // ... additional fields +} +``` + +#### LocationChange +```go +type LocationChange struct { + Method string `json:"method"` + BeaconRef Beacon `json:"beacon_info"` + Name string `json:"name"` + PreviousLocation string `json:"previous_location"` + NewLocation string `json:"new_location"` + Timestamp int64 `json:"timestamp"` +} +``` + +## 🐳 Docker Services + +### Available Services + +| Service | Image | Ports | Description | +|---------|-------|-------|-------------| +| kafka | apache/kafka:3.9.0 | 9092, 9093 | Apache Kafka message broker | +| kafdrop | obsidiandynamics/kafdrop | 9000 | Kafka monitoring UI | +| presence-bridge | local/build | - | MQTT to Kafka bridge | +| presence-decoder | local/build | - | BLE beacon decoder | +| presence-location | local/build | - | Location calculation service | +| presence-server | local/build | 8080, 8081 | HTTP API and WebSocket server | + +### Volumes + +- `./volumes/presence.db` - BoltDB database file +- `./volumes/node-red/` - Node-RED configuration and flows +- `./volumes/kafka-data/` - Kafka persistent data + +## 🔧 Development + +### Local Development Setup + +1. **Install dependencies**: + ```bash + go mod download + ``` + +2. **Run individual services**: + ```bash + # Run bridge service + go run cmd/bridge/main.go + + # Run decoder service + go run cmd/decoder/main.go + + # Run location algorithm + go run cmd/location/main.go + + # Run API server + go run cmd/server/main.go + ``` + +3. **Run tests**: + ```bash + go test ./... + ``` + +### Building Docker Images + +```bash +# Build all services +docker build -t presence-system . + +# Build individual service +docker build -f build/package/Dockerfile.bridge -t presence-bridge . +``` + +### Project Structure + +``` +/ +├── cmd/ # Main application entry points +│ ├── server/ # HTTP API & WebSocket server +│ ├── bridge/ # MQTT to Kafka bridge +│ ├── decoder/ # BLE beacon decoder +│ ├── location/ # Location calculation algorithm +│ └── testbench/ # Testing/development utilities +├── internal/ # Private application code +│ ├── app/ # Application components +│ └── pkg/ # Shared internal packages +│ ├── model/ # Data structures and types +│ ├── kafkaclient/ # Kafka producer/consumer +│ ├── config/ # Configuration management +│ ├── persistence/ # BoltDB operations +│ ├── redis/ # Redis client +│ └── bridge/ # MQTT bridge logic +├── build/ # Build artifacts and Docker configs +│ ├── package/ # Dockerfile for each component +│ └── docker-compose.yaml # Complete system deployment +├── web/ # Web interface files +├── volumes/ # Persistent data and configurations +├── scripts/ # Utility scripts +└── docs/ # Documentation +``` + +## 📈 Monitoring and Debugging + +### Kafka Topics + +- `rawbeacons` - Raw BLE beacon advertisements +- `alertbeacons` - Processed beacon events (battery, buttons) +- `locevents` - Location change notifications + +### Health Checks + +- **Kafka**: Topic creation and broker connectivity +- **Services**: Automatic restart on failure +- **Database**: BoltDB integrity checks + +### Logs + +```bash +# View service logs +docker-compose logs -f [service-name] + +# View all logs +docker-compose logs -f +``` + +## 🔌 Integrations + +### Node-RED Flows + +Pre-configured Node-RED flows are available in `/volumes/node-red/`: + +- **Beacon monitoring dashboards** +- **Location-based automations** +- **Battery level alerts** +- **Notification systems** + +### MQTT Topics + +System publishes to the following MQTT topics: + +- `Presence/Beacons/{beacon_id}/location` - Current beacon location +- `Presence/Beacons/{beacon_id}/battery` - Battery level +- `Presence/Beacons/{beacon_id}/button` - Button press events +- `Presence/Beacons/{beacon_id}/distance` - Distance from nearest gateway + +## 🤝 Contributing + +1. Fork the repository +2. Create a feature branch (`git checkout -b feature/amazing-feature`) +3. Commit your changes (`git commit -m 'Add amazing feature'`) +4. Push to the branch (`git push origin feature/amazing-feature`) +5. Open a Pull Request + +### Development Guidelines + +- Follow Go conventions and best practices +- Write comprehensive tests for new features +- Update documentation for API changes +- Use meaningful commit messages + +## 📄 License + +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. + +## 🆘 Support + +- **Issues**: [GitHub Issues](https://github.com/AFASystems/presence/issues) +- **Documentation**: [Project Wiki](https://github.com/AFASystems/presence/wiki) +- **Discussions**: [GitHub Discussions](https://github.com/AFASystems/presence/discussions) + +## 🔮 Roadmap + +### Upcoming Features + +- [ ] Enhanced location algorithms with machine learning +- [ ] Support for additional beacon types (iBeacon, AltBeacon) +- [ ] Mobile application for beacon management +- [ ] Advanced analytics and reporting dashboard +- [ ] Multi-tenant architecture +- [ ] Cloud deployment templates + +### Current Development Status + +- ✅ **Bridge Service**: Complete and stable +- ✅ **Decoder Service**: Core functionality implemented +- ✅ **Location Algorithm**: Basic algorithm functional +- ✅ **Server Service**: API and WebSocket implementation +- 🚧 **Web Interface**: Basic UI, enhancements in progress +- 🚧 **Documentation**: Comprehensive documentation being created +- 📋 **Testing**: Automated tests being expanded + +--- + +**AFA Systems Presence Detection** - Real-time BLE beacon tracking and location intelligence for modern IoT environments. \ No newline at end of file diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..0310d38 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,748 @@ +# API Documentation + +## Overview + +The AFA Systems Presence Detection API provides RESTful endpoints for managing beacons, settings, and real-time WebSocket communication for live updates. + +## Base URL + +``` +http://localhost:8080 +``` + +## Authentication + +Currently, the API does not implement authentication. This should be added for production deployments. + +## REST API Endpoints + +### Beacon Management + +#### Get All Beacons +Retrieves a list of all registered beacons with their current status and location information. + +```http +GET /api/beacons +``` + +**Response:** +```json +{ + "beacons": [ + { + "name": "Conference Room Beacon", + "beacon_id": "beacon_001", + "beacon_type": "ingics", + "beacon_location": "conference_room", + "last_seen": 1703078400, + "distance": 2.5, + "location_confidence": 85, + "hs_button_counter": 42, + "hs_button_battery": 85, + "hs_button_random": "abc123", + "hs_button_mode": "normal" + } + ] +} +``` + +#### Create Beacon +Registers a new beacon in the system. + +```http +POST /api/beacons +Content-Type: application/json +``` + +**Request Body:** +```json +{ + "name": "Meeting Room Beacon", + "beacon_id": "beacon_002", + "beacon_type": "eddystone", + "beacon_location": "meeting_room", + "hs_button_counter": 0, + "hs_button_battery": 100 +} +``` + +**Response:** +```json +{ + "message": "Beacon created successfully", + "beacon": { + "name": "Meeting Room Beacon", + "beacon_id": "beacon_002", + "beacon_type": "eddystone", + "beacon_location": "meeting_room", + "last_seen": 0, + "distance": 0, + "location_confidence": 0, + "hs_button_counter": 0, + "hs_button_battery": 100 + } +} +``` + +#### Update Beacon +Updates an existing beacon's information. + +```http +PUT /api/beacons/{id} +Content-Type: application/json +``` + +**Path Parameters:** +- `id` (string): The beacon ID to update + +**Request Body:** +```json +{ + "name": "Updated Conference Room Beacon", + "beacon_location": "main_conference", + "location_confidence": 90 +} +``` + +**Response:** +```json +{ + "message": "Beacon updated successfully", + "beacon": { + "name": "Updated Conference Room Beacon", + "beacon_id": "beacon_001", + "beacon_type": "ingics", + "beacon_location": "main_conference", + "last_seen": 1703078400, + "distance": 2.5, + "location_confidence": 90, + "hs_button_counter": 42, + "hs_button_battery": 85 + } +} +``` + +#### Delete Beacon +Removes a beacon from the system. + +```http +DELETE /api/beacons/{id} +``` + +**Path Parameters:** +- `id` (string): The beacon ID to delete + +**Response:** +```json +{ + "message": "Beacon deleted successfully" +} +``` + +### Settings Management + +#### Get System Settings +Retrieves current system configuration settings. + +```http +GET /api/settings +``` + +**Response:** +```json +{ + "settings": { + "location_confidence": 80, + "last_seen_threshold": 300, + "beacon_metrics_size": 10, + "ha_send_interval": 60, + "ha_send_changes_only": true, + "rssi_min_threshold": -90, + "enforce_rssi_threshold": true + } +} +``` + +#### Update System Settings +Updates system configuration settings. + +```http +POST /api/settings +Content-Type: application/json +``` + +**Request Body:** +```json +{ + "location_confidence": 85, + "last_seen_threshold": 600, + "beacon_metrics_size": 15, + "ha_send_interval": 30, + "rssi_min_threshold": -85 +} +``` + +**Response:** +```json +{ + "message": "Settings updated successfully", + "settings": { + "location_confidence": 85, + "last_seen_threshold": 600, + "beacon_metrics_size": 15, + "ha_send_interval": 30, + "ha_send_changes_only": true, + "rssi_min_threshold": -85, + "enforce_rssi_threshold": true + } +} +``` + +### Location Information + +#### Get Beacon Locations +Retrieves current location information for all beacons. + +```http +GET /api/locations +``` + +**Response:** +```json +{ + "beacons": [ + { + "method": "location_update", + "previous_confident_location": "reception", + "distance": 3.2, + "id": "beacon_001", + "location": "conference_room", + "last_seen": 1703078450 + }, + { + "method": "location_update", + "previous_confident_location": "office_a", + "distance": 1.8, + "id": "beacon_002", + "location": "meeting_room", + "last_seen": 1703078440 + } + ] +} +``` + +#### Get Specific Beacon Location +Retrieves location information for a specific beacon. + +```http +GET /api/locations/{id} +``` + +**Path Parameters:** +- `id` (string): The beacon ID + +**Response:** +```json +{ + "method": "location_update", + "previous_confident_location": "reception", + "distance": 3.2, + "id": "beacon_001", + "location": "conference_room", + "last_seen": 1703078450 +} +``` + +### Health Check + +#### System Health +Check if the API server is running and basic systems are operational. + +```http +GET /api/health +``` + +**Response:** +```json +{ + "status": "healthy", + "timestamp": "2024-12-20T10:30:00Z", + "services": { + "database": "connected", + "kafka": "connected", + "redis": "connected" + } +} +``` + +## WebSocket API + +### WebSocket Connection +Connect to the WebSocket endpoint for real-time updates. + +``` +ws://localhost:8080/ws/broadcast +``` + +### WebSocket Message Format + +#### Beacon Update Notification +```json +{ + "type": "beacon_update", + "data": { + "method": "location_update", + "beacon_info": { + "name": "Conference Room Beacon", + "beacon_id": "beacon_001", + "beacon_type": "ingics", + "distance": 2.5 + }, + "name": "conference_room", + "beacon_name": "Conference Room Beacon", + "previous_location": "reception", + "new_location": "conference_room", + "timestamp": 1703078450 + } +} +``` + +#### Button Press Event +```json +{ + "type": "button_event", + "data": { + "beacon_id": "beacon_001", + "button_counter": 43, + "button_mode": "normal", + "timestamp": 1703078460 + } +} +``` + +#### Battery Alert +```json +{ + "type": "battery_alert", + "data": { + "beacon_id": "beacon_002", + "battery_level": 15, + "alert_level": "warning", + "timestamp": 1703078470 + } +} +``` + +#### Fall Detection Event +```json +{ + "type": "fall_detection", + "data": { + "beacon_id": "beacon_001", + "event_type": "fall_detected", + "confidence": 92, + "timestamp": 1703078480 + } +} +``` + +#### System Status Update +```json +{ + "type": "system_status", + "data": { + "active_beacons": 12, + "total_locations": 8, + "kafka_status": "connected", + "redis_status": "connected", + "timestamp": 1703078490 + } +} +``` + +## Data Models + +### Beacon Model +```typescript +interface Beacon { + name: string; + beacon_id: string; + beacon_type: "ingics" | "eddystone" | "minew_b7" | "ibeacon"; + beacon_location: string; + last_seen: number; // Unix timestamp + distance: number; // Distance in meters + previous_location?: string; + previous_confident_location?: string; + expired_location?: string; + location_confidence: number; // 0-100 + location_history: string[]; + beacon_metrics: BeaconMetric[]; + + // Handshake/Button specific fields + hs_button_counter: number; + hs_button_prev: number; + hs_button_battery: number; + hs_button_random: string; + hs_button_mode: string; +} +``` + +### BeaconMetric Model +```typescript +interface BeaconMetric { + location: string; + distance: number; + rssi: number; + timestamp: number; +} +``` + +### Settings Model +```typescript +interface Settings { + location_confidence: number; // Minimum confidence level (0-100) + last_seen_threshold: number; // Seconds before beacon considered offline + beacon_metrics_size: number; // Number of RSSI measurements to keep + ha_send_interval: number; // Home Assistant update interval (seconds) + ha_send_changes_only: boolean; // Only send updates on changes + rssi_min_threshold: number; // Minimum RSSI for detection + enforce_rssi_threshold: boolean; // Filter weak signals +} +``` + +### LocationChange Model +```typescript +interface LocationChange { + method: string; // "location_update" | "beacon_added" | "beacon_removed" + beacon_ref: Beacon; // Complete beacon information + name: string; // Beacon name + beacon_name: string; // Beacon name (duplicate) + previous_location: string; // Previous location + new_location: string; // New location + timestamp: number; // Unix timestamp +} +``` + +## Error Responses + +### Standard Error Format +```json +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "Invalid request data", + "details": { + "field": "beacon_id", + "reason": "Beacon ID is required" + } + } +} +``` + +### Common Error Codes + +| Code | HTTP Status | Description | +|------|-------------|-------------| +| `VALIDATION_ERROR` | 400 | Request data validation failed | +| `NOT_FOUND` | 404 | Resource not found | +| `CONFLICT` | 409 | Resource already exists | +| `INTERNAL_ERROR` | 500 | Internal server error | +| `SERVICE_UNAVAILABLE` | 503 | Required service is unavailable | + +### Validation Error Example +```http +POST /api/beacons +Content-Type: application/json +``` + +**Invalid Request:** +```json +{ + "name": "", + "beacon_type": "invalid_type" +} +``` + +**Response:** +```json +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "Invalid request data", + "details": { + "name": "Name cannot be empty", + "beacon_type": "Invalid beacon type. Must be one of: ingics, eddystone, minew_b7, ibeacon" + } + } +} +``` + +## Rate Limiting + +Currently, the API does not implement rate limiting. Consider implementing rate limiting for production deployments: + +- Suggested limits: 100 requests per minute per IP address +- WebSocket connections: Maximum 50 concurrent connections +- Consider authentication-based rate limiting + +## CORS Configuration + +The API server is configured with CORS enabled for development. Production deployments should restrict CORS origins to specific domains. + +## Integration Examples + +### JavaScript/TypeScript Client + +```typescript +class PresenceAPIClient { + private baseURL: string; + + constructor(baseURL: string = 'http://localhost:8080') { + this.baseURL = baseURL; + } + + async getBeacons(): Promise { + const response = await fetch(`${this.baseURL}/api/beacons`); + if (!response.ok) { + throw new Error(`HTTP error! status: ${response.status}`); + } + const data = await response.json(); + return data.beacons; + } + + async createBeacon(beacon: Partial): Promise { + const response = await fetch(`${this.baseURL}/api/beacons`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + }, + body: JSON.stringify(beacon), + }); + + if (!response.ok) { + const error = await response.json(); + throw new Error(error.error?.message || 'Failed to create beacon'); + } + + const data = await response.json(); + return data.beacon; + } + + connectWebSocket(onMessage: (message: any) => void): WebSocket { + const ws = new WebSocket(`${this.baseURL.replace('http', 'ws')}/ws/broadcast`); + + ws.onmessage = (event) => { + try { + const message = JSON.parse(event.data); + onMessage(message); + } catch (error) { + console.error('Failed to parse WebSocket message:', error); + } + }; + + ws.onerror = (error) => { + console.error('WebSocket error:', error); + }; + + ws.onclose = () => { + console.log('WebSocket connection closed'); + }; + + return ws; + } +} + +// Usage example +const client = new PresenceAPIClient(); + +// Get all beacons +const beacons = await client.getBeacons(); +console.log('Active beacons:', beacons); + +// Create a new beacon +const newBeacon = await client.createBeacon({ + name: 'Test Beacon', + beacon_id: 'test_beacon_001', + beacon_type: 'eddystone', + beacon_location: 'test_room' +}); + +// Connect to WebSocket for real-time updates +const ws = client.connectWebSocket((message) => { + switch (message.type) { + case 'beacon_update': + console.log('Beacon location updated:', message.data); + break; + case 'button_event': + console.log('Button pressed:', message.data); + break; + case 'battery_alert': + console.log('Low battery warning:', message.data); + break; + } +}); +``` + +### Python Client + +```python +import requests +import websocket +import json +from typing import List, Dict, Any + +class PresenceAPIClient: + def __init__(self, base_url: str = "http://localhost:8080"): + self.base_url = base_url + self.ws_url = base_url.replace("http", "ws") + + def get_beacons(self) -> List[Dict[str, Any]]: + """Get all registered beacons.""" + response = requests.get(f"{self.base_url}/api/beacons") + response.raise_for_status() + data = response.json() + return data["beacons"] + + def create_beacon(self, beacon_data: Dict[str, Any]) -> Dict[str, Any]: + """Create a new beacon.""" + response = requests.post( + f"{self.base_url}/api/beacons", + json=beacon_data + ) + response.raise_for_status() + data = response.json() + return data["beacon"] + + def update_beacon(self, beacon_id: str, beacon_data: Dict[str, Any]) -> Dict[str, Any]: + """Update an existing beacon.""" + response = requests.put( + f"{self.base_url}/api/beacons/{beacon_id}", + json=beacon_data + ) + response.raise_for_status() + data = response.json() + return data["beacon"] + + def delete_beacon(self, beacon_id: str) -> None: + """Delete a beacon.""" + response = requests.delete(f"{self.base_url}/api/beacons/{beacon_id}") + response.raise_for_status() + + def get_settings(self) -> Dict[str, Any]: + """Get system settings.""" + response = requests.get(f"{self.base_url}/api/settings") + response.raise_for_status() + return response.json()["settings"] + + def update_settings(self, settings_data: Dict[str, Any]) -> Dict[str, Any]: + """Update system settings.""" + response = requests.post( + f"{self.base_url}/api/settings", + json=settings_data + ) + response.raise_for_status() + return response.json()["settings"] + +# Usage example +client = PresenceAPIClient() + +# Get all beacons +beacons = client.get_beacons() +print(f"Found {len(beacons)} beacons") + +# Create a new beacon +new_beacon = client.create_beacon({ + "name": "Python Test Beacon", + "beacon_id": "python_test_001", + "beacon_type": "eddystone", + "beacon_location": "python_room" +}) +print(f"Created beacon: {new_beacon['name']}") + +# Update settings +settings = client.update_settings({ + "location_confidence": 85, + "ha_send_interval": 30 +}) +print(f"Updated settings: {settings}") +``` + +## Testing + +### Unit Testing Example (Go) + +```go +package api_test + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + "github.com/stretchr/testify/assert" +) + +func TestGetBeacons(t *testing.T) { + // Setup test server + router := setupTestRouter() + + req, _ := http.NewRequest("GET", "/api/beacons", nil) + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + + var response map[string]interface{} + err := json.Unmarshal(w.Body.Bytes(), &response) + assert.NoError(t, err) + assert.Contains(t, response, "beacons") +} + +func TestCreateBeacon(t *testing.T) { + router := setupTestRouter() + + beaconData := map[string]interface{}{ + "name": "Test Beacon", + "beacon_id": "test_001", + "beacon_type": "eddystone", + "beacon_location": "test_room", + } + + jsonData, _ := json.Marshal(beaconData) + req, _ := http.NewRequest("POST", "/api/beacons", bytes.NewBuffer(jsonData)) + req.Header.Set("Content-Type", "application/json") + + w := httptest.NewRecorder() + router.ServeHTTP(w, req) + + assert.Equal(t, http.StatusCreated, w.Code) + + var response map[string]interface{} + err := json.Unmarshal(w.Body.Bytes(), &response) + assert.NoError(t, err) + assert.Contains(t, response, "beacon") +} +``` + +## Security Considerations + +For production deployments, consider implementing: + +1. **Authentication**: JWT tokens or API key authentication +2. **Authorization**: Role-based access control (RBAC) +3. **Rate Limiting**: Prevent API abuse +4. **Input Validation**: Comprehensive input sanitization +5. **HTTPS**: TLS encryption for all API communications +6. **CORS**: Restrict origins to trusted domains +7. **Logging**: Comprehensive audit logging +8. **Security Headers**: Implement security HTTP headers + +## API Versioning + +The current API is version 1. Future versions will be: + +- Version 1: `/api/v1/...` (current, implied) +- Version 2: `/api/v2/...` (future breaking changes) + +Backward compatibility will be maintained within major versions. \ No newline at end of file diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..f8259a1 --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,1039 @@ +# Deployment Guide + +## Overview + +This guide covers the deployment of the AFA Systems Presence Detection system in various environments, from development setups to production deployments. + +## System Requirements + +### Minimum Requirements + +- **CPU**: 2 cores +- **Memory**: 4GB RAM +- **Storage**: 20GB available space +- **Network**: Reliable MQTT broker connection +- **Operating System**: Linux (Ubuntu 20.04+, CentOS 8+, RHEL 8+), macOS, or Windows 10+ + +### Recommended Requirements + +- **CPU**: 4 cores +- **Memory**: 8GB RAM +- **Storage**: 50GB SSD +- **Network**: Gigabit ethernet, low-latency MQTT connection +- **Operating System**: Ubuntu 22.04 LTS or RHEL 9+ + +### Software Dependencies + +- **Docker**: 20.10+ with Docker Compose v2.0+ +- **Git**: For source code management +- **Go**: 1.24+ (for local development) +- **MQTT Broker**: Compatible with BLE gateway devices (Mosquitto, EMQX, HiveMQ) + +## Deployment Options + +### 1. Docker Compose (Recommended) + +The Docker Compose deployment provides a complete, production-ready environment with all services included. + +#### Quick Start + +1. **Clone the repository**: + ```bash + git clone https://github.com/AFASystems/presence.git + cd presence + ``` + +2. **Create environment configuration**: + ```bash + cp .env.example .env + # Edit .env with your configuration + ``` + +3. **Start all services**: + ```bash + cd build + docker-compose up -d + ``` + +4. **Verify deployment**: + ```bash + docker-compose ps + docker-compose logs -f + ``` + +#### Environment Configuration + +Create a `.env` file in the project root: + +```bash +# Web Server Configuration +HTTP_HOST_PATH=:8080 +HTTP_WS_HOST_PATH=:8081 + +# MQTT Configuration +MQTT_HOST=tcp://your-mqtt-broker:1883 +MQTT_USERNAME=your_mqtt_username +MQTT_PASSWORD=your_mqtt_password +MQTT_CLIENT_ID=presence_system + +# Kafka Configuration +KAFKA_URL=kafka:29092 +KAFKA_BOOTSTRAP_SERVERS=kafka:29092 +KAFKA_GROUP_ID=presence_group + +# Database Configuration +DB_PATH=./volumes/presence.db + +# Redis Configuration +REDIS_URL=redis:6379 +REDIS_PASSWORD= + +# Security +CORS_ORIGINS=http://localhost:3000,http://localhost:8080 +API_SECRET_KEY=your-secret-key-here + +# Logging +LOG_LEVEL=info +LOG_FORMAT=json +``` + +#### Docker Compose Services + +The default `docker-compose.yaml` includes: + +| Service | Description | Ports | Resources | +|---------|-------------|-------|------------| +| **kafka** | Apache Kafka message broker | 9092, 9093 | 2GB RAM | +| **kafdrop** | Kafka monitoring UI | 9000 | 512MB RAM | +| **presence-bridge** | MQTT to Kafka bridge | - | 256MB RAM | +| **presence-decoder** | BLE beacon decoder | - | 512MB RAM | +| **presence-location** | Location calculation service | - | 512MB RAM | +| **presence-server** | HTTP API & WebSocket server | 8080, 8081 | 1GB RAM | +| **redis** | Caching and session storage | 6379 | 512MB RAM | + +#### Production Docker Compose + +For production deployment, create `docker-compose.prod.yaml`: + +```yaml +version: "3.8" + +services: + kafka: + image: apache/kafka:3.9.0 + restart: always + environment: + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092,CONTROLLER://:9093 + KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://kafka.example.com:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2 + KAFKA_NUM_PARTITIONS: 6 + KAFKA_DEFAULT_REPLICATION_FACTOR: 3 + KAFKA_MIN_INSYNC_REPLICAS: 2 + KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" + KAFKA_DELETE_TOPIC_ENABLE: "true" + KAFKA_LOG_RETENTION_HOURS: 168 + KAFKA_LOG_RETENTION_BYTES: 1073741824 + volumes: + - kafka-data:/var/lib/kafka/data + deploy: + resources: + limits: + memory: 4G + reservations: + memory: 2G + + presence-bridge: + build: + context: .. + dockerfile: build/package/Dockerfile.bridge + restart: always + environment: + - HTTP_HOST_PATH=${HTTP_HOST_PATH} + - MQTT_HOST=${MQTT_HOST} + - MQTT_USERNAME=${MQTT_USERNAME} + - MQTT_PASSWORD=${MQTT_PASSWORD} + - KAFKA_URL=${KAFKA_URL} + - LOG_LEVEL=${LOG_LEVEL} + volumes: + - ../volumes:/app/volumes + depends_on: + kafka: + condition: service_healthy + deploy: + resources: + limits: + memory: 512M + reservations: + memory: 256M + + presence-decoder: + build: + context: .. + dockerfile: build/package/Dockerfile.decoder + restart: always + environment: + - HTTP_HOST_PATH=${HTTP_HOST_PATH} + - KAFKA_URL=${KAFKA_URL} + - REDIS_URL=${REDIS_URL} + - LOG_LEVEL=${LOG_LEVEL} + volumes: + - ../volumes:/app/volumes + depends_on: + kafka: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 1G + reservations: + memory: 512M + + presence-location: + build: + context: .. + dockerfile: build/package/Dockerfile.location + restart: always + environment: + - HTTP_HOST_PATH=${HTTP_HOST_PATH} + - KAFKA_URL=${KAFKA_URL} + - REDIS_URL=${REDIS_URL} + - LOG_LEVEL=${LOG_LEVEL} + volumes: + - ../volumes:/app/volumes + depends_on: + kafka: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 1G + reservations: + memory: 512M + + presence-server: + build: + context: .. + dockerfile: build/package/Dockerfile.server + restart: always + environment: + - HTTP_HOST_PATH=${HTTP_HOST_PATH} + - HTTP_WS_HOST_PATH=${HTTP_WS_HOST_PATH} + - KAFKA_URL=${KAFKA_URL} + - REDIS_URL=${REDIS_URL} + - DB_PATH=/app/volumes/presence.db + - CORS_ORIGINS=${CORS_ORIGINS} + - API_SECRET_KEY=${API_SECRET_KEY} + - LOG_LEVEL=${LOG_LEVEL} + ports: + - "8080:8080" + - "8081:8081" + volumes: + - ../volumes:/app/volumes + - ../web:/app/web + depends_on: + kafka: + condition: service_healthy + redis: + condition: service_healthy + deploy: + resources: + limits: + memory: 2G + reservations: + memory: 1G + + redis: + image: redis:7-alpine + restart: always + command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD} + volumes: + - redis-data:/data + deploy: + resources: + limits: + memory: 1G + reservations: + memory: 512M + + kafdrop: + image: obsidiandynamics/kafdrop + restart: always + environment: + KAFKA_BROKERCONNECT: "kafka:29092" + JVM_OPTS: "-Xms256M -Xmx512M" + ports: + - "9000:9000" + depends_on: + kafka: + condition: service_healthy + + nginx: + image: nginx:alpine + restart: always + ports: + - "80:80" + - "443:443" + volumes: + - ./nginx.conf:/etc/nginx/nginx.conf + - ./ssl:/etc/nginx/ssl + depends_on: + - presence-server + deploy: + resources: + limits: + memory: 256M + reservations: + memory: 128M + +volumes: + kafka-data: + redis-data: +``` + +### 2. Kubernetes Deployment + +For larger-scale deployments, use Kubernetes with Helm charts. + +#### Prerequisites + +- Kubernetes cluster (v1.24+) +- Helm 3.0+ +- kubectl configured +- Persistent storage provisioner + +#### Installation + +1. **Create namespace**: + ```bash + kubectl create namespace presence-system + ``` + +2. **Add Helm repository**: + ```bash + helm repo add presence-system https://charts.afasystems.com/presence + helm repo update + ``` + +3. **Install with custom values**: + ```bash + helm install presence presence-system/presence-system \ + --namespace presence-system \ + --values values.prod.yaml + ``` + +#### Custom Values (`values.prod.yaml`) + +```yaml +global: + imageRegistry: your-registry.com + imagePullSecrets: [your-registry-secret] + + mqtt: + host: "tcp://mqtt-broker:1883" + username: "your_username" + password: "your_password" + +kafka: + enabled: true + replicaCount: 3 + persistence: + enabled: true + size: 100Gi + resources: + requests: + memory: "2Gi" + cpu: "1" + limits: + memory: "4Gi" + cpu: "2" + +redis: + enabled: true + auth: + enabled: true + password: "your-redis-password" + master: + persistence: + enabled: true + size: 20Gi + resources: + requests: + memory: "512Mi" + cpu: "0.5" + limits: + memory: "1Gi" + cpu: "1" + +bridge: + replicaCount: 2 + resources: + requests: + memory: "256Mi" + cpu: "0.25" + limits: + memory: "512Mi" + cpu: "0.5" + +decoder: + replicaCount: 3 + resources: + requests: + memory: "512Mi" + cpu: "0.5" + limits: + memory: "1Gi" + cpu: "1" + +location: + replicaCount: 3 + resources: + requests: + memory: "512Mi" + cpu: "0.5" + limits: + memory: "1Gi" + cpu: "1" + +server: + replicaCount: 2 + service: + type: LoadBalancer + ports: + http: 80 + https: 443 + websocket: 8081 + ingress: + enabled: true + className: "nginx" + annotations: + nginx.ingress.kubernetes.io/rewrite-target: / + hosts: + - host: presence.example.com + paths: + - path: / + pathType: Prefix + tls: + - secretName: presence-tls + hosts: + - presence.example.com + resources: + requests: + memory: "1Gi" + cpu: "0.5" + limits: + memory: "2Gi" + cpu: "1" + +monitoring: + enabled: true + prometheus: + enabled: true + grafana: + enabled: true + +autoscaling: + enabled: true + bridge: + minReplicas: 2 + maxReplicas: 10 + targetCPUUtilizationPercentage: 70 + decoder: + minReplicas: 3 + maxReplicas: 20 + targetCPUUtilizationPercentage: 70 + location: + minReplicas: 3 + maxReplicas: 20 + targetCPUUtilizationPercentage: 70 + server: + minReplicas: 2 + maxReplicas: 10 + targetCPUUtilizationPercentage: 70 +``` + +### 3. Manual Deployment + +For custom environments or specific requirements. + +#### Building from Source + +1. **Prerequisites**: + ```bash + # Install Go 1.24+ + wget https://go.dev/dl/go1.24.linux-amd64.tar.gz + sudo tar -C /usr/local -xzf go1.24.linux-amd64.tar.gz + export PATH=$PATH:/usr/local/go/bin + + # Install Node.js (for web assets) + curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - + sudo apt-get install -y nodejs + ``` + +2. **Clone and build**: + ```bash + git clone https://github.com/AFASystems/presence.git + cd presence + + # Build web assets + cd web + npm install + npm run build + + # Build Go applications + cd .. + go mod download + go build -o bin/bridge ./cmd/bridge + go build -o bin/decoder ./cmd/decoder + go build -o bin/location ./cmd/location + go build -o bin/server ./cmd/server + ``` + +3. **System service files** (`/etc/systemd/system/presence-bridge.service`): + ```ini + [Unit] + Description=Presence Bridge Service + After=network.target + + [Service] + Type=simple + User=presence + Group=presence + WorkingDirectory=/opt/presence + ExecStart=/opt/presence/bin/bridge + Restart=always + RestartSec=10 + Environment=HTTP_HOST_PATH=:8080 + Environment=MQTT_HOST=tcp://mqtt-broker:1883 + Environment=MQTT_USERNAME=your_username + Environment=MQTT_PASSWORD=your_password + Environment=KAFKA_URL=kafka:29092 + Environment=LOG_LEVEL=info + + [Install] + WantedBy=multi-user.target + ``` + + Create similar service files for decoder, location, and server services. + +4. **Enable and start services**: + ```bash + sudo systemctl daemon-reload + sudo systemctl enable presence-bridge + sudo systemctl enable presence-decoder + sudo systemctl enable presence-location + sudo systemctl enable presence-server + + sudo systemctl start presence-bridge + sudo systemctl start presence-decoder + sudo systemctl start presence-location + sudo systemctl start presence-server + ``` + +## Network Configuration + +### Firewall Rules + +Ensure these ports are open: + +| Port | Service | Description | +|------|---------|-------------| +| 80 | HTTP | Web interface (if using nginx) | +| 443 | HTTPS | Secure web interface | +| 8080 | HTTP API | REST API server | +| 8081 | WebSocket | Real-time updates | +| 9000 | Kafdrop | Kafka monitoring UI | +| 1883 | MQTT | MQTT broker (if external) | +| 9092 | Kafka | Kafka broker | + +### Load Balancer Configuration + +#### HAProxy Example + +``` +frontend presence_frontend + bind *:80 + default_backend presence_backend + +backend presence_backend + balance roundrobin + server presence1 10.0.1.10:8080 check + server presence2 10.0.1.11:8080 check + +frontend presence_ws_frontend + bind *:8081 + default_backend presence_ws_backend + +backend presence_ws_backend + balance roundrobin + server presence1 10.0.1.10:8081 check + server presence2 10.0.1.11:8081 check +``` + +#### Nginx Example + +```nginx +upstream presence_backend { + server 10.0.1.10:8080; + server 10.0.1.11:8080; +} + +upstream presence_ws_backend { + server 10.0.1.10:8081; + server 10.0.1.11:8081; +} + +server { + listen 80; + server_name presence.example.com; + + location / { + proxy_pass http://presence_backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + location /ws { + proxy_pass http://presence_ws_backend; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +## SSL/TLS Configuration + +### Let's Encrypt with Certbot + +1. **Install Certbot**: + ```bash + sudo apt-get update + sudo apt-get install certbot python3-certbot-nginx + ``` + +2. **Generate certificates**: + ```bash + sudo certbot --nginx -d presence.example.com + ``` + +3. **Auto-renewal**: + ```bash + sudo crontab -e + # Add: 0 12 * * * /usr/bin/certbot renew --quiet + ``` + +### Nginx SSL Configuration + +```nginx +server { + listen 443 ssl http2; + server_name presence.example.com; + + ssl_certificate /etc/letsencrypt/live/presence.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/presence.example.com/privkey.pem; + + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512; + ssl_prefer_server_ciphers off; + ssl_session_cache shared:SSL:10m; + + location / { + proxy_pass http://presence_backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + location /ws { + proxy_pass http://presence_ws_backend; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +## Monitoring and Logging + +### Prometheus Configuration + +Create `prometheus.yml`: + +```yaml +global: + scrape_interval: 15s + evaluation_interval: 15s + +scrape_configs: + - job_name: 'presence-server' + static_configs: + - targets: ['presence-server:8080'] + metrics_path: /metrics + scrape_interval: 5s + + - job_name: 'kafka' + static_configs: + - targets: ['kafka:9092'] + + - job_name: 'redis' + static_configs: + - targets: ['redis:6379'] +``` + +### Grafana Dashboard + +Import the provided Grafana dashboard or create custom visualizations: + +1. **System Metrics**: CPU, Memory, Disk usage +2. **Kafka Metrics**: Throughput, lag, topic sizes +3. **Beacon Metrics**: Active beacons, location updates, battery levels +4. **API Metrics**: Request rates, response times, error rates +5. **WebSocket Metrics**: Active connections, message rates + +### Log Aggregation with ELK Stack + +#### Elasticsearch Configuration + +```yaml +version: '3.8' +services: + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.5.0 + environment: + - discovery.type=single-node + - "ES_JAVA_OPTS=-Xms1g -Xmx1g" + ports: + - "9200:9200" + volumes: + - elasticsearch-data:/usr/share/elasticsearch/data + + logstash: + image: docker.elastic.co/logstash/logstash:8.5.0 + ports: + - "5044:5044" + volumes: + - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf + depends_on: + - elasticsearch + + kibana: + image: docker.elastic.co/kibana/kibana:8.5.0 + ports: + - "5601:5601" + environment: + - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 + depends_on: + - elasticsearch + +volumes: + elasticsearch-data: +``` + +## Backup and Recovery + +### Database Backup + +```bash +#!/bin/bash +# backup.sh + +BACKUP_DIR="/backups/presence" +DATE=$(date +%Y%m%d_%H%M%S) +DB_FILE="/opt/presence/volumes/presence.db" + +mkdir -p $BACKUP_DIR + +# Backup BoltDB +cp $DB_FILE $BACKUP_DIR/presence_$DATE.db + +# Compress old backups +find $BACKUP_DIR -name "presence_*.db" -mtime +7 -exec gzip {} \; + +# Keep only last 30 days +find $BACKUP_DIR -name "presence_*.db.gz" -mtime +30 -delete +``` + +### Restore from Backup + +```bash +#!/bin/bash +# restore.sh + +BACKUP_FILE=$1 +DB_FILE="/opt/presence/volumes/presence.db" + +if [ -z "$BACKUP_FILE" ]; then + echo "Usage: $0 " + exit 1 +fi + +# Stop services +sudo systemctl stop presence-server presence-decoder presence-location + +# Restore database +sudo cp $BACKUP_FILE $DB_FILE +sudo chown presence:presence $DB_FILE + +# Start services +sudo systemctl start presence-server presence-decoder presence-location +``` + +### Kafka Backup + +```bash +#!/bin/bin/bash +# kafka_backup.sh + +BACKUP_DIR="/backups/kafka" +DATE=$(date +%Y%m%d_%H%M%S) + +mkdir -p $BACKUP_DIR + +# Backup Kafka topics +kafka-topics.sh --bootstrap-server localhost:9092 --list > $BACKUP_DIR/topics_$DATE.txt + +# Backup topic data (example) +for topic in rawbeacons alertbeacons locevents; do + kafka-console-consumer.sh --bootstrap-server localhost:9092 \ + --topic $topic --from-beginning > $BACKUP_DIR/${topic}_$DATE.json +done +``` + +## Performance Tuning + +### Kafka Optimization + +```bash +# Kafka server.properties +num.network.threads=8 +num.io.threads=16 +socket.send.buffer.bytes=102400 +socket.receive.buffer.bytes=102400 +socket.request.max.bytes=104857600 +log.retention.hours=168 +log.segment.bytes=1073741824 +log.retention.check.interval.ms=300000 +``` + +### Redis Optimization + +```bash +# redis.conf +maxmemory 2gb +maxmemory-policy allkeys-lru +save 900 1 +save 300 10 +save 60 10000 +``` + +### Application Tuning + +Environment variables for performance: + +```bash +# Go runtime +GOMAXPROCS=4 +GOGC=100 + +# Application settings +BEACON_METRICS_SIZE=50 +LOCATION_CONFIDENCE=90 +KAFKA_CONSUMER_FETCH_MAX_BYTES=1048576 +REDIS_POOL_SIZE=50 +``` + +## Security Hardening + +### Network Security + +1. **Network Segmentation**: + ```bash + # Create dedicated network for presence system + docker network create --driver bridge presence-network + docker network connect presence-network presence-server + ``` + +2. **Port Security**: + ```bash + # Only expose necessary ports + ufw allow 80/tcp + ufw allow 443/tcp + ufw allow 8080/tcp + ufw deny 9092/tcp # Kafka internal only + ``` + +### Application Security + +1. **Environment Variable Security**: + ```bash + # Use secrets management + export API_SECRET_KEY=$(openssl rand -base64 32) + export MQTT_PASSWORD_FILE=/run/secrets/mqtt_password + ``` + +2. **Container Security**: + ```yaml + # Docker Compose security settings + services: + presence-server: + security_opt: + - no-new-privileges:true + read_only: true + tmpfs: + - /tmp + user: "1000:1000" + cap_drop: + - ALL + cap_add: + - NET_BIND_SERVICE + ``` + +## Troubleshooting + +### Common Issues + +#### Service Won't Start + +```bash +# Check logs +docker-compose logs presence-server +sudo journalctl -u presence-bridge -f + +# Check configuration +docker-compose config +``` + +#### Kafka Connection Issues + +```bash +# Check Kafka health +docker-compose exec kafka kafka-topics.sh --bootstrap-server localhost:9092 --list + +# Check network connectivity +docker network ls +docker network inspect presence_build +``` + +#### High Memory Usage + +```bash +# Monitor memory usage +docker stats + +# Check Go garbage collection +docker-compose exec presence-server pprof http://localhost:6060/debug/pprof/heap +``` + +#### WebSocket Connection Issues + +```bash +# Test WebSocket connection +wscat -c ws://localhost:8080/ws/broadcast + +# Check logs for connection issues +docker-compose logs presence-server | grep websocket +``` + +### Performance Diagnostics + +#### System Monitoring + +```bash +# System resources +htop +iostat -x 1 +netstat -i + +# Application performance +curl http://localhost:8080/api/health +docker-compose exec presence-server curl http://localhost:8080/api/health +``` + +#### Database Performance + +```bash +# BoltDB inspection +boltstats ./volumes/presence.db + +# Redis monitoring +redis-cli info memory +redis-cli info stats +``` + +## Migration and Upgrades + +### Version Upgrade Procedure + +1. **Backup current system**: + ```bash + ./scripts/backup.sh + ``` + +2. **Update source code**: + ```bash + git fetch origin + git checkout v2.0.0 + ``` + +3. **Build new images**: + ```bash + docker-compose build --no-cache + ``` + +4. **Run database migrations**: + ```bash + docker-compose run --rm presence-server ./migrate up + ``` + +5. **Restart services with zero downtime**: + ```bash + docker-compose up -d --scale presence-server=2 + # Wait for health checks + docker-compose up -d --scale presence-server=1 + ``` + +### Configuration Migration + +Create migration scripts for configuration changes: + +```bash +#!/bin/bash +# migrate_v1_to_v2.sh + +# Backup old config +cp .env .env.backup + +# Add new configuration variables +echo "NEW_FEATURE_ENABLED=true" >> .env +echo "API_RATE_LIMIT=100" >> .env + +# Update deprecated settings +sed -i 's/OLD_SETTING/NEW_SETTING/g' .env + +echo "Migration completed. Please review .env file." +``` + +This comprehensive deployment guide should help you successfully deploy and manage the AFA Systems Presence Detection system in various environments. \ No newline at end of file