Data Management

Learn how to manage, analyze, and export data collected by USV.

Data Types

1. Telemetry Data

Real-time system status data:

• GPS position
• Speed and direction
• Battery level
• System status

Storage Format: Time-series database (InfluxDB)

2. Sensor Data

Environmental data collected by various sensors:

• Water quality parameters
• Weather data
• Images/videos
• Sonar/radar data

Storage Format: SQLite / PostgreSQL + file system

3. Log Data

System operation logs:

• System events
• Error messages
• User operations
• Mission records

Storage Format: JSON / plain text

Data Collection

Configure Collection Parameters

data_collection:
  # Telemetry data
  telemetry:
    enabled: true
    rate: 1                # Hz
    buffer_size: 1000

  # Sensor data
  sensors:
    water_quality:
      enabled: true
      rate: 0.5            # Hz
      parameters:
        - name: "ph"
          unit: "pH"
        - name: "dissolved_oxygen"
          unit: "mg/L"
        - name: "temperature"
          unit: "°C"

    camera:
      enabled: true
      mode: "video"        # video, photo, timelapse
      resolution: [1920, 1080]
      fps: 30
      encoding: "h264"
      quality: 85

  # Storage settings
  storage:
    local: true
    cloud: true
    compression: true

Start Data Collection

# Start collection
jhusv data start

# Stop collection
jhusv data stop

# View status
jhusv data status

Data Storage

Local Storage Structure

~/jhusv/data/
├── telemetry/
│   ├── 2024-01-15/
│   │   ├── 10-00-00.db
│   │   └── 14-30-00.db
│   └── 2024-01-16/
├── sensors/
│   ├── water_quality/
│   │   └── 2024-01-15.csv
│   ├── images/
│   │   └── 2024-01-15/
│   └── videos/
│       └── 2024-01-15/
├── logs/
│   ├── system.log
│   └── mission.log
└── missions/
    ├── completed/
    └── reports/

Data Query

Using CLI

# Query recent telemetry data
jhusv data query telemetry --since "1 hour ago"

# Query specific sensor data
jhusv data query sensors \
  --type water_quality \
  --parameter ph \
  --start "2024-01-15 00:00" \
  --end "2024-01-15 23:59"

# Export as CSV
jhusv data export \
  --type sensors \
  --format csv \
  --output data.csv

Using Python API

from jhusv import DataManager
from datetime import datetime, timedelta

# Create data manager
dm = DataManager()

# Query telemetry data
end_time = datetime.now()
start_time = end_time - timedelta(hours=1)

telemetry = dm.query_telemetry(
    device_id="USV-2024-001",
    start_time=start_time,
    end_time=end_time
)

# Query sensor data
water_data = dm.query_sensors(
    device_id="USV-2024-001",
    sensor_type="water_quality",
    parameters=["ph", "dissolved_oxygen"],
    start_time=start_time,
    end_time=end_time
)

# Convert to Pandas DataFrame
import pandas as pd
df = pd.DataFrame(water_data)
print(df.describe())

Data Analysis

Basic Statistics

from jhusv.analysis import Statistics

stats = Statistics()

# Calculate statistics
result = stats.calculate(
    data=water_data,
    parameters=["ph", "dissolved_oxygen", "temperature"]
)

print(result)

Data Visualization

from jhusv.visualization import Plotter

plotter = Plotter()

# Time series plot
plotter.time_series(
    data=water_data,
    parameters=["ph", "dissolved_oxygen"],
    title="Water Quality Over Time",
    output="water_quality.png"
)

# Track map
plotter.track_map(
    telemetry=telemetry,
    waypoints=waypoints,
    title="Mission Track",
    output="track_map.html"
)

Data Export

Export Formats

Supported formats:

• CSV
• JSON
• Excel
• GeoJSON
• KML (for GPS tracks)
• HDF5 (for large datasets)

Export Example

# Export as CSV
jhusv data export \
  --type sensors \
  --format csv \
  --output water_data.csv

# Export as GeoJSON (with geographic info)
jhusv data export \
  --type telemetry \
  --format geojson \
  --output track.geojson

Cloud Sync

Configure Cloud Storage

cloud_sync:
  enabled: true
  provider: "aws"          # aws, aliyun, tencent

  aws:
    region: "ap-southeast-1"
    bucket: "jhusv-data"
    credentials:
      access_key: "${AWS_ACCESS_KEY}"
      secret_key: "${AWS_SECRET_KEY}"

  sync:
    mode: "auto"           # auto, manual
    interval: 3600         # seconds
    compress: true
    encrypt: true

Manual Sync

# Upload data
jhusv cloud upload --path ~/jhusv/data --recursive

# Download data
jhusv cloud download --path missions/mission_001 --output ./local

# Sync status
jhusv cloud status

Data Cleanup

Auto Cleanup Policy

data_retention:
  enabled: true

  rules:
    - type: "telemetry"
      keep_days: 90

    - type: "sensor_readings"
      keep_days: 180

    - type: "videos"
      keep_days: 30
      compress_after_days: 7

    - type: "logs"
      keep_days: 30

  schedule: "0 2 * * *"    # 2 AM daily

Manual Cleanup

# Clean old data
jhusv data cleanup --older-than 90d

# Clean specific type
jhusv data cleanup --type videos --older-than 30d

# Compress old data
jhusv data compress --older-than 7d

Best Practices

Data Collection

✅ Recommendations:

• Set reasonable sampling rates (avoid excessive)
• Regularly calibrate sensors
• Monitor storage space
• Enable data compression

Data Storage

✅ Recommendations:

• Regularly backup important data
• Use cloud sync
• Implement retention policies
• Encrypt sensitive data

Data Analysis

✅ Recommendations:

• Process collected data promptly
• Validate data quality
• Generate regular reports
• Establish data archives

Next Steps

• 🔌 Explore Data API
• 📊 Learn about advanced analysis
• 🛠️ Check troubleshooting