K
Klivvr Blog

From Device to Dashboard: The Analytics Data Pipeline

Trace the full journey of analytics data from event creation on an iOS device through ingestion, processing, storage, and visualization on dashboards.

technical7 min readBy Klivvr Engineering
Share:

When a user taps a button in your iOS app, an analytics event is born. But the journey from that tap to a chart on your dashboard involves a sophisticated pipeline spanning the device, the network, ingestion servers, stream processors, data warehouses, and visualization tools. Understanding this end-to-end pipeline is essential for building reliable analytics and diagnosing data issues when they arise.

This article traces the complete lifecycle of an analytics event, from creation on the device to rendering on a dashboard, and explains the architectural decisions at each stage.

Stage 1: Event Creation and On-Device Processing

The pipeline begins the moment your app calls track(). On the device, the event passes through several processing stages before it ever touches the network.

// Event creation with automatic metadata
struct AnalyticsEvent: Codable {
    let id: String
    let name: String
    var properties: [String: AnyCodable]
    let timestamp: Date
    let deviceTimestamp: Date
    let sequenceNumber: Int
 
    init(name: String, properties: [String: Any]) {
        self.id = UUID().uuidString
        self.name = name
        self.properties = properties.mapValues { AnyCodable($0) }
        self.timestamp = Date()
        self.deviceTimestamp = Date()
        self.sequenceNumber = SequenceGenerator.shared.next()
    }
}
 
// Sequence generator ensures event ordering even with identical timestamps
final class SequenceGenerator {
    static let shared = SequenceGenerator()
    private var counter: Int = 0
    private let lock = NSLock()
 
    func next() -> Int {
        lock.lock()
        defer { lock.unlock() }
        counter += 1
        return counter
    }
}

After creation, the event flows through the enrichment and validation pipeline described in our architecture article. The enriched event is then serialized and written to the persistent queue. This entire process happens synchronously on a background queue, typically completing in under a millisecond.

Stage 2: Batching and Network Transmission

Events accumulate in the persistent queue until a flush trigger fires. The SDK batches events together and compresses them for transmission.

// Batch creation and compression
struct EventBatch: Codable {
    let batchId: String
    let events: [AnalyticsEvent]
    let sentAt: Date
    let sdkVersion: String
 
    var compressedPayload: Data? {
        guard let jsonData = try? JSONEncoder().encode(self) else { return nil }
        return try? (jsonData as NSData).compressed(using: .zlib) as Data
    }
}
 
// Network transmission with background task support
final class EventTransmitter {
    private let endpoint: URL
    private let apiKey: String
    private let session: URLSession
 
    func transmit(batch: EventBatch) async throws -> TransmissionResult {
        guard let payload = batch.compressedPayload else {
            throw TransmissionError.compressionFailed
        }
 
        var request = URLRequest(url: endpoint)
        request.httpMethod = "POST"
        request.httpBody = payload
        request.setValue("application/json", forHTTPHeaderField: "Content-Type")
        request.setValue("gzip", forHTTPHeaderField: "Content-Encoding")
        request.setValue(apiKey, forHTTPHeaderField: "X-API-Key")
        request.setValue(batch.batchId, forHTTPHeaderField: "X-Batch-ID")
 
        let (data, response) = try await session.data(for: request)
 
        guard let httpResponse = response as? HTTPURLResponse else {
            throw TransmissionError.invalidResponse
        }
 
        switch httpResponse.statusCode {
        case 200...299:
            return .success(eventsDelivered: batch.events.count)
        case 429:
            let retryAfter = httpResponse.value(forHTTPHeaderField: "Retry-After")
            throw TransmissionError.rateLimited(retryAfter: retryAfter)
        case 400...499:
            // Client errors: don't retry, discard batch
            return .dropped(reason: "Client error: \(httpResponse.statusCode)")
        case 500...599:
            throw TransmissionError.serverError(statusCode: httpResponse.statusCode)
        default:
            throw TransmissionError.unexpectedStatus(httpResponse.statusCode)
        }
    }
}

Compression typically reduces payload size by 70-80%, significantly reducing bandwidth consumption. The batch ID enables idempotent processing on the server side, preventing duplicate events when retries occur.

Stage 3: Ingestion and Validation

On the server side, the ingestion layer receives batches, decompresses them, validates the schema, and writes events to a message queue for downstream processing.

// Client-side schema validation before sending
struct EventSchemaValidator: EventValidator {
    private let schemas: [String: EventSchema]
 
    struct EventSchema {
        let requiredProperties: Set<String>
        let propertyTypes: [String: PropertyType]
    }
 
    enum PropertyType {
        case string
        case number
        case boolean
        case array
    }
 
    func validate(_ event: AnalyticsEvent) throws {
        // Check event name is not empty
        guard !event.name.isEmpty else {
            throw ValidationError.emptyEventName
        }
 
        // Check event name format (snake_case)
        guard event.name.range(of: "^[a-z][a-z0-9_]*$", options: .regularExpression) != nil else {
            throw ValidationError.invalidEventNameFormat(event.name)
        }
 
        // Validate against schema if one exists
        if let schema = schemas[event.name] {
            let propertyKeys = Set(event.properties.keys)
            let missingKeys = schema.requiredProperties.subtracting(propertyKeys)
            guard missingKeys.isEmpty else {
                throw ValidationError.missingRequiredProperties(Array(missingKeys))
            }
        }
 
        // Check property count limit
        guard event.properties.count <= 100 else {
            throw ValidationError.tooManyProperties(count: event.properties.count)
        }
    }
}

Server-side ingestion typically uses a technology like Apache Kafka or Amazon Kinesis as the message queue. Events written to the queue are durable and can be consumed by multiple downstream processors independently.

Stage 4: Stream Processing and Transformation

Raw events need transformation before they are useful for analysis. Stream processors enrich events with server-side data, compute derived metrics, and route events to appropriate storage systems.

// Client-side event transformation example
// (mirrors what happens server-side for consistency)
struct EventTransformer {
    func transform(_ event: AnalyticsEvent) -> TransformedEvent {
        var transformed = TransformedEvent(
            eventId: event.id,
            eventName: event.name,
            timestamp: event.timestamp,
            receivedAt: Date()
        )
 
        // Compute derived properties
        if event.name == "purchase_completed" {
            if let price = event.properties["price"]?.value as? Double,
               let quantity = event.properties["quantity"]?.value as? Int {
                transformed.derivedProperties["total_revenue"] = price * Double(quantity)
            }
        }
 
        // Categorize the event
        transformed.eventCategory = categorize(eventName: event.name)
 
        // Compute time-based properties
        let calendar = Calendar.current
        transformed.derivedProperties["hour_of_day"] = calendar.component(.hour, from: event.timestamp)
        transformed.derivedProperties["day_of_week"] = calendar.component(.weekday, from: event.timestamp)
 
        return transformed
    }
 
    private func categorize(eventName: String) -> String {
        if eventName.hasPrefix("screen_") { return "navigation" }
        if eventName.hasPrefix("purchase_") { return "revenue" }
        if eventName.hasPrefix("error_") { return "errors" }
        if eventName.hasPrefix("button_") { return "engagement" }
        return "other"
    }
}

Stream processing also handles real-time alerting. If error event rates spike or conversion rates drop below a threshold, the system can trigger notifications to the engineering or product team without waiting for batch processing.

Stage 5: Storage and Querying

Processed events land in a data warehouse optimized for analytical queries. The storage layer is typically columnar (like BigQuery, Snowflake, or ClickHouse) because analytics queries scan many rows but only a few columns.

// Client-side query builder for analytics APIs
struct AnalyticsQuery {
    let eventName: String
    var filters: [QueryFilter] = []
    var groupBy: [String] = []
    var timeRange: TimeRange
    var aggregation: Aggregation = .count
 
    enum Aggregation {
        case count
        case sum(property: String)
        case average(property: String)
        case uniqueCount(property: String)
    }
 
    struct QueryFilter {
        let property: String
        let `operator`: FilterOperator
        let value: Any
 
        enum FilterOperator: String {
            case equals = "eq"
            case notEquals = "neq"
            case greaterThan = "gt"
            case lessThan = "lt"
            case contains = "contains"
        }
    }
 
    struct TimeRange {
        let start: Date
        let end: Date
 
        static func lastDays(_ days: Int) -> TimeRange {
            TimeRange(
                start: Calendar.current.date(byAdding: .day, value: -days, to: Date())!,
                end: Date()
            )
        }
    }
 
    func toJSON() -> [String: Any] {
        var json: [String: Any] = [
            "event": eventName,
            "time_range": [
                "start": ISO8601DateFormatter().string(from: timeRange.start),
                "end": ISO8601DateFormatter().string(from: timeRange.end)
            ]
        ]
        if !groupBy.isEmpty {
            json["group_by"] = groupBy
        }
        return json
    }
}

Data Quality Monitoring

At every stage of the pipeline, things can go wrong. Events can be malformed, batches can be dropped, and processing can introduce errors. Monitoring data quality is as important as the pipeline itself.

// Pipeline health monitoring from the SDK side
final class PipelineHealthMonitor {
    private var metrics = PipelineMetrics()
 
    struct PipelineMetrics {
        var eventsCreated: Int = 0
        var eventsEnqueued: Int = 0
        var eventsTransmitted: Int = 0
        var eventsDropped: Int = 0
        var transmissionErrors: Int = 0
        var validationErrors: Int = 0
        var averageLatency: TimeInterval = 0
    }
 
    func recordEventCreated() { metrics.eventsCreated += 1 }
    func recordEventEnqueued() { metrics.eventsEnqueued += 1 }
    func recordEventTransmitted(count: Int) { metrics.eventsTransmitted += count }
    func recordEventDropped(count: Int) { metrics.eventsDropped += count }
    func recordTransmissionError() { metrics.transmissionErrors += 1 }
 
    func reportHealth() -> PipelineMetrics {
        metrics
    }
 
    var deliveryRate: Double {
        guard metrics.eventsCreated > 0 else { return 0 }
        return Double(metrics.eventsTransmitted) / Double(metrics.eventsCreated)
    }
}

Practical Tips

Always include a sent_at timestamp alongside the device timestamp so the server can compute clock skew. Use batch IDs for idempotent processing to prevent duplicates on retries. Compress payloads before transmission to reduce data usage. Monitor your delivery rate -- anything below 99% warrants investigation. Build a debug mode that logs the full event lifecycle locally so developers can trace events from creation to dashboard.

Conclusion

The analytics data pipeline is a complex system with many failure points. By understanding each stage -- from on-device event creation through network transmission, server-side ingestion, stream processing, and storage -- you can build a pipeline that reliably delivers accurate data to your dashboards. KlivvrAnalyticsKit handles the critical first stages of this pipeline, ensuring events are created correctly, persisted reliably, and transmitted efficiently, giving the rest of the pipeline clean data to work with.

Related Articles

business

Ensuring Data Quality in Mobile Analytics

Establish data quality practices for mobile analytics, including validation, monitoring, testing, and governance to maintain trustworthy analytics data.

7 min read