K
Klivvr Blog

A/B Testing in Mobile Apps: A Practical Guide

Learn how to implement robust A/B testing in iOS apps using analytics SDKs, covering experiment design, variant assignment, and statistical analysis.

technical7 min readBy Klivvr Engineering
Share:

A/B testing is the bridge between product intuition and evidence-based decision making. In mobile apps, experimentation comes with unique challenges: you cannot update the experience instantly like on the web, users carry state across sessions, and network connectivity is unreliable. Building a robust experimentation framework on iOS requires careful design of variant assignment, event correlation, and statistical rigor.

This article covers how to implement A/B testing in iOS apps using KlivvrAnalyticsKit as the measurement layer, from experiment configuration to result analysis.

Experiment Configuration and Variant Assignment

The foundation of any A/B test is deterministic variant assignment. A user must always see the same variant for a given experiment, even across app restarts. We achieve this with a hash-based assignment algorithm.

struct Experiment: Codable {
    let id: String
    let name: String
    let variants: [Variant]
    let isActive: Bool
    let startDate: Date
    let endDate: Date?
    let targetAudience: AudienceFilter?
 
    struct Variant: Codable {
        let id: String
        let name: String
        let weight: Double // 0.0 to 1.0, all variants must sum to 1.0
        let config: [String: AnyCodable]
    }
 
    struct AudienceFilter: Codable {
        let minAppVersion: String?
        let maxAppVersion: String?
        let countries: [String]?
        let userSegments: [String]?
    }
}
 
final class ExperimentManager {
    private var experiments: [Experiment] = []
    private var assignments: [String: String] = [:] // experimentId -> variantId
    private let userId: String
 
    init(userId: String) {
        self.userId = userId
        loadCachedAssignments()
    }
 
    // Deterministic variant assignment using consistent hashing
    func getVariant(for experimentId: String) -> Experiment.Variant? {
        guard let experiment = experiments.first(where: { $0.id == experimentId }),
              experiment.isActive else {
            return nil
        }
 
        // Check cached assignment first
        if let cachedVariantId = assignments[experimentId],
           let variant = experiment.variants.first(where: { $0.id == cachedVariantId }) {
            return variant
        }
 
        // Deterministic assignment based on user ID and experiment ID
        let assignmentKey = "\(userId):\(experimentId)"
        let hash = assignmentKey.stableHash
        let normalizedHash = Double(hash % 10000) / 10000.0
 
        var cumulativeWeight = 0.0
        for variant in experiment.variants {
            cumulativeWeight += variant.weight
            if normalizedHash < cumulativeWeight {
                assignments[experimentId] = variant.id
                persistAssignments()
                trackExposure(experiment: experiment, variant: variant)
                return variant
            }
        }
 
        // Fallback to last variant
        let fallback = experiment.variants.last
        if let fallback {
            assignments[experimentId] = fallback.id
            persistAssignments()
        }
        return fallback
    }
 
    private func trackExposure(experiment: Experiment, variant: Experiment.Variant) {
        KlivvrAnalytics.shared.track("experiment_exposure", properties: [
            "experiment_id": experiment.id,
            "experiment_name": experiment.name,
            "variant_id": variant.id,
            "variant_name": variant.name
        ])
    }
}
 
// Stable hash extension for deterministic assignment
extension String {
    var stableHash: UInt64 {
        var hasher = FNV1aHasher()
        for byte in self.utf8 {
            hasher.combine(byte)
        }
        return hasher.finalValue
    }
}
 
struct FNV1aHasher {
    private var hash: UInt64 = 14695981039346656037
 
    mutating func combine(_ byte: UInt8) {
        hash ^= UInt64(byte)
        hash &*= 1099511628211
    }
 
    var finalValue: UInt64 { hash }
}

The FNV-1a hash ensures consistent assignment without needing server calls. The same user-experiment combination always produces the same hash, which maps to the same variant.

Feature Flags and Experiment-Driven UI

Once variant assignment is in place, you need a clean API for the UI layer to consume experiment configurations. Property wrappers make this seamless in SwiftUI.

@propertyWrapper
struct ExperimentVariant<Value> {
    let experimentId: String
    let configKey: String
    let defaultValue: Value
 
    var wrappedValue: Value {
        guard let variant = ExperimentManager.shared.getVariant(for: experimentId),
              let value = variant.config[configKey]?.value as? Value else {
            return defaultValue
        }
        return value
    }
}
 
// Usage in a view model
class OnboardingViewModel: ObservableObject {
    @ExperimentVariant(
        experimentId: "onboarding_flow_v2",
        configKey: "show_social_proof",
        defaultValue: false
    )
    var showSocialProof: Bool
 
    @ExperimentVariant(
        experimentId: "onboarding_flow_v2",
        configKey: "cta_text",
        defaultValue: "Get Started"
    )
    var ctaText: String
 
    @ExperimentVariant(
        experimentId: "pricing_page_test",
        configKey: "layout",
        defaultValue: "vertical"
    )
    var pricingLayout: String
}
 
// SwiftUI view consuming experiment variants
struct OnboardingView: View {
    @StateObject private var viewModel = OnboardingViewModel()
 
    var body: some View {
        VStack {
            if viewModel.showSocialProof {
                SocialProofBanner()
            }
 
            // Content varies by experiment
            Text("Welcome to the app")
 
            Button(viewModel.ctaText) {
                // Track conversion event
                KlivvrAnalytics.shared.track("onboarding_cta_tapped", properties: [
                    "cta_text": viewModel.ctaText
                ])
            }
        }
    }
}

Tracking Experiment Metrics

The analytics layer must correlate every user action with their active experiment variants. This is where enrichers become critical -- every event should carry experiment context.

// Experiment enricher attaches active variants to all events
final class ExperimentEnricher: EventEnricher {
    private let experimentManager: ExperimentManager
 
    init(experimentManager: ExperimentManager) {
        self.experimentManager = experimentManager
    }
 
    func enrich(_ event: inout AnalyticsEvent) {
        let activeAssignments = experimentManager.activeAssignments
        guard !activeAssignments.isEmpty else { return }
 
        // Attach all active experiment assignments
        var experimentContext: [[String: String]] = []
        for (experimentId, variantId) in activeAssignments {
            experimentContext.append([
                "experiment_id": experimentId,
                "variant_id": variantId
            ])
        }
 
        event.properties["experiments"] = experimentContext
    }
}
 
// Goal tracking for experiments
struct ExperimentGoal {
    let experimentId: String
    let goalName: String
    let eventName: String
    let eventFilter: ((AnalyticsEvent) -> Bool)?
}
 
final class GoalTracker {
    private var goals: [ExperimentGoal] = []
    private let analytics: KlivvrAnalytics
 
    func registerGoal(_ goal: ExperimentGoal) {
        goals.append(goal)
    }
 
    func evaluateEvent(_ event: AnalyticsEvent) {
        for goal in goals where goal.eventName == event.name {
            if let filter = goal.eventFilter, !filter(event) {
                continue
            }
 
            analytics.track("experiment_goal_reached", properties: [
                "experiment_id": goal.experimentId,
                "goal_name": goal.goalName,
                "original_event": event.name
            ])
        }
    }
}

Remote Experiment Configuration

Experiments should be configurable without app updates. A remote configuration system allows you to start, stop, and modify experiments in real time.

final class RemoteExperimentProvider {
    private let configURL: URL
    private let session: URLSession
    private let cache: ExperimentCache
 
    func fetchExperiments() async throws -> [Experiment] {
        var request = URLRequest(url: configURL)
        request.cachePolicy = .reloadIgnoringLocalCacheData
 
        // Include last-modified header for conditional requests
        if let lastModified = cache.lastModified {
            request.setValue(lastModified, forHTTPHeaderField: "If-Modified-Since")
        }
 
        let (data, response) = try await session.data(for: request)
 
        guard let httpResponse = response as? HTTPURLResponse else {
            throw ExperimentError.invalidResponse
        }
 
        switch httpResponse.statusCode {
        case 200:
            let experiments = try JSONDecoder().decode([Experiment].self, from: data)
            cache.store(experiments: experiments)
            cache.lastModified = httpResponse.value(forHTTPHeaderField: "Last-Modified")
            return experiments
        case 304:
            // Not modified, use cache
            return cache.loadExperiments() ?? []
        default:
            // Fall back to cache on error
            return cache.loadExperiments() ?? []
        }
    }
 
    // Fetch on app launch with fallback to cache
    func initialize() async {
        let experiments: [Experiment]
        do {
            experiments = try await fetchExperiments()
        } catch {
            experiments = cache.loadExperiments() ?? []
            Logger.warning("Failed to fetch experiments, using cache: \(error)")
        }
 
        ExperimentManager.shared.updateExperiments(experiments)
    }
}

Statistical Significance and Analysis

Running an experiment is meaningless without proper statistical analysis. While heavy computation happens server-side, the SDK can support basic analysis for debugging purposes.

// Basic experiment statistics (for debugging/preview)
struct ExperimentResults {
    let experimentId: String
    let variants: [VariantResult]
 
    struct VariantResult {
        let variantId: String
        let variantName: String
        let sampleSize: Int
        let conversionCount: Int
        let conversionRate: Double
 
        var standardError: Double {
            guard sampleSize > 0 else { return 0 }
            let p = conversionRate
            return sqrt(p * (1 - p) / Double(sampleSize))
        }
 
        var confidenceInterval: (lower: Double, upper: Double) {
            let z = 1.96 // 95% confidence
            return (
                lower: max(0, conversionRate - z * standardError),
                upper: min(1, conversionRate + z * standardError)
            )
        }
    }
 
    // Check if the difference between two variants is statistically significant
    func isSignificant(controlIndex: Int = 0, treatmentIndex: Int = 1) -> Bool {
        guard variants.count > max(controlIndex, treatmentIndex) else { return false }
 
        let control = variants[controlIndex]
        let treatment = variants[treatmentIndex]
 
        let pooledSE = sqrt(
            control.standardError * control.standardError +
            treatment.standardError * treatment.standardError
        )
 
        guard pooledSE > 0 else { return false }
 
        let zScore = abs(treatment.conversionRate - control.conversionRate) / pooledSE
        return zScore > 1.96 // 95% confidence level
    }
}

Practical Tips

Never change variant assignments mid-experiment -- it invalidates your results. Always track an "exposure" event when a user first sees a variant, not when they are assigned. This distinction matters because assignment might happen at app launch but exposure only when the user navigates to the relevant screen. Run experiments for a full week minimum to account for day-of-week effects. Use guardrail metrics (crash rate, session length, retention) alongside your primary metric to detect unintended harm. Clean up old experiments promptly to reduce code complexity and runtime overhead.

Conclusion

A/B testing on mobile requires deterministic variant assignment, clean UI integration, comprehensive metric tracking, and statistical rigor. With KlivvrAnalyticsKit providing the measurement backbone, you can build an experimentation framework that delivers reliable results and drives data-informed product decisions. The patterns outlined here -- hash-based assignment, property wrapper integration, experiment enrichers, and remote configuration -- form a production-ready experimentation system for iOS apps.

Related Articles

business

Ensuring Data Quality in Mobile Analytics

Establish data quality practices for mobile analytics, including validation, monitoring, testing, and governance to maintain trustworthy analytics data.

7 min read