AR in Mobile: Building with ARCore and ARKit

The AR Revolution in Mobile Apps

Augmented reality has moved far beyond novelty filters and game demos. In 2026, AR is a core feature in retail, education, healthcare, industrial maintenance, and navigation apps. Users expect immersive experiences that blend digital content with the physical world — and the tools to build those experiences are more mature than ever.

Google’s ARCore and Apple’s ARKit are the two dominant SDKs powering mobile AR across Android and iOS respectively. Whether you’re placing furniture in a living room, overlaying navigation arrows on a sidewalk, or inspecting a 3D engine model, these platforms handle the heavy lifting of environment understanding, motion tracking, and rendering.

This guide takes a practical, senior-developer approach to building real AR experiences. We’ll cover the core capabilities of both platforms, walk through code examples, and discuss the performance considerations that separate production-ready AR apps from weekend prototypes.

---

Prerequisites

Before diving into AR development, you should have:

• Flutter 3.x or native development experience (Swift/Kotlin)
• A physical Android device (ARCore requires Android 8.0+, Google Play Services)
• A physical iOS device with an A9 chip or later (ARKit requires iOS 11+)
• Android Studio or Xcode installed
• Basic understanding of 3D coordinate systems and transforms

Note: AR emulation works in Android Studio’s AR emulator (limited), but iOS AR testing requires a real device. Always test on physical hardware for accurate results.

---

ARCore vs ARKit: What You’re Working With

Feature	ARCore (Google)	ARKit (Apple)
Platform	Android (8.0+)	iOS (11+)
Language	Java/Kotlin + Sceneform/OpenGL	Swift + RealityKit/SceneKit
Flutter Plugin	ar_flutter_plugin, google_arcore	arkit_plugin, ar_flutter_plugin
Plane Detection	Horizontal + Vertical	Horizontal + Vertical
Image Tracking	Yes	Yes (World Tracking)
Object Detection	Cloud Anchors, Geospatial API	Object Detection (iOS 14+)
Light Estimation	Ambient intensity, color correction	Ambient intensity, color temperature
Depth API	Yes (native)	LiDAR-based (Pro models)

Choosing a Cross-Platform Strategy

For Flutter projects, ar_flutter_plugin (by CariusLabs) wraps both ARCore and ARKit behind a unified API. If you need platform-specific features like ARKit’s RealityKit or ARCore’s Geospatial API, you’ll drop into platform channels.

For native projects, each platform has its own idiomatic approach — ARKit leans on RealityKit and SwiftUI, while ARCore integrates with Sceneform or raw OpenGL ES.

---

Plane Detection: Understanding the Physical World

Plane detection is the foundation of most AR experiences. It identifies horizontal surfaces (floors, tables) and vertical surfaces (walls, doors) so you can anchor virtual content to real-world geometry.

Flutter Implementation

import 'package:ar_flutter_plugin/ar_flutter_plugin.dart';
import 'package:ar_flutter_plugin/datatypes/config_planar_detection.dart';

class ARPlaneScreen extends StatefulWidget {
  @override
  _ARPlaneScreenState createState() => _ARPlaneScreenState();
}

class _ARPlaneScreenState extends State<ARPlaneScreen> {
  ARSessionManager? sessionManager;
  ARObjectManager? objectManager;

  @override
  void initState() {
    super.initState();
    initAR();
  }

  void initAR() {
    sessionManager = ARSessionManager("YOUR_ANDROID_APP_ID", "YOUR_IOS_BUNDLE_ID", this, SessionType.world);
    objectManager = ARObjectManager(sessionManager!);
  }

  void onARViewCreated(ARSessionManager arSessionManager, ARObjectManager arObjectManager) {
    sessionManager = arSessionManager;
    objectManager = arObjectManager;
    sessionManager!.initialize(onPlaneFound: onPlaneFound);
  }

  void onPlaneFound(List<ARPlaneAnchor> planes) {
    if (planes.isNotEmpty) {
      final plane = planes.first;
      addModelToPlane(plane);
    }
  }

  void addModelToPlane(ARPlaneAnchor plane) {
    objectManager!.addTransformableNode(
      plane: plane,
      modelPath: "assets/models/chair.glb",
      scale: Vector3(0.3, 0.3, 0.3),
    );
  }
}

Native ARKit (Swift)

import RealityKit
import ARKit

class ViewController: UIViewController, ARSessionDelegate {
    @IBOutlet var arView: ARView!

    override func viewDidLoad() {
        super.viewDidLoad()
        arView.session.delegate = self
        startPlaneDetection()
    }

    func startPlaneDetection() {
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal, .vertical]
        arView.session.run(config, options: [.resetTracking, .removeExistingAnchors])
    }

    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        guard let planeAnchor = anchors.first as? ARPlaneAnchor else { return }
        placeObject(on: planeAnchor)
    }

    func placeObject(on anchor: ARPlaneAnchor) {
        let mesh = MeshResource.generatePlane(width: anchor.extent.x, depth: anchor.extent.z)
        let material = SimpleMaterial(color: .blue.withAlphaComponent(0.3), isMetallic: false)
        let planeEntity = ModelEntity(mesh: mesh, materials: [material])
        planeEntity.generateCollisionShapes(recursive: true)
        arView.installGestures([.rotation, .scale], for: planeEntity)
        let anchorEntity = AnchorEntity(anchor: anchor)
        anchorEntity.addChild(planeEntity)
        arView.scene.addAnchor(anchorEntity)
    }
}

Native ARCore (Kotlin)

import com.google.ar.sceneform.SceneView
import com.google.ar.sceneform.ux.ArFragment
import com.google.ar.core.Plane

class ARActivity : AppCompatActivity() {
    private lateinit var arFragment: ArFragment

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_ar)
        arFragment = supportFragmentManager.findFragmentById(R.id.ar_fragment) as ArFragment

        arFragment.arSceneView.scene.addOnUpdateListener {
            val frame = arFragment.arSceneView.arFrame ?: return@addOnUpdateListener
            val planes = frame.getUpdatedTrackables(Plane::class.java)
            if (planes.isNotEmpty()) {
                val plane = planes.first()
                placeModelOnPlane(plane)
            }
        }
    }

    private fun placeModelOnPlane(plane: Plane) {
        val anchor = plane.createAnchor(plane.centerPose)
        ModelRenderable.builder()
            .setSource(this, Uri.parse("chair.sfb"))
            .build()
            .thenAccept { model ->
                val node = AnchorNode(anchor)
                node.setParent(arFragment.arSceneView.scene)
                val modelNode = TransformableNode(arFragment.transformationSystem)
                modelNode.setParent(node)
                modelNode.renderable = model
            }
    }
}

---

Image Tracking: Recognizing Real-World Markers

Image tracking detects predefined images in the camera feed and attaches virtual content to them. This is powerful for product visualization, museum guides, and interactive print media.

Flutter Implementation

void initImageTracking() {
  sessionManager!.initialize(
    onPlaneFound: onPlaneFound,
    onImageTracking: onImageTracked,
  );

  sessionManager!.addTrackingImage(
    imageAssetPath: "assets/images/target_product.jpg",
    physicalWidthInMeters: 0.2,
  );
}

void onImageTracked(ARImageAnchor anchor) {
  objectManager!.addTransformableNode(
    anchor: anchor,
    modelPath: "assets/models/product_model.glb",
    scale: Vector3(0.15, 0.15, 0.15),
  );
}

Native ARKit (Swift)

func startImageTracking() {
    guard let referenceImage = ARReferenceImage(
        named: "product_marker",
        physicalSize: CGSize(width: 0.2, height: 0.2)
    ) else { return }

    let config = ARWorldTrackingConfiguration()
    config.detectionImages = [referenceImage]
    config.maximumNumberOfTrackedImages = 4
    arView.session.run(config)
}

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    guard let imageAnchor = anchors.first as? ARImageAnchor else { return }
    let imageName = imageAnchor.referenceImage.name ?? "unknown"
    print("Detected image: \(imageName)")

    let overlayEntity = ModelEntity(
        mesh: MeshResource.generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .cyan, isMetallic: true)]
    )
    overlayEntity.transform = Transform(scale: SIMD3(repeating: 1),
                                         rotation: simd_quatf(),
                                         translation: SIMD3(0, 0.05, 0))

    let anchorEntity = AnchorEntity(anchor: imageAnchor)
    anchorEntity.addChild(overlayEntity)
    arView.scene.addAnchor(anchorEntity)
}

Native ARCore (Kotlin)

private fun startImageTracking() {
    val config = AugmentedImageDatabase(arFragment.arSceneView.session)
    val inputStream = assets.open("target_product.jpg")
    val bitmap = BitmapFactory.decodeStream(inputStream)
    config.addImage("product_marker", bitmap, 0.2f)

    val sessionConfig = com.google.ar.core.Config(arFragment.arSceneView.session)
    sessionConfig.augmentedImageDatabase = config
    arFragment.arSceneView.session.configure(sessionConfig)
}

---

3D Object Placement and Interaction

Placing and manipulating 3D models is where AR apps become truly engaging. Users should be able to tap to place, drag, rotate, and scale objects naturally.

Loading GLB/GLTF Models in Flutter

void loadAndPlaceModel(String modelPath, Vector3 position) {
  objectManager!.addLocalObject(
    modelPath: modelPath,
    position: position,
    scale: Vector3(0.5, 0.5, 0.5),
    onObjectPlaced: (ARNode node) {
      sessionManager!.addTapHandler((node, tapPosition) {
        animateObject(node);
      });
    },
  );
}

void animateObject(ARNode node) {
  node.playAnimation(
    name: "rotate",
    duration: Duration(milliseconds: 500),
    repeatCount: 1,
  );
}

Gesture Handling in RealityKit

func placeObject(at position: SIMD3<Float>) {
    let modelEntity = try! ModelEntity.loadModel(named: "furniture")
    modelEntity.generateCollisionShapes(recursive: true)
    modelEntity.position = position
    modelEntity.scale = SIMD3(repeating: 0.01)

    let anchor = AnchorEntity()
    anchor.position = position
    anchor.addChild(modelEntity)
    arView.scene.addAnchor(anchor)

    arView.installGestures([.rotation, .scale, .translation], for: modelEntity)

    UIView.animate(withDuration: 0.3) {
        modelEntity.scale = SIMD3(repeating: 1.0)
    }
}

---

Light Estimation: Making AR Look Real

Light estimation makes virtual objects blend seamlessly with the real environment by matching lighting conditions.

ARKit Light Estimation

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    guard let lightEstimate = frame.lightEstimate else { return }

    let ambientIntensity = Float(lightEstimate.ambientIntensity)
    let ambientColorTemperature = Float(lightEstimate.ambientColorTemperature)

    updateSceneLighting(intensity: ambientIntensity, temperature: ambientColorTemperature)
}

func updateSceneLighting(intensity: Float, temperature: Float) {
    let directionalLight = arView.scene.findEntity(named: "sunLight") as? DirectionalLight
    directionalLight?.light?.intensity = intensity

    let colorTemp = UIColor(white: CGFloat(temperature / 6500.0), alpha: 1.0)
    directionalLight?.light?.color = colorTemp
}

ARCore Light Estimation

arFragment.arSceneView.scene.addOnUpdateListener { frameTime ->
    val frame = arFragment.arSceneView.arFrame
    val lightEstimate = frame?.lightEstimate ?: return@addOnUpdateListener

    val intensity = lightEstimate.pixelIntensity
    val colorCorrection = lightEstimate.colorCorrection

    val directionalLight = DirectionalLightComponent(
        Intensity(intensity * 1000),
        temperatureToColor(lightEstimate.colorCorrection)
    )
}

---

Performance Considerations

AR is computationally expensive. Here’s how to keep your app running at 60fps:

1. Optimize 3D Models

• Use glTF/GLB format for the best compression
• Target under 100K polygons per model
• Use compressed textures (ASTC for mobile, max 1024x1024)
• Bake lighting into textures when possible

2. Manage AR Session Lifecycle

@override
void dispose() {
  sessionManager?.dispose();
  super.dispose();
}

@override
void didChangeAppLifecycleState(AppLifecycleState state) {
  if (state == AppLifecycleState.paused) {
    sessionManager?.pause();
  } else if (state == AppLifecycleState.resumed) {
    sessionManager?.resume();
  }
}

3. Object Pooling for Multiple Placements

class ObjectPool {
  final int maxSize;
  final List<ARNode> _pool = [];

  ObjectPool({this.maxSize = 10});

  ARNode acquire(String modelPath) {
    if (_pool.isNotEmpty) {
      return _pool.removeLast();
    }
    return ARNode(modelPath: modelPath);
  }

  void release(ARNode node) {
    if (_pool.length < maxSize) {
      node.reset();
      _pool.add(node);
    } else {
      node.remove();
    }
  }
}

4. Frame Rate Monitoring

func measurePerformance() {
    let startTime = CFAbsoluteTimeGetCurrent()
    arView.snapshot { image in
        let elapsed = CFAbsoluteTimeGetCurrent() - startTime
        print("Snapshot render time: \(elapsed * 1000)ms")
    }
}

5. Platform-Specific Tips

• Android: Use Sceneform’s ViewRenderable for UI overlays instead of separate UI layers. Disable plane detection once you’ve found your surface.
• iOS: Use LiDAR-equipped devices for depth-based occlusion. Pre-load RealityKit scenes during app launch, not during AR session.

---

Cloud Anchors and Shared Experiences

Both platforms support shared AR experiences where multiple users see the same virtual content.

ARCore Cloud Anchors

private fun hostCloudAnchor(anchor: Anchor) {
    anchor.cloudAnchorApi.host(7).addOnSuccessListener { cloudAnchorId ->
        shareAnchorIdWithPeers(cloudAnchorId)
    }
}

private fun resolveCloudAnchor(cloudAnchorId: String) {
    anchor.cloudAnchorApi.resolve(cloudAnchorId).addOnSuccessListener { anchor ->
        placeResolvedContent(anchor)
    }
}

RealityKit Shared Experience (iOS)

func startMultipeerSession() {
    arView.session.multipeerSessionDelegate = self
}

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    guard let data = try? NSKeyedArchiver.archivedData(withRootObject: frame.transform, requiringSecureCoding: true) else { return }
    try? arView.session.update(frame: frame)
}

---

Testing and Debugging AR Apps

Essential Debug Tools

• ARCore Depth API visualizer — see depth maps in real-time
• Xcode RealityKit debug options — show wireframes, collision shapes, and anchors
• Sceneform Inspector (Android) — view anchor planes and hit results
• Unity as a prototyping tool — rapid AR concept validation before native implementation

Common Pitfalls

1. Forgetting to handle AR session interruption — phone calls and app switches break tracking
2. Using high-poly models on low-end devices — always profile on minimum-spec hardware
3. Ignoring world origin drift — re-localize periodically for long sessions
4. Over-requesting permissions — explain why camera access is needed in your UI

---

Conclusion & Next Steps

Building AR experiences in mobile requires understanding both the platform capabilities and the practical constraints of mobile hardware. ARCore and ARKit provide robust foundations, but the difference between a good and great AR app lies in attention to detail — proper light estimation, optimized models, smooth gesture handling, and graceful session management.

Next steps to explore:

• LiDAR integration — depth mesh and people occlusion on supported devices
• ARCore Geospatial API — place content at real-world GPS coordinates
• Reality Composer Pro — build complex AR scenes without code
• WebXR — bring AR experiences to mobile browsers
• Hand tracking — gesture-based interactions without screen touch

Start with a single feature — plane detection with object placement — and iterate from there. The most successful AR apps do one thing extremely well rather than trying to showcase every capability.

---

This guide references Google’s ARCore developer documentation for the latest platform APIs and best practices.