Matthieu Scharffe

Engineer

🇫🇷 France | matthieu.scharffe@gmail.com

Profile Photo

TempleRun-RL

Teaching a reinforcement-learning agent to play the real Temple Run — on a physical phone

DoneSolo project · code and policy released
PyTorchPPO (Stable-Baselines3)FridaIL2CPP reversingAndroid
Pixel-art of a runner fleeing a snake through ancient ruins

Learning RL through a childhood favourite

I wanted to learn reinforcement learning by teaching an agent to play Temple Run. The hard part was getting enough practice at the speeds where it kept failing. I reverse-engineered the game, built a fast simulator, and trained a PPO policy there before testing it on my Android phone.

16,782 m
Best device run · separate recording session
4,736 m
Median distance · 14 device runs
256
Simulated environments training in parallel

The median and best come from different evaluation sessions. The 14-run benchmark had a 4,000-frame limit; one run was still alive beyond 15 km when it was cut off.

First, I tried building the game

My first approach was to build an endless runner with Claude Code, so I could train without depending on a phone. I spent too much time on graphics and still didn't have the mechanics I needed. Matching how the real game behaves turned out to be much harder than making something that looked like it.

The early Panda3D prototype: useful experiments, but too much work before training could begin.

Read the state, then simulate the mechanics

I switched to reverse-engineering the Android APK. A Frida gadget reads the game's state from memory and calls its native movement methods. ADB forwards the connection to my computer. The agent receives compact observations about speed, position, obstacles and upcoming turns, rather than screenshots.

That let me control the game, but one phone still runs at real-time speed. The ARM64 game also made instrumentation on my x86_64 computer awkward. Scaling PPO training by adding phones wasn't practical.

Instead, I reproduced the relevant game logic in a headless Python simulator, then moved it to C++ for faster training. It models the observations, jump timing, turns, narrow paths and death rules. Each failure on the phone gives me a way to check where that approximation is wrong.

The phone is the test that matters. I classify device deaths by obstacle, speed and action. The recurring failures guide the next training stage; disagreements with the simulator guide its corrections.

Practice the situations that end runs

Temple Run accelerates as you survive: speed rises from 100 to 210 over roughly 4 km. Starting every episode at zero means a weak policy spends most of its time practising the slow opening, then dies before it can learn the fast sections.

Half of the episodes start at a random distance between 0 and 5,000 m, with the corresponding speed and late-game obstacles. That gives the agent practice at high speed without first having to survive several kilometres.

The 256 environments also mix ordinary tracks with targeted drills in every batch. Roughly half are normal play; the others make difficult situations appear much more often. Training one skill in isolation helped initially, but could make the policy forget another—especially turns.

The environment mix
Episode profiles, sampled at reset from sim_world.json. Shares are rounded from weights totalling 110.
ShareTrackPractice
45.5%Normal gameAll skills together
12.7%Obstacle-dense90% obstacle probability; repeated reactions
10.9%CrevassesLevel changes every 9 cells
10.9%Wide gaps90% wide-gap probability; long-jump timing
5.5%Stacked obstaclesSequences such as narrow → jump → narrow
5.5%Narrow pathsNarrow strips available from the start
9.1%Hardcore mixDense obstacles, combos, gaps and level changes
View the simulator configuration

A model built through repeated device tests

The released policy is the result of a long sequence of refinements. I started each stage from the previous best checkpoint, trained for 10 million steps, and tested it on the phone. The next stage targeted the weakness I could actually see.

Some fixes changed training: more chained jumps, a penalty for dangerous landing spots, or practice turning before a corner. Others changed the interface: buffering turns through the native swipe queue and delivering fresher observations. Timing the action correctly mattered as much as choosing it.

Training history and policy details
  1. Turns first. Learn corners without obstacles, then introduce the full track.
  2. Narrow paths. Focused practice, followed by a return to mixed play.
  3. Crevasses. Model the ledge as a two-cell step with a speed-dependent jump arc.
  4. Stacked obstacles. Practise narrow → jump → narrow transitions.
  5. More capacity. Widen the network from 256 to 512 units per layer while retaining the learned policy.
  6. Post-turn hazards and repeated gaps. Learn to land, then jump again.
  7. Early turns. Encourage exploration of turning before the corner to account for device delay.
  8. Landing hazards. Add a predicted-landing cue and penalise jumps toward dangerous cells.
  9. Chained jumps at speed. The final “chain coast” stage produced the released policy.

Stages use a small decreasing learning rate, fading exploration and the best checkpoint. The released Stable-Baselines3 PPO policy has two 512-unit hidden layers and a 67-value observation. Recent actions provide context for avoiding repeated commands.

Model and evaluation notes

Reward distance, shape the timing

The base reward measures forward distance and penalises death. Extra terms encourage staying on narrow strips, reacting to obstacles and turning early. They also discourage unnecessary actions and jumps that clear one obstacle only to land on the next.

The reward equation and definitions
rt  =  0.1 max⁡ ⁣(0,  dt−dt−1)⏟distance (m)  −  1[death]⏟death  +  0.18  ct  1narrow⏟stay on the strip  +  ρ(at,s) r_t \;=\; \underbrace{0.1\,\max\!\big(0,\; d_t - d_{t-1}\big)}_{\text{distance (m)}} \;-\; \underbrace{\mathbf{1}[\text{death}]}_{\text{death}} \;+\; \underbrace{0.18\; c_t\;\mathbf{1}_{\text{narrow}}}_{\text{stay on the strip}} \;+\; \rho(a_t, s)
ρ(a,s)  =  {−0.02+0.1 1obs↑−0.30 1land−8 1step−0.08 1airif a=jump−0.02+0.1 1obs↓−0.08 1slideif a=slide−0.02+0.1 αt 1junc+0.5 1early−0.1 1spamif a∈{turn left,turn right}−0.02 (1−1narrow)−0.05 1straightif a∈{tilt left,tilt right}0if a=no-op \rho(a, s) \;=\; \begin{cases} -0.02 + 0.1\,\mathbf{1}_{\text{obs}}^{\uparrow} - 0.30\,\mathbf{1}_{\text{land}} - 8\,\mathbf{1}_{\text{step}} - 0.08\,\mathbf{1}_{\text{air}} & \text{if } a = \text{jump} \\[6pt] -0.02 + 0.1\,\mathbf{1}_{\text{obs}}^{\downarrow} - 0.08\,\mathbf{1}_{\text{slide}} & \text{if } a = \text{slide} \\[6pt] -0.02 + 0.1\,\alpha_t\,\mathbf{1}_{\text{junc}} + 0.5\,\mathbf{1}_{\text{early}} - 0.1\,\mathbf{1}_{\text{spam}} & \text{if } a \in \{\text{turn left}, \text{turn right}\} \\[6pt] -0.02\,(1 - \mathbf{1}_{\text{narrow}}) - 0.05\,\mathbf{1}_{\text{straight}} & \text{if } a \in \{\text{tilt left}, \text{tilt right}\} \\[6pt] 0 & \text{if } a = \text{no-op} \end{cases}

At each decision step (10 per second), d is distance in metres and each indicator is 1 when its condition holds, otherwise 0. The action term is ρ.

  • Obstacle cues: ↑ means a jump obstacle in the current or next cell; ↓ means a slide obstacle. A safe step down does not count as a jump obstacle.
  • Jump penalties: land flags a hazardous landing cell; step flags an unnecessary jump over a safe step down; air flags a jump while already airborne. The slide flag means already sliding.
  • Turn cues: junc means a valid turn on the junction; early means a valid turn one cell before it; spam means the previous action was also a turn.
  • Path cues: narrow means on or about to enter a narrow strip; straight means no narrow path ahead.
p∈[0,1),v∈[100,210],ℓ∈[−7,7]p\in[0,1),\quad v\in[100,210],\quad \ell\in[-7,7]
σ=0.0143v−0.11,k=⌊p+σ⌋\sigma=0.0143v-0.11,\qquad k=\lfloor p+\sigma\rfloor
ct=clip⁡(ℓ sign⁡(ℓ⋆)/3,0,1)c_t=\operatorname{clip}(\ell\,\operatorname{sign}(\ell^\star)/3,0,1)
αt=max⁡(0,1−∣p+v/600−0.44∣/0.16)\alpha_t=\max(0,1-|p+v/600-0.44|/0.16)

p is progress through the current cell, v is game speed, and ℓ is lateral position. A jump spans σ cells; k identifies its landing cell (0–3). ℓ★ marks the side of the safe strip. Strip closeness c is set to 1 on its last cell. Turn aim α peaks when the predicted position after turning is centred in the cell.

Policy snapshots

Three stages of learning, running on the real phone.

Early policy
Learning the basics
Intermediate policy
~1,960 m device run
Released policy
16,782 m best run · 30-second excerpt

Try it, then beat my policy

The code and pretrained policy are available on GitHub. Run the baseline on your phone, train your own agent, and compare its best distance and median across device runs.

You'll need a non-rooted ARM64 Android phone, USB debugging, ADB and your own Temple Run APK. The repository includes the instrumentation tools; the game itself isn't included.

Send me your results. There isn't a public submission board yet. Email matthieu.scharffe@gmail.com with your agent's best distance, median, number of evaluation runs and a code or policy link. Include the phone, game version, any run limits and a recording of the best run, so we can compare the results fairly.