The promise sounds like marketing: film a game from the fence, and a coach hands back the three things that will win you the most points this week. It is fair to be skeptical. So let's open the hood. Here is exactly what Rally's AI coach does with your video, step by step, and — just as importantly — what it does not pretend to do.
Step one: it finds the people and the ball
The moment your clip lands, Rally runs pose estimation on every player in frame. That means it locates roughly two dozen points on each body — wrists, elbows, shoulders, hips, knees, ankles — in every single frame of video. At the same time, a separate model tracks the ball, which is the hard part: a pickleball is small, fast, and spends a lot of its life as a blur.
This is why framing matters. Rally asks for one wide angle of the whole court because it needs to see feet and ball at once. Prop the phone against the fence at roughly net height, capture both kitchens, and the tracking has everything it needs. No special camera, no wearable, no markers on your body.
Step two: it turns motion into shots
Raw skeletons and ball positions are just numbers. The next stage is segmentation: Rally slices the continuous stream into discrete events — a serve here, a return there, a third-shot drop, a dink exchange, a putaway. Each event gets tagged with who hit it, where they were standing, where the ball went, and what happened next.
The coach is not grading your swing against an ideal. It is grading your swing against what happened on the next three shots.
That last part is the whole game. A shot is not good or bad in isolation — it is good or bad because of the point it produced. By linking each shot to its outcome, the coach can tell the difference between a drive that won you the rally and an identical-looking drive that got blocked back for a winner.
Step three: it compares you to thousands of rallies
Your tagged rallies are then measured against a large library of labeled points — games annotated by real coaches at every level. This is how the system knows that your paddle is consistently late on the backhand, or that you drift back off the line after your third shot, or that your resets float high enough to get attacked. It is pattern-matching your habits against what separates each rating band from the next.
Crucially, it does not surface everything it finds. If it dumped forty observations on you, you would fix none of them. Instead it ranks the patterns by how many points they are costing you and hands back the top few. Small, specific, and yours.
What it will not do
Rally's coach is honest about its limits, and you should be too. It will not replace a great in-person pro who can feel the timing of your swing and put a hand on your shoulder to fix your grip. It does not see intention, nerves, or the conversation you had with your partner between points. What it does is make the time between lessons count — so you show up to your next session already working on the right thing.
- It runs first on your device, so your everyday clips stay private by default.
- It grades shots by outcome, not by how pretty the swing looks.
- It hands you a short, ranked list — not a data dump you'll ignore.
That is the entire trick. No magic, no hand-waving — just careful tracking, honest labeling, and the discipline to tell you the two or three things that actually matter this week. Prop the phone against the fence and try it. The feedback is specific enough that you will recognize yourself in it immediately.