Covid shut down the tennis tournament at Wimbledon this year, but a new model simulates showdowns between the sport’s greatest players.
What’s new: Stanford researchers Kayvon Fatahalian and Maneesh Agrawala developed Vid2Player, a system that simulates the footwork, positioning, and strokes of tennis pros like Roger Federer, Serena Williams, and Novak Djokovic. Users choose players and, if they want, control their location and stroke at the beginning of each shot cycle, creating a realistic match.
How it works: Vid2Player has three main parts: a behavioral model for each player, a clip search, and a renderer.
- The authors extracted sprites, or two-dimensional bitmap graphics, that represent various players from an annotated dataset of matches recorded at Wimbledon in 2018 and 2019. They used Mask R-CNN to find every pixel in each video frame that corresponds to the player and his or her racket. Then Deep Image Matting turned the pixels into sprites. Additional preprocessing made the sprites more consistent.
- Rather than control the players manually, users can leave it to the behavioral models. These models are made up of kernel density estimators, a type of non-parametric model similar to a histogram.
- The clip search finds a video segment that best matches the selected movements and shots. The rendering module paints the appropriate sprite onto a virtual tennis court.
- Vid2Player continues to generate player motions until one player makes a mistake or fails to return the ball.
Realistic rallies: Five tennis experts who evaluated Vid2Player said it produced more realistic action than previous efforts. Similar systems like Tennis Real Play and Vid2Game draw from video databases to simulate on-court action under user control. However, they don’t search for clips that most closely match what a player would do in a given situation.
Why it matters: Apart from filling the hole in this year’s tennis season, simulations like this could allow fans to create never-before-seen matchups like Federer versus Williams. Or our favorite: Federer vs. Federer. (Federer won)
We’re thinking: We’re in love-love with this model!