Take chess, for instance. Obviously, people compete at it. Obviously, artistic merit doesn't count—no panel of judges scores players on how prettily they move the pieces. But chess is a game, not a sport because it doesn't require anything physical. You don't even need to be human. Computers play it—better than we do.
Now about that notion of sport requiring some form of physical activity: this old ACF bulletin (from way back in 2001, in fact) has an interesting letter from one Jeremy Gilling. Says he:
Why not accept that all sports involve a combination of physical and intellectual effort, with marathon running (perhaps) at or near one end of the spectrum and chess at the other? You should also be aware that chess played at the elite level is physically exhausting. Physical fitness is rewarded by better results across the board.