If each RGB channel got more than 8 bits, you wouldn't need gamma.... you'd be able to represent the brightness directly.
Wasteful? Perhaps. But it'd simplify the whole problem and things would start to look the same everywhere. (in, oh, a dozen years)