When I started researching to origin of this phrase I clicked on one site that was talking about the origin being in a game of marbles and I thought, "No way."
The origin of this phrase is found in the oldest game in recorded, and unrecorded apparently, history! Marbles! Click here to read all about it.
Today the phrase is used beyond a game of marbles and means that things are serious and for real and there's no playing around.