The law defines marriage, assigns conditions on who can get married and under what circumstances, licenses marriages, defines who can administer the oaths and witness the signing of the license, and has processes for the dissolution of the marriage. It also provides legal acknowledgement of the marriage, enforces some of its obligations, and provides certain privileges (tax and otherwise). The law is, of course, written by the various legislative bodies, enacted by the various executive office holders, and interpreted by the courts.
So when someone wants the government to get its fingers out of marriage, what would that mean? Can the government license marriage without defining it? Well, no, not really - how would you know what had been licensed? How about without regulating it? Are there any examples of anything that's licensed but not regulated? None I can think of.
Assume that the government chose to no longer define, regulate, or license marriage. Marriages would become a matter of personal contract. While there might be boilerplate that's common to many such contracts, the specifics would be infinitely variable. Weddings would be a matter for lawyers, divorce would be a matter of contract law. Woe to the aggrieved spouse without a signed, witnessed copy of the marriage contract (as amended) with all pertaining codicils.
How about those legal privileges? They are arbitrary, based on the expected definition of marriage. If any arrangement can be called marriage, would the people (who are the ultimate authority behind the government, after all) be willing to give special consideration to, well, anyone who says they're married?