I’m not too interested in talking at length about the evolution of political parties, but it has to be said that things got pretty weird in the 1850s.
The Democratic-Republican party of Jefferson and Monroe started out as ultra-small government, partially due to the remaining distaste for monarchy and the big programs of John Adams. However, as the need for centralized authority in the running of a large country became more apparent, the Democratic-Republicans started putting more and more power into the federal government, to the point that John Quincy Adams, formerly a hated Federalist, became a natural member of their party.
But then Andrew Jackson was like, we need to be EVEN MORE POWERFUL, so although he claimed Jefferson and Monroe as his heroes, he is considered the first Democratic president because he broke with so much of their philosophy (like diplomacy).
So then the anti-Jackson faction kind of didn’t know what to do, because they considered themselves in opposition to him, even though he had kind of come from their party. So the only thing to do was to start a new party that was obviously not his.
The Anti-Masonic party started as a vote-getter for John Quincy Adams. A small group of supporters wanted Adams back in power, and they thought the most effective way to get rid of Jackson was to stoke people’s fear of Freemasons, of whom Jackson was a powerful member. It didn’t work, so the Anti-Masons mostly all became Whigs, which had a broader political platform.
The Whigs came to power with the presidency of William Henry Harrison, but his successor John Tyler abandoned the party while still president. Then the Whigs coalesced around opposition to the Mexican-American War and James K. Polk, its Jacksonian-protégé leader, and got Zachary Taylor elected as the anti-Polk.
The Whigs, though, had a hard time keeping it together. In order to be a powerful national party, they had to appeal to be North and South, and this was next to impossible, because everyone had a strong view on slavery. Abolitionists kept breaking away from the Whigs and starting their own parties, but they never got strong enough to win anything besides local elections.
Then there was the Native American Party, otherwise known as the Know-Nothing Party, or Nativists. This party was started by the secret society Order of the Star-Spangled Banner, a group of New England white guys who hated immigrants and Catholics. If any of them were ever asked about their secret society, they would say “I know nothing about it,” which is how they got their nickname.
The people in the Know-Nothing Party were valuable to Whigs, though, because they agreed on sectional compromise as a policy, so they eventually got folded back into the Whigs as well.
But, in general, party-hopping was rampant among non-Democrats. There were Free-Soilers, National Whigs, "Silver-Grey" Whigs, Nativists, Anti-Nebraskans, and the Liberty Party. Old Millf started out as an Anti-Mason, then a Whig, and then remained a Whig when the rest of the party became Republicans. It’s hard to keep track of, because everybody keeps switching. As Rayback says in his biography of Millf, a lot of politicians in the 1850s were “experimenting with new political loyalties,” and everything got very unpredictable.