Dedicated server: is only running server-side things. i.e. it's a computer sitting there running the server.
Non-Dedicated server: a server that isn't "dedicated" to running just the server, which basically means it's running a client as well. This would be when you create a server out of the same PC you're playing on.
I'm pretty sure that's it, someone else feel free to correct