I realise that for most people, there isn't a choice for them between having a Virtual Private Server (VPS) or Shared Hosting for a new site. Shared hosting is the easiest way to get a self hosted site, all the standard configuration is set up; some packages even come with options to install content management systems, blog platforms or forum software on demand; and any security requirements are dealt with by the hosting provider.
For me it's not so straight forward. I came to webhosting after experimenting with Linux, and running my own personal webserver at home. Obviously, this is firewalled off from the rest of the internet, so some security aspects are greatly simplified by this obscurity, but it is possible for me to set up a perfectly functional apache webserver and start serving pages to the world.
The choice then comes down to two main areas: security, and flexibility. <!-- more -->
With shared hosting, security of your site comes down to the security of your logins to the hosting platform, and the security of the software you run; the hosting provider looks after their end.
With a VPS, you are in charge of your own processor instance, so you have to make sure that all the services running on it are locked down tight. This is an involved job, and means you'll need experience of the operating system you're using and all the programs you've got on it. I'm sure I learn about a different program every time I want to do something new in Linux!
With shared hosting, part of your subscription pays for the technical expertise to keep your account running, If you're running your own box, you might need to know or hire that support, depending on how critical/valuable your web presence is to you.
When I started looking at hosting, I thought that shared hosting was the poor choice. I'd thought once you'd ordered it, all you got was unencrypted ftp access to deal with your files, and running a VPS was the only way to get the flexibility of ssh, command line access, scripting and cron jobs. As it turns out, I was wrong.
Good shared hosting is remarkably flexible; you'll get ssh access to a jailshell for your account, so you get the ability to use sftp for file trandser and can play at the command line to your heart's content. You'll probably even get some of the same visual tools to help set-up your site, like phpmyadmin for your MySQL databases in addition to the web-based configuration tools to help you get your site configured without having to tackle the command prompt.
The only real lack of flexibility with shared hosting is the lack of root access. And for almost every web-hosting use, root access is not required.
Now choosing your web host is even less simple than choosing what type to have. But, like choosing a builder, getting a recommendation from a satisfied customer that you trust is always a good place to start!
 -- Yes, security through obscurity is not good good practice when used alone, but it will greatly reduce the risk of intrusion. I can make it less likely to have the server compromised by using a NAT firewall (which is not solely "security by obscurity" anyway) at the internet connection.  -- where: Risk = Probability of intrusion × Severity of damage that intrusion would cause  -- This isn't a bug, it's a feature. One of the tenets of Linux programming is to do "one thing well". Unfortunately, it means you need help to find the tool you want when you get started!  -- A jailshell means you can't view the contents of world readable files on other accounts in the same server, and they can't read yours. So, for instance, they can't see the plain text database passwords in your CMS's scripts, that have to be readable by the server.