In SSMS 2005 and SSMS 2008 the method to clear the server name from "connect to server" dialog is to close SSMS and remove mru.dat ( in case of 2005) or SqlStudio.bin ( in case of 2008 and 2008 R2).  This method has been documented here.

Below are the default location of these files
C:\Documents and Settings\<USER>\Application Data\Microsoft\Microsoft SQL Server\90\Tools\Shell\mru.dat
C:\Documents and Settings\<USER>\Application Data\Microsoft\Microsoft SQL Server\100\Tools\Shell\SqlStudio.bin

The biggest problem of deleting this file is we also loose the server names which we do not want to remove.  In SSMS 2012 there is documented way to delete the server name from "connect to server" dialog. Now, we can remove the server name by selecting it and pressing DELETE. 

Now in SSMS "connect to server"  dialog, navigate to the particular server with keyboard

Press “DELETE”. Now this server will be removed from list.

Please note that when you again connect to the server you removed, it will again added to MRU list and you again need to delete it.

References:-

1. http://sqlblog.com/blogs/aaron_bertrand/archive/2009/03/18/managing-the-mru-list-for-the-connect-to-server-dialog-in-ssms.aspx

2. http://blogs.msdn.com/b/managingsql/archive/2011/07/13/deleting-old-server-names-from-quot-connect-to-server-quot-dialog-in-ssms.aspx