You can also mess around so you have separate shortcuts for windowed and fullscreen, if you want.
You can right-click either the original shortcut or a copy and in the Properties menu, put +set r_fullscreen 0 (substituting '1' for '0' if you want a fullscreen shortcut) at the end of your Target line. Example:
"F:\Program Files\Tremulous\tremulous.exe" +set r_fullscreen 0
I think -window or -windowed also did it. Not sure.