If your monitor does not support a particular resolution/refresh combination then you have to use a different resolution and/or refresh rate that your monitor does support. Or buy a new monitor.
If you're asking how to set a custom mode then use the following commands:
seta r_customwidth 860
seta r_customheight 640
seta r_mode -1