Well first you have to make sure that you have the necessary tools to build tremulous from source. Then you should get the tremulous sourcecode, apply any patches that you need (backport/other qvms).
Then you just call make BUILD_GAME_QVM=1 and you should get a new qvm (first without any mods of course).
To make any modifications you have to read and understand the source code in src/game and then change it to what you want it to do.
One example for a small silly mod: the development version of tremulous has barricades that shrink when an alien jumps on them. Now you want to extend this so that the barricade can work like a catapult when an alien jumps from it. The barricade should need to be half a second shrunk before the catapult can work.
The functions that you have to edit are in g_buildable.c namely ABarricade_Shrink and ABarricade_Touch.
In ABarricade_Shrink you have to remember the time when the barricade was shrunk. The barricade already has an attribute self->shrunktime, but unfortunately this is updated every frame, what you need is the first time that it was shrunk, so you have to edit g_local.h and add a new attribute to the gentity_s structure. This holds all the data that the server keeps for every entity in the game. Time is measured in milliseconds and stored as integer, so you add:
int firstShrunkTime; // time when a barricade started to shrink
after the shrunktime and in ABarricade_Shrink you add the line self->firstShrunkTime = self->shrunkTime = level.time;
at the place where the shrink animation is started.
Then you can add the following to the end of ABarricade_Touch:
if ( &g_entities[ other->s.groundEntityNum ] == self && // other entity has to stand on barricade
client->pers.cmd.upmove >= 10 && // other entity must be pressing up
level.time >= self->firstShrunkTime + 500 && // barricade must be shrunk for at least 500 ms
self->spawned && self->powered ) { // barricade should be fully built and OM must exist
vec3_t base, dir;
// calculate base point of barricade
VectorCopy( self->s.origin, base );
base[ 2 ] -= self->r.mins[ 2 ];
// Calculate direction of the other entity from the base of the barricade
VectorSubtract( other->s.origin, base, dir);
VectorNormalize( dir );
// Set other entities velocity, fat aliens are pushed less.
VectorScale( dir, 40000 / BG_FindHealthForClass( client-pers.classSelection ), client->ps.velocity );
}
This is of course completely untested and may fail in various ways, but you should get an idea how modding works.
PS: I have actually looked at the tremfusion sources when I wrote this, so the tremulous source may work slightly different.