Crap, I wish I could remember the name of that game.
You are some kind of ball that can travel along the outer lines of the square. By pressing A, if I remember correctly, you could draw your own path. As long as you were drawing, the scribbling thing you see on the picture can kill you by hitting the line. As soon as you reach a 'solid' line, the line you were drawing becomes solid as well, closing a part of the map as well, making the square smaller and thus highering the difficulty. You could get extra points by enclosing the scribbling thing.