gam_server belongs to the gamin file alteration monitor, a part of FAM. By default, it monitors all /mnt/* and /media/* mounts. We happen to have an NFS share for common configs, sources and ISOs and such. So it makes some sense.
Searching Google turns up a surprising amount hits for the same problem, dating back as far as 2004. There were some memory leaks with versions older than 0.1.7 but most problems should be fixed starting with that version. So update gamin if you don't have that version. The next issue is the default config for gamin. Two seperate solutions (1|2) both outline the solution: adjust the /etc/gamin/.gaminrc and set appropriate poll intervals for the file systems of your choice. In our case, we'll adjust NFS polls to every 30 seconds, ext3 to 10 secs.
fsset nfs poll 30
fsset ext3 poll 10
You could also adjust the mount points being monitered. Might try that later.