Path to HxComp.exe
HxComp.exe is compiler of Help 2 documentation which is used by Help 2 template when you generate Help 2 output. For VS .NET 2002 it is part of free VSHIK 2002 (Visual Studio Help Integration Kit v2.1) from Microsoft. For VS .NET 2003 it is part of VSHIK 2003. For VS 2005/2008 it is part of Visual Studio 2005/2008 SDK (formally the VSIP SDK) which contains the Help 2 SDK. This program is usually located in C:\Program Files\Microsoft Help 2.0 SDK folder.
VSdocman tries to find that path automatically when started for the first time. You can also use Auto-find button anytime later. If it fails you can do it manually. If you cannot find that file you probably have not installed VSHIK or VS SDK. You can download it from http://www.helixoft.com/vsdocman-faqs/where-can-i-download-hxcomp.exe.html
You do not need to install VSHIK or VS SDK if you do not want to generate Help 2 output.
Custom variable 1-3
These variables are used for your own purposes. You can specify your paths or texts here and then access them in templates using macros $CUSTOM-VAR1$, $CUSTOM-VAR2$ and $CUSTOM-VAR3$.