This basically involves two steps:

1. Client-side support, aka adding the required decoding libraries on the local computer. See official documentation: Enabling the H.264 codec on the NoMachine client host. Here is an example of following the steps on my Macbook Air:

   
   brew update
   brew install ffmpeg && brew upgrade ffmpeg
   cd /usr/local/Cellar/ffmpeg/3.0.2/lib
   sudo cp libavcodec.dylib /Applications/NoMachine/Contents/Frameworks/lib
   sudo cp libavutil.dylib /Applications/NoMachine/Contents/Frameworks/lib
   

Setting a custom resolution to that of the guest monitor

Say you have two monitors physically connected to the server, supporting by default maximum resolutions of 1440x900 and 1600x900. If you are logging in to the remote server from a machine that has a larger display, it may be difficult to add the new resolution. I have attempted to follow dozens of similar instructions I found online, with little luck. The only thing that worked for me seems like a hack, but it works.

➜  xrandr  | grep -i primary 
DVI-D-0 connected primary 1440x900+0+0 (normal left inverted right x axis y axis) 410mm x 257mm

We’ll be using the scale option of randr to change our resolution in this case, i.e. (x resolution)(x scale factor) = (desired x resolution); (y resolution)(y scale factor) = (desired y resolution). You’ll want to use at least a few decimal places for non-truncating decimal numbers unless your scaling factor is a rational number:

xrandr --output DVI-D-0 --scale 1.3333333x1.33333333

Afterwards, you should see this represented if you run the initial xrandr command above.

➜ xrandr  | grep -i primary 
DVI-D-0 connected primary 1920x1200+0+0 (normal left inverted right x axis y axis) 410mm x 257mm