A simple explanation of the steps involved in applying a compressor in Reaper:
1) Add the ReaComp effect
2) Set a Ratio of about 4:1
3) Select Auto make-up gain & Limit Output checkboxes on the right
4) Slide the Threshold down until it is causing about 2 or 3 dB of Gain Reduction (GR) on the biggest peaks of the audio.
Note that since "Auto make-up gain" checkbox is on that is why the volume gets louder as the threshold is brought down.