32-bit versions of Windows are capable of addressing up to 4GB of memory, but 2GB are reserved for the OS, so only 2GB remain for apps like Quartus.
By default, machines with 4GB running 32-bit Windows will not use space above 2GB.
I found various app notes and web articles on editing the boot.ini file to add the "/3GB" setting in Windows XP, which increases the address space for apps to 3GB and reduces the OS portion to 1GB.
Since Windows Vista and Windows 7 don't use a boot.ini file, you have to use a different approach.
These newer versions of Windows get their boot information from a particular "store" which is edited by a command-line program called "bcdedit" (Boot Configuration Data Editor?) under C:\Windows\System32.
If you simply run the command "bcdedit' from a "DOS" command window
(Start -> All Programs -> Accessories -> Command Prompt) you should see a list of current settings.
If the command prompt returns a message "Configuration Data Store could not be opened. Access is denied." or opens another command window which immediately disappears, those are indications you need to run the program with explicit administrator privileges.
To run a command window with admin privileges, click on
(Start -> All Programs -> Accessories -> Command Prompt)
but MAKE SURE TO RIGHT-CLICK ON COMMAND PROMPT
and then select "RUN AS ADMINISTRATOR"
Once the command window opens, the option switch to edit is called "IncreaseUserVA" (Increase User Virtual Address Space), and it is done by typing
bcdedit /set IncreaseUserVA 3072
It should return "The Operation Completed Successfully."
After that, you can simply type "bcdedit" and the returned data should show "IncreaseUserVA 3072" under the "Windows Boot Loader" section.
Reboot the machine and it should work for you. This fixed my "Out of Memory" error when compiling a Stratix III (EP3LS150) on a 4GB Windows Vista-32 machine. I monitored the Quartus processes (quartus_fit) under task manager and verified in the post-compile reports that over 2GB were used.
I hope this helps!