I think the main culprit is Google Toolbar. So uninstall it and see if it makes a difference.
Another reason is fragmentation in the Firefox internal database. There is an add-on that “cleans” the Database.
Please post a comment if this helped you. Thanks.
Also try this tool memoryfox.blogspot.com
you have to have a sandbox account to be able to install it.