Ah, missed that part (rental)...
For caulking there is also temporary/removable stuff:
I use it on some of my windows in the fall to stop drafts, remove it in the spring to open windows for summer... It works well for me.
The real sound panels are actually not all that expensive:
auralex.com
US prices on that site but Long & McQuade has them, prices are for a box not a single panel.
But they will do little to stop low bass from getting outside. You really need vibration isolation and ideally mass for that.
Don't rule out really heavy drapes (maybe with the caulking above on the windows for the drafts), they act as thermal insulation, they help deaden the room, won't usually cause mold problems, block light, are rental friendly, usually fire friendly (no poisonous gas) and don't stop you from getting out but again not much for low bass getting out.
OR go old school with egg cartons...