Well, I've experienced both, and for me, anyway, noisy is the worst. My house is usually extremely organized and well-kept. Things have a place, and those things are in their places. I hate disorganization, even at my office. It's like if you want to save yourself tons of time, just organize your stuff, so you don't have to spend an hour looking for it when you need it! But…I can take disorganization 100 times more than noise.
I grew up living out in the country. Like 1/4 mile from anyone else. It was dead quiet. The only thing at night was the occasional owl and chirping crickets. When I was 18 I moved to the city, and I had to get used to having the loud noises at all times of the day and night. Cars, trains, buses, people, etc., drove me crazy for a while, until I got more accustomed to them.
A few years after that, I moved back to the country, although we had next-door neighbors who had kids with motorcycles. I was pretty used to the noise, although there were times when we had to go next door and tell them to turn down their music because we could hear it in our house.
Now, I live next door to someone who feels it's okay to have throw-down parties at all hours of the day and night, regardless that she lives in a very nice $300K house in a very nice neighborhood.
So, noise it tops of annoyance for me. I can take just about anything but noise.