Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

PIVOTAL LABS
Standup 08/20/2008

Ask for Help

“We are getting 504 Gateway errors and we thing it is because our mongrels are freezing up do to inability to allocate memory, what to do?”

Without more info on the problem a few possibilities were suggested, such as the OS might be swap thrashing or the OS has no more memory to allocate.

One suggestion is to cut down your swap space to 0 in an attempt to verify that your mongrels are asking for too much, basically remove to OS swapping memory to disk from the equation.

Another suggestion is to boost your swap up to some insane size, also to take it out of the equation, the theory being that we know mongrel can leak memory, we trust the OS to keep the used memory in RAM, and we have plenty of disk space, so why put your OS in the position of not grating a mongrel what it is asking for.

Both solutions above don’t seem ideal but, whatever, we are pragmatists, and if we combine those with periodic monitoring of the system using top/ps/vmstat, at least your mongrel can keep running and this may give you time to figure out why mongrel may be so memory hungry

Comments
  1. edbond says:

    We had problems with ‘freezed’ mongrels when used ferret and had wrong permissions on index directory.
    Mongrels cpu usage was at 100%. strace to mongrel process and see what it doing.

  2. Jeff Dean says:

    I’ve had success debugging similar issues using FiveRuns – they’ve got great alerts, too – like “Holy Sh*t the memory is low!”. Combining a huge swap size and a solution like FiveRuns/NewRelic might be a good place to start.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *