High Server Load

[Deep]

Hi,
We are facing High Server load problem for last few days..

and it happens at perticular time only (around 1 AM IST ) but not everyday..

we checked the cron jobs running but did not find any job running at that time...

somtimes mysql connections goes upto 150 and sometimes 20 but the server load is always 1+ at that time...

any ideas about it ?

can it be bcoz of zombie processes ? I think DDOS attack is not possible coz server is totally managed and we have our own hardware firewall too...

i googled a lot about it but couldnt find proper reason for it...

Server config :
OS - Redhat Linux 3 Enterprise
AMD 2400+
1 Gig RAM
73 GB SCSI HDD i suppose (Not very sure about the HDD size but its SCSI for sure)

CICSO Hardware firewall

Tape backups are being taken twice a week by Rackapce.com (can that cause the high load? )

ideas please...
Deep

[GNUrag]

could you get the output of this command [ # ps aux ]when the load is at its highest ?

ps: Rackspace should be having competant sys admins ... did you talk to them also ? Tape backups dont cause any server overload.... our HBCSE server is also backed up every week.... never faced any sluggishness...

[Deep]

actually i had done that when it had happend in the day time and that time we knew the reason...

that was one mysql script which was causing the problem...

but this is kind of odd time...and we have given ssh access to only One IP i.e. our office IP for security reasons

I havent asked rackspace about it coz before asking then I want to check what can be the possibilties....

I am going to write them tmrw...and will also find out the things from our side...

u know any possibilities which can cause this kind of problem.. ?

Deep

Turn on system accouting and use sa command to find out this is the only way to get snapshot of today, y’day etc!

[Deep]

Originally Posted by Anonymous

Turn on system accouting and use sa command to find out this is the only way to get snapshot of today, y’day etc!

thanks for the info..

gonna try it soon

Regards
Deep

[firewall]

hi, Deep u have a nice config. ! btw Server load is nothing to do with Zombie. As Zombies are already dead... they dont have much to do with server loadd.

now on the reason of that.. well i think i cant say anything on that by not seeing the server...
Btw you ca nchec kthe mySQL.. coz thwere is a latest exploits is released.

A .diff file, applied to the MySQL 5.0.0-alpha source distribution will allow building a MySQL client that can be used to connect to a remote MySQL server with no password.

This could be a reason.. some one may applying this.... as mySQL AB hasn't released a patch you may try to close the remote port to get rid of that...

btw here is that exploit code....

1941,1956c1941,1942
<   if (passwd[0])
<   {
<     if (mysql->server_capabilities & CLIENT_SECURE_CONNECTION)
<     {
<       *end++= SCRAMBLE_LENGTH;
<       scramble(end, mysql->scramble, passwd);
<       end+= SCRAMBLE_LENGTH;
<     }
<     else
<     {
<       scramble_323(end, mysql->scramble, passwd);
<       end+= SCRAMBLE_LENGTH_323 + 1;
<     }
<   }
<   else
<     *end++= '\0';                               /* empty password */
---
>   sprintf(end,"\x14\x00");
>   end+=2;

[Deep]

I suppose remote port is closed coz when u write telnet ip 3306 it shows connection to the server lost

i said zombie coz i have heard that if too many dead processes are stored in zombie then it might take more ram and make the server slow or something...

for example 2 many unclosed mysql connection in zombie might lead fo more use of memory or something but i am not very sure about it..just a guess..

anyways i am going to write to racksapce tmrw i.e. after checking few things on server today and see what they are saying..

EDIT : woha you posted code also..lol we will try it somtime on our server..haha

Deep

[firewall]

kool... you have ur solution man.... ! Rackspace....