Huge memory leaks(httpd)

[Ruslan U. Zakirov]

Hello, Jesse and other.
We have problem with RT 3.0.4 mod_perl 1.28 Apache 1.3.28 mysql 4.0.14
perl 5.8.0
One httpd process get more and more memory and then memory end up.
MaxRequestPerChild didn't help.
Is it mod_perl related problem?
May be we need to go to FastCGI for some time?
Jesse, please give suggestions or solutions, now RT3 on our production
server.
Best regards. Ruslan.

_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Michael van Elst]

Post by Ruslan U. Zakirov
One httpd process get more and more memory and then memory end up.

Is there a large mail (large attachment) being sent to RT ?

Post by Ruslan U. Zakirov
MaxRequestPerChild didn't help.

Setting MaxRequestPerChild to 1 will handle all memory leaks (but of
course it slows down apache). If you still run out of memory then
you simply do not have enough memory.

Post by Ruslan U. Zakirov
Is it mod_perl related problem?

Apache growing is a mod_perl related problem.

Post by Ruslan U. Zakirov
May be we need to go to FastCGI for some time?

This will only move to problem from apache to the fastcgi process.
The fastcgi process will grow was large and fast as the apache process
does now.

Greetings,
--
,eM""=. a"-. Michael van Elst
dWWMWM" - :GM==; ***@dev.de.cw.net
:WWMWMw=--. "W=' cable & wireless
9WWMm==-.
"-Wmw-" CABLE & WIRELESS
_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Ruslan U. Zakirov]

Post by Michael van Elst

Post by Ruslan U. Zakirov
One httpd process get more and more memory and then memory end up.

Is there a large mail (large attachment) being sent to RT ?

No. Not sure. Max attach size 16MB

Post by Michael van Elst

Post by Ruslan U. Zakirov
MaxRequestPerChild didn't help.

Setting MaxRequestPerChild to 1 will handle all memory leaks (but of
course it slows down apache). If you still run out of memory then
you simply do not have enough memory.

I'v sad didn't help. My current MaxRequestPerChild is 10.

Post by Michael van Elst

Post by Ruslan U. Zakirov
Is it mod_perl related problem?

Apache growing is a mod_perl related problem.

Only one process grow to 250MB. I know that mod_perl uses memory, but
not 250 MB.

Post by Michael van Elst

Post by Ruslan U. Zakirov
May be we need to go to FastCGI for some time?

This will only move to problem from apache to the fastcgi process.
The fastcgi process will grow was large and fast as the apache process
does now.

Do you have same behavior of apache?

Post by Michael van Elst
Greetings,

_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Michael van Elst]

Post by Ruslan U. Zakirov
No. Not sure. Max attach size 16MB

This is fairly large. Do you really use attachments that large ?

Post by Ruslan U. Zakirov
Only one process grow to 250MB. I know that mod_perl uses memory, but
not 250 MB.

It's not mod_perl that uses memory but RT which is run by mod_perl.
Most comes from trying to handle unicode.

As an example, when I run the following script on my FreeBSD machine,
the perl interpreter first requires about 1.7MB, after reading the
kernel file (about 2.8MB) it grows to 7.7MB, after changing the
variable to unicode mode it grows to 17MB.

----- snip -----
sleep 5;
$x = `cat /kernel`;
sleep 10;
substr($x,0,1) = "\x{100}";
sleep 60;
----- snap -----

perl required about 5.5 times the size of the actual data. The
same (and worse) happens when RT handles a 16MB attachment. The
_best_ it could do would grow the process to about 90MB. But since
RT keeps a few copies of the message in memory it grows even larger.

Going to fastcgi doesn't change this, except that now its not
the apache process growing but the fastcgi process growing.

Post by Ruslan U. Zakirov
Do you have same behavior of apache?

I would get this behaviour if I'd try to send a 16MB ticket to RT.

Greetings,
--
,eM""=. a"-. Michael van Elst
dWWMWM" - :GM==; ***@dev.de.cw.net
:WWMWMw=--. "W=' cable & wireless
9WWMm==-.
"-Wmw-" CABLE & WIRELESS
_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Ruslan U. Zakirov]

Post by Michael van Elst

Post by Ruslan U. Zakirov
No. Not sure. Max attach size 16MB

This is fairly large. Do you really use attachments that large ?

Additional info on problem:
Now we have in mqueue letter 5MB
When rt-mailgate starts it eat ~70MB of memory with this mail.
Apache size become ~250M and rss ~125M,
but rt-mailgate return EX_TEMPFAIL
Questions:
1) Why rt-mailgate uses so much memory? Does it convert encodings?
2) If mail size 5MB Base64 encoded(orig mail enc is ISO-8895-1) then
what size of data will be inserted in mySQL?
3.1) $MaxAttachmentSize - it's UTF-8 Encoded or Base64 or original size?
3.2) Should $MaxAttachmentSize(RT) be equal to
max_allowed_packet(mySQL)?
Best regards. Ruslan.

_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Michael van Elst]

Post by Ruslan U. Zakirov
1) Why rt-mailgate uses so much memory? Does it convert encodings?

rt-mailgate sends the raw mail via HTTP to the webserver, to do this
it has to convert it to the 'application/x-www-form-urlencoded' format.

I had a glance at the LWP modules that do this and these keep at least
6 copies of the mail in memory plus 2 copies of the converted string,
which is slightly larger. But seeing that rt-mailgate grew to 14 times
the size of the mail, as you wrote, I have probably overlooked a few
copies.

Post by Ruslan U. Zakirov
2) If mail size 5MB Base64 encoded(orig mail enc is ISO-8895-1) then
what size of data will be inserted in mySQL?

Depends. The base64 encoded part will be decoded and shrink, if it
is text inside it will be converted to UTF8 and grow slightly (depending
on the number of non-ASCII characters). If it's all non-ASCII characters
it could double in size.

But I doubt that you have 5MB _text_. It's probably more like 5MB _data_
which is therefore stored in about 3.75MB.

With Postgres the situation is worse as everything is again Base64 converted
if there is a single '\0'-Byte in the data.

Post by Ruslan U. Zakirov
3.1) $MaxAttachmentSize - it's UTF-8 Encoded or Base64 or original size?

AFAIK it is the size of the attachment, i.e. Base64 if the mail is
Base64 encoded.

Post by Ruslan U. Zakirov
3.2) Should $MaxAttachmentSize(RT) be equal to
max_allowed_packet(mySQL)?

Regarding MySQL, yes. I am not sure if "exactly equal" works, so I would
take the safe road and set it slightly smaller to avoid overflows.

Seeing the above issues of RT handling huge mails, I would set it much
smaller. But that depends on your requirements, if you absolutely must
handle such huge mails with RT then you have no other choice.

Greetings,
--
,eM""=. a"-. Michael van Elst
dWWMWM" - :GM==; ***@dev.de.cw.net
:WWMWMw=--. "W=' cable & wireless
9WWMm==-.
"-Wmw-" CABLE & WIRELESS
_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Ruslan U. Zakirov]

Hello again.
Today was similar situation
1) Mail with two *binary* attachments jpg(~200K) and mov(~4.5M)
2) Now we got 3 new instances of this mail, but 1st with all
attachments, 2nd without, 3rd with jpg only
3) mailq continue to send this email because rt-mailgate returns
EX_TEMPFAIL
I think that amount of instances grow if I didn't rm that mail.
My machine still under memory pressure.
RT config:
...
Set($MailCommand , 'sendmail');
Set($SendmailArguments , "-oi -t");
Set($UseFriendlyFromLine , 1);
Set($MaxAttachmentSize , 6000000);
...
Mysql conf:
max_allowed_packet = 16M

May be problem related to some timeout issues?
Need your help. Ruslan.

Post by Michael van Elst

Post by Ruslan U. Zakirov
1) Why rt-mailgate uses so much memory? Does it convert encodings?

rt-mailgate sends the raw mail via HTTP to the webserver, to do this
it has to convert it to the 'application/x-www-form-urlencoded' format.
I had a glance at the LWP modules that do this and these keep at least
6 copies of the mail in memory plus 2 copies of the converted string,
which is slightly larger. But seeing that rt-mailgate grew to 14 times
the size of the mail, as you wrote, I have probably overlooked a few
copies.

Post by Ruslan U. Zakirov
2) If mail size 5MB Base64 encoded(orig mail enc is ISO-8895-1) then
what size of data will be inserted in mySQL?

Depends. The base64 encoded part will be decoded and shrink, if it
is text inside it will be converted to UTF8 and grow slightly (depending
on the number of non-ASCII characters). If it's all non-ASCII characters
it could double in size.
But I doubt that you have 5MB _text_. It's probably more like 5MB _data_
which is therefore stored in about 3.75MB.
With Postgres the situation is worse as everything is again Base64 converted
if there is a single '\0'-Byte in the data.

Post by Ruslan U. Zakirov
3.1) $MaxAttachmentSize - it's UTF-8 Encoded or Base64 or original size?

AFAIK it is the size of the attachment, i.e. Base64 if the mail is
Base64 encoded.

Post by Ruslan U. Zakirov
3.2) Should $MaxAttachmentSize(RT) be equal to
max_allowed_packet(mySQL)?

Regarding MySQL, yes. I am not sure if "exactly equal" works, so I would
take the safe road and set it slightly smaller to avoid overflows.
Seeing the above issues of RT handling huge mails, I would set it much
smaller. But that depends on your requirements, if you absolutely must
handle such huge mails with RT then you have no other choice.
Greetings,

_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

[Michael van Elst]

Post by Ruslan U. Zakirov
3) mailq continue to send this email because rt-mailgate returns
EX_TEMPFAIL

You may run rt-mailgate manually with the --debug option. It will then
print on stderr the answer of the web server which may include some
useful error message.
--
,eM""=. a"-. Michael van Elst
dWWMWM" - :GM==; ***@dev.de.cw.net
:WWMWMw=--. "W=' cable & wireless
9WWMm==-.
"-Wmw-" CABLE & WIRELESS
_______________________________________________
rt-users mailing list
rt-***@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm