COllaborative DEcision MAking (codema.in)

There was a problem with prosody, fixed by a restart of prosody

Pirate Praveen Wed 17 Aug 2016 2:03PMPublicSeen by 412
Aug 17 08:59:27 datamanager     error   Unable to write to offline storage ('/var/lib/prosody/poddery%2ecom/offline/praveen.list: Too many open files') for user: praveen@poddery.com 
Aug 17 09:10:23 datamanager     error   Unable to write to offline storage ('/var/lib/prosody/poddery%2ecom/offline/bsc.list: Too many open files') for user: bsc@poddery.com

We got to dig deeper and find out more, anyone interested?

Pirate PraveenSun 27 Nov 2016 6:12PM

This keeps repeating often, I had to restart again today.

Pirate PraveenSun 27 Nov 2016 6:18PM

@balasankarchelamat @jayaura @akshay @anisha @fayadfami @tvm @isaagar @mintojoseph can any of you help with this issue?

Minto JosephMon 28 Nov 2016 1:47AM

@praveenarimbrathod Are you sure it is not caused by hitting open file descriptor limit? Checked ulimit -n ? If not fixed, will check when I am back on 30th..

Pirate PraveenThu 1 Dec 2016 12:38PM

We have added two cron jobs to log open files and sockets every day.

1 1 * * * lsof -u prosody > /root/debug/"lsof-`date --rfc-3339=date`"
1 1 * * * netstat -taupe > /root/debug/"netstat-`date --rfc-3339=date`"

When this happens next time, we have better data to analyze and find the root cause.

VidyutFri 2 Dec 2016 2:36AM

Why are logged out users allowed access to view information like this?

Pirate PraveenFri 2 Dec 2016 3:55AM

If you meant the addresses, only our own addresees (both addresses mentioned are podmins) are copied.

Pirate PraveenMon 12 Dec 2016 10:28AM

We are hitting the limits again. I was not able to login some time back and log had same error message. I'm thinking there is a problem with prosody configuration as it should be using the database and not file system for offline messages.

Pirate PraveenMon 12 Dec 2016 3:08PM

@mintojoseph was analyzing the situation and figured out the issue using strace -xvtto /tmp/lua.st -p 29152 (pid of prosody process). We have set MAXFDS to a higher value in /etc/defaults/prosody and also modified /etc/prosody/prosody.cfg.lua to use sql backend for writing offline messages. Hope this fixes the issues. We are still monitoring the situation, we'll update here if there are more issues.