Disk is full

I have changed database pool size and sidekiq concurrency to 25. Hopefully this fixes the issue.

@praveenarimbrathod It was 15. I increased it to 20. The max limit is 25.

Comrades Again we have trouble connecting with other pods. Already more than 24 hours with the problem.

Sorry to be once again the bearer of bad news.

@juansantiago thanks for the heads up, we got another report of the same problem from another user as well. I have increased sidekiq workers to 2 and which means it can handle 50 connections now. But it seems mysql is not able handle such load. There is some discussion here https://poddery.com/posts/1304598 anyone familiar with mysql tuning please suggest.

following http://www.hardwaresecrets.com/printpage/How-to-Optimize-a-MySQL-Server/1747

diaspora@poddery:~$ ./mysqltuner.pl
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_IN.utf8",
LC_MONETARY = "en_IN.utf8",
LC_NUMERIC = "en_IN.utf8",
LC_MEASUREMENT = "en_IN.utf8",
LC_TIME = "en_IN.utf8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

MySQLTuner 1.3.0 - Major Hayden major@mhtx.net
Bug reports, feature requests, and downloads at http://mysqltuner.com/
Run with '--help' for additional options and output filtering
[!!] Successfully authenticated with no password - SECURITY RISK!
[OK] Currently running supported MySQL version 5.5.37-0+wheezy1
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[!!] InnoDB is enabled but isn't being used
[OK] Total fragmented tables: 0

-------- Security Recommendations -------------------------------------------
ERROR 1142 (42000) at line 1: SELECT command denied to user ''@'localhost' for table 'user'
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 76d 14h 46m 3s (247M q [37.402 qps], 3M conn, TX: 282B, RX: 44B)
[--] Reads / Writes: 95% / 5%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (15% of installed RAM)
[OK] Slow queries: 0% (2K/247M)
[OK] Highest usage of available connections: 39% (60/151)
[!!] Cannot calculate MyISAM index size - re-run script as root user
[OK] Query cache efficiency: 65.4% (148M cached / 226M selects)
[!!] Query cache prunes per day: 798431
[OK] Sorts requiring temporary tables: 4% (26K temp sorts / 609K sorts)
[!!] Temporary tables created on disk: 34% (245K on disk / 722K total)
[OK] Thread cache hit rate: 99% (780 created / 3M connections)
[!!] Table cache hit rate: 18% (400 open / 2K opened)
[OK] Open file limit used: 0% (0/1K)
[OK] Table locks acquired immediately: 100% (85M immediate / 85M locks)

-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
Variables to adjust:
query_cache_size (> 16M)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
table_open_cache (> 400)

query_cache_size is now increased to 24MB.

I can see mysql is loading the server very heavily. We have to switch to postgresql immediately.

Another action that can help things better is to move out all other sites on the server. @manukrishnantv what can we do here?

I remember having a copy of an old DB also active in the mysql server, but not being used, which if exists will surely take more than 3GB. Removing that might also increase mysql performance for now.

poddery is now offline :(