{"id":2971,"date":"2015-08-08T23:37:02","date_gmt":"2015-08-08T21:37:02","guid":{"rendered":"http:\/\/michlstechblog.info\/blog\/?p=2971"},"modified":"2015-08-07T10:59:04","modified_gmt":"2015-08-07T08:59:04","slug":"esxi-no-space-left-on-device-ramdisk-root-is-full","status":"publish","type":"post","link":"https:\/\/michlstechblog.info\/blog\/esxi-no-space-left-on-device-ramdisk-root-is-full\/","title":{"rendered":"ESXi: No space left on device, ramdisk (root) is full"},"content":{"rendered":"<div class=\"twoclick_social_bookmarks_post_2971 social_share_privacy clearfix 1.6.4 locale-en_US sprite-en_US\"><\/div><div class=\"twoclick-js\"><script type=\"text\/javascript\">\/* <![CDATA[ *\/\njQuery(document).ready(function($){if($('.twoclick_social_bookmarks_post_2971')){$('.twoclick_social_bookmarks_post_2971').socialSharePrivacy({\"services\":{\"flattr\":{\"uid\":\"Michl\",\"status\":\"on\",\"the_title\":\"ESXi%3A%20No%20space%20left%20on%20device%2C%20ramdisk%20%28root%29%20is%20full\",\"the_excerpt\":\"Hi%2C%0D%0A%0D%0Aa%20VMware%20ESXi%20becomes%20unresponsive%20when%20the%20root%20ramdisk%20is%20full.%0D%0A%0D%0ASymptoms%3A%0D%0A%0D%0A%09Can%27t%20move%20VMs%20away%20from%20an%20ESXi%20host%20%28vMotion%29%0D%0A%09vSphere%20HA%20is%20not%20working%0D%0A%09ESXi%20Host%20does%20not%20join%20the%20cluster%0D%0A%09Could%20not%20start%20vpxa%20daemon%0D%0A%0D%0A%0D%0A~%20%23%C2%A0%20cat%20%2Fvar%2Flog%2Fvmkwarning.log%0D%0A%0D%0A%20%28more%26hellip%3B%29\",\"txt_info\":\"2 clicks for more data protection:\\r\\n\\r\\nOnly when you click here, the button will be come active and you can send your recommendation to Flattr. When activating, data are transmitted to third parties. \",\"perma_option\":\"off\"}},\"txt_help\":\"When you activate these fields by clicking, information to Flattr may be transferred abroad, and probably may also stored there.\",\"settings_perma\":\"Enable permanently and accept data transmission. \",\"info_link\":\"http:\\\/\\\/www.heise.de\\\/ct\\\/artikel\\\/2-Klicks-fuer-mehr-Datenschutz-1333879.html\",\"uri\":\"https:\\\/\\\/michlstechblog.info\\\/blog\\\/esxi-no-space-left-on-device-ramdisk-root-is-full\\\/\",\"post_id\":2971,\"post_title_referrer_track\":\"ESXi%3A+No+space+left+on+device%2C+ramdisk+%28root%29+is+full\",\"display_infobox\":\"on\"});}});\n\/* ]]> *\/<\/script><\/div><p>Hi,<\/p>\n<p>a VMware ESXi becomes unresponsive when the root ramdisk is full.<\/p>\n<p>Symptoms:<\/p>\n<ul>\n<li>Can&#8217;t move VMs away from an ESXi host (vMotion)<\/li>\n<li>vSphere HA is not working<\/li>\n<li>ESXi Host does not join the cluster<\/li>\n<li>Could not start vpxa daemon<\/li>\n<\/ul>\n<p><code><br \/>\n~ #\u00a0 <strong>cat \/var\/log\/vmkwarning.log<\/strong><br \/>\n<\/code><br \/>\n<!--more--><br \/>\nshows warnings like<br \/>\n<code><br \/>\nWARNING: VisorFS<br \/>\n....<br \/>\nObj: 1940: Cannot create file \/var\/run\/vmware\/tickets\/vmtck-526c7464-61d9-f8 for process hostd-worker because the inode table of its ramdisk (root) is full.<br \/>\n....<br \/>\n<\/code><br \/>\n<code><br \/>\n~ # <strong>cat \/var\/log\/vpxa.log<\/strong><br \/>\n....<br \/>\n[FFA691A0 warning 'Libs'] Cannot make directory \/var\/run\/vmware\/root_0\/1438931685015351_5476781: No space left on device<br \/>\n...<br \/>\n<\/code><br \/>\nThere are muliple eventualities:<\/p>\n<ul>\n<li>No space is left<\/li>\n<li>inode table is full<\/li>\n<\/ul>\n<p>I checked the (visorfs) freespace :<\/p>\n<pre><code>\r\n~ # vdf\r\n......\r\n-----\r\nRamdisk\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1k-blocks\u00a0\u00a0\u00a0\u00a0\u00a0 Used Available Use% Mounted on\r\nroot\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 32768\u00a0\u00a0\u00a0\u00a0\u00a017401     15367  53% --\r\netc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 28672\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 288\u00a0\u00a0\u00a0\u00a0 28384\u00a0\u00a0 1% --\r\ntmp\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 196608\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 12\u00a0\u00a0\u00a0 196596\u00a0\u00a0 0% --\r\nhostdstats\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 822272\u00a0\u00a0\u00a0\u00a0 16628\u00a0\u00a0\u00a0 805644\u00a0\u00a0 2% --\r\nsnmptraps\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1024\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 1024\u00a0\u00a0 0% --\r\n<\/code>\r\n<\/pre>\n<p>It seems all is ok. Also the inodes:<\/p>\n<pre><code>\r\n\/var\/log # stat -f \/\r\n  File: \"\/\"\r\n    ID: 100000000 Namelen: 127     Type: visorfs\r\nBlock size: 4096\r\nBlocks: Total: 443834     Free: 266260     Available: 266260\r\nInodes: Total: 524288     Free: 519045\r\n<\/code><\/pre>\n<p>but a<br \/>\n<code><br \/>\n~ # mkdir \/var\/test<br \/>\n<\/code><br \/>\nfails with <strong>not enough space on disk<\/strong><br \/>\nThis is very strange behaviour. I looked for files in the root filesystem, starting in \/var<br \/>\n<code><br \/>\n~ # \/var # find \/var\/|wc -l<br \/>\n4217<br \/>\n<\/code><br \/>\nA deeper look into the directory shows that there a lot of files in \/var\/log, all with the prefix log.svs.???????.<br \/>\n<code><br \/>\n~ # ls -l \/var\/log\/log.svs.*<br \/>\n<\/code><br \/>\ncan&#8217;t list all files \ud83d\ude41 , too many arguments&#8230;.<\/p>\n<p>I decided to delete all of them which are older then 1 day<br \/>\n<code><br \/>\n~ # find \/var\/log -mtime +1 |grep log.svs.[0-9]|xargs rm<br \/>\n<\/code><br \/>\nafter deleting the files I could start the vpxa daemon, and the host joined the cluster.<br \/>\n<code><br \/>\n~ # \/etc\/init.d\/vpxa start<br \/>\n<\/code><br \/>\nroot filesystem is now at 5% usage<\/p>\n<pre><code>\r\n~ # vdf\r\nRamdisk              1k-blocks      Used Available Use% Mounted on\r\nroot                     32768      1844     30924   5% --\r\netc                      28672       288     28384   1% --\r\ntmp                     196608       108    196500   0% --\r\nhostdstats              822272     13044    809228   1% --\r\n<\/code><\/pre>\n<p>A usefull full <a title=\"Investigate disk space \" href=\"http:\/\/kb.vmware.com\/kb\/1003564\" target=\"_blank\">link <\/a>to VMwares support page.<\/p>\n<p>Michael<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi, a VMware ESXi becomes unresponsive when the root ramdisk is full. Symptoms: Can&#8217;t move VMs away from an ESXi host (vMotion) vSphere HA is not working ESXi Host does not join the cluster Could not start vpxa daemon ~ #\u00a0 cat \/var\/log\/vmkwarning.log<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[457,730,731,14,732],"class_list":["post-2971","post","type-post","status-publish","format-standard","hentry","category-vmware","tag-esxi","tag-inode-table-of-its-ramdisk-root-is-full","tag-no-space-left-on-device","tag-vmware-2","tag-vpxa-could-not-start"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/posts\/2971","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/comments?post=2971"}],"version-history":[{"count":8,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/posts\/2971\/revisions"}],"predecessor-version":[{"id":2979,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/posts\/2971\/revisions\/2979"}],"wp:attachment":[{"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/media?parent=2971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/categories?post=2971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/michlstechblog.info\/blog\/wp-json\/wp\/v2\/tags?post=2971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}