Resolving 'Root Partition Is Filling Up' Issue on Sophos UTM Firewall

from:

https://wandersick.blogspot.com/2016/06/resolving-root-partition-is-filling-up.html

This is a quick guide on how to resolve an occasional problem that could happen to neglected Sophos UTM firewall with default settings, where updates are automatically downloaded to device, filling up storage after some time.

An email alert similar to the below is sent to administrator when it happens:

From: Firewall Notification System [do-not-reply@fw-notify.net]
Sent: 18 June 2016 11:58 PM
To: Administrator [admin@wandersick.com]
Subject: [SOPHOS][INFO-150] Root partition is filling up - please check

Root partition is filling up - please check. Current usage: 86%
--
System Uptime      : 362 days 1 hour 2 minutes
System Load        : 0.05
System Version     : Sophos UTM 9.210-20

Please refer to the manual for detailed instructions.

To solve it, one way is to

1. SSH into Sophos UTM firewall using root login
2. Remove content under Sophos update directory (Main focus of this blog post)
3. Configure firmware download interval for System Up2Date packages from Automatic to Manual

1. SSH into Sophos UTM firewall using root with public key authentication

In order for root
login to be made possible, it is required to first generate a private
key and set up public key authentication. You may refer to any guide
found on the Internet such as this one.

After enabling root login, we can then authenticate using
the private key file using SCP/SSH clients such as WinSCP, PuTTy by
specifying private key file as shown below.

a. Specifying private key file in WinSCP SCP client:

b. Specifying private key file in PuTTy SSH client:

2. Remove content under Sophos update directory

Commands for determining storage utilization on directory /var

• cd /var
• du -shx  *

After root SSH login, executing the above directory size checking commands revealed the /var/up2date folder (containing downloaded firmware updates) took 2GB of space on the device.

Mitigate the situation by removing the update files under /var/up2date/sys. Below screenshot shows a way to do it using WinSCP. (Backing up the files is recommended prior to deletion.)

Command for listing space utilization statistics on each file system mounted on device

• df -h

Before removing the files, the disk utilization was 86%:

After removing the files, the disk utilization lowered to 46%:

3. Configure download interval for System Up2Date packages as Manual

To prevent this problem from recurring, firmware download interval can be set to Manual
on the firewall. After configuring the setting, firewall updates will
not be automatically downloaded, hence device storage won’t be filled up
as easily.

Tip: Be sure to do this quickly after
moving/deleting the files. Otherwise, the device storage will be filled
up again by the update download next time

Hope this post provides a quick way to resolve this problem for those who encounter it.

Update in April 2017:
Fix it as early as we can! It was found that if we leave the problem
unresolved until the firewall disk partition becomes 100% full, web
administration console (httpd) will fail with the error below, and we
must recover the firewall via console logon (or using Windows PE/Linux
boot disc USB).

Internal Server Error

The server encounted an internal error or misconfiguration and was unable to complete your request.

Please
contact the server administrator at admin to inform them of the time
this error occurred, and the actions you performed just before this
error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Troubleshooting firewall 100% disk full – can't believe we can boot up #firewall with Windows PE or Linux boot disc (USB) like a PC #Sophos pic.twitter.com/AhGFOzeMqj

— a wandersick (@wandersick) April 5, 2017

Reference

• Root partition filling up…
  https://community.sophos.com/products/unified-threat-management/f/52/t/30237

let's find where the problem is, from the command line as root:

cd /
du -shx *

And then go into the largest in the list and do du -shx * again.

• Up2Date SSH
  https://community.sophos.com/products/unified-threat-management/f/52/t/74866

It looks like you're now on 9.318, so I would set the Download interval to "Manual" and delete everything in /var/up2date/sys