Objective: I goal of this document is meant to outline a few steps that will allow you to troubleshoot the cause behind why users were unable to access the internet while behind a Fortigate 100A device.
Problem: I received a ticket today stating that users in one our computer labs were unable to access the internet.
After arriving onsite I discounted the device and plugged directly into the ISP link and confirmed that they were no issues with the ISP connection. Now is time to open a ticket with support and start the series of troubleshooting to figure out the root cause of the issue.
Step one: Information gathering
After opening the ticket I was told that the issue could have possibly been caused by a bad firmware image, or a corrupted configuration. I needed to log into the device to find out more information and the only way to do so was via the console port.
Connecting to the Fortigate 100A console port:
- Start your favorite terminal emulator program use the following settings:
- Baud rate: 9600
- Data bits: 8
- Parity: None
- Stop bit: 1
- Flow control: None
- Next, reboot the device and watch the screen for any error message.
After rebooting the first time I got the following message, “You must format the boot device”, I then rebooted a second time and got the following message “The config file may contain errors, Please see details by the command ‘diagnose debug config-error-log read”. This was somewhat good news because I would have had to RMA the device if the system RAM was corrupted.
At this point I know I had the following options:
- Backup the current configs
- Reset the device to it’s default state
- Reload a new configs
Backing up the device to USB via the console port:
- Plug a usb device into one of the ports on the back of the device
- Login to the device, if you are unable to use your admin login you can login with the maintainer account, this account is only valid for 30 sec after the device has been rebooted; so copy the username and password to a text file then cut/paste to the console. Username: maintainer Password: bcpb<input device serial number using uppercase letters>
- Issue the following command: execute backup configs usb filename<use any name here>
FG100 # execute backup full-config usb fg100a-10-15-11
Copy onfigs fg100a-10-15-11 to USB disk …
Copy onfigs file to USB disk OK.
Before reloading a new configs its best to run a few diagnostic commands to try and understand what happened:
FG100 # diag sys top –> Look at CPU load and any processor that running hot
FG100 # diag debug crashlog –> Look for clues as to what service crashed
89: 2011-11-14 16:06:42 <04434> application cmdbsvr
290: 2011-11-14 16:06:42 <04434> *** signal 7 (Bus error) received ***
291: 2011-11-14 16:06:42 <04434> Register dump:
298: 2011-11-14 16:06:42 <04434> Backtrace:
299: 2011-11-14 16:06:42 <04434> [0x0893277b] => /bin/cmdbsvr
300: 2011-11-14 16:06:42 <04434> [0x08932a96] => /bin/cmdbsvr
301: 2011-11-14 16:06:42 <04434> [0x08910473] => /bin/cmdbsvr
Reload new configs via console port:
- Rename your most recent backup configs file to fgt_system.conf, then place file on the root of your USB drive.
- Plug the USB into one of USB ports on the back of the unit and reboot the unit and you should see a similar output:
Reading boot image 1370111 bytes.
System is started.
Get image from USB disk …Can not get image from USB disk.
Get config file from USB disk OK.
File check OK.
The system is going down NOW !!
If you are not certain and you leave the drive in after the system reboots you will see the following message indication that the configs file on the disk is the same as the file on the system.
Get config file from USB disk OK.
Checksum check synced! Don’t need restore config.
After I restored the config file I was still unable to connect out to the internet, so I issued the following command to verify my IP address setting:
FG100A # get system interface
After discovering that the IP address for my external interface WAN1 had a different subnet than the one I wrote down previously when I connected directly to my ISP modem with my PC. I decided to change the interface type to DHCP.
Configuring external interface for DHCP:
FG100A # configs system interface –> To enter into Interface config mode
FG100A (interface) # edit wan1 –> Choose your external interface in our case it was wan1
FG100A (wan1) # unset ip –> Removing current static IP entry
FG100A (wan1) # set mode dhcp –> To change mode to DHCP from static
FG100A (wan1) # show –> To confirm that the change was made
config system interface
set vdom “root”
set mode dhcp
set allowaccess https ssh fgfm
set type physical
FG100A (wan1) # end
I was now able to access the internet!!
Discovering what happened:
Given that the logs were lost due the fact the FortiGate was reset and the unit is storing it’s logs in RAM, I can’t diagnose the exact cause. But we did see in the Crashlog “diag debug crashlog read”, which is written to flash that the cmdbsvr was crashing. We have identified a issue with cmdbsvr on the version of fortios on your fortigate.
Bug #’s : 117281, 144277
Summary : cmdbsvr crash in conserve mode may cause configuration loss
I updated the device to MR3 patch 2, since this version addressed the issue.