Troubleshooting FortiGate 100A Connectivity Issue

Objective:  I goal of this document is meant to outline a few steps that will allow you to troubleshoot the cause behind why users were unable to access the internet while behind a Fortigate 100A device.

Problem: I received a ticket today stating that users in one our computer labs were unable to access the internet.

After arriving onsite I discounted the device and plugged directly into the ISP link and confirmed that they were no issues with the ISP connection. Now is time to open a ticket with support and start the series of troubleshooting to figure out the root cause of the issue.

Step one: Information gathering

After opening the ticket I was told that the issue could have possibly been caused by a bad firmware image, or a corrupted configuration. I needed to log into the device to find out more information and the only way to do so was via the console port.

Connecting to the Fortigate 100A console port:

  • Start your favorite terminal emulator program use the following settings:
    • Baud rate: 9600
    • Data bits: 8
    • Parity: None
    • Stop bit: 1
    • Flow control: None
  • Next, reboot the device and watch the screen for any error message.

 

After rebooting the first time I got the following message, “You must format the boot device”,  I then rebooted a second time and got the following message “The config file may contain errors, Please see details by the command ‘diagnose debug config-error-log read”.  This was somewhat good news because I would have had to RMA the device if the system RAM was corrupted.

At this point I know I had the following options:

  • Backup the current configs
  • Reset the device to it’s default state
  • Reload a new configs

Backing up the device to USB via the console port:

  • Plug a usb device into one of the ports on the back of the device
  • Login to the device, if you are unable to use your admin login you can login with the maintainer account, this account is only valid for 30 sec after the device has been rebooted; so copy the username and password to a text file then cut/paste to the console. Username: maintainer Password: bcpb<input device serial number using uppercase letters>
  • Issue the following command: execute backup configs usb filename<use any name here>
    FG100 # execute backup full-config usb fg100a-10-15-11
    Please wait…
    Copy onfigs fg100a-10-15-11 to USB disk …
    Copy onfigs file to USB disk OK.
    Setting timestamp

Before reloading a new configs its best to run a few diagnostic commands to try and understand what happened:

FG100 #  diag sys top –> Look at CPU load and any processor that running hot

FG100 # diag debug crashlog –> Look for clues as to what service crashed

          Ex Output:

89: 2011-11-14 16:06:42 <04434> application cmdbsvr

 290: 2011-11-14 16:06:42 <04434> *** signal 7 (Bus error) received ***

 291: 2011-11-14 16:06:42 <04434> Register dump:

298: 2011-11-14 16:06:42 <04434> Backtrace:

 299: 2011-11-14 16:06:42 <04434> [0x0893277b] => /bin/cmdbsvr 

 300: 2011-11-14 16:06:42 <04434> [0x08932a96] => /bin/cmdbsvr 

 301: 2011-11-14 16:06:42 <04434> [0x08910473] => /bin/cmdbsvr 

 

Reload new configs via console port:

  • Rename your most recent backup configs file to fgt_system.conf, then place file on the root of your USB drive.
  • Plug the USB into one of USB ports on the back of the unit and reboot the unit and you should see a similar output:

 

Reading boot image 1370111 bytes.
Initializing firewall…
System is started.
Get image from USB disk …Can not get image from USB disk.
Get config file from USB disk OK.
File check OK.

The system is going down NOW !!

 If you are not certain and you leave the drive in after the system reboots you will see the following message indication that the configs file on the disk is the same as the file on the system.

 Get config file from USB disk OK.
Checksum check synced! Don’t need restore config.

 

 Additional Troubleshooting:

After I restored the config file I was still unable to connect out to the internet, so I issued the following command to verify my IP address setting:

FG100A # get system interface  

After discovering that the IP address for my external interface WAN1 had a different subnet than the one I wrote down previously when I connected directly to my ISP modem with my PC. I decided to change the interface type to DHCP.

 

Configuring external interface for DHCP:

FG100A #  configs system interface –> To enter into Interface config mode

FG100A (interface) # edit wan1 –> Choose your external interface in our case it was wan1

FG100A (wan1) # unset ip –>  Removing current static IP entry

FG100A (wan1) # set mode dhcp –> To change mode to DHCP from static

FG100A (wan1) # show –> To confirm that the change was made

config system interface
edit “wan1”
set vdom “root”
set mode dhcp
set allowaccess https ssh fgfm
set type physical
next
end

 FG100A (wan1) # end

 I was now able to access the internet!!

Discovering what happened:

Wrap-up:

Given that the logs were lost due the fact the FortiGate was reset and the unit is storing it’s logs in RAM, I can’t diagnose the exact cause. But we did see in the Crashlog “diag debug crashlog read”, which is written to flash that the cmdbsvr was crashing. We have identified a issue with cmdbsvr on the version of fortios on your fortigate.

Bug #’s : 117281, 144277
Summary : cmdbsvr crash in conserve mode may cause configuration loss

I updated the device to MR3 patch 2, since this version addressed the issue.
Reference Documentation:

http://emea.fortinet.net/fortinet/bht/index.php

http://bit.ly/uRSdYJ

http://docs.fortinet.com/fscan/fortiscan_cli_40.pdf

Advertisements

Logging and Troubleshooting with FortiGate 100A

I needed a way to effectively troubleshoot a given traffic issue on our FortiGate 100A device and the out of the box configuration didn’t have this configured. This can be accomplish using the web GUI or via an SSH session, and since I already have SSH configure I went with this option for most of the configuration.

Step 1: Enable Logging

SSH to the device with your favorite client and issue the following commands:

FG100A # show firewall policy (this command will display the details for all policies)

edit 1
set srcintf “internal”
set dstintf “wan1”
set srcaddr “all”
set dstaddr “all”
set action accept
set schedule “always”
set service “ANY”
set logtraffic disable
set nat enable

next
end

Once you know which policy you would like to troubleshoot it’s now time to enable logging

FG100A # edit 1
(to edit policy id 1)

FG100A (1) # set logtraffic enable (to enable logging for this policy)

Once you are finish you can always review your changes on the GUI if you would to do so click on Firewall tab –> Policy –> select the policy in question and click edit on the top –> Then click the check box for “Log Violation Traffic”

Next, click on the “Log&Report” tab –>Log Setting –> and check off “Local Logging & Archiving”, “Memory” –> Minimum log level “Information” –> check off “Enable IPS Packet Archive”

Apply settings!

Step 2:  Diagnose Sniffer Packet

Launch another SSH session and issue the follow command just makes sure you put the filter expressions in quotes:

diag sniffer packet <interface> <’filter’> <verbose> <count>

Simple test do a continuous ping from your workstation to an IP on the internet and then issue the command below and keep the windows open:

diag sniff packet any ‘host <workstation ip> and icmp’ 4

Step 2:  Enabling debug mode

Lastly in a another SSH window enable debug mode, set the source IP address and watch as a series of useful information flows across the console to help you narrow down the problem at hand.

diag debug enable
diag debug flow show console enable
diag debug flow show function-name enable
diag debug flow filter proto 1
(protocol 1 is for icmp)

diag debug flow filter addr <workstation ip>
diag debug flow trace start 400
(400 is a random number it’ the number of traces)

To disable the debug when you are finish run the following command:
diag debug disable

References:

http://docs.fortinet.com/fgt/techdocs/fortigate-admin.pdf

http://docs.fortinet.com/fgt/techdocs/fortigate-cli.pdf

Port forwarding with Fortigate 100A

I needed a way to remotely access and manage an admin workstation at one of our  satellite locations with low bandwidth. We didn’t want to use the VPN function because the last time we tried this it was really slow when pulling over the user profile.

Configuring SSH Access:

You can configure this device via https, telnet, or SSH. Since I prefer SSH I will begin by showing you how to enable SSH.

  • Log into your Fortigate appliance via the https://ipaddress then click on the “Dashboard” sub-menu on the left.
  • Scroll down and click on “CLI Console” from there enter the following command to enable SSH access on your WAN/DMZ/Internal interface:


FG100A #config system interface
edit WAN1
set allowacess ssh https
end

Now you can fire up your favorite SSH client and connect!

Adding a virtual IP to enable port forwarding:

To forward TCP or UDP ports received by your FortiGate unit external interface to an internal server, you need to follow two steps.

=> Add a virtual IP
=>Add a firewall policy with a virtual IP

This example describes how to configure port forwarding to allow access to an internal Windows XP PC (IP 172.29.19.10) with the Dameware remote access utility on port of 5555 externally and 6129 internally.

Adding a virtual IP

First run the following command to see if any VIP are already configured on the firewall.

FG100A #  show firewall vip

Once you verify you don’t have any old rule to update you can simply add one of your own (Test-VIP) with the following command:

FG100A # config firewall vip
FG100A (vip) # edit Test-VIP
FG100A (Test-VIP) # set extip 192.168.10.10
=> setting up the external IP
FG100A (Test-VIP) # set extintf wan2
=> binding to an external interface
FG100A (Test-VIP) # set portforward enable
=> enable port forwarding
FG100A (Test-VIP) #  set mappedip 172.29.19.10
=> mapping to an internal IP
FG100A (Test-VIP) # set extport 5555
=> setting up external port
FG100A (Test-VIP) # set mappedport 6129
=> mapping to internal destination port
FG100A (Test-VIP) # end

Now you can do a “show firewall vip “ to verify your virtual IP configuration:


Now all that’s left is to define a firewall policy that accepts DameWare traffic from the Internet on port 5555 and forwards it to the internal Windows PC.

To add a firewall policy with a virtual IP
You can do so following these steps:

1-  Set Source Interface to WAN1.
2-  Set Source Address to all.
3- Set Destination Interface to internal.
4- Set Destination Address to the name of the virtual IP.
Usually, the remainder of the options in this firewall policy does not need to be changed. For example, Service can remain ANY, because the virtual IP only forwards packets using port 5555.

FG100A # config firewall policy
FG100A # show
=> view all configured policy
FG100A  (policy) # edit 2 => to create new policy
FG100A (2) # set ?
=> notice some are require others  are optional.
FG100A (2) # set srcintf wan2
=> set source interface name
FG100A (2) # set dstintf internal
=> set destination interface name
FG100A (2) # set srcaddr all

FG100A (2) # set dstaddr Test-VIP
FG100A (2) # set action accept

FG100A (2) # end
FG100A # show firewall policy
=> review changes and test

Reference

http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=12945&sliceId=1&docTypeID=DT_KCARTICLE_1_1&dialogID=10078518&stateId=0%200%2010080217