External Log Collection for UCS Fabric Interconnects

I’ve been troubleshooting some pretty annoying bugs in our Cisco UCS environments. Most are easily solved by collecting some techsupport files, opening a TAC case and working through the glitch or config issue. However one has me really stumped and frustrated. 

When we collect these techsupport files, more specifically techsupport files directed at a specific chassis – an IO Module will randomly disconnect, reboot, or reconfigure – dropping half or both connections to the fabric (we have one uplink per IOM, two per chassis).

As we continue to troubleshoot with Cisco TAC, we mostly find out later that the tech support file we generate after the issue doesn’t contain the information they need, the logs have rolled over… or over written due to activity in the domain. We gather syslogs religiously but the information necessary isn’t sent out via syslog when it happens. Feature request?

After pressing one of the TAC engineers on my fifth case this year on this issue, he clued me in on a feature for exporting logs to an external server. Click away if you know this, I certainly didn’t.

Here’s a quick and dirty on how to do it with a generic Ubuntu server.

I’m going to write this soup to nuts for someone who’s a novice and never setup a Linux server. By no means will this be hardened and secured for public visibility – just a place for your FIs to dump their logs. Chime in with a comment if you have improvements or suggestions.

1. Deploy Ubuntu Server on a VM with a few gigs of space
   
2. Update your server once its online with these two commands:
   1. sudo apt-get update
   2. sudo apt-get upgrade
3. Reboot your server once upgrades are complete
4. Create a dedicated user
   • sudo adduser ucsloguser
   • provide password info but nothing else matters
5. Let’s assume you’ll be dumping logs to the user’s home directory, so the path will be /home/ucsloguser
   • For a permanent home, you could add a second disk and mount it under a dedicated path or get your neckbeard on and use LVM to create a logical volume you can add disks and grow later. For now we’ll keep it simple, stupid.
6. Using your favorite SSH client (Terminal, PuTTY, XTerm, etc) connect to your new server using the ucsloguser account to verify you can SSH to your Ubuntu server.
7. Ok, Linux server is ready to go.

Configuring the Fabric Interconnects to dump logs onto your Ubuntu server

1. SSH to your FI – doesn’t matter which, both will respect the monitoring change. 
   
2. Run the follow commands:
   1. scope monitoring
   2. scope sysdebug
   3. scope log-export-policy
3. Now we set the log export policy
   1. set hostname [Linux server IP or FQDN if your DNS is updated]
   2. set user ucsloguser
   3. set passwd [press enter, then enter the password of the ucsloguser]
   4. set admin-state yes
   5. set proto scp
   6. set path /home/ucsloguser/
   7. commit-buffer
4. That’s it. Now log into your Linux server and see if log file .tgz bundles are showing up in your home directory.

Tips:

Use the command sudo ls -thl to sort by newest to oldest files with a human readable size.

Use the command sudo df -h to show the space consumed. 

Use an SCP utility like WinSCP to retrieve files from your log server so you can send them to Cisco TAC now.