CCNP Security: SENSS Part 1-1

Notes for the SENSS exam, mapped against Cisco's blueprint for the 300-206.  As there are no books for this exam, my sources for this primarily came from various manuals and documentation available on Cisco's site.  Whether a blueprint item is listed as "implement" or "describe" determined how far in depth I studied the particular topic.  Notes focus on CLI 

Threat Defense: 1.1 Implement firewall (ASA or IOS depending on which supports the implementation)

1.1.a Implement ACLs

ACL Types

Extended ACLs-main type used, used for access rules, traffic matching for service policies, AAA rules, WCCP, botnet traffic filter, VPN group, and DAP policies

EtherType ACLs- apply to non-IP layer 2 traffic for transparent mode

WebType ACLs - Used for filtering clientless SSL VPN traffic

Standard ACLs - identify traffic based on destination address only.  Used by route maps and VPN filters.

ACLs and NAT

Important to note distinction on when Real IP addresses (the untranslated IP address) and Mapped IP addresses (NAT translated IPs).  Major distinction occurs between 8.2 code and 8.3+ code for the ASA.

Use Real IP Adddresses

-Access rules (referenced by access-group command)

-Service Policy Rules (referenced by match access-list command

-Botnet Traffic Filter classification (referenced by dynamic-filter enable classify-list command)

-AAA rules (aaa match command)

-WCCP (wccp redirect-list group-list command)

Use Translated IP Addresses

-IPSec ACLs

-capture command ACLs

-Per-user ACLs

-Routing protocol ACLs

-All other feature ACLs

Network mask is different from IOS access-list command. ASA uses a network mask.  IOS uses wildcard bits.

Configuration and Management

Show ACL contents, line numbers, and hit counts

asa# show access-list outside_access_in

Add an ACE- access-list [name] [line line-num] type

If no line num is specified, ACE is appended to the end of the ACL

Add comments to an ACL (does not apply to webtype)- access-list [name] [line line-num] remark

Best practice is to insert the remark before the ACE.  Remark is limited to 100 char. If a line number is not specified, remark is added to the end of the ACL.

ACE's and remarks cannot be edited or moved.  Instead a new one must be created at the right location via line number.  The old ACE or remark should then be deleted.  ASDM is easier to use for this purpose.

Delete an ACE or remark- no access-list parameters 

Parameters must match the ACE or remark exactly to remove it

Delete an entire ACL including remarks - clear configure access-list name

If no name is specified, all ACL's are deleted.  There is no confirmation with this command.

Rename an ACL- access-list name rename new_name

Apply an ACL to a policy- must use the access-group command to apply an extended ACL to an interface.  ACL's must be applied to a policy to have any effect on traffic.

Configure Extended ACLs

Syntax:

access-list access_list_name [line line_number] extended {deny | permit}
protocol_argument source_address_argument dest_address_argument
[log [[level] [interval secs] | disable | default]]
[time-range time_range_name]
[inactive]

1.1.b Implement static/dynamic NAT/PAT

Nat Types

Dynamic NAT- group of real IP addresses mapped to a group of mapped IP addresses

Dynamic PAT- group of real IP addresses mapped to a single IP address using a unique source port

Static NAT - consistent mapping between a real and mapped IP address, allows bidirectional traffic

Identity NAT- reall address statically translated to itself.  Used to bypass NAT

ASA can implement NAT by network object NAT and twice NAT.  Recommended to use network object NAT unless extra twice NAT features required.

Network Object NAT

Defines all NAT rules configured as a parameter of a network object.  

Twice NAT

Identifies source and destination address in a single rule.  Can allow both source and destination to be translated

Identity NAT

Map an IP to itself, used for bypassing NAT

Configure Dynamic Network Object NAT

-Create a host or range network object for the mapped addresses

-Define target real addresses to be translated

-Configure dynamic NAT for object IP addresses.  

Nat config to translate inside subnet 192.168.1.0/24 dynamically to 10.1.1.1-100 pool

ciscoasa(config)# object network in-out-nat

ciscoasa(config-network-object)# range 10.1.1.1 10.1.1.100

ciscoasa(config)# object network inside-network

ciscoasa(config-network-object)# subnet 192.168.1.0 255.255.255.0

ciscoasa(config-network-object)# nat (inside,outside) dynamic in-out-nat

Configure Dynamic Twice NAT

-Create host or range network objects  for the source real, source mapped, destination real, and destination mapped addresses

-Create service objects for destination real and destination mapped ports

Configure Dynamic PAT

ciscoasa(config)# nat (inside,outside) source dynamic inside-subnet interface 

Configure Static NAT

ciscoasa(config)# nat (inside, dmz) source static inside-net inside-net-mapped destination static Server1 Server1 service real-service mapped-service

Configure Identity NAT

ciscoasa#(config-network-object) nat (inside, outside) static host-obj host-obj-identity

-mapped ip should match the real IP.  host-obj ip = host-obj-identity

1.1.c Implement object groups

Object groups allow the resuse and grouping of IP addresses, which can greatly shorten ACE entries in an ACL.  The syntax is object-group network [name]

Example

ciscoasa(config)# object-group network malicious

ciscoasa(config-network)# network-object host 10.1.1.101

ciscoasa(config-network)# network-object host 10.1.1.102

ciscoasa(config-network)# network-object host 10.1.1.103

ciscoasa(config-network)# object-group network inside-hosts

ciscoasa(config-network)# network-object host 192.168.1.4

ciscoasa(config-network)# network-object host 192.168.1.5

ciscoasa(config-network)# access-list ACL_IN extended deny ip object-group inside-hosts object-group malicious

ciscoasa(config-network)# access-list ACL_IN extended permit ip any any

ciscoasa(config-network)# access-group ACL_IN in interface inside

1.1.d Describe threat detection features

Works at layers 3 and 4 to develop a baseline for traffic on the device, checking packet drop stats and accumulating reports based on traffic patterns.  Three main types-basic, advanced, and scanning.  Only scanning can be configured to be reactive, via the shunning of hosts.

Basic Threat Detection

Monitors rate of dropped packets and security events due to:

-ACL drops

-Bad packet format

-Connection limits exceeded

-DoS attack

-Basic firewall checks failed

-Suspicious ICMP packets detected

-Packets failed application inspection

-Interface overload

-Scanning attack detected

-Incomplete session detection

When threat is detected, ASA generates system log message 733100.  Basic threat detection measures the rate the drops occur over a period of time, called the average rate interval (ARI).  Range is 600 seconds to 30 days.  If the number events within the ARI exceed the thresholds, the events are considered a threat.  Basic threat detection does not take action and is informational only.  Burst rate interval (BRI) is always smaller then the ARI.  It looks at smaller periods of snapshot data. BRI is 1/30th of the ARI.

Advanced Threat Detection

More granular, such as for tracking stats for host IPs, ports, protocols, individual ACLs, etc.  Default setting, is enabled only for ACL statistics.  Time periods are 20 min, 1 hour, 8 hours, and 24 hours. Advanced threat detection is purely informational.

Scanning Threat Detection

Used to keep track of suspected attackers which create connections on too many hosts in a subnet, or ports on a host/subnet.  Disabled by default.

The ARI and BRI settings are shared between Basic and Scanning Threat detection.  Scanning Threat Detection maintains a database of attacker and target IP addresses.  Traffic is only considered if it passes an ACL, whereas basic threat detection will consider traffic valid even if it is dropped by an ACL.

Can be reactive by shunning an attacker's IP address.  If an attack is detected, a syslog message 733101 is logged.  If configured to react, a syslog message 733102 is logged.  When the shun expires, syslog message 733103 is logged. 

1.1.e Implement botnet traffic filtering

Workflow

-Enable used of dynamic database

ciscoasa(config)# dynamic-filter updater-client enable

ciscoasa(config)# dynamic-filter use-database

-Add static entries to database, optional

ciscoasa(config)# dynamic-filter blacklist

ciscoasa(config-llist)# name malicious.site.example.com

ciscoasa(config)# dynamic-filter whitelist

ciscoasa(config-llist)# name good.site.example.com

-Enable DNS snooping

Default config inspects all UDP DNS traffic on all interfaces, but DNS snooping is disabled

Recommendation-enable DNS snooping only on interfaces where external DNS requests are traveling

ciscoasa(config)# class-map dynamic-filter_snoop_class

ciscoasa(config-cmap)# match port udp eq domain

ciscoasa(config)# policy-map dynamic-filter_snoop_policy

ciscoasa(config-pmap)# class dynamic-filter_snoop_class

ciscoasa(config-pmap-c)# inspect dns preset_dns_map dynamic-filter-snoop

ciscoasa(config)# service-policy dynamic-filter_snoop_policy interface outside

-Enable traffic classification and actions

ciscoasa(config)# access-list dynamic-filter_acl extended permit tcp any any eq 80

ciscoasa(config)# dynamic-filter enable interface outside

ciscoasa(config)# dynamic-filter drop blacklist interface outside 

-Block traffic manually based on syslog msg info, optional

1.1.f Configure application filtering and protocol inspection

Inspection engines are required for services which embed IP addresses into the user data packet, or which open secondary connections on dynamic ports.  Requires the ASA to do DPI.  

Configured via service policies.  Uses the MPF.

-(optional) Create an ACL to identify the traffic which inspections will be applied to in a l3/l4 class map.  Default inspects all traffic

-specify additional optional parameters for applying inspection to the traffic

-Add or edit a l3/l4 policy map that sets the actions to enact on the class map traffic

ciscoasa(config)# policy-map [name]

ciscoasa(config-pmap)# class [name]

ciscoasa(config-pmap-c)# inspect [protocol]

ciscoasa(config)# service-policy policymap_name [global | interface int-name]

default config:

class-map inspection_default

match default-inspection-traffic

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

dns-guard

protocol-enforcement

nat-rewrite

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225 _default_h323_map

inspect h323 ras _default_h323_map

inspect ip-options _default_ip_options_map

inspect netbios

inspect rsh

inspect rtsp

inspect skinny

inspect esmtp _default_esmtp_map

inspect sqlnet

inspect sunrpc

inspect tftp

inspect sip

inspect xdmcp

1.1.g Describe ASA security contexts

The ASA can be partitioned into multiple contexts, each of which acts independently with separately defined administrators, security policies, and interfaces.  Similiar to running multiple ASA's within a single box, with some limitations.

Features not supported in multiple context mode:

-RIP

-OSPFv3 (OSPFv2 supported)

-Multicast Routing

-Threat Detection

-Unified Communications

-QoS

-Remote Access VPN (site to site supported)

Config files

Context configurations: separate file for each context, identifies security policy, interfaces.  Context configs stored in flash memory, or downloaded from external source (ext, TFTP server)

System configuration: context configuration locations, interfaces, and other context operations parameters are defined by the system admin in the system configuration.  Acts simliar to the startup configuration.  System config does not define interfaces for itself, uses the admin context config for this purpose.  Includes a failover interface for failover traffic

Admin context configuration: Admin context can access both the system and all other contexts.  Lives on flash memory and cannot be remote.  Admin context is created automatically when converted from single mode, on flash memory, and is named admin.cfg.

Packet Classification

To determine which context is used to send/receive packets, the ASA uses 

Unique interfaces:This method is always used for transparent mode as unique interfaces are always required for contexts.  If only one context (non transparent mode) is associated with the ingress interface, the traffic is classified to the associated context

Unique MAC Addresses:Auto generation of MAC addresses for context interfaces is enabled by default, or they can be configured manually.  Context interface MAC's must be unique if multiple contexts share an interface.

NAT Configuration:  If unique MAC addresses are disabled, then the ASA uses mapped addresses via the NAT config to classify packets to the appropriate context.  Cisco recommendation is to use unique MAC addresses over NAT traffic classification.

If the destination MAC is a multicast or broadcast MAC address, the packet is duplicated and sent to each context.

Cascading Contexts

Contexts can be placed in front of other contexts.  The outside interface of one context is the same interface as the inside interface of a different context.

Context User Management

System Administrator-accessed via the console port, or via the admin context via remote management.  System administrators can access all contexts.  

Context Administrator-these admins can only manage the context the user account is assigned to.  Accessible via telnet, ssh, ASDM.

Resource Management

Default setting-all security contexts have unlimited access to ASA resources.  VPN resources are disabled by default.  Resource management can be configured to limit use per context.  

Resource classes-contexts are assigned to resource classes to determine the limits.  All contexts belong to the default class if they are not assigned to a new resource class.  Contexts can only be assigned to one resource class.  

Resource Limits-Can be defined as a percentage or an absolute value.  Services (except VPN resources) can be oversubscribed, however if another context uses too many resources, the situation could result in other contexts dropping connections due to oversubscription.

Default Class-if a context belongs to a class other than the default, the other class settings ovveride the default class settings.  If settings are undefined in the new class, and the context is assigned to both the default class and the new class (the only multi-class membership permitted), then the default class values will take effect for the undefined values in the custom class.  Default class has unlimited access to resources for all contexts, except the following:

Telnet, SSH, IPSec- 5 sessions each max per context

MAC addresses- 65,535 entries

VPN site-to-site tunnels - 0 sessions.  Class must be configured manually to allow VPN sessions

Licensing

5506-X - not supported

5512-X - Requires Security Plus license, 2 contexts.  Additional context license-5 contexts

5515-X - Base license 2 contexts, additional context license, 5 contexts

5525-X - Base license 2 contexts, additional context license, 5, 10, 20 contexts

5545-X - Base license 2 contexts, additional context license, 5,10,20,50 contexts

5555-X - Base license 2 contexts, additional context license, 5,10,20,50,100

5585-X SSP-10- Base 2 contexts, additional 5,10,20,50,100 contexts

SSP-20, 40, 60 - Base 2 contexts, additional 5,10,20,50,100,250 contexts

ASAv - Not supported

High Availability-HSRP, VRRP, GLBP

For this post, I'm going to provide a brief overview of each of the first hop redundancy protocols covered in CCNP Switch.

Hot Standby Router Protocol (HSRP)- Cisco proprietary.  Active/Standby design.  There is 1 active router, 1 standby router, and any other routers in the group are "other routers" in the listening state.  The set of routers are called the Virtual Router, which represents a shared IP and MAC address, which is used by the endpoints as the default gateway.  Virtual MAC format: 0000.0c07.acxx .  xx represents the HSRP group number.  HSRP requires layer 2 connectivity between routers.

16 max HSRP groups

HSRP Packet Flow: Workstation uses ARP to resolve MAC for default gateway.  Active router responds to ARP and replies with virtual MAC.  In the event of failure, hosts do not need to update their ARP cache, as the standby router when it transitions to the active state, sends a gratuitous ARP reply to a broadcast MAC destination address, for the virtual IP address to force a CAM table update on the L2 switches, so L2 forwarding correctly points to the new active HSRP router.

HSRP States

Initial-Starting state, entered when an interface comes up or a configuration change occurs
Listen-Router is aware of the virtual IP, but is not active or standby.  Listens for hello messages
Speak-Sends hello messages and participates in election of active and standby routers.
Standby-Ready to become the active router should active router fail.  Sends hello messages.  Max 1 standby router per group.
Active-Actively forwards packets sent to the HSRP virtual mac address.  Sends hello messages. Max 1 active router per group.

HSRP Timers

Hello time- 3 seconds
Hold time - 10 seconds
Both timers are configurable.  Active HSRP router will advertise timers, advertised timers from the active router will take higher precedence then statically configured values.
HSRP hello packets are sent to multicast destination IP 224.0.0.2

HSRP Priority
Ranges from 0-255, default value 100
Router with the highest priority becomes the active router during an election.  In the event of a tie, the router with the highest IP address wins the election.

HSRP Preemption
Enabling preemption allows a standby router with a higher priority to take over an active router's role as the active router for the HSRP group.  Preemption is disabled by default

HSRP Tracking
Allows the priority of a HSRP router to be adjusted automatically based on tracked interface or object availability. If a tracked interface drops, the priority can be decremented to a configured value to be lower then the current active router, allowing the standby to take over. Note that preemption needs to be configured for this work properly.  Tracking can be coupled with IP SLA.

HSRP Caveats- Ensure HSRP active router is also the root STP bridge for the associated VLAN.  Ensure HSRP timers match across all routers in an associated HSRP group.  Ensure a L2 connection exists amongst HSRP routers. Timers configured on the active HSRP router are advertised to other members of the group and override manually configured timer values.

Virtual Router Redundancy Protocol (VRRP)
Essentially IEEE compatible version of HSRP, with some different features.  One router is active, and handles all traffic forwarding directed to the virtual IP address.  This router is called the master router.  Any other routers in the VRRP group are called backup routers.  Compared to HSRP, a real router's IP address can be assigned as the virtual IP address for the VRRP group.  If the real address is used, the router using this IP becomes the master address.  If a virtual IP is used, master is elected based on highest priority.

255 max VRRP groups

VRRP Packet Flow: Same as HSRP.

VRRP States:
Initialize
Backup-Backup VRRP routers do not sent advertisements, compared to HSRP.  Instead, they continue to listen for master advertisements.  If the master advertisements stop and the master down interval is exceeded, master state transition begins.
Master-While in the master state, the router actively forwards frames sent to the virtual IP and sends advertisements once every second, by default.

VRRP Timers:
Three timers are used:
The advertisement timer-default 1 second
The master down interval.  This is calculated via 3 * advertistement interval + skew time
The skew time.  Calculated 256 - priority / 256.  The skew timer ensures the router with the higher priority becomes the next VRRP master.
Only the VRRP group master sends advertisements, and it sends it by default every second to multicast address 224.0.0.18.

VRRP Priority:
Priority is configurable from 1-254, with 100 being the default priority.  0 is a special value used in the event of a master router powering down in a controlled manner to inform backup routers that it is going down.  In the event of a tie, the router with the highest IP address becomes the master.

VRRP Preemption
Unlike HSRP, preemption is enabled by default.

VRRP Tracking:
VRRP cannot track interfaces like HSRP, but can track objects

VRRP Caveats:
L2 link amongst VRRP members required for operation
Master router should be the STP root for the corresponding VLAN in a switched environment to ensure optimal pathing
VRRP group timers must match-unlike HSRP, timers are not advertised from the master router.

Gateway Load Balancing Protocol (GLBP)
Cisco proprietary protocol which allows for an active/active network with multiple routers actively forwarding traffic, comapred to HSRP and VRRP which only have one active or master router per configured group.  Can actively forward and share traffic across multiple gateways.  Routers in a GLBP group fall into one of two categories.  Active virtual gateway (AVG).  This router is elected to be the AVG for the group and assigned a virtual mac address to each remaining member of the GLBP group.  Active virtual forwarder (AVF).  The remaining gateways in the GLBP group, which are assigned virtual MACs by the AVG.  Compared to HSRP and VRRP, AVFs forward traffic.

There can be a max of 1024 virtual grouters, or GLBP groups, per physical interface of a router.
There can be a max of 4 virtual forwarders per group.

GLBP Packet Flow
AVG replies to ARP requests from hosts, and depending on the selected load balancing mode, replies with the virtual mac address for GLBP group members.

*GLBP States- still looking for more detailed information of the exact GLBP states

GLBP Load Balancing Modes
Round-robin load balancing algorithm (default)- Reply to each client ARP request used the virtual MAC address of next possible GLBP group member in a round robin fashion.
Weighted load-balancing algorithm- traffic load directed to a router is based on a weighted value assigned to the router
Host-dependent load-balancing algorithm- Host will use the same virtual mac address assuming the virtual mac address is a participant of the GLBP group

GLBP Timers
Hello- default 3 seconds
Hold time- default 10 seconds
Redirect - determines when AVG stops responding to ARP for a failed virtual mac of an AVF
Secondary Hold - Amount of time before an AVF will accept packets from an assumed virtual mac taken from a failed AVF
The AVG sends hello messages by default every 3 seconds to GLBP members via multicast 224.0.0.102.
The AVG will advertise timer values to GLBP group members

GLBP Priority
Priority determines the AVG for a group.  Priority is 1-255,with 100 as the default.  If preemption is enabled (it is off by default), as with HSRP and VRRP, a router can take over the role of AVG if it has a higher priority.

GLBP Weighting 
Weighting is used to determine AVF redundancy priority in the event of an AVF failure.  Additionally, a threshold is created to determine when an AVF may or may not be active.  The default weight value is 100. All AVFs backup on another-if an AVF fails, out of the remaining AVFs, the one with the highest weighting value wins this secondary election and accepts packets sent to two virtual macs-its own, and the one it assumed via the election.  This is where the redirect and secondary hold timers come into play.

GLBP Preemption:
Disabled by default but supported

GLBP Tracking:
Tracks objects and interfaces

GLBP Caveats:
L2 link necessary between GLBP group members
Active gateways should be configured as STP root for corresponding VLANs

Catalyst Switching Tables

General notes/review material for basic L2 switching with Catalyst switches, along with notes about the CAM and TCAM tables.

L2 Switching Review

L2 switches learn mac addresses via incoming frames, and stores the MAC address, switchport, and VLAN on which the frame arrives in the CAM table.  Frames are forwarded by destination MAC address, and frames with errors are not sent out of the switch.  Frames are received, inspected, and regenerated-this is store-and-forward switching.

CAM Tables

All Catalyst switches use a content-addressable memory (CAM) table for L2 switching.  MAC addresses are stored as keys, which the switch uses to consult then to determine egress port and VLAN ID.  If the MAC is not present, the flooding process begins to learn the MAC address.  If the mac address learned on one port presents itself on another port, the MAC address and time-stamp are saved for the new port and the prior entry deleted.

MAC addresses are learned dynamically by default but may be set statically.  The default aging period for the CAM table is 300 seconds.   If a MAC address keeps alternating between learned switch ports, a syslog error for MAC address flapping between interfaces is generated.

Change mac address table aging:
switch(config)# mac address-table aging-time [seconds]

Clear CAM table entries manually:
switch#clear mac address-table dynamic address [mac] | interface [type] | vlan [id]

Set static CAM entries
switch(config)#mac address-table static [mac] vlan [id] interface [type]

Show CAM table size:
switch#show mac address-table dynamic address [mac] | interface [type] | vland [id]

TCAM Tables

Switches capable of processing QoS and security ACLs use a Ternary Content-Addressable Memory Table (TCAM).  This processing is implemented in hardware.  Multi-layer switches and some L2 switches are equipped with TCAM table(s).  TCAM evaluates inbound packets against an entire access list using one table lookup.  Most switches have multiple TCAM tables for both in/out QoS and Security ACLs, allowing these lookups to be performed simultaneously or in parallel with L2 and L3 forwarding decisions.

The structure of the TCAM table is essentially an extension of the CAM table.  Like the CAM table, a lookup operation is performed, but TCAM entries consist of three input values for keys-Value, Mask, and Result (VMR) combinations.  Values and Masks are 134 bits.  The Value consists of source and destination addresses and protocol information.  The Masks choose value bits of interest.  Results are numeric values which show what action to take after the TCAM table lookup operation.  Frame or packet headers are compared against the value and mask pairs to yield a desired result.  Compared to a traditional permit/deny/log response, TCAM results can translate into QoS policing, pointing to a next hop routing table, etc.

The TCAM table consists of two primary parts, the Feature Manager (FM) and Switching Database Manager (SDM).

FM-This compiles ACE entries into the TCAM table, allowing TCAM to be consulted at full frame forwarding speed.
SDM-This can partition the TCAM on some switches (excluding 4500 and 6500) to tune the TCAM partitions.


TCAM Value Patterns
Ethernet-Source MAC 48, Destination MAC 48, Ethertype 16
ICMP- Source IP 32, Destination IP 32, Protocol 16, ICMP code 8, ICMP type 4, ToS 8
Extended IP TCP/UDP- Source IP 32, Dest IP 32, Protocol 16, IP ToS 8, src port 16, src operator 4, dest port 16, dest operator 4
Other IP- source IP 32, dest IP 32, protocol 16, ToS 8
IGMP- Source IP 32, Dest IP 32, protocol 16, ToS 8, IGMP message type 8
IPX- Source IPX network 32, Dest IPX network 32, dest node 48, IPX packet type 16

Port Operators
If an ACE uses a port operator such as range, neq, gt, or lt, when the FM software compiles the TCAM entry it uses a logical operation unit (LOU) register.  LOU's are limited and only so many can be used.

vCenter and Nexus 1000V Extension Key Mismatch

More troubleshooting notes-after I removed the VEMs, and then re-installed the VSM's, I encountered another error-the vCenter extension key for the nexus dvs mismatched with the VSM extension key:

I appeared to then have an orphaned DVS switch.  I resolved this by following a series of steps to match up the VSM and vCenter extension keys.  Note that when you perform these steps, and reconnect the VSM to the dvs switch, the VSM will sync its port profile configuration with the dvs.  If you have the running config backed up, now would be a good time to load that config into the VSM.  If its a fresh VSM, you'll obviously lose all of the port profiles.

1. SSH into Nexus 1000V.  Issue a show svs connections to get the svs connection name to the vCenter server.
   
2. Enter config mode.  Type in svs connections [vcenter connection name]
3. Type in no connect
4. Exit back into conf t
5. Locate the extension key for the orphaned dvs switch in vcenter.  This is under the summary page for the dvs switch. In the screenshot, you can see it is Cisco_Nexus_1000V_1351344349
   
6. In config mode on the Nexus 1000V, enter vmware vc extension-key Cisco_Nexus_1000V_1351344349 (this number will obviously vary)
7. Save config, reboot the nexus.
8. Delete the current vcenter extension key.  Browse to https://vcenterIP/mob/?moid=ExtensionManager
   
9. You should see a key ID which is different on the page, then the key from the isolated dvs switch.  Copy this key so you can unregister it.
10. Click unregister extension.  Copy and paste in the key from step 9.  Click Invoke Method.  It should say void if this is successfull.
11. Restart vCenter.
12. Now you need to get the extension key from the Nexus 1000V for the dvs switch you are recovering and import it into vCenter.  Browse to http://VSM IP address.  Right click save on the Cisco Nexus 1000V Extension xml link
    
13. Open the vcenter client.  Click on plugins, manage plug ins.  Right click, add new plugin and upload the xml file. 
    
14. Go back to the Nexus 1000V.  In config mode, type in svs connections [vcenter connection name].  Type in connect.  The connection to vcenter should go up now that the extension key matches on both sides.  If you browse back to the vcenter extensions page, you'll now see the correct extension key ID.  
    

You can also confirm your nexus 1000V connection to vcenter.  You should now have regained control over the previously orphaned DVS. The VSM and dvs should also have now synced configurations.

At this point, if you want to entirely remove the vds, and start by scratch by pulling the vem software off esxi, deleting the VSMs, and starting over fresh, you'll want to get the vds out first.

On the Nexus 1000V, go into conf t, svs connection [vcenter connection name] then enter in no vmware vds.  This will delete the vds from the vmware side.

Vsphere and Nexus 1000V-Removing Orphaned VEMs

A few days ago, I had been working with my lab on setting up the Cisco Nexus 1000V virtual distributed switches.  After finishing with my lab, I decided it was time to remove the Nexus 1000V-it doesnt work to great when you can only use it on one host.  One of my servers has disk drives, the other is diskless and uses a SAN. I only have 2 nics one of which is used by the SAN, so it appears impossible to migrate it over to the Nexus with only 1 nic, or vcenter loses connectivity during the process.  I found this out the hard way and had to reset my virtual networking a few times using DCUI..

So, I deleted my VSM VM's, before removing the VEM modules from the hosts.  I learned that this is definitely not the correct procedure for removing the Nexus 1000V.  The VEMs can't be deleted from the vsphere inventory, and re-installing the VSM's does not appear to be possible through the Cisco installer app as it detects the VEM presence on the host!  This was an incredibly frustrating process until I learned how to forcefully remove the VEM from the host.  While I was following the vmware KB's on the subject, I kept receiving an error that a cisco tar file was active, the vib could not be removed, and that a reboot was required.  Additionally, as I found out later, I ended up with an orphaned vds object in vCenter.  Deleting the VSM's first is not the way to go.

So, this procedure is for removing orphaned VEM's from ESXi hosts.  It requires use of the ESXi console.  First, make sure that all physical adapters are migrated off the orphaned VEM, and that there are no virtual machines still attached to it.  Put the host into maintenance mode.  Open up the DCUI on the ESXi host, and enable the command line/service console under troubleshooting options.  Once this is enabled, press ALT-F1 to open the console.

First, check the VEM status by entering in the command vem status

If you see that it is running, enter in vem stop

Now let's verify that there are no more vem processes running and using system resources-if there are, the next series of commands will fail and you'll get to reboot your ESXi host and start these all over again!

By making use of the linux lsof command, you can see if there are any vem processes using system files or network resources.  If you don't see anything on your screen, there are no more vem processes active, using system file resources.  If you do see any, its time to force kill these processes.

lsof | grep vem

kill -9 [vem pid]

Now, enter the command esxcli software vib list | grep Cisco

This command lists the Cisco vib files installed on the host for the VEM.  When you have the output, enter in the next command to remove it.

esxicli software vib remove -n [vib name]

This could take some time to complete, it will look like the screen has hung.  This should successfully finish.  Reboot the host and bring it out of maintenance mode.  You should no longer have the VEM installed on this specific host anymore.  After this process, I was able to run the Cisco installer app for the VSM and re-install the VSM and VEMs-however you will be greeted with a new problem.  The vds object in vCenter will not connect with the new VSM-see my next post on how to clear up this issue!

As a note, I am using vsphere 5.5 and had the Nexus 1000v version 4.2.1.SV2.2.1.a and vCenter is running as the VCSA.

Lab Overhaul

Its been some time since I last posted here, however in the time that has passed, I have finished overhauling my lab.  I have now setup a vsphere 5.5 cluster utilizing two diskless dell c1100 servers, each with 72GB of RAM and dual quad core xenon processors.  The servers boot esxi via flash drives.  The servers I have tied to a SAN via iSCSI with a total of 3.6TB of usable storage (split into smaller sized LUNs for use as data stores) in a RAID 10 configuration, giving me plenty of storage room for a lab with decent performance.  I also finally bought a proper rack to install all of my equipment into!

I wanted to experiment more with network virtualization, so I obtained the Cisco Nexus 1000k essentials virtual switches (these are free, you just need a valid CCO login) and replaced the standard v switches.  The servers data NICs are setup via trunks to my 3550 L3 switches and from there route through the rest of my lab network.  The servers have dedicated gig uplinks to the SAN.

All of my routing practice will be done now in a GNS3 VM that I have built-the physical routers I have are just too outdated now to really be of much use.  I'm probably going to sell all of my routers and buy myself a nice gigabit Cisco L3 switch-anyone want to donate a Nexus? :)

Future Posts

At this point, I've mostly been studying for classes as a part of my bachelor's degree program (and I already have all of the Cisco courses completed), so I haven't put much emphasis at this point in knocking out another Cisco cert.  But I still want to keep posting my work and labs here, so I will continue to do so, it may just not necessarily be Cisco related.

Lately I've been very interested in security as it applies to virtual infrastructure, so the content of my future posts will likely be orientated in that area-trying out different deployments for securing and monitoring virtual environments, such as Cisco's virtual secure gateway, vmware's vshield, etc.