VMware NSX :Underlying Gear & Physical Networking.

Hello Friends,

I am back on VMware NSX  track and This is my first post NSX. Now  going to do revised my NSX study, Hands-On and re-skill myself.   So I would love to hear you for any suggestion, correction that will help myself and other also.

Before begin want to give all credit to Blogger, VMware and Partner for their great write-up and sharing to enhance other knowledge also. 

It is my insight view on VMware NSX in reference of planning to deploy NSX for any Enterprise class environment.  Their few of requirement must to check and few of must to addressed, so here below those are:

  1. ESXi: Minimum Two (02) Host (ESXi) in each vSphere DRS cluster. Stop!!! Thinking because I am not saying that you cannot have a single standalone host within a cluster, absolutely its work but for HA (ESXi High availability) you have at least 02 Host in cluster.  I highly recommend to having three separate clusters with minimum three (03) Host.
  2. NSX License : NSX come with 3 different license editions; Standard, Advanced, and Enterprise. For more info check this NSX Licensing
  3. Max Minimum: It is bare minimum on the stand-alone Host clusters, the entire minimum to be one compute cluster, one edge and one management NSX. It is highly not recommended for any Production except PoC (Proof of concept).
  4. Network Minimum: With 1 Gig speed all just work well means if think that having 10 gigabit is a requirement for NSX that is not a true declaration.
  5. JUMBO Frame: Jumbo frames must be enabled and must to enabled on end to end communicated device like Physical switch and vNIC.
  6. TCP/IP Stack: TCP/IP stack got introduced in vSphere 5.5!!! Question what it does? Sure it is allow to create separate routing table for every different network that we create with default gateway per instance stack. So when you think of your default TCP/IP stack, you can think of things such as your Management kernel port, your vMotion kernel port, NFS, and iSCSI.
  7. VTEP & VXLAN: Cool VTEP is middle MAN who seat between L2 Fabric and Virtual Machine. VTEP remember connectivity flow of VMs with L2 logical networking. Its act as bridge.

Even though VXLAN is VLAN of VLANs to overcome of it limitation 4000 VLANs and its create logical VXLAN network through combing two or more VLANs. It support Layer 3 routing like OSPF, BGP & IS-IS, Also support multicast like IGMP & PIM.

Check this for more deep into VXLAN and VTEP

  1. VTEP HA : Now wish to share my environment which is running on a 1 gigabit fabric that have four NICs per host and I used two to have HA VTEPs, and two for HA management configuration. Let’s talk in details, if anyone running a 10G NIC configuration. I wish to have HA VTEPs however because it help to load balance my VXLAN tunnels between the two VTEPs on my hypervisor. So the recommendation is to have at least two…10 gigabit NIC.
  2. EDGE Racks: Yup!!! Most of flow going through edge racks even most of the tenants and customer connectivity will land here, so this major area there must have 10G speed and remain like management and Host networking OK with 1G speed.

Also shared L2 connectivity between management rack to edge rack to do fail over between the two and lets vSphere HA switch that.

  1. LEAF & SPINE: Not all but few folks are confuse between Leaf and spine, Let see what is Leaf and Spine. Leaf has access switches that connect to end point devices like servers, firewalls, load balancers, and edge routers etc. Spine nodes are aggregation nodes that provide connectivity between different racks in your datacenter.

Spine has leaf in it, like Network backbone where every Leaf switch is interconnected with each and every Spine switch. Leaf nodes are going to be your top-of-rack switch.

For more info click here Spin & Leaf

  1. STP: Do not use Spanning Tree Protocols (STP) not supported even not going to help because it is only help to avoid getting into any sort of networking loop.
  2. Layer 2 fabric: Let’s think about huge environment because Scale, so need to design carefully. L2 Fabric learn MAC address and use MAC to do communication and it can learn and save thousands or millions of MAC address entries.

Precisely you got it, also known as your ARP and MAC table.L2 Fabric has a limitation of just over 4000 VLANs in most of environments.

So I am going to take pause here and I’ll resume myself with next post soon. Once again Please do comment as your feedback and share your thought foe better understanding of mine and other also.

Thank you

Kuver P Singh

Advertisements

VMware Command Alphabet : A to Z

Hi,

I have tested below ESXi command line in my Lab and documented for my VMware friends. Hope you all like it and it would help you.

Important Tips: Before connect or do putty of ESXi, set the logging output to get all command history saved.

# auto-backup.sh [To take ESXi configuration backup, see the KB2043048]

# df [To get filesystem info]

# env [To see ESXi basic connection setings]

# ethtool -i vmnic0 [To see Network Card driver and firmware version]

# ethtool -S vmnic0 [To see Network TX & RX packet stats]

# ethtool -c vmnic0 [To see Network Coalesce parameters]

(Coalesce: Help tune network latency and performance, refer vSphere performance guide)

# esxcfg-info | grep -i xxxx [To see ESXi component detailed information]

# firmwareConfig.sh [To troubleshoot hardware firmware issue]

# fdisk -l [To see device list and partition table.

# hostd -v [To check ESXi Agent info and version]

# hostname [To know ESXi Name]

# hwclock [To get ESXi Date & Time]

# irqinfo [To see shared IRQ]

# kill  [To kill process]

# lspci [To check PCI hardware manufacture vendor info]

# lsof [To see locked file map with process]

# localcli [Similar to ESXCLi but can be used when hostd service not starting or for troubleshooting]

# memstats [To see memory stats because webclient does not show all information, see the KB2043413]

# net-cdp | net-dvs | net-lacp | net-lbt | net-stats [To use for network Troubleshooting and see the stats]

# pktcap-uw [To capture the packet, see the KB2051814]

# partedUtil [To manage, create anfd delete disk partition (VMFS), see the KB1036609]

# passwd [To reset ESXi password]

# ps [To get ESXi process list]

# sched-stats [To see ESXi vmkernel stats for each event]

# services.sh [To troubleshooting issue, start or restarting ESXi service will not impact running VM]

# tcpdump-uw [To capture TCP dump for network issue investigation]

[Thank you @ Viren – http://www.virten.net/2015/10/esxi-network-troubleshooting-with-tcpdump-uw-and-pktcap-uw/]

# uwstats {To see user world stats]

# vdf [To see VMFS space usages in detailed, see the KB1003564]

# vdu [VMware disk utility]

# vdq [VMware vSAN utility]

# vmkping [To ping IP and test MTU packet using -l switch]

# vmware -lv [To see VMware ESXi version and build]

# vmkchdev -l | grep vmnic0 [ VM kernel change device, see the KB1027206]

# vm-support [To generate VMware support bundle log “.tgz” that can shared with VMware support]

# voma -m vmfs -d /vmfs/devices/disks/xxxxx [vSphere On-disk Metadata Analyzer (VOMA), see the KB2036767]

# vmkfstools [To perform VMFS operation, see this http://pubs.vmware.com/esx254/admin/wwhelp/wwhimpl/common/html/wwhelp.htm?context=admin&file=esx25admin_disks.10.13.html]

Thank you

ESXCLi for VMWARE Administration

Hello Friend,

On your demand, I have listed down few useful ESXCli command and hoping it will help you to make your operation easy.

I would like to Thank you and recommend to read 

Automated Report of ESXi HW Driver&Firmware info

vDocumentation : Automate VMware Config documentation

If you face any difficulties to execute above script comment me below with your issue.

####ESXCLi Operation ################################
# Below statement I found very useful for VMware Administration#
######### ESXi Version Info #################################
vmware -vl

#Output: VMware ESXi 6.0.0 build-5224934
#VMware ESXi 6.0.0 Update 3

esxcli system version get

#[root@localhost:~] esxcli system version get
#Product: VMware ESXi
#Version: 6.0.0
#Build: Releasebuild-5224934
#Update: 3
#Patch: 58

####### Network Adapter Info ###############################
esxcli network nic list
ethtool -i vmnic0
esxcli network nic get -n vmnic0
vmkchdev -l | grep vmnic0
vmkchdev -l | grep vmnic5
esxcli network nic get -n vmnic5

####### HBA Adapter Info #################################
esxcfg-scsidevs -a # Choose your driver in my case Qlogic
vmkload_mod -s qlnativefc | grep -i version
vmkchdev -l | grep vmhba1
vmkload_mod -s mpt2sas | grep -i version
vmkchdev -l | grep vmhba0

########## ESXCLi system ############################################
esxcli system # To use for esxi operation and output give all option

#Show ESXi version and build
esxcli system version get

#Enter Maintenance Mode
esxcli system maintenanceMode set –-enable yes

#Exit maintenance Mode
esxcli system maintenanceMode set –enable no

#List only advanced settings that have been changed from the system defaults
esxcli system settings advanced list

#List only kernel settings that have been changed from the system defaults
esxcli system settings kernel list –d

#List / Change / Test SNMP
esxcli system snmp get # Use these switch “| hash | set | test”

############# ESXCLi VM ######################

#List VMs on the ESXi server with the World ID
esxcli vm process list

#Kill a VM
esxcli vm process kill –t soft -w WorldID

##########ESXCLi software####################################

Install updates and drivers on a ESXi host (make sure the ESXI host is MaintenanceMode)
esxcli software vib install -d /vmfs/volumes/VMFS01/patches/nameoftheupdate.zip #Install will replace old file/driver
esxcli software vib update -d /vmfs/volumes/VMFS01/patches/nameoftheupdate.zip #Update will keep and overwrite old file & driver safe to use.

##############ESXCLi network #################################

#List physical NICs and connection state
esxcli network nic list

#List network information for the VM
esxcli network vm list

#############ESXCLi storage ######################################

#List the devices currently controlled by the VMware NMP Multi path Plugin and show the SATP and PSP information associated with that device
esxcli storage nmp device list

#List VAAI properties for devices currently registered with the PSA.
esxcli storage core device vaai status get

#Change the default pathing policy to Round Robin and choose yout Storage Vendor
esxcli storage nmp satp set –default-psp VMW_PSP_RR –satp youre_satp_policy

Thank You

 

 

Lets meet at Delhi VMUG Event 23-Sep-17

Hello Friends,

Do register to your self for Delhi VMUG event, Announced date is Saturday, 23-Sep-2017.  It is first time in Delhi any event organized for local VMware user community.

So don’t missed it, You will have a chance to make new friends to expand your professional networking, you will learn about new technology such as NSX, vROPS & vRA and get free vSphere Tips & Tricks along with Career guidance.

Click here for Register yourself

If anyone failed to register or facing any issue. Please comment below this post.

Thanks to VMware VMUG Team @MyVMUG to design cool logo for Delhi_VMUG.

c678f0ba-bac4-4339-9879-5653d861ad02

Creating a RDM Disk

Hi,

Creating RDM quick recap and steps to configure RDM disk. Hope its help.

What is RDM? : Raw Disk Mapping (RDM) configured through bypassing Hypervisior, RDM can talk directly SAN device (Storage). Once you configured RDM first time its create mapping file in VMFS Datastore where .VMX file reside on same DataStore and RDM has metadata for managing, configuring and control on disk access to the SAN.
RDM has two compatibility type:

1. Physical compatibility RDM (pRDM).

2. Virtual compatibility RDM (vRDM).

1. Physical RDM (pRDM) : Guest OS can send SCSI commands to SAN device without involving Hypervisior and pRDM has limition VM’s Snapshots although SAN (LUN) level sanpshost can be used as alternative solution also FT  will not work for pRDM. SIOC and vCloud Directior does not support RDM (Any compatibility)
pRDM good for Cluster use case between Physical to Virtual Cluster Node and Cluster VM (Node) distribute running on different ESXi Host (Across the box)

2. Virtual RDM (vRDM) : Here Hypervisior responsible for SCSI commands and Snapshot of VM can be taken. SIOC and vCloud Directior does not support RDM (Any compatibility) and FT will not work for vRDM.
vRDM can use Across the cluster and cluster in box, if your all cluster VM (Node) running on same.
RDM Prerequisite: A raw LUN [ max 62TB (vRDM) & 64TB (pRDM)]
RDM Creation :
Step-1 : Unused LUN with identical naa.id to choose correct one.
Step-2 : Re-Scan for new device for all HBA prior to attaching VM.
Step-3 : Select Correct VM which you want to add RDM.
Step-4 : Click on Edit setting and choose new device from bottom and select RDM Disk.
Step-5 : Select RDM Compatibility mode choosing drop down option.
Step-6 : VM should be shut down or power-off before sharing RDM disk.
Step-7 : Select Multi writer flag for shared RDM disk.
Step-8 : Shared RDM should be configured iSCSI Controller SCSI 1, 2 and 3 because SCSI 0  used for Local disk only.
Step-9 : Check with application owner if they require DISK UUID, So enable it using VM advanced option and add config parameter [disk.enableUUID=true].

Step-10 : Power on all VM that should be power on without any error.

Feel free share your suggestion for me and comment below if any thing I missed or you want to highlights.

Thanks

Kuver P Singh

A Thinker | Practitioner | Photographer

 

Learn vROPS like Expert

Hi,

Before moving I would like to thanks to  Iwan’e1′ Rahabok & Sunny Dua for there great job and Remarkable blog post for vRealize Operation Manager.

Got an opportunity to listen Iwan’e1′ Rahabok after that workshop my way of thinking and handling operation got changed, So once again thanks to you all Legend.

iWAN e1

In pic from Right to Left, Iwan ‘e1’, ME (Kuver) and RAM, clicked during an workshop.

Continue reading

VMware Cloud Foundation

Hello,

As show in below Pic (Pic source : VMware)and VMware Cloud Foundation <vCF> is bundle product include vSphere ESXi, vCenter , vSAN and NSX . Nice and user friendly GUI to start installation using VCF image bundle.

vCF

VMware Cloud Foundation Overview

VMware Cloud Foundation in Nutshell

Use below HOL for your hands on and Technical understanding .

VMware Cloud Foundation Hands-On Lab

For Any Query or Doubt see the below KB:

VMware Cloud Foundation FAQ

IBM has solution for Hybrid Cloud deployment

IBM for VCF Cloud Solution