Stream Flow Data from ExtremeAnalytics into Splunk


ExtremeAnalytics includes the ability to stream flow data from an Analytics engine to Splunk. To help you use Splunk with ExtremeAnalytics, we added a Splunk directory to the ExtremeCloud IQ - Site Engine NetSight/appdata/Purview directory.

The Splunk directory contains the following:

  • A PDF with instructions describing how to add Extreme’s enterprise IPFIX fields into the Splunk vocabulary and adjust the Splunk streaming app to process the Extreme IPFIX format.
  • Files that you can copy to the Splunk server to facilitate integration, instead of manually editing the files.

Use the procedures in this section to send Splunk-enriched network flow data using IPFIX.

Environment

  • ExtremeCloud IQ - Site Engine 22.3.10
  • Extreme Management Center 8.2 and later
  • Splunk 7.2.6 (single server deployment) and later
  • Splunk Stream 7.1.3 and later

Overview

You can configure the Splunk Stream app to process Netflow/IPFIX flow records and add the data into the Splunk data store. Configure this partly by editing text files on the file system, and partly by using the web UI.

The instance of Splunk Stream at any site can already be configured to import one or more flow sources. Because of this, you must take care to merge the needed changes for ExtremeAnalytics with the existing file contents. After you make the file system changes, restart Splunk. Then, define a new “stream” using the user interface. Finally, enable and deploy the IPFIX exporter of ExtremeAnalytics from the ExtremeCloud IQ - Site Engine user interface.

Part 1 – Making File Level Splunk Modifications

  1. Connect to the Splunk server via SSH.
  2. Enter cd $SPLUNK_HOME/etc/apps/splunk_app_stream/local.
  3. Copy the streamfwd.conf file. (If there is no streamfwd.conf file present, skip this step.)
  4. Copy Extreme's version of the streamfwd.conf file and paste it into streamfwd.conf. Alternately, merge Extreme's version of streamfwd.conf settings into the existing streamfwd.conf file.
  5. Enter cd $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local.
  6. Copy the streamfwd.conf file. (If there is no streamfwd.conf file present, skip this step.)
  7. Copy the streamfwd.conf file from the splunk_app_stream/local directory to this directory.
  8. Enter cd $SPLUNK_HOME/etc/apps/splunk_app_stream/default/vocabulary.
  9. Copy the extr.xml file to this directory.
  10. Enter cd $SPLUNK_HOME/etc/apps/splunk_app_stream/default/vocabulary/streams.
  11. Make a copy of the netflow file.
  12. Merge the contents of our extr.netflow file to the netflow file.

Part 2 – Creating a New Stream using the Splunk web UI

  1. Log in to Splunk (by default, the web server is on port 8000).
  2. Navigate to the Splunk Stream App.
  3. Select Configure Streams from the Configuration menu.
  4. Optionally, disable all existing streams if you installed Splunk Stream solely to integrate Analytics flow data.
  5. Create a new stream.
    1. Select New Stream > Metadata Stream to start creating a stream for ExtremeAnalytics.
    2. Select the Netflow protocol.
    3. Type a name and description for your stream.



    4. If you are an advanced Splunk user, you can choose an appropriate Aggregation method. This can be changed later, as well. We will leave aggregate off in this topic.





    5. In the Fields dialog box, you do not have to deselect any filters. You can deselect any selected filters you do not need later.





    6. In the Filters dialog box, do not deselect any filters. You can deselect any selected filters you do not need later.



    7. In the Settings dialog box, select Enabled for Status.



    8. In the Groups dialog box, leave the default, and select Create Stream.


    9. After you select Create Stream, Splunk confirms that your stream was created.


Part 3 – Configuring each Analytics Engine to Export IPFIX Data to the Splunk Server

  1. Log in to ExtremeCloud IQ - Site Engine.
  2. Navigate to Analytics > Configuration.
  3. Use the following steps for each engine you want exporting flows to Splunk,
    1. Select its Configuration page.
    2. Expand the IPFIX/Netflow Exporter section and fill out the required fields.
    3. Ensure that Export Enabled is checked.
    4. Set the Export IP to the Splunk server IP address.
    5. Set the Export Port to 2055 unless this has been customized in Splunk.
    6. Set the Protocol to either IPFIX or IPFIX with Padded Strings.
    7. Select Save.

    The Metadata contains some protocol-specific data for the analysis of DNS, HTTP, etc. This additional data can double the size of the records.

  4. Enforce the changes by expanding Engines in the left-panel of Analytics > Configuration, selecting the engine in the left-panel menu, and selecting the Enforce button. The Splunk Search displays "netflowdata" within a minute or two.



Appendix

Files

$SPLUNK/etc/apps/splunk_app_stream/local/streamfwd.conf

$SPLUNK/etc/apps/Splunk_TA_stream/local/streamfwd.conf

[streamfwd]

port = 8889

ipAddr = 127.0.0.1

netflowReceiver.0.ip = 10.54.116.45

netflowReceiver.0.port = 2055

netflowReceiver.0.decoder = netflow

#netflowElement.997.enterpriseid = 1916

netflowElement.997.id = 96

netflowElement.997.termid = extr.appName

netflowElement.998.enterpriseid = 1916

netflowElement.998.id = 371

netflowElement.998.termid = extr.userName

netflowElement.999.enterpriseid = 1916

netflowElement.999.id = 372

netflowElement.999.termid = extr.appGroupName

netflowElement.1000.enterpriseid = 1916

netflowElement.1000.id = 1000

netflowElement.1000.termid = extr.srcHostName

netflowElement.1001.enterpriseid = 1916

netflowElement.1001.id = 1001

netflowElement.1001.termid = extr.dstHostName

netflowElement.1002.enterpriseid = 1916

netflowElement.1002.id = 1002

netflowElement.1002.termid = extr.netResponseTime

netflowElement.1003.enterpriseid = 1916

netflowElement.1003.id = 1003

netflowElement.1003.termid = extr.appResponseTime

netflowElement.1004.enterpriseid = 1916

netflowElement.1004.id = 1004

netflowElement.1004.termtype = ipaddress

netflowElement.1004.termid = extr.serverAddress

netflowElement.1005.enterpriseid = 1916

netflowElement.1005.id = 1005

netflowElement.1005.termid = extr.nacProfile

netflowElement.1006.enterpriseid = 1916

netflowElement.1006.id = 1006

netflowElement.1006.termid = extr.detailedLocation

netflowElement.1007.enterpriseid = 1916

netflowElement.1007.id = 1007

netflowElement.1007.termid = extr.oneSidedFlow

netflowElement.1008.enterpriseid = 1916

netflowElement.1008.id = 1008

netflowElement.1008.termid = extr.clientLocation

netflowElement.1009.enterpriseid = 1916

netflowElement.1009.id = 1009

netflowElement.1009.termid = extr.serverLocation

netflowElement.1010.enterpriseid = 1916

netflowElement.1010.id = 1010

netflowElement.1010.termid = extr.metaData

$SPLUNK/etc/apps/splunk_app_stream/default/vocabulary/extreme.xml

<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="7.1.1">
	<Vocabulary id="extr">
	  <Locked>true</Locked>
	  <Name>ExtremeAnalytics Netflow Protocol Vocabulary</Name>
		<Term id="extr.appName">
			<Type>string</Type>
			<Comment>Application Name - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.userName">
			<Type>string</Type>
			<Comment>User Name - ExtremeAnalytics</Comment>
		</Term>	
		<Term id="extr.appGroupName">
			<Type>string</Type>
			<Comment>Application Group Name - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.srcHostName">
			<Type>string</Type>
			<Comment>Source Host Name - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.dstHostName">
			<Type>string</Type>
			<Comment>Destination Host Name - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.netResponseTime">
			<Type>uint64</Type>
			<Comment>TCP Response Time - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.appResponseTime">
			<Type>uint64</Type>
			<Comment>Application Response Time - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.serverAddress">
			<Type>shortstring</Type>
			<Comment>Server IP Address - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.nacProfile">
			<Type>string</Type>
			<Comment>Client NAC Profile - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.detailedLocation">
			<Type>string</Type>
			<Comment>Client Detailed Location - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.oneSidedFlow">
			<Type>uint8</Type>
			<Comment>One Sided Flow Boolean - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.clientLocation">
			<Type>string</Type>
			<Comment>Client Location - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.serverLocation">
			<Type>string</Type>
			<Comment>Server Location - ExtremeAnalytics</Comment>
		</Term>
		<Term id="extr.metaData">
			<Type>string</Type>
			<Comment>Extra Data - ExtremeAnalytics</Comment>
		</Term>
	</Vocabulary>
</CmConfig>

$SPLUNK/etc/apps/splunk_app_stream/default/streams/netflow (additions)

{
	"aggType": "value",
	"desc": "Application Name - ExtremeAnalytics",
	"enabled": true,
	"name": "app-name",
	"term": "extr.appName"
},
{
	"aggType": "value",
	"desc": "Application Group Name - ExtremeAnalytics",
	"enabled": true,
	"name": "application-group",
	"term": "extr.appGroupName"
},
{
	"aggType": "value",
	"desc": "User Name - ExtremeAnalytics",
	"enabled": true,
	"name": "x-user-name",
	"term": "extr.userName"
},
{
	"aggType": "value",
	"desc": "Source Host Name - ExtremeAnalytics",
	"enabled": true,
	"name": "src_host_name",
	"term": "extr.srcHostName"
},
{
	"aggType": "value",
	"desc": "Destination Host Name - ExtremeAnalytics",
	"enabled": true,
	"name": "dst_host_name",
	"term": "extr.dstHostName"
},
{
	"aggType": "value",
	"desc": "TCP Response Time - ExtremeAnalytics",
	"enabled": true,
	"name": "tcp-response-time",
	"term": "extr.netResponseTime"
},
{
	"aggType": "value",
	"desc": "Application Response Time - ExtremeAnalytics",
	"enabled": true,
	"name": "app-response-time",
	"term": "extr.appResponseTime"
},
{
	"aggType": "value",
	"desc": "Server IP Address - ExtremeAnalytics",
	"enabled": true,
	"name": "server-ip-address",
	"term": "extr.serverAddress"
},
{
	"aggType": "value",
	"desc": "Client NAC Profile - ExtremeAnalytics",
	"enabled": true,
	"name": "client-nac-profile",
	"term": "extr.nacProfile"
},
{
	"aggType": "value",
	"desc": "Client Detailed Location - ExtremeAnalytics",
	"enabled": true,
	"name": "client-detailed-location",
	"term": "extr.detailedLocation"
},
{
	"aggType": "value",
	"desc": "One Sided Flow Boolean - ExtremeAnalytics",
	"enabled": true,
	"name": "one-sided-flow",
	"term": "extr.oneSidedFlow"
},
{
	"aggType": "value",
	"desc": "Client Location - ExtremeAnalytics",
	"enabled": true,
	"name": "client-location",
	"term": "extr.clientLocation"
},
{
	"aggType": "value",
	"desc": "Server Location - ExtremeAnalytics",
	"enabled": true,
	"name": "server-location",
	"term": "extr.serverLocation"
},
{
	"aggType": "value",
	"desc": "Extra Meta Data - ExtremeAnalytics",
	"enabled": true,
	"name": "meta-data",
	"term": "extr.metaData"
}