Integrating Netflow

Netflow Overview

Netflow is a network protocol developed by Cisco Systems to harvest information about IP traffic. It provides the ability to collect IP network traffic as it enters or exits an interface. The data provided by netflow can be analyzed to determine things such as the source and destination of traffic, class of service, the cause of congestion and many other factors. Netflow consists of data templates, options templates and flowsets containing the data respect to the incoming templates. In s-Server, the implementation is done for the collector, which processes the data received from the exporter. Netflow has evolved through the following versions :

  • Netflow v5
  • Netflow v9
  • IPFIX ( IP Flow Information Export ), aka Netflow v10

In s-Server we support all versions of netflow.

Netflow v9 and IPFIX

The Netflow v9/IPFIX UDF (netflowCollector) is used to access the IP Flow information from the data networks; these network data are gathered by network elements like router and switches in the form of flow data and exported to collectors for further processing. The collected data provides fine-grained metering for highly flexible and detailed resource usage accounting.

A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device. These collected flows are exported to an external device, the NetFlow collector. Network flows are highly granular; for example, flow records include details such as IP addresses, packet and byte counts, timestamps, Type of Service (ToS), application ports, input and output interfaces, etc. Exported NetFlow data is used for a variety of purposes, including enterprise accounting and departmental chargebacks, ISP billing, data warehousing, network monitoring, capacity planning, application monitoring and profiling, user monitoring and profiling, security analysis, and data mining for marketing purposes.

Transport Protocol Support

As of now s-Server only supports UDP Protocol for Netflow v9 and IPFIX version. As per the lastest RFC, the supported protocols for IPFIX version (v10) are as follows :

  • UPD (supported by s-Server)
  • SCTP (not yet supported by s-Server)
  • TCP (not yet supported by s-Server)

Security Consideration

  • TLS (Transport layer Security)
  • DTLS (Datagram Transaport Layer Security)
  • SSL (Secure Socket layer)
  • UTF-8 Encoding of String type data

s-Server aleady supports UTF-8 encoding for string type as part of the collector.

Template Withdrawal and Redefinition

As of now, the template withdrawal functionality is not included in s-Server version of Netflow IPFIX Collector. Withdrawal of template is specific to the IPFIX version (v10) only. This mechanism is not applicable for UPD Protocol. Template withdrawal will be supported as part of TCP/SCTP Protocol implementation.

In case of UDP Protocol, the life cycle of the template is maintained as per the received time. As soon as a new template comes in to the collector, the old template is discarded and the new one is taken into consideration. The templates are considered in the sequence they are received.

Fixed Field Details

The netflow message data contains a few fields as part of the message and a message header in each and every message. The details are as follows:

Field Name v5 Supported v9 Supported IPFIX Supported ** Field Details**
REPORTER Yes Yes Yes Source IP Deatils
ROWTIME Yes Yes Yes Time of parsing the data
NETFLOW_VERSION Yes Yes Yes Netflow version details
FLOW_COUNT Yes Yes No Number of Flowset records, both template and data
FLOW_LENGTH No No Yes Total number of bytes present in the flowset
SYSTEM_UPTIME Yes Yes No Time in millisecond, the device was first booted
EXPORT_TIME No No Yes Time the message left the exporter system expresed in seconds since the UNIX epoch of 1 January 1970 at 00:00 UTC
UNIX_SECS Yes Yes No Seconds since 0000 Coordinated Universal Time (UTC) 1970
UNIX_NSECS Yes No No Residual nanoseconds since 0000 Coordinated Universal Time 1970
FLOW_SEQUENCE Yes No No Sequence counter of total flows seen
SEQUENCE_NUMBER No Yes Yes Incremental sequence counter of all export packets sent by this export device; this value is cumulative, and it can be used to identify whether any export packets have been missed
SOURCE_ID No Yes No The Source ID field is a 32-bit value that is used to guarantee uniqueness for all flows exported from a particular device.
OBSERVATION_DOMAIN_ID No No Yes A 32-bit identifier of the Observation Domain that is locally unique to the Exporting Process. The Exporting Process uses the Observation Domain ID to uniquely identify to the Collecting Process the Observation Domain that metered the Flows.
SCOPE_SYSTEM No Yes No Netflow v9 Specific Scope Type Detail
SCOPE_INTERFACE No Yes No Netflow v9 Specific Scope Type Detail
SCOPE_LINE_CARD No Yes No Netflow v9 Specific Scope Type Detail
SCOPE_NETFLOW_CACHE No Yes No Netflow v9 Specific Scope Type Detail
SCOPE_TEMPLATE No Yes No Netflow v9 Specific Scope Type Detail

Template Field Details

Note: These fields are added as part of the IANA_STD_DEF.csv, file present in the path plugin/IANA_STD_DEF.csv. For adding any missing fields, please reach out to the s-Server tech support team or replace the existing file to support the missing fields as in the file above on top of the existing entries. Make sure, no 2 field name should be associated with the same element Id. Field details are same for both Netflow v9 and Netflow Ipfix versions, and the details are as follows :

FIELD NAME ELEMENT/FIELD ID ABSTRACT DATA TYPE
OCTETDELTACOUNT 1 unsigned64
PACKETDELTACOUNT 2 unsigned64
DELTAFLOWCOUNT 3 unsigned64
PROTOCOLIDENTIFIER 4 unsigned8
IPCLASSOFSERVICE 5 unsigned8
TCPCONTROLBITS 6 unsigned16
SOURCETRANSPORTPORT 7 unsigned16
SOURCEIPV4ADDRESS 8 ipv4Address
SOURCEIPV4PREFIXLENGTH 9 unsigned8
INGRESSINTERFACE 10 unsigned32
DESTINATIONTRANSPORTPORT 11 unsigned16
DESTINATIONIPV4ADDRESS 12 ipv4Address
DESTINATIONIPV4PREFIXLENGTH 13 unsigned8
EGRESSINTERFACE 14 unsigned32
IPNEXTHOPIPV4ADDRESS 15 ipv4Address
BGPSOURCEASNUMBER 16 unsigned32
BGPDESTINATIONASNUMBER 17 unsigned32
BGPNEXTHOPIPV4ADDRESS 18 ipv4Address
POSTMCASTPACKETDELTACOUNT 19 unsigned64
POSTMCASTOCTETDELTACOUNT 20 unsigned64
FLOWENDSYSUPTIME 21 unsigned32
FLOWSTARTSYSUPTIME 22 unsigned32
POSTOCTETDELTACOUNT 23 unsigned64
POSTPACKETDELTACOUNT 24 unsigned64
MINIMUMIPTOTALLENGTH 25 unsigned64
MAXIMUMIPTOTALLENGTH 26 unsigned64
SOURCEIPV6ADDRESS 27 ipv6Address
DESTINATIONIPV6ADDRESS 28 ipv6Address
SOURCEIPV6PREFIXLENGTH 29 unsigned8
DESTINATIONIPV6PREFIXLENGTH 30 unsigned8
FLOWLABELIPV6 31 unsigned32
ICMPTYPECODEIPV4 32 unsigned16
IGMPTYPE 33 unsigned8
SAMPLINGINTERVAL 34 unsigned32
SAMPLINGALGORITHM 35 unsigned8
FLOWACTIVETIMEOUT 36 unsigned16
FLOWIDLETIMEOUT 37 unsigned16
ENGINETYPE 38 unsigned8
ENGINEID 39 unsigned8
EXPORTEDOCTETTOTALCOUNT 40 unsigned64
EXPORTEDMESSAGETOTALCOUNT 41 unsigned64
EXPORTEDFLOWRECORDTOTALCOUNT 42 unsigned64
IPV4ROUTERSC 43 ipv4Address
SOURCEIPV4PREFIX 44 ipv4Address
DESTINATIONIPV4PREFIX 45 ipv4Address
MPLSTOPLABELTYPE 46 unsigned8
MPLSTOPLABELIPV4ADDRESS 47 ipv4Address
SAMPLERID 48 unsigned8
SAMPLERMODE 49 unsigned8
SAMPLERRANDOMINTERVAL 50 unsigned32
CLASSID 51 unsigned8
MINIMUMTTL 52 unsigned8
MAXIMUMTTL 53 unsigned8
FRAGMENTIDENTIFICATION 54 unsigned32
POSTIPCLASSOFSERVICE 55 unsigned8
SOURCEMACADDRESS 56 macAddress
POSTDESTINATIONMACADDRESS 57 macAddress
VLANID 58 unsigned16
POSTVLANID 59 unsigned16
IPVERSION 60 unsigned8
FLOWDIRECTION 61 unsigned8
IPNEXTHOPIPV6ADDRESS 62 ipv6Address
BGPNEXTHOPIPV6ADDRESS 63 ipv6Address
IPV6EXTENSIONHEADERS 64 unsigned32
MPLSTOPLABELSTACKSECTION 70 octetArray
MPLSLABELSTACKSECTION2 71 octetArray
MPLSLABELSTACKSECTION3 72 octetArray
MPLSLABELSTACKSECTION4 73 octetArray
MPLSLABELSTACKSECTION5 74 octetArray
MPLSLABELSTACKSECTION6 75 octetArray
MPLSLABELSTACKSECTION7 76 octetArray
MPLSLABELSTACKSECTION8 77 octetArray
MPLSLABELSTACKSECTION9 78 octetArray
MPLSLABELSTACKSECTION10 79 octetArray
DESTINATIONMACADDRESS 80 macAddress
POSTSOURCEMACADDRESS 81 macAddress
INTERFACENAME 82 string
INTERFACEDESCRIPTION 83 string
SAMPLERNAME 84 string
OCTETTOTALCOUNT 85 unsigned64
PACKETTOTALCOUNT 86 unsigned64
FLAGSANDSAMPLERID 87 unsigned32
FRAGMENTOFFSET 88 unsigned16
FORWARDINGSTATUS 89 unsigned8
MPLSVPNROUTEDISTINGUISHER 90 octetArray
MPLSTOPLABELPREFIXLENGTH 91 unsigned8
SRCTRAFFICINDEX 92 unsigned32
DSTTRAFFICINDEX 93 unsigned32
APPLICATIONDESCRIPTION 94 string
APPLICATIONID 95 octetArray
APPLICATIONNAME 96 string
POSTIPDIFFSERVCODEPOINT 98 unsigned8
MULTICASTREPLICATIONFACTOR 99 unsigned32
CLASSNAME 100 string
CLASSIFICATIONENGINEID 101 unsigned8
LAYER2PACKETSECTIONOFFSET 102 unsigned16
LAYER2PACKETSECTIONSIZE 103 unsigned16
LAYER2PACKETSECTIONDATA 104 octetArray
BGPNEXTADJACENTASNUMBER 128 unsigned32
BGPPREVADJACENTASNUMBER 129 unsigned32
EXPORTERIPV4ADDRESS 130 ipv4Address
EXPORTERIPV6ADDRESS 131 ipv6Address
DROPPEDOCTETDELTACOUNT 132 unsigned64
DROPPEDPACKETDELTACOUNT 133 unsigned64
DROPPEDOCTETTOTALCOUNT 134 unsigned64
DROPPEDPACKETTOTALCOUNT 135 unsigned64
FLOWENDREASON 136 unsigned8
COMMONPROPERTIESID 137 unsigned64
OBSERVATIONPOINTID 138 unsigned64
ICMPTYPECODEIPV6 139 unsigned16
MPLSTOPLABELIPV6ADDRESS 140 ipv6Address
LINECARDID 141 unsigned32
PORTID 142 unsigned32
METERINGPROCESSID 143 unsigned32
EXPORTINGPROCESSID 144 unsigned32
TEMPLATEID 145 unsigned16
WLANCHANNELID 146 unsigned8
WLANSSID 147 string
FLOWID 148 unsigned64
OBSERVATIONDOMAINID 149 unsigned32
FLOWSTARTSECONDS 150 dateTimeSeconds
FLOWENDSECONDS 151 dateTimeSeconds
FLOWSTARTMILLISECONDS 152 dateTimeMilliseconds
FLOWENDMILLISECONDS 153 dateTimeMilliseconds
FLOWSTARTMICROSECONDS 154 dateTimeMicroseconds
FLOWENDMICROSECONDS 155 dateTimeMicroseconds
FLOWSTARTNANOSECONDS 156 dateTimeNanoseconds
FLOWENDNANOSECONDS 157 dateTimeNanoseconds
FLOWSTARTDELTAMICROSECONDS 158 unsigned32
FLOWENDDELTAMICROSECONDS 159 unsigned32
SYSTEMINITTIMEMILLISECONDS 160 dateTimeMilliseconds
FLOWDURATIONMILLISECONDS 161 unsigned32
FLOWDURATIONMICROSECONDS 162 unsigned32
OBSERVEDFLOWTOTALCOUNT 163 unsigned64
IGNOREDPACKETTOTALCOUNT 164 unsigned64
IGNOREDOCTETTOTALCOUNT 165 unsigned64
NOTSENTFLOWTOTALCOUNT 166 unsigned64
NOTSENTPACKETTOTALCOUNT 167 unsigned64
NOTSENTOCTETTOTALCOUNT 168 unsigned64
DESTINATIONIPV6PREFIX 169 ipv6Address
SOURCEIPV6PREFIX 170 ipv6Address
POSTOCTETTOTALCOUNT 171 unsigned64
POSTPACKETTOTALCOUNT 172 unsigned64
FLOWKEYINDICATOR 173 unsigned64
POSTMCASTPACKETTOTALCOUNT 174 unsigned64
POSTMCASTOCTETTOTALCOUNT 175 unsigned64
ICMPTYPEIPV4 176 unsigned8
ICMPCODEIPV4 177 unsigned8
ICMPTYPEIPV6 178 unsigned8
ICMPCODEIPV6 179 unsigned8
UDPSOURCEPORT 180 unsigned16
UDPDESTINATIONPORT 181 unsigned16
TCPSOURCEPORT 182 unsigned16
TCPDESTINATIONPORT 183 unsigned16
TCPSEQUENCENUMBER 184 unsigned32
TCPACKNOWLEDGEMENTNUMBER 185 unsigned32
TCPWINDOWSIZE 186 unsigned16
TCPURGENTPOINTER 187 unsigned16
TCPHEADERLENGTH 188 unsigned8
IPHEADERLENGTH 189 unsigned8
TOTALLENGTHIPV4 190 unsigned16
PAYLOADLENGTHIPV6 191 unsigned16
IPTTL 192 unsigned8
NEXTHEADERIPV6 193 unsigned8
MPLSPAYLOADLENGTH 194 unsigned32
IPDIFFSERVCODEPOINT 195 unsigned8
IPPRECEDENCE 196 unsigned8
FRAGMENTFLAGS 197 unsigned8
OCTETDELTASUMOFSQUARES 198 unsigned64
OCTETTOTALSUMOFSQUARES 199 unsigned64
MPLSTOPLABELTTL 200 unsigned8
MPLSLABELSTACKLENGTH 201 unsigned32
MPLSLABELSTACKDEPTH 202 unsigned32
MPLSTOPLABELEXP 203 unsigned8
IPPAYLOADLENGTH 204 unsigned32
UDPMESSAGELENGTH 205 unsigned16
ISMULTICAST 206 unsigned8
IPV4IHL 207 unsigned8
IPV4OPTIONS 208 unsigned32
TCPOPTIONS 209 unsigned64
PADDINGOCTETS 210 octetArray
COLLECTORIPV4ADDRESS 211 ipv4Address
COLLECTORIPV6ADDRESS 212 ipv6Address
EXPORTINTERFACE 213 unsigned32
EXPORTPROTOCOLVERSION 214 unsigned8
EXPORTTRANSPORTPROTOCOL 215 unsigned8
COLLECTORTRANSPORTPORT 216 unsigned16
EXPORTERTRANSPORTPORT 217 unsigned16
TCPSYNTOTALCOUNT 218 unsigned64
TCPFINTOTALCOUNT 219 unsigned64
TCPRSTTOTALCOUNT 220 unsigned64
TCPPSHTOTALCOUNT 221 unsigned64
TCPACKTOTALCOUNT 222 unsigned64
TCPURGTOTALCOUNT 223 unsigned64
IPTOTALLENGTH 224 unsigned64
POSTNATSOURCEIPV4ADDRESS 225 ipv4Address
POSTNATDESTINATIONIPV4ADDRESS 226 ipv4Address
POSTNAPTSOURCETRANSPORTPORT 227 unsigned16
POSTNAPTDESTINATIONTRANSPORTPORT 228 unsigned16
NATORIGINATINGADDRESSREALM 229 unsigned8
NATEVENT 230 unsigned8
INITIATOROCTETS 231 unsigned64
RESPONDEROCTETS 232 unsigned64
FIREWALLEVENT 233 unsigned8
INGRESSVRFID 234 unsigned32
EGRESSVRFID 235 unsigned32
VRFNAME 236 string
POSTMPLSTOPLABELEXP 237 unsigned8
TCPWINDOWSCALE 238 unsigned16
BIFLOWDIRECTION 239 unsigned8
ETHERNETHEADERLENGTH 240 unsigned8
ETHERNETPAYLOADLENGTH 241 unsigned16
ETHERNETTOTALLENGTH 242 unsigned16
DOT1QVLANID 243 unsigned16
DOT1QPRIORITY 244 unsigned8
DOT1QCUSTOMERVLANID 245 unsigned16
DOT1QCUSTOMERPRIORITY 246 unsigned8
METROEVCID 247 string
METROEVCTYPE 248 unsigned8
PSEUDOWIREID 249 unsigned32
PSEUDOWIRETYPE 250 unsigned16
PSEUDOWIRECONTROLWORD 251 unsigned32
INGRESSPHYSICALINTERFACE 252 unsigned32
EGRESSPHYSICALINTERFACE 253 unsigned32
POSTDOT1QVLANID 254 unsigned16
POSTDOT1QCUSTOMERVLANID 255 unsigned16
ETHERNETTYPE 256 unsigned16
POSTIPPRECEDENCE 257 unsigned8
COLLECTIONTIMEMILLISECONDS 258 dateTimeMilliseconds
EXPORTSCTPSTREAMID 259 unsigned16
MAXEXPORTSECONDS 260 dateTimeSeconds
MAXFLOWENDSECONDS 261 dateTimeSeconds
MESSAGEMD5CHECKSUM 262 octetArray
MESSAGESCOPE 263 unsigned8
MINEXPORTSECONDS 264 dateTimeSeconds
MINFLOWSTARTSECONDS 265 dateTimeSeconds
OPAQUEOCTETS 266 octetArray
SESSIONSCOPE 267 unsigned8
MAXFLOWENDMICROSECONDS 268 dateTimeMicroseconds
MAXFLOWENDMILLISECONDS 269 dateTimeMilliseconds
MAXFLOWENDNANOSECONDS 270 dateTimeNanoseconds
MINFLOWSTARTMICROSECONDS 271 dateTimeMicroseconds
MINFLOWSTARTMILLISECONDS 272 dateTimeMilliseconds
MINFLOWSTARTNANOSECONDS 273 dateTimeNanoseconds
COLLECTORCERTIFICATE 274 octetArray
EXPORTERCERTIFICATE 275 octetArray
DATARECORDSRELIABILITY 276 boolean
OBSERVATIONPOINTTYPE 277 unsigned8
NEWCONNECTIONDELTACOUNT 278 unsigned32
CONNECTIONSUMDURATIONSECONDS 279 unsigned64
CONNECTIONTRANSACTIONID 280 unsigned64
POSTNATSOURCEIPV6ADDRESS 281 ipv6Address
POSTNATDESTINATIONIPV6ADDRESS 282 ipv6Address
NATPOOLID 283 unsigned32
NATPOOLNAME 284 string
ANONYMIZATIONFLAGS 285 unsigned16
ANONYMIZATIONTECHNIQUE 286 unsigned16
INFORMATIONELEMENTINDEX 287 unsigned16
P2PTECHNOLOGY 288 string
TUNNELTECHNOLOGY 289 string
ENCRYPTEDTECHNOLOGY 290 string
BASICLIST 291 basicList
SUBTEMPLATELIST 292 subTemplateList
SUBTEMPLATEMULTILIST 293 subTemplateMultiList
BGPVALIDITYSTATE 294 unsigned8
IPSECSPI 295 unsigned32
GREKEY 296 unsigned32
NATTYPE 297 unsigned8
INITIATORPACKETS 298 unsigned64
RESPONDERPACKETS 299 unsigned64
OBSERVATIONDOMAINNAME 300 string
SELECTIONSEQUENCEID 301 unsigned64
SELECTORID 302 unsigned64
INFORMATIONELEMENTID 303 unsigned16
SELECTORALGORITHM 304 unsigned16
SAMPLINGPACKETINTERVAL 305 unsigned32
SAMPLINGPACKETSPACE 306 unsigned32
SAMPLINGTIMEINTERVAL 307 unsigned32
SAMPLINGTIMESPACE 308 unsigned32
SAMPLINGSIZE 309 unsigned32
SAMPLINGPOPULATION 310 unsigned32
SAMPLINGPROBABILITY 311 float64
DATALINKFRAMESIZE 312 unsigned16
IPHEADERPACKETSECTION 313 octetArray
IPPAYLOADPACKETSECTION 314 octetArray
DATALINKFRAMESECTION 315 octetArray
MPLSLABELSTACKSECTION 316 octetArray
MPLSPAYLOADPACKETSECTION 317 octetArray
SELECTORIDTOTALPKTSOBSERVED 318 unsigned64
SELECTORIDTOTALPKTSSELECTED 319 unsigned64
ABSOLUTEERROR 320 float64
RELATIVEERROR 321 float64
OBSERVATIONTIMESECONDS 322 dateTimeSeconds
OBSERVATIONTIMEMILLISECONDS 323 dateTimeMilliseconds
OBSERVATIONTIMEMICROSECONDS 324 dateTimeMicroseconds
OBSERVATIONTIMENANOSECONDS 325 dateTimeNanoseconds
DIGESTHASHVALUE 326 unsigned64
HASHIPPAYLOADOFFSET 327 unsigned64
HASHIPPAYLOADSIZE 328 unsigned64
HASHOUTPUTRANGEMIN 329 unsigned64
HASHOUTPUTRANGEMAX 330 unsigned64
HASHSELECTEDRANGEMIN 331 unsigned64
HASHSELECTEDRANGEMAX 332 unsigned64
HASHDIGESTOUTPUT 333 boolean
HASHINITIALISERVALUE 334 unsigned64
SELECTORNAME 335 string
UPPERCILIMIT 336 float64
LOWERCILIMIT 337 float64
CONFIDENCELEVEL 338 float64
INFORMATIONELEMENTDATATYPE 339 unsigned8
INFORMATIONELEMENTDESCRIPTION 340 string
INFORMATIONELEMENTNAME 341 string
INFORMATIONELEMENTRANGEBEGIN 342 unsigned64
INFORMATIONELEMENTRANGEEND 343 unsigned64
INFORMATIONELEMENTSEMANTICS 344 unsigned8
INFORMATIONELEMENTUNITS 345 unsigned16
PRIVATEENTERPRISENUMBER 346 unsigned32
VIRTUALSTATIONINTERFACEID 347 octetArray
VIRTUALSTATIONINTERFACENAME 348 string
VIRTUALSTATIONUUID 349 octetArray
VIRTUALSTATIONNAME 350 string
LAYER2SEGMENTID 351 unsigned64
LAYER2OCTETDELTACOUNT 352 unsigned64
LAYER2OCTETTOTALCOUNT 353 unsigned64
INGRESSUNICASTPACKETTOTALCOUNT 354 unsigned64
INGRESSMULTICASTPACKETTOTALCOUNT 355 unsigned64
INGRESSBROADCASTPACKETTOTALCOUNT 356 unsigned64
EGRESSUNICASTPACKETTOTALCOUNT 357 unsigned64
EGRESSBROADCASTPACKETTOTALCOUNT 358 unsigned64
MONITORINGINTERVALSTARTMILLISECONDS 359 dateTimeMilliseconds
MONITORINGINTERVALENDMILLISECONDS 360 dateTimeMilliseconds
PORTRANGESTART 361 unsigned16
PORTRANGEEND 362 unsigned16
PORTRANGESTEPSIZE 363 unsigned16
PORTRANGENUMPORTS 364 unsigned16
STAMACADDRESS 365 macAddress
STAIPV4ADDRESS 366 ipv4Address
WTPMACADDRESS 367 macAddress
INGRESSINTERFACETYPE 368 unsigned32
EGRESSINTERFACETYPE 369 unsigned32
RTPSEQUENCENUMBER 370 unsigned16
USERNAME 371 string
APPLICATIONCATEGORYNAME 372 string
APPLICATIONSUBCATEGORYNAME 373 string
APPLICATIONGROUPNAME 374 string
ORIGINALFLOWSPRESENT 375 unsigned64
ORIGINALFLOWSINITIATED 376 unsigned64
ORIGINALFLOWSCOMPLETED 377 unsigned64
DISTINCTCOUNTOFSOURCEIPADDRESS 378 unsigned64
DISTINCTCOUNTOFDESTINATIONIPADDRESS 379 unsigned64
DISTINCTCOUNTOFSOURCEIPV4ADDRESS 380 unsigned32
DISTINCTCOUNTOFDESTINATIONIPV4ADDRESS 381 unsigned32
DISTINCTCOUNTOFSOURCEIPV6ADDRESS 382 unsigned64
DISTINCTCOUNTOFDESTINATIONIPV6ADDRESS 383 unsigned64
VALUEDISTRIBUTIONMETHOD 384 unsigned8
RFC3550JITTERMILLISECONDS 385 unsigned32
RFC3550JITTERMICROSECONDS 386 unsigned32
RFC3550JITTERNANOSECONDS 387 unsigned32
DOT1QDEI 388 boolean
DOT1QCUSTOMERDEI 389 boolean
FLOWSELECTORALGORITHM 390 unsigned16
FLOWSELECTEDOCTETDELTACOUNT 391 unsigned64
FLOWSELECTEDPACKETDELTACOUNT 392 unsigned64
FLOWSELECTEDFLOWDELTACOUNT 393 unsigned64
SELECTORIDTOTALFLOWSOBSERVED 394 unsigned64
SELECTORIDTOTALFLOWSSELECTED 395 unsigned64
SAMPLINGFLOWINTERVAL 396 unsigned64
SAMPLINGFLOWSPACING 397 unsigned64
FLOWSAMPLINGTIMEINTERVAL 398 unsigned64
FLOWSAMPLINGTIMESPACING 399 unsigned64
HASHFLOWDOMAIN 400 unsigned16
TRANSPORTOCTETDELTACOUNT 401 unsigned64
TRANSPORTPACKETDELTACOUNT 402 unsigned64
ORIGINALEXPORTERIPV4ADDRESS 403 ipv4Address
ORIGINALEXPORTERIPV6ADDRESS 404 ipv6Address
ORIGINALOBSERVATIONDOMAINID 405 unsigned32
INTERMEDIATEPROCESSID 406 unsigned32
IGNOREDDATARECORDTOTALCOUNT 407 unsigned64
DATALINKFRAMETYPE 408 unsigned16
SECTIONOFFSET 409 unsigned16
SECTIONEXPORTEDOCTETS 410 unsigned16
DOT1QSERVICEINSTANCETAG 411 octetArray
DOT1QSERVICEINSTANCEID 412 unsigned32
DOT1QSERVICEINSTANCEPRIORITY 413 unsigned8
DOT1QCUSTOMERSOURCEMACADDRESS 414 macAddress
DOT1QCUSTOMERDESTINATIONMACADDRESS 415 macAddress
POSTLAYER2OCTETDELTACOUNT 417 unsigned64
POSTMCASTLAYER2OCTETDELTACOUNT 418 unsigned64
POSTLAYER2OCTETTOTALCOUNT 420 unsigned64
POSTMCASTLAYER2OCTETTOTALCOUNT 421 unsigned64
MINIMUMLAYER2TOTALLENGTH 422 unsigned64
MAXIMUMLAYER2TOTALLENGTH 423 unsigned64
DROPPEDLAYER2OCTETDELTACOUNT 424 unsigned64
DROPPEDLAYER2OCTETTOTALCOUNT 425 unsigned64
IGNOREDLAYER2OCTETTOTALCOUNT 426 unsigned64
NOTSENTLAYER2OCTETTOTALCOUNT 427 unsigned64
LAYER2OCTETDELTASUMOFSQUARES 428 unsigned64
LAYER2OCTETTOTALSUMOFSQUARES 429 unsigned64
LAYER2FRAMEDELTACOUNT 430 unsigned64
LAYER2FRAMETOTALCOUNT 431 unsigned64
PSEUDOWIREDESTINATIONIPV4ADDRESS 432 ipv4Address
IGNOREDLAYER2FRAMETOTALCOUNT 433 unsigned64
MIBOBJECTVALUEINTEGER 434 signed32
MIBOBJECTVALUEOCTETSTRING 435 octetArray
MIBOBJECTVALUEOID 436 octetArray
MIBOBJECTVALUEBITS 437 octetArray
MIBOBJECTVALUEIPADDRESS 438 ipv4Address
MIBOBJECTVALUECOUNTER 439 unsigned64
MIBOBJECTVALUEGAUGE 440 unsigned32
MIBOBJECTVALUETIMETICKS 441 unsigned32
MIBOBJECTVALUEUNSIGNED 442 unsigned32
MIBOBJECTVALUETABLE 443 subTemplateList
MIBOBJECTVALUEROW 444 subTemplateList
MIBOBJECTIDENTIFIER 445 octetArray
MIBSUBIDENTIFIER 446 unsigned32
MIBINDEXINDICATOR 447 unsigned64
MIBCAPTURETIMESEMANTICS 448 unsigned8
MIBCONTEXTENGINEID 449 octetArray
MIBCONTEXTNAME 450 string
MIBOBJECTNAME 451 string
MIBOBJECTDESCRIPTION 452 string
MIBOBJECTSYNTAX 453 string
MIBMODULENAME 454 string
MOBILEIMSI 455 string
MOBILEMSISDN 456 string
HTTPSTATUSCODE 457 unsigned16
SOURCETRANSPORTPORTSLIMIT 458 unsigned16
HTTPREQUESTMETHOD 459 string
HTTPREQUESTHOST 460 string
HTTPREQUESTTARGET 461 string
HTTPMESSAGEVERSION 462 string
NATINSTANCEID 463 unsigned32
INTERNALADDRESSREALM 464 octetArray
EXTERNALADDRESSREALM 465 octetArray
NATQUOTAEXCEEDEDEVENT 466 unsigned32
NATTHRESHOLDEVENT 467 unsigned32
HTTPUSERAGENT 468 string
HTTPCONTENTTYPE 469 string
HTTPREASONPHRASE 470 string
MAXSESSIONENTRIES 471 unsigned32
MAXBIBENTRIES 472 unsigned32
MAXENTRIESPERUSER 473 unsigned32
MAXSUBSCRIBERS 474 unsigned32
MAXFRAGMENTSPENDINGREASSEMBLY 475 unsigned32
ADDRESSPOOLHIGHTHRESHOLD 476 unsigned32
ADDRESSPOOLLOWTHRESHOLD 477 unsigned32
ADDRESSPORTMAPPINGHIGHTHRESHOLD 478 unsigned32
ADDRESSPORTMAPPINGLOWTHRESHOLD 479 unsigned32
ADDRESSPORTMAPPINGPERUSERHIGHTHRESHOLD 480 unsigned32
GLOBALADDRESSMAPPINGHIGHTHRESHOLD 481 unsigned32
VPNIDENTIFIER 482 octetArray
BGPCOMMUNITY 483 unsigned32
BGPSOURCECOMMUNITYLIST 484 basicList
BGPDESTINATIONCOMMUNITYLIST 485 basicList
BGPEXTENDEDCOMMUNITY 486 octetArray
BGPSOURCEEXTENDEDCOMMUNITYLIST 487 basicList
BGPDESTINATIONEXTENDEDCOMMUNITYLIST 488 basicList
BGPLARGECOMMUNITY 489 octetArray
BGPSOURCELARGECOMMUNITYLIST 490 basicList
BGPDESTINATIONLARGECOMMUNITYLIST 491 basicList

Declaring Netflow Collector UDF

To use the Netflow v9/IPFIX UDF, you first need to define it as a function. See the topics Writing a Java UDF in the Integration Guide and CREATE FUNCTION in the SQLstream Streaming SQL Reference Guide for more details on defining functions.

A simple declaration for determining the netflow v9/IPFIX parameters is as follows:

Netflow v9 example :

  1. Create the UDF netflowCollector as below in the below example.

  2. Configure the router/switch to dump the required CFLOW data to particular IP and Port.

  3. Execute the UDF to listen to the IP and Port configured in step 2 above.

Values Passed to the UDF as Parameter

To execute the UDF in the sqllineClient, three parameters are required to be passed, the details are as followed :

  • bind_address : The IP address from the router or switch from which the packet data will be received.
  • port : The port configured along with the IP to get the packet data.
  • file_path : Need to be set to '' in case of v9 and for IPFIX set to the requird path where the enterprise template is stored.

Example netflowCollector UDF

CREATE OR REPLACE SCHEMA test;
DROP SCHEMA test CASCADE;
CREATE OR REPLACE SCHEMA test;
SET SCHEMA 'test';
SET PATH 'test';

create or replace function netflowCollector(bind_address varchar(128), port varchar(64), file_path varchar(250))
returns table(
ROWTIME TIMESTAMP NOT NULL,
REPORTER BINARY(16),
SCOPE_SYSTEM BINARY(4),
OCTETDELTACOUNT BIGINT,
PACKETDELTACOUNT BIGINT,
PROTOCOLIDENTIFIER TINYINT,
SOURCETRANSPORTPORT INT,
SOURCEIPV4ADDRES BINARY(4),
SOURCEIPV4PREFIXLENGTH TINYINT,
INGRESSINTERFACE INT,
DESTINATIONTRANSPORTPORT INT,
DESTINATIONIPV4PREFIXLENGTH TINYINT,
EGRESSINTERFACE INT,
IPNEXTHOPIPV4ADDRESS BINARY(4),
BGPSOURCEASNUMBER INT,
BGPDESTINATIONASNUMBER INT,
BGPNEXTHOPIPV4ADDRESS BINARY(4),
FLOWENDSYSUPTIME BIGINT,
FLOWSTARTSYSUPTIME BIGINT,
POSTOCTETDELTACOUNT BIGINT,
POSTPACKETDELTACOUNT BIGINT,
SOURCEIPV6ADDRESS BINARY(16),
DESTINATIONIPV6ADDRESS BINARY(16),
SOURCEIPV6PREFIXLENGTH TINYINT,
DESTINATIONIPV6PREFIXLENGTH TINYINT,
FLOWLABELIPV6 BIGINT,
ICMPTYPECODEIPV4 INT,
SAMPLINGALGORITHM TINYINT,
EXPORTEDOCTETTOTALCOUNT BIGINT,
EXPORTEDMESSAGETOTALCOUNT BIGINT,
EXPORTEDFLOWRECORDTOTALCOUNT BIGINT,
SAMPLERID TINYINT,
SAMPLERMODE TINYINT,
SAMPLERRANDOMINTERVAL BIGINT,
SAMPLERNAME VARCHAR(6),
FORWARDINGSTATUS TINYINT
)
LANGUAGE EXTERNAL
NO SQL
NO STATE
EXTERNAL NAME 'plugin/netflow';

--Create the View by calling the created UDF
create or replace view nf as select stream * from stream(test.netflowCollector('127.0.0.1', '2058',''));

--Select the data from the stream for the selected columns mentioned at the time of creating the UDF
select stream rowtime, * from nf ;

Netflow v9 Sample Output :

ROWTIME                       2022-05-10 03:53:04.635
REPORTER
SCOPE_SYSTEM
OCTETDELTACOUNT               273
PACKETDELTACOUNT              3
PROTOCOLIDENTIFIER            17
SOURCETRANSPORTPORT           58271
SOURCEIPV4ADDRES
SOURCEIPV4PREFIXLENGTH
INGRESSINTERFACE              416
DESTINATIONTRANSPORTPORT      443
DESTINATIONIPV4PREFIXLENGTH
EGRESSINTERFACE               829
IPNEXTHOPIPV4ADDRESS
BGPSOURCEASNUMBER             0
BGPDESTINATIONASNUMBER        0
BGPNEXTHOPIPV4ADDRESS
FLOWENDSYSUPTIME              2359276866
FLOWSTARTSYSUPTIME            2359256525
POSTOCTETDELTACOUNT
POSTPACKETDELTACOUNT
SOURCEIPV6ADDRESS             26000001928B2BAF6C89B0EE81DB3793
DESTINATIONIPV6ADDRESS        2607F8B04017000A0000000000000018
SOURCEIPV6PREFIXLENGTH        44
DESTINATIONIPV6PREFIXLENGTH   32
FLOWLABELIPV6                 0
ICMPTYPECODEIPV4
SAMPLINGALGORITHM
EXPORTEDOCTETTOTALCOUNT
EXPORTEDMESSAGETOTALCOUNT
EXPORTEDFLOWRECORDTOTALCOUNT
SAMPLERID                     4
SAMPLERMODE
SAMPLERRANDOMINTERVAL
SAMPLERNAME
FORWARDINGSTATUS              64

Netflow IPFIX Example

CREATE OR REPLACE SCHEMA test;
DROP SCHEMA test CASCADE;
CREATE OR REPLACE SCHEMA test;
SET SCHEMA 'test';

create or replace function netflowCollector(bind_address varchar(128), port varchar(64), file_path varchar(250))
returns table(
ROWTIME TIMESTAMP NOT NULL,
NETFLOW_VERSION TINYINT,
EXPORT_TIME BIGINT,
SESSION_ID BIGINT,
TIMESTAMPS BIGINT,
NAT_BINDING_TIMER INT,
GMT_OFFSET CHAR(5),
PORT_RNG_ALLOC_TIMESTAMP BIGINT,
PORT_RNG_DEALLOC_TIMESTAMP BIGINT,
LAST_FLOW_TIMESTAMP BIGINT,
RAT_TYPE SMALLINT,
NETWORK_INST_NAME VARCHAR(65535),
ROUTE_CNXT_NAME VARCHAR(65535),
IMEI CHAR(16),
BND_TERMINATION_CAUSE SMALLINT,
UPF_NAME VARCHAR(65535),
SESSION_EVENT SMALLINT,
SESSION_PER_PUB_IP SMALLINT,
CHARGING_ID VARCHAR(65535),
SERVC_NW_ID VARCHAR(65535),
SGWC_IPV4_ADDR BINARY(4),
SGWC_IPV6_ADDR BINARY(16),
PGWC_IPV4_ADDR BINARY(4),
PGWC_IPV6_ADDR BINARY(16),
SUPI VARCHAR(65535),
GPSI VARCHAR(65535),
APN_NAME VARCHAR(65535),
NAT_POOL_NAME VARCHAR(65535),
SOURCEIPV4ADDRESS BINARY(4),
SOURCEIPV6ADDRESS BINARY(16),
MOBILEIMSI VARCHAR(65535),
MOBILEMSISDN VARCHAR(65535),
NATEVENT SMALLINT,
POSTNATSOURCEIPV4ADDRESS BINARY(4),
PORTRANGESTART INT,
PORTRANGEEND INT,
NATINSTANCEID BIGINT
)
LANGUAGE EXTERNAL
NO SQL
NO STATE
EXTERNAL NAME 'plugin/netflow';

--Create the view by calling the UDF
create or replace view nf as select stream * from stream(test.netflowCollector('127.0.0.1', '2058','unitsql/plugins/netflow'));

--Execute the select statement to get the stream of data
select stream rowtime,* from nf ;

Netflow IPFIX Sample output :

ROWTIME                     2022-05-10 03:55:35.217
NETFLOW_VERSION             10
EXPORT_TIME                 1596798127
SESSION_ID                  8358680908399640577
TIMESTAMPS                  1596798127602
NAT_BINDING_TIMER
GMT_OFFSET                  +0200
PORT_RNG_ALLOC_TIMESTAMP
PORT_RNG_DEALLOC_TIMESTAMP
LAST_FLOW_TIMESTAMP
RAT_TYPE                    0
NETWORK_INST_NAME
ROUTE_CNXT_NAME
IMEI                        999900000000030
BND_TERMINATION_CAUSE
UPF_NAME
SESSION_EVENT               1
SESSION_PER_PUB_IP
CHARGING_ID
SERVC_NW_ID
SGWC_IPV4_ADDR              00000000
SGWC_IPV6_ADDR              00000000000000000000000000000000
PGWC_IPV4_ADDR              00000000
PGWC_IPV6_ADDR              00000000000000000000000000000000
SUPI
GPSI
APN_NAME
NAT_POOL_NAME
SOURCEIPV4ADDRESS           31000005
SOURCEIPV6ADDRESS           00000000000000000000000000000000
MOBILEIMSI                  226041000000003
MOBILEMSISDN                40700000003
NATEVENT
POSTNATSOURCEIPV4ADDRESS
PORTRANGESTART
PORTRANGEEND
NATINSTANCEID

Note: The file_path should be set in order to read the enterprise related CSV file. The file path needs to be given here.