Network Management

Week 7, Mar 14
LT-412, Mon 4:15-6:45 pm

Exam notes

It is important that you understand
See the study guide, #5 and #9 in particular.


NMS discovery
cisco ping mib / row creation
augmentation
Enterprise (private) mibs; cisco examples
sysObjectID
RMON


Network Management Systems

The first step is to discover nodes. This can be done via manual configuration, but is more often done with ping, or possibly with some other form of TCP scan (eg connection requests to  port 139, the netbios port), or possibly probes to UDP port 161, the SNMP port. The range of desired hosts is almost always specified manually.

The next phase is to probe the detected host to figure out "what it is" and "what it is doing". This, especially for the latter sense, is also called capability detection. One common way to proceed is with a TCP port scan, coupled with actual connections to the active ports.

However, we can also get a lot of information from SNMP. (Actually, we can get the effect of a TCP port scan by retrieving the SNMP TCP Group, in particular the Connections Table).  Some particular things to look at include the SysOID value, which the NMS can use as a lookup key to figure out what kinds of "private" SNMP queries to make. For example, a cisco switch would send back a SysOID that began 1.3.6.1.4.1.9, and a net-snmp agent returns 1.3.6.1.4.1.8072.3.2.10.   
From NET-SNMP-MIB.txt:
     
1.3.6.1.4.1 private
private.8072 net-snmp
net-snmp.1
 netSnmpObjects
net-snmp.3 netSnmpEnumerations
netSnmpEnumerations.2 netSNMPAgentOIDs
netSNMPAgentOIDs.10 not specifically mentioned

Cisco ping mib

The cisco ping mib contains a real example of SNMPv2-style row-creation. The idea here is for the manager to create a row that tells the agent what neighbors to ping. Once the row is ready, the agent goes ahead and does the pinging, and leaves the results in that row.

In the ping mib, the manager chooses a pseudorandom value to use as the row index (ciscoPingSerialNumber). Also, the row is large enough that it might be reasonable to be built piecemeal, although createAndGo is explicitly mentioned.

column name
 type
 additional information
ciscoPingSerialNumber INTEGER mandatory
ciscoPingProtocol CiscoNetworkProtocol mandatory
ciscoPingAddress CiscoNetworkAddress mandatory
ciscoPingPacketCount INTEGER default 5
ciscoPingPacketSize INTEGER default 100
ciscoPingPacketTimeout INTEGER default 2000ms
ciscoPingDelay INTEGER default 0
ciscoPingTrapOnCompletion TruthValue default false
ciscoPingSentPackets Counter set by agent
ciscoPingReceivedPackets Counter set by agent
ciscoPingMinRtt INTEGER set by agent
ciscoPingAvgRtt INTEGER set by agent
ciscoPingMaxRtt INTEGER set by agent
ciscoPingCompleted TruthValue set by agent
ciscoPingEntryOwner OwnerString an identifier, eg hostname, for the manager
ciscoPingEntryStatus RowStatus SNMPv2 type
ciscoPingVrfName OCTET STRING default ""; used for VPNs

The manager sets the lines in bold in the usual way, after picking a tentative serial number (and having it succeed!). The manager then sets the ciscoPingEntryStatus column of the new row to active, which enables the whole thing and causes the agent to:
Note the existence of default values for some columns. These would be implemented by having the agent fill in the default values for any of these columns that are not specified in the initial row-creating SET request.

TestAndIncr semaphore versus table creation

The RowStatus values createAndGo and createAndWait act as semaphores: if another create has been executed on that index, they fail. The TestAndIncr is another, more explicit, semaphore: a SET request includes a value for the TestAndIncr. The agent response is to verify that the value set matches the TestAndIncr's current value; if so, the value is incremented. Like all SETs, if this SET command fails (because someone else TestAndIncred the object just before), then all the other SET commands in the same SET_REQUEST also fail. Note, however, that the semantics for updating the particular TestAndIncr column in question are not standard.

Here's how we would use this.
Note that to update a value in a row that is subject to contention, we must:
The last two can be done in parallel.

Note that use of this kind of semaphore is "voluntary"; managers agree to it as a way of maintaining agent consistency. The agent may not actually enforce it. Enforcement would mean disallowing any SET actions to the row unless the semaphore had the value inuse.

Table Expansion

SNMPv2 allows a new MIB to expand an existing table, without simply replacing the definition. Table expansion can take one of three forms. Augmentation means the new table simply adds more columns to the base table , as ifXTable does. The new table explicitly uses the same INDEX as the base table, meaning that it does not contain its own INDEX. Other options are "dense tables", which expand the base table but which may add rows as well as columns, and "sparse tables", which expand the base table but only in selected rows. In the ETHERLIKE-MIB example above, the StatsTable is an example of a sparse table: we extend only those rows of the interfaces table that represent Ethernet-like devices. The CollisionCount table, on the other hand, is an example of a dense table: we extend selected rows of the interfaces table with multiple values (one corresponding to each CollisionCount recorded).

Table expansion does not correspond to the database join operation, except in limited cases (eg augmentation can be thought of as the join of two tables that have the same index).

Augmented tables

The keyword AUGMENTS often appears in MIB files, but usually as a comment; that is, the actual definition doesn't use the SNMPv2 extension. However, a genuine use appears in HOST-RESOURCES-MIB.txt. First is the definition of hrSWRunTable,
with the following Entry definition. The table is indexed by hrSWRunIndex, its first column.

hrSWRunEntry OBJECT-TYPE
    SYNTAX     HrSWRunEntry
    MAX-ACCESS not-accessible
    STATUS     current
    ....
    INDEX { hrSWRunIndex }
    ::= { hrSWRunTable 1 }

Then the hrSWRunPerfTable table is defined to augment the above through the addition of additional columns (hrSWRunPerfCPU and hrSWRunPerfMem), indexed the same way:

    hrSWRunPerfEntry OBJECT-TYPE
        SYNTAX HrSWRunPerfEntry           -- SEQUENCE { hrSWRunPerfCPU, hrSWRunPerfMem }
        ...
        AUGMENTS { hrSWRunEntry }
        ::= {hrSWRunPerfTable 1}

The AUGMENTS keyword means that the index of this new hrSWRunPerfTable is the same as that of hrSWRunTable, that is, hrSWRunIndex. The AUGMENTS { hrSWRunEntry } could be thought of as a standin for INDEX { hrSWRunIndex }, except that the AUGMENTS keyword is intended to suggest that every row of the base table is extended by the new table, and the latter (INDEX) form does not. Note, however, that in the (SNMPv1) extended interfaces table ifXTable in IF-MIB-V1SMI.MIB, the definition is:

ifXEntry OBJECT-TYPE
    SYNTAX IfXEntry
    ...
    INDEX { ifIndex }
    ::= { ifXTable 1 }

(in the SNMPv2 form of the ifXEntry table, the AUGMENTS keyword is used.) Note that the INDEX clause directly refers to the ifIndex column of the original ifTable (a field not present in ifXTable).

Recall that identifiers such as hrSWRunEntry and hrSWRunIndex represent OIDs:
    hrSWRunIndex = hrSWRunEntry.1
The OID in the AUGMENTS clause is that of the other table's Entry; the OID in the INDEX clause is that of the actual column.

Note that hrSWRunEntry is also defined in this same MIB file (though perhaps hrSWRunPerfEntry was added later in the version history).

dense tables

It is also possible to add a "dense" table, where more than one row of the new table may correspond to one row of the base table. One possible example: the original table is a table of cards (hardware modules) in a switch, and the new table is a table of ports on cards. Some cards may contain multiple ports. Another example is the CollisionCount table of ETHERLIKE-MIB; each interface of the right kind is extended in multiple ways. In both of these examples, it is possible that, while some rows of the base table might be extended with multiplicity, other rows might not be extended at all.

The bottom line is that the new table is viewed as being "tacked on" to the base table, but is indexed by a combination of the base-table index and at least one new column of its own.

By way of example, suppose the base table is cardTable, for expansion cards added to a switch, with an Entry as follows:
    cardEntry OBJECT-TYPE
       SYNTAX   CardEntry      -- SEQUENCE { cardNumber, cardWhatever, ...}
       ...
       INDEX   {cardNumber}
      ::= {cardTable 1}

Then the new table might be portTable, to add information about all the ports on each with an Entry as follows:
    portEntry OBJECT-TYPE
       SYNTAX PortEntry       -- SEQUENCE {portNumber, portStats, ...}
       ...
       INDEX {cardNumber, portNumber }
       ::= {portTable 1}
Note that we use INDEX, rather than AUGMENTS, and we simply list the two fields used to index the additional data, even though the first field is part of the base table but not part of the second table. Rows on the first table would be slot1, slot2, slot3, etc; rows in the augmenting table might be
     (slot1, port1) -- single-port card
     (slot2, port1), (slot2, port2), (slot2, port3)   -- three-port card
     (slot3, port1), (slot3, port2)      -- two-port card
      ...

sparse tables

Adding a "sparse" table to a base table is like augmentation except that some added rows are empty. In keeping with the slot example above, suppose we have a new table cardExtraTable, with information only for certain cards. We might then define
    cardExtraTable OBJECT-TYPE
          SYNTAX CardExtraEntry      -- SEQUENCE {cardExtraInfo1, cardExtraInfo2, ...}
          ...
          INDEX {cardEntry }
          ::= {cardExtraTable 1}
Note the INDEX claue, not AUGMENTS, and the fact that the index is defined to be the entire cardEntry, not just the cardNumber (though it is hard to be sure, since this SNMPv2 option was seldom used). This is actually very similar to the SNMPv1 formulation of an AUGMENTed table.

The ETHERLIKE-MIB StatsTable is an example of a sparse table extension: we extend those rows in the interfaces table that correspond to Ethernet interfaces, and not the other rows. In the actual ETHERLIKE-MIB file, the index for the StatsTable is specified as dot3StatsIndex, and it is only later, in the definition of this field, that it is made clear that this is supposed to be the same as the ifIndex of the interfaces table; this slightly muddies the exposition.
 

Enterprise (private) mibs


I have a large directory of various mib files; one can search in it for "enterprises"
     egrep '{ *enterprises  *[0-9]+' *    # note the pattern here, in bold, between single quotes
 
2
 ibm
9
 cisco
11
 hp
16
 hpnr
23
 novell
36
 dec
77
 lanmanager
119
 nec
224
 lanOptics
232
 compaq
311
 microsoft
353
 atmForum
494
 madge
711
 lightstream
795
 adaptec
1123
 symbios
1608
 mylex
2021
 ucd-snmp
2636
 juniper
8072
 net-snmp
 
     
Getting MIB files: Lots of these are available from RFCs, sometimes in need of minor patches.  There are software tools available to extract the MIB file from a published RFC. Private ones are usually available from the company involved, though often with a surprising amount of difficulty.
 

A few cisco mibs

cisco BRIDGE-MIB.MIB (1994; you know this is old because it says "bridge" rather than "switch")
    spanning-tree info (dot1dStp group)
          lots of scalars
          dot1dStpPortTable
    forwarding table ?

Nothing on queue overflows!
    

cisco Catalyst 2900 ethernet switch
     CISCO-C2900-MIB-V1SMI.MIB
    
This is for this specific series of switch
c2900SysInfo:
    BoardVersion
    PeakBuffersUsed
    TotalBufferDepth
    AddrCapacity (forwarding table size)
    SNMP versions of the various LED displays

c2900SysConfig:
what to do for:
    security issues
    broadcast-packet storms

c2900ModuleTable: table of all included hardware modules
   
c2900Port:
    table of port entries. Each module may have several ports.
    address-learning stuff
    PortBufferCongestionControl

c2900BandwidthUsage:


Hardware environment:
  OLD-CISCO-ENV-MIB.MIB
    sample data for monitoring temperature
Test points:
  1. temperature of entering air
  2. temperature of air leaving the router
  3. +5 v power-supply voltage
  4. +12 v power-supply voltage
  5. -12 v power-supply voltage
  6. -5 v power-supply voltage
Also shutdown thresholds for the above. #1: 43° C, #2: 58° C (typical),
Also information on the stats as of the last shutdown (more important than last reboot!)
No table of history values!
Info on the environment-monitoring card itself.


Cisco TCP: CISCO-TCP-MIB-V1SMI.MIB

Notice that the intent here is to extend the existing TCP table. The additional fields:
Nothing on the current CongestionWindow size, though, or even the flavor of TCP congestion management (Tahoe, Reno, newReno)


Cisco queue management: CISCO-QUEUE-MIB

Queue type and subqueues
for subqueues:
    cQStatsQNumber	Integer32 (0..2147483647),
    cQStatsDepth	Gauge32,
    cQStatsMaxDepth	Integer32,
    cQStatsDiscards	Counter32
Is this all you need? There is no history table for recent queue bursts, for example. There is no average queue length, even.

Note the implicit problem with SNMP here: if you want something and the agent is not configured to provide it, it can be very difficult to add support. This is true of all software, however.

Here is a table of all the MIBs supported by the cisco 7200-series routers (it may be a couple years old). Note that some are cisco-specific, and some are generic extensions of mib-2.

ATM-MIB (RFC 1695) CISCO-SYSLOG-MIB
BGP4-MIB (RFC 1657) CISCO-TC
CISCO-AAA-SERVER-MIB CISCO-TCP-MIB
CISCO-AAL5-MIB CISCO-VLAN-IFTABLE-RELATIONSHIP-MIB
CISCO-ACCESS-ENVON-MIB CISCO-VPDN-MGMT-MIB
CISCO-ATM-EXT-MIB CISCO-VPDN-MGMT-EXT-MIB
CISCO-BULK-FILE-MIB ENTITY-MIB (RFC 2737)
CISCO-CDP-MIB ETHERLIKE-MIB (RFC 2665)
CISCO-CLASS-BASED-QOS-MIB EVENT-MIB (RFC 2981)
CISCO-CONFIG-COPY-MIB EXPRESSION-MIB
CISCO-CONFIG-MAN-MIB IF-MIB (RFC 2233)
CISCO-ENTITY-ALARM-MIB IP-LOCALPOOL-MIB
CISCO-ENTITY-ASSET-MIB MPLS-LDP-MIB
CISCO-ENTITY-EXT-MIB MPLS-LSR-MIB
CISCO-ENTITY-FRU-CONTROL-MIB MPLS-TE-MIB
CISCO-ENTITY-PFE -MIB MPLS-VPN-MIB
CISCO-ENTITY-SENSOR-MIB NOTIFICATION-LOG-MIB (RFC3014)
CISCO-ENTITY-VENDORTYPE-OID-MIB OLD-CISCO-CHASSIS-MIB
CISCO-ENVMON-MIB OLD-CISCO-CPU-MIB
CISCO-FLASH-MIB OLD-CISCO-INTERFACES-MIB
CISCO-FRAME-RELAY-MIB OLD-CISCO-IP-MIB
CISCO-FTP-CLIENT-MIB OLD-CISCO-MEMORY-MIB
CISCO-HSRP-EXT-MIB PIM-MIB (RFC 2934)
CISCO-HSRP-MIB RFC1213-MIB (MIB II)
CISCO-IETF-IP-MIB RFC1243-MIB (AppleTalk)
CISCO-IMAGE-MIB RFC1253-MIB (OSPF)
CISCO-IPMROUTE-MIB RFC1315-MIB (FRAME RELAY MIB)
CISCO-IP-STAT-MIB RFC2495-MIB (DS1)
CISCO-MEMORY-POOL-MIB RFC2496-MIB (DS3)
CISCO-NBAR-PROTOCOL-DISCOVERY-MIB RMON-MIB (RFC 1757)
CISCO-PING-MIB SNMP-FRAMEWORK-MIB (RFC 2571)
CISCO-PPPOE-MIB SNMPv2-MIB (RFC 1907)
CISCO-PROCESS-MIB SNMP-NOTIFICATION-MIB (RFC 2573)
CISCO-PRODUCTS-MIB SNMP-TARGET-MIB (RFC 2573)
CISCO-QUEUE-MIB SNMP-USM-MIB (RFC 2574)
CISCO-RTTMON-MIB SNMP-VACM-MIB (RFC 2575)
CISCO-SMI SONET MIB
CISCO-SSG-MIB TCP-MIB (RFC 2012)
CISCO-NETFLOW-MIB UDP-MIB (RFC 2013)
CISCO-AAA-SESSION-MIB ---


 
OSI mgmt scheme: CMIP: Common Mgmt Info Protocol
    contains CMIS (CMI Service) as a subset
    addresses all 7 layers of OSI model, including the 2 that do not exist
     
    CMIP is rather large; needs lots of resources.
    Object-oriented (later versions of SNMP have some support here)

SNMP uses OIDs to name everything; CMIP uses Distinguished Names (DNs) similar to X.500. Here is a possible DN format for me at Loyola; the CN values are "domain components":
dn: cn=Peter Dordal,dc=luc,dc=edu
The OU (Organizational Unit) component is also used; in the above, it might be ou=Computer Science Department. Also, sometimes the "dotted" syntax is used to join DC's.
 
Distinguished Names are a little verbose, but they do serve the purpose of "labeling every attribute" at least as well as OIDs, and they provide for slightly more natural naming of table attributes.

Throughout the development of SNMP, most participants seemed to believe that CMIP was better and was coming Real Soon Now. That didn't happen, and basically still has not.   

SNMP & CMIP use a combination of polling and async notification (SNMP "traps")
 
CMIP view of the world: 
 
    Network Mgmt:
  1. Organization model:  how NMS is to be organized
  2. Information model (SNMP mostly here): the data itself
  3. Communication model: how components communicate
  4. Functional model:
               5 functional areas: fault, configuration, performance, security, accounting
     
To compare:
 
    SNMP focuses mostly on the Information Model part of this.
 
Why specify an organization model? Shouldn't the software focus on mechanism, and leave the organization to the end users?
Consider SAP software.
It contains modules for ERP, Customer mgmt, accounting mgmt, etc. Customers who buy SAP are buying a way of organizing their business, not just a way of implementing their existing organization!
 
Organization model: 
    simple:
    manager, MDB, agents, unmanaged objects
     
    3-tier:
        MoM: manager of managers
        Local Managers
         
These kinds of models are used in the SNMP world as well, sometimes on a more ad hoc basis.
 
 
Information model:
 
ASN.1 naming: hierarchical, numbered levels (why weren't strings used?)
 
MIB: naming info and TYPE info
 
MIB v MDB: 
    MIB contains schema for each device; MDB contains actual results
    Adding a new kind of device requires updating the MIB
 
Implementation of an agent: you know the names of everything, but how to get the statistics is an entirely separate issue.
 
Much of MIB info is about device statistics, but some is about people, or about software.
 
Specifications:
    OID & descriptor
    Syntax
    Access
    Status
    Definition
     
GET/SET apply only to atomic (scalar) values; structures and arrays need GET-NEXT.

Communication model:
    Largely this is the SNMP protocol itself, including UDP transport. However, sometimes the arcitectural model and administrative models are lumped in here, and also module-block-diagrams for agents and managers.

Functional model:
What it does.

 
SNMP OID trees versus true objects
 
    cisco_router extends router
     
    cisco_7400  extends cisco_router
     
Object inheritance makes a lot of sense here. Note that SNMP cisco-specific information is under 1.3.6.1.4.1.9, and general router info is under 1.3.6.1.2.1.4 (IP)
 
Still, OID trees do offer a relatively natural perspective on inheritance, and CMIP is not much more natural.
    

sysObjectID


Earlier we discussed that the sysObjectID value returned by an agent can be used by an NMS to look up something of that agent's capabilities, especially (though not necessarily exclusively) under the private 1.3.6.1.4.1 OID tree.

However, note that netSNMP returns private.8072.3.2.10, described in the following table:
     
1.3.6.1.4.1 private
private.8072 net-snmp
net-snmp.3 netSnmpEnumerations
netSnmpEnumerations.2 netSNMPAgentOIDs
netSNMPAgentOIDs.10 not specifically mentioned

If we look this up in the NET-SNMP mibs, however, there is nothing to be found. If the NET-SNMP project does not define what this last OID level, the 10, stands for, then there is precious little chance that an off-the-shelf NMS will figure it out.

Also, note that if we try to do data retrieval from a net-snmp agent below private.8072, we get entries ranging from net-snmp.1.2.1.1.4 to net-snmp.1.9.1.1.5. Not net-snmp.3! (Although this in and of itself is not unexpected; it simply means that agents identify themselves with one OID, and keep their agent-specific data collections under another.)

Brief look at NET-SNMP-AGENT-MIB.txt, defining
nsVersion              OBJECT IDENTIFIER ::= {netSnmpObjects 1}
nsMibRegistry          OBJECT IDENTIFIER ::= {netSnmpObjects 2}
nsExtensions           OBJECT IDENTIFIER ::= {netSnmpObjects 3}
nsDLMod                OBJECT IDENTIFIER ::= {netSnmpObjects 4}
nsCache                OBJECT IDENTIFIER ::= {netSnmpObjects 5}
nsErrorHistory         OBJECT IDENTIFIER ::= {netSnmpObjects 6}
nsConfiguration        OBJECT IDENTIFIER ::= {netSnmpObjects 7}
nsTransactions         OBJECT IDENTIFIER ::= {netSnmpObjects 8}


NET-SNMP-SYSTEM-MIB.txt defines:
nsMemory                    OBJECT IDENTIFIER ::= {netSnmpObjects 31}
nsSwap                      OBJECT IDENTIFIER ::= {netSnmpObjects 32}
nsCPU                       OBJECT IDENTIFIER ::= {netSnmpObjects 33}
nsLoad                      OBJECT IDENTIFIER ::= {netSnmpObjects 34}
nsDiskIO                    OBJECT IDENTIFIER ::= {netSnmpObjects 35}

Similarly, there is no clear-cut OID (either key or value) that identifies "ethernet switches" as a class. However, there isn't much of anything that identifies them as a class; there are all sorts of special cases (VPNs and "bridge-routers" come to mind). On the other hand,  it's probably enough for an NMS to match on the proper prefix 1.3.6.1.4.1.8072. Bottom line, an NMS had best be prepared to doa lot of prefix matching on the sysObjectID value.
    

RMON

= RMON 1, containing Ethernet information only. Later, RMON2 added monitoring at the IP and TCP (port) layers.

See RMON-MIB

Basic idea: remote agents do some limited monitoring of subnets. They become "mini-managers", freeing the manager from probing every host on the subnet. Managers can create rows to specify what monitoring is to be done.

Note that implementing RMON is a significant undertaking for an agent: it will have to put its interfaces into promiscuous mode and analyze every packet. Of course, tcpdump & wireshark do this too.

Note the OwnerString defined at the beginning of RMON-MIB (line 31). Note that adherence to the conventions here is voluntary, and unenforced.

The next type defined is EntryStatus, for row creation. RMON introduced its own row-creation mechanism; this was later expanded by SNMPv2 which added two additional values. The RMON row-status column can have one of the four (not six as in SNMPv2) values valid, createRequest, underCreation, invalid.

How row-creation works (from RMON-MIB).
   
Look at the transitions table in the MIB:

To:valid create Request under Creationinvalid
From:
valid OK NO OK OK
createRequest    N/A N/A N/A N/A
underCreation OK NO OK OK
invalid NO NO NO OK
nonExistent NO OK NO OK

createRequest supplies a value for etherStatsIndex. In the unlikely event that this value is now in use (unlikely at least if the management station checked first that it was unused, which would mean another manager created the row at about the same time), then we try again.

Ethernet Statistics Group (etherStats)

This is not indexed by ifIndex! The index etherStatsIndex is arbitrary (pseudorandom); it is the second column of the table, etherStatsDataSource, that contains the OID of the actual interface to be measured. Note that we can have more than one row for the same interface! (perhaps started at different times?)

etherStatsIndex, etherStatsDataSource, data, etherStatsOwner, etherStatsStatus

Here's the definition of etherStatsDataSource

In order to identify a particular interface, this object shall identify the instance of the ifIndex object... for the desired interface.  For example, if an entry were to receive data from interface #1, this object would be set to ifIndex.1.
                 
In other words, this value is the OID index for that particular interface in the interfaces table ifTable. Note two managers might create entries for the same interface (though this is not intended to happen) A manager chooses an etherStatsIndex value pseudorandomly, and creates the entry.

Also note that we have a coarse size histogram available: etherStatsPkts65to127Octets to etherStatsPkts1024to1518Octets

History group

This group has two tables: control & data

The control entry specifies the interface (historyControlDataSource), the bucket count, and the interval. The data table then creates one bucket for each time interval, containing a summary of the usage during that interval. Actual history stats: packet counts, byte counts, error counts of various types, etc.

we create a control entry, and that directs how the data table will be built. Note how the data table is indexed, and how rows get deleted (recycled?) as new buckets are created.

When the manager creates a row, it supplies:

Note that we might create a 30-second history table and a 30-minute history table, with possibly different bucket counts for each.

The etherHistorySampleIndex is initialized to 1 and is incremented each interval. Old bucket entries are deleted as the buckets are recycled: from historyControlBucketsGranted (starting line 645):
                  When the number of buckets reaches the value of
                  this object and a new bucket is to be added to the
                  media-specific table, the oldest bucket associated
                  with this historyControlEntry shall be deleted by
                  the agent so that the new bucket can be added.

Look at the indexing here. The etherHistoryTable is indexed by etherHistoryIndex and etherHistorySampleIndex. The latter is a bucket serial number; the former is the same as historyControlIndex, the index to the historyControlTable.




The Hosts group has three tables hostControlTable, hostTable, and hostTimeTable.
The hostControlTable specifies the interface. Note hostControlTableSize (line 1462), which determines the number of hosts to be recorded.

The hostControlIndex:
DESCRIPTION: An index that uniquely identifies an entry in the hostControl table.  Each such entry defines a function that discovers hosts on a particular interface and places statistics about them in the hostTable and the hostTimeTable on behalf of this hostControlEntry."

The hostTable is indexed by hostIndex and hostAddress; the latter is a particular node's physical address, and contains per-host statistics such as in and out packets, bytes, and errors. hostIndex is the same as hostControlIndex.

Read the description of the hostCreationOrder column.

The hostTimeTable is the same except indexed serially by discovery time. The actual index is { hostTimeIndex, hostTimeCreationOrder }. The hostTimeIndex is again the hostControlIndex. See "The hostTimeTable has two important uses.", line 1355:
The hostTimeTable has two important uses.  The first is the fast download of this potentially large table.  Because the index of this table runs from 1 to the size of the table, inclusive, its values are predictable.  This allows very efficient packing of variables into SNMP PDU's and allows a table transfer to have multiple packets outstanding. These benefits increase transfer rates tremendously.

The second use of the hostTimeTable is the efficient discovery by the management station of new entries added to the table. After the management station has downloaded the entire table, it knows that new entries will be added immediately after the end of the current table.  It can thus detect new entries there and retrieve them easily.

I am not aware of any evidence that having contiguous index values allows for more efficient PDU packing! But it is an intriguing idea.


TopN group

hostTopNControlTable
Indexed by hostTopNControlIndex
The N is supposed to be the hostTopNRequestedSize/hostTopNGrantedSize.
The hostTopNRateBase specifies what variable we are to use for the ranking. You get to pick from a list of seven.

Collection runs for the management-supplied hostTopNTimeRemaining.

The actual data table is the hostTopNTable, with columns
               hostTopNReport                INTEGER (1..65535),  -- matches control-table entry
              hostTopNIndex                 INTEGER (1..65535),
              hostTopNAddress               OCTET STRING,       -- host physical address
              hostTopNRate                      INTEGER                   -- numeric value of change in the selected variable
Index columns are underlined. In this table, the hostTopNIndex specifies the 1..N rank of a particular host (the hostTopNAddress). This may change during the collection period. Alternatively, no data at all may be available until the collection period has ended.


Matrix Group

MatrixControlTable
SD and DS tables: why both? Because one is indexed by source and the other by destination.

The SDTable has fields:
              matrixSDSourceAddress       OCTET STRING,
              matrixSDDestAddress         OCTET STRING,
              matrixSDIndex               INTEGER (1..65535),
              matrixSDPkts                Counter,
              matrixSDOctets              Counter,
              matrixSDErrors              Counter

Again, index fields are underlined. The matrixSDIndex corresponds to the control-table entry.