NetMgmt Week 10

Network Management Week 10

SNMP v3
LARTC
BGP
OpenNMS and SysOID matching

Non-snmp managing tools

ping

both individual and subnet pings are widely used for new-node detection, although subnet pings are not necessary.

        use of individual pings
        use of "subnet ping"
          ping -b 10.213.119.0

Other methods of host discovery
        ARP (or IP physaddr) table
        TCP Connection table
        Packet sniffing / RMON
        Port scanning

traceroute

mostly used for manual analysis of problems

tcpdump/wireshark

often used for manual analysis. Some problems:

switched Ethernets often mean you can't see more than one host!
massive amounts of data

Note that wireshark "understands" a whole lot of different packet formats.

nmap (port scanner)

Good tool for scanning subnets to determine device type
Excellent port-scanning features, which helps greatly with service/capability detection
Excellent at "OS fingerprinting"

spong

spong.sourceforge.net
"spong is not quite dead yet"; recent work in 2005

spong is a non-snmp-based Network Management System. It consists of a collection of monitor scripts, typically in perl. The spong.hosts file lists hosts and services that need monitoring. It works best for modest networks. OpenNMS is also a collection of monitor scripts, but note spong's different discovery method.

            spong.interfaces:     snmp interfaces check
                        checks for any that are "down"

        client-side installation, too

mon

        mon service monitoring daemon
        www.kernel.org/software/mon

recently has added SNMP to checking: uptime, HP printers, processes, disk free space, disk quotas


like spong, mon is based on manual configuration of a number of "probe" programs. SNMP discovery is not used. Instead there are "hostgroup" entries.

        smtp: get quickie response from server
            <220
            >QUIT
            <221

        http: actually get a page
        mysql: gets list of tables

        email probing: what can go wrong?

not enough server disk space (sendmail pauses automatically)
too high server processor load
authentication tool failure
insufficient randomness for SSL
dns trouble
deferred connection

Neither of these has explicit monitors for checking actual transfers: sending data, seeing if it is delivered ok.

openNMS: auto-detects new hosts, monitoring profile is based on sysOID; also supports manual config (probably the only feasible approach!) for servers & services (can probe for services)

One current theme for network monitoring is that of compliance monitoring:

Are there any non-Windows7 nodes?
Are there any windows nodes our IT dept hasn't authorized?
Are their any Windows nodes missing patch 12-3789?

Note that nmap-style OS fingerprinting does some of this quite well. (The specific-patch part, not so well.)

(There has been some recent convergence in the marketplace of NMS-style applications and "intrusion detection"-style applications; the latter monitor all packets and use that as the basis for host discovery.

OpenNMS uses "adaptive polling": once an outage is discovered through NMS, the service is polled more frequently. Typical generic polling interval: 300 sec; typical polling interval for a down service: 30 sec.

Actual OIDs collected: see datacollection-config.xml (substitutes for mib compiler)

See: opennms.org/index.php/Data_Collection_Configuration_How-To

datacollection-config.xml
    Each package in the collectd configuration file points to an
    snmp-collection     definition in this file. Each snmp-collection
    defines what information to collect via SNMP, and it is pretty
    powerful as far as configuration goes. The default configuration
    is fairly complete for basic purposes, and will probably not
    require much changing initially.

Defines groups, systemDefs.

<groups>
<group name = "mib2-interfaces" ifType = "all">
<mibObj oid=".1.3.6.1.2.1.2.2.1.10" instance="ifIndex" alias="ifInOctets" type="counter"/>
<mibObj oid=".1.3.6.1.2.1.2.2.1.13" instance="ifIndex" alias="ifInDiscards" type="counter"/>
<mibObj oid=".1.3.6.1.2.1.2.2.1.14" instance="ifIndex" alias="ifInErrors" type="counter"/>
<mibObj oid=".1.3.6.1.2.1.2.2.1.16" instance="ifIndex" alias="ifOutOctets" type="counter"/>
<mibObj oid=".1.3.6.1.2.1.2.2.1.19" instance="ifIndex" alias="ifOutDiscards type="counter"/>
<mibObj oid=".1.3.6.1.2.1.2.2.1.20" instance="ifIndex" alias="ifOutErrors" type="counter"/>
</group>

But who do you ask all this information of?
Define SYSTEMS,
    also in datacollection-config.xls

With the definition below, any device with a system OID that is being used for
SNMP data collection whose systemOID starts with ".1.3.6.1.4.1.2021.250."
will have the specified information collected.

This assumes that you have assigned sysOID names in a sensible manner.

<systems>
   <systemDef name = "Net-SNMP (UCD)">
           <sysoidMask>.1.3.6.1.4.1.2021.250.</sysoidMask>
           <collect>
                <includeGroup>mib2-interfaces-net-snmp</includeGroup>
                <includeGroup>mib2-host-resources-storage</includeGroup>
                <includeGroup>mib2-host-resources-system</includeGroup>
                <includeGroup>mib2-host-resources-memory</includeGroup>
                <includeGroup>ucd-loadavg</includeGroup>
           </collect>
   </systemDef>

   <systemDef name = "Net-SNMP">
    <sysoidMask>.1.3.6.1.4.1.8072.3.</sysoidMask>
    <collect>
        <includeGroup>mib2-host-resources-storage</includeGroup>
        <includeGroup>mib2-host-resources-system</includeGroup>
        <includeGroup>mib2-host-resources-memory</includeGroup>
        <includeGroup>net-snmp-disk</includeGroup>
        <includeGroup>ucd-loadavg</includeGroup>
        <includeGroup>ucd-memory</includeGroup>
        <includeGroup>ucd-sysstat</includeGroup>
    </collect>
   </systemDef>

Now look at router info:
    <sysoid> matches an exact systemOID
    <sysoidMask> matches a *prefix*

        <systemDef name = "Cisco PIX1">
            <sysoid>.1.3.6.1.4.1.9.1.390</sysoid>
            <collect>
                <includeGroup>cisco-pix</includeGroup>
            </collect>
        </systemDef>

        <systemDef name = "Cisco PIX2">
            <sysoid>.1.3.6.1.4.1.9.1.417</sysoid>
            <collect>
                <includeGroup>cisco-pix</includeGroup>
            </collect>
        </systemDef>

        <systemDef name = "Cisco PIX3">
            <sysoid>.1.3.6.1.4.1.9.1.451</sysoid>
            <collect>
                <includeGroup>cisco-pix</includeGroup>
            </collect>
        </systemDef>

        <systemDef name = "Cisco Routers">
            <sysoidMask>.1.3.6.1.4.1.9.1.</sysoidMask>
            <collect>
         <includeGroup>cisco-memory</includeGroup>
                <includeGroup>cisco-router</includeGroup>
                <includeGroup>cisco-router-if-stats</includeGroup>
                <includeGroup>cisco-router-env</includeGroup>
            </collect>
        </systemDef>

This is the part that might be easier with CMIP, which allegedly has better support for true inheritance.

Anyway, between <groups>, <systems>, and <sysoidMask>, we can program openNMS to retrieve specific oids from specific devices.

NetSNMP and adding new OIDs to 1.3.6.1.4.1.2021

See the section beginning "Executables/scripts" in /etc/snmp/snmpd.conf.

I have enabled echotest and shelltest.

Here is the output: (ucd = 1.3.6.1.4.1.2021)

UCD-SNMP-MIB::extIndex.1	INTEGER	1	ucd.8.1.1.1
UCD-SNMP-MIB::extIndex.2	INTEGER	2	ucd.8.1.1.2
UCD-SNMP-MIB::extNames.1	STRING	echotest	ucd.8.1.2.1
UCD-SNMP-MIB::extNames.2	STRING	shelltest	ucd.8.1.2.2
UCD-SNMP-MIB::extCommand.1	STRING	/bin/echo	ucd.8.1.3.1
UCD-SNMP-MIB::extCommand.2	STRING	/bin/sh	ucd.8.1.3.2
UCD-SNMP-MIB::extResult.1	INTEGER	0	ucd.8.1.100.1
UCD-SNMP-MIB::extResult.2	INTEGER	35	ucd.8.1.100.2
UCD-SNMP-MIB::extOutput.1	STRING	peters snmp data	ucd.8.1.101.1
UCD-SNMP-MIB::extOutput.2	STRING	hello world	ucd.8.1.101.2
UCD-SNMP-MIB::extErrFix.1	INTEGER	noError(0)	ucd.8.1.102.1
UCD-SNMP-MIB::extErrFix.2	INTEGER	noError(0)	ucd.8.1.102.2
UCD-SNMP-MIB::extErrFixCmd.1	STRING		ucd.8.1.103.1
UCD-SNMP-MIB::extErrFixCmd.2	STRING		ucd.8.1.103.2

SNMP v3

snmp v3: the main purpose was to add authentication, and eliminate community-based security.

Look again at the snmpd.conf file (for net-snmp).
We had a series of mappings
community name => security_name (using the com2sec directive)
security_name => group (using the group directive)
group => views for read and write (using the access directive)

views are defined using the view directive.

SNMP v3 switched from the community model to the user service model (USM). It also applied more fully the View Access Control Model (VACM), though elements of that existed before.

Here is our previous example, but now with connections to VACM spelled out:

com2sec table; the key (index) is community. In this and following examples, the indexes are underlined. This is not a VACM table in the sense of SNMP v3, since it refers to community names. In v3, usernames are mapped to groups directly in the group table following.

security_name	source	community
foocom	default	foo

group / vacmSecurityToGroup (1.3.6.1.6.3.16.1.2)

group_name	version	security_name
MyFOOgroup	v1	foocom

In SNMPv3, the table above is known as the vacmSecurityToGroup table; see Mauro & Schmidt p 81

security_name (or vacmSecurityName): the secname from the com2sec table for v1/v2, or the username (USM) for v3.
version is really vacmSecurityModel: community v USM. But community models are specified in NetSNMP as v1 or v2.
vacmGroupName is used as the link to the access table.
The real table also has two other columns, StorageType (volatile v nonvolatile) and RowStatus.

Finally there is the access table, which in SNMPv3 becomes known as the vacmAccessTable. But notice that the version below is what we've been using in SNMPv1! (Column numbering is slightly different in SNMP v3)

group_name	prefix	sec_model	sec_level	match	read	write	notify
MyFOOgroup	""	any	noauth	exact	fooview	none	none
MyROSystem	""	any	noauth	exact	sysview	none	none

The vacmAccessTable is spelled out on pp 81-82 of Mauro & Schmidt. Sec_model refers to community versus USM. Every column save the first begins with "vacmAccess", abbreviated below to "vA". The list below contains the "official" VACM column names, versus the "informal" names above used by NetSNMP.

vacmGroupName: as in the vacmSecurityToGroup table
vAContextPrefix: matches a supplied ContextName used for lookup; depending on the match column, the match is either exact or by prefix. The empty string is used for v1/v2.
vASecurityModel: v1, v2, USM, any
vASecurityLevel: noAuth, authNoPriv, AuthPriv. Only noAuth makes sense for v1 & v2
vAContextMatch: "exact" or "prefix"; used with the ContextPrefix column
vAReadViewName: the name of a view from the view table below, or "none"
vaWriteViewName: ditto
vaNotifyViewName: ditto
The real table again has StorageType and RowStatus columns.

The sec_level is (noauth | authNoPriv | AuthPriv), for cleartext unsigned, cleartext digitally signed, and encrypted, respectively.

view / vacmViewFamilyTable table

view_name	incl/excl	OID subtree	mask
fooview	included	.1.3.6.1.2.1.1
fooview	included	.1.3.6.1.2.1.2
sysview	included	.1.3.4.1.2.1.1
all	included	.1

The mask (a bitstring in hex) can specify a set of levels in the subtree OID that are significant; the remaining levels are "wildcarded". This is only relevant for granting views to specific rows of a table, where the columns (which come before the row indexes) are wildcarded. Otherwise, all levels of the OID_subtree specified are considered significant, and all further levels are wildcarded.

Recall that in v1/v2 at least some of this configuration had to be done outside the SNMP framework. Actually, in v1/v2, essentially all this configuration has to be done outside of SNMP. SNMPv3 defines VACM as formal tables within the OID hierarchy, so that they can be updated within SNMP.

There is a good picture of this in M&S Fig 3-3, p 83. However, the OIDs involved (at the right edge of the picture) should really be shown as inputsto the vacmViewFamilyTable, rather than to the oval immediately below it (labeled "Yes/no decision").

snmp v3 itself

framework for multiple versions
Fig 3-1 in M&S, p 76

SNMP engine:

Dispatcher

Message
Processing
System

Security
System

Access
Control
System

Below the SNMP entity are six "applications", together with suggestions as to whether they represent traditional manager or agent roles.

command generator (manager)
notification receiver (manager)
proxy forwarder (manager, sometimes)
command responder (agent!)
notification originator (agent)
other

Within the SNMP engine proper, the "Dispatcher" first sees incoming messages and does initial version-detection, handing the message off to the "Message Processing System" if the latter includes a module supporting the version in question. The Dispatcher is an attempt at universal compatibility.

SNMPv3 dropped the terms "manager" and "agent", but not of course the reality. All SNMP nodes are now called SNMP entities; each entity is an SNMP "engine" (the core) plus one or more of six "applications". Note that a typical agent needs the "Command responder" application, and perhaps the "Notification originator". Agents (in most roles) are now to be called "authoritative SNMP engines."

Each SNMP node has an SNMPEngineID. This idea goes back to SNMPv1, but is more central in v3. The SNMPEngineID can be computed a variety of ways, typically from some form of address (IP, mac, admin). Note that typical routers have multiple addresses (often the "smallest" address is used for the SNMPEngineID), and most switch ports don't necessarily have MAC addresses at all (they really do, but they are not so visible).

1-bit + EnterpriseID (total 4 bytes) (= private enterprise #, eg 8072 for NetSNMP)
+ Format Indicator (1 byte)
+ Format

Format Indicator:

IPV4 addr
IPv6 addr
Lowest nonspecial IP addr
Phys addr / lowest phys addr
Text string, admin assigned
binary string, admin assigned
misc

Names: principal / securityName; only the latter is "visible". But this isn't entirely a security issue: the "principal" is an abstraction; the securityName is a "handle".

SNMPv3 MIB: new snmp entries under 1.3.6.1.6.3, as before

Here is a diagram of the SNMPv3 packet format (see also http://www.insanum.com/docs/usm.html)

                    SNMPv3 Packet Format

                  -------------------------
    /|\           | msgVersion            |
     |            |-----------------------|
     |            | msgID                 |
     |            |-----------------------|         USM Security Parameters
     |            | msgMaxSize            |
     |            |-----------------------|    /-------------------------------
     |            | msgFlags              |   / | msgAuthoritativeEngineID    |
  scope of        |-----------------------|  /  |-----------------------------|
authentication    | msgSecurityModel      | /   | msgAuthoritativeEngineBoots |
     |            |-----------------------|/    |-----------------------------|
     |            |                       |     | msgAuthoritativeEngineTime  |
     |            | msgSecurityParameters |     |-----------------------------|
     |            |                       |     | msgUserName                 |
     |            |-----------------------|\    |-----------------------------|
     |     /|\    |                       | \   | msgAuthenticationParameters |
     |      |     |                       |  \  |-----------------------------|
     |      |     |                       |   \ | msgPrivacyParameters        |
     |  scope of  | scopedPDU             |    \-------------------------------
     | encryption |                       |
     |      |     |                       |
     |      |     |                       |
     |      |     |                       |
    \|/    \|/    |                       |
                  -------------------------

Four Threats

DISCLOSURE: eavesdropping, either of messages in transit or by unauthorized requests for SNMP info from agents.

MODIFICATION of the contents of messages or information, including "man-in-the-middle" attacks.

MASQUERADE: an endpoint pretending to be someone else. This overlaps with DISCLOSURE, in that an endpoint masquerading as a legitimate requester may be violating the DISCLOSURE policy

MESSAGE STREAM MODIFICATION: could the packets be reordered by a bad guy in such a way that the message is changed? The most common is the "replay attack", eg to reboot the machine at noon by replaying the normal system maintenance reboot messages sent last midnight. (MSM is generally different from MODIFICATION).

Another (more obscure) possibility is that one might change
YES NO
to
NO YES
with this technique. Secure sequence numbers solve the problem.

"man-in-the-middle" and "replay" attacks are standard security issues. Note that public-key cryptosystems are vulnerable to man-in-the-middle attacks if they are too trusting of the keys they send:

Alice----------------------Eve---------------------Bob

Normal: Alice asks Bob for his public key, and encrypts her message to Bob with it. Only Bob can decrypt it.

MitM (WitM? Or Yves?): Alice thinks she's asking Bob for his public key, but Eve sees to it that Alice gets Eve's public key. Alice again encrypts her message to Bob with it. Eve intercepts it, decrypts it, does whatever else she wants with it, and then uses Bob's public key to re-encrypt it and send it on to Bob. Bob thinks the message came from Alice.

(Historically, the name Eve was supposedly reserved for evesdroppers; sometimes the name Mal is used for attackers. Firefly fans take note.)

other security non-concerns:
DOS: Denial Of Service

Traffic analysis
password example

Encryption is used to prevent DISCLOSURE of messages in transit. The others will all be addressed in SNMPv3 with sequence numbers, timestamps, and digital signatures.

SNMP v3 security

Prolog:
    MD5 and SHA1 secure checksums (128 and 160 bit respectively) for signatures
    DES encryption (56-bit key)

Auth means digital signatures, the first group. Priv (privacy) uses encryption.

Questions to keep in mind

Question 1: what advance config is needed to get secure communication to work?

Question 2: if an agent is compromised, what else is compromised?

Authoritative SNMP engine: this is typically the agent. Specifically, in any exchange the side that responds to GET/SET commands is the authoritative engine. The other end is non-authoritative. The non-authoritative side must keep track of the <engineID, timestamp> of the authoritative side.

This authoritative/nonauthoritative is in keeping with the idea that agents (the authoritative nodes) have privileged data, and they are responsible for demanding credentials before giving it up. The nonauthoritative side (the NMS) just sends requests, with its credentials.

Authoritative end: gets requests, sends responses, sends traps/informs.

engineID Discovery: a manager can ask an agent for its engineID by sending a kind of "null" message.

Timestamps

These are used to prevent replay and to guarantee timeliness of messages. Messages must arrive within a 150-second window. This is thus "loose" time synchronization. Timestamps are actually <bootcount, time>; bootcount is incremented on reboots and when time wraps around. Bootcount values must match exactly; time can be +/- 150 seconds. This prevents replay more than 150 sec later.

compare CHAN protocol of blast/chan/select in Peterson&Davie, with BID & MID.

Another "null" message can request the engineBoots and EngineTime.

When an authoritative engine receives a message, it is rejected if:

local-boots = 2³¹-1
local-boots != message-boots
local-time and message-time not within +/- 150 sec

The first one is kind of peculiar: if you've rebooted your agent more than 2 billion times, you probably have to buy a new one. Or else reconfigure it to be a "new" SNMP engine, with a new local-boots value.

Note that message-boots refers to the nonauthoritative engine's idea of the authoritative engine's engineBoots count, not to the nonauthoritative engine's own engineBoots count (nonauthoritative engines don't really even keep the latter).

If a message is rejected due to clock drift of more than 150 sec, it is logged but the sender can resynchronize clocks and try again.

When the non-authoritative engine (ie the manager) receives a message, it updates its values if:
    remote boots > stored local value of boots for that host
    (in other words, agent rebooted again since we last heard)
        Action: update stored local value

    boots are equal, but remote time > local stored time.
        Action: update stored local time

The message is rejected if:

remote boots value = 2³¹ - 1
remote boots < stored local value (the authoritative side can never lower its engineBoots value!)
boots equal, but clocks differ by > 150 sec

timestamps are only used with content protection (Auth or Priv). They are easily forged otherwise.

The nonauthoritative side, when looking at responses, will disallow duplicate messageIDs within the 150-second window. (This means that all messageIDs must be saved for that long.) When looking at a response, the nonauthoritative side also verifies that the messageID is the one expected!! Duplicate messageIDs might indicate a replay attack.

For the authoritative side, things are more complicated, given that we can't be sure the request was received. So duplicate messages may be sent by the nonauthoritative side simply because an earlier response was lost.

Duplicate messages present the risk that the same request will be processed multiple times. However, most GET requests are idempotent, meaning that executing it twice has the same effect as executing it once. Re-execution of idempotent requests is harmless.

Many SET operations (but not all, not, for example, row-creation) are also idempotent. Sometimes a TestAndIncrement semaphore can be used. The fact that SETS are atomic (all are done or none are done) is also helpful in many settings.

With encryption or authentication, the only risk is that the entire GET/SET block will be replayed. So our semaphore cannot be removed!

Three security modes:
    noAuthNoPriv
    authNoPriv    = hash signatures
    authPriv    = encryption
Why not use authPriv for everything?? Overhead and legal concerns, mainly.

Notation:
    md5(string)
    str1 ^ str2:    concatenation
Privacy v Authentication:
    The latter can be achieved using secure hashes of message^shared_key.

User-based security model:

Users are analogues of v1 communities, except that users now have usernames and passwords. Both must be presented.

A User may be a person or an NMS (or something in between). Generally, WRITE access is probably only assigned to persons. But an NMS will need copious READ access, and WRITE access to RMON/PING stuff (as needed)

A user (manager) identifies itself to an agent. Agents keep key info and passwords for each USER (manager). But agents keep only an encrypted password; if they are compromised then the user/manager is not (except for bad info from that agent). This means that a user/manager can use the same password on multiple agents, with reasonable security. (If any of these agents is compromised at the time the account is first set up, however, then the original password is compromised.)

Outline:

managers identify themselves with a key based on their password
agent keeps key info for each user
user & agent can negotiate KEY UPDATE, securely
key is never actually sent

The agent must be MANUALLY CONFIGURED to know an initial master user key (below)

The agent can then be SNMP-configured to know the initial user key, sort of like it can be SNMP-configured to know updated user key (except initial-user-key generation involves cloning the MASTER entry)

We can also configure a MIB VIEW for each user, using snmp.

Note re discovery:
SNMPv1: if you have the wrong "community" password, you get no response back. So you can't probe a network to find the SNMPv1 agents.

SNMPv3:
A nonauthoritative engine can send a message with:

security_model	noAuthNoPriv
username	"initial" or ""
engineID	null string
varBindList	empty

(See Mauro & Schmidt fig 3-2, p 79, for a full list of all message parameters).

The authoritative engine response to a message such as this contains its engineID and basic security parameters. This is how engineID discovery works.

Now, for the nonauthoritative engine to get time/boots params, it sends a second message:

engineID	engineID of Authoritative end, as just discovered
username	any valid username known by the Authoritative end
authParams	user authentication data associated with username above
boots	0
time	0

For this second discovery query, credentials must be supplied.

Basic key management

An authoritative engine keeps a secret key for each user (manager).

The nonAuthoritative side first generates a user key ku, based on the password:

    password => repeated to 1MB => take hash => ku

(The repetition to 1MB is intended to make the calculation slower, to make it harder to guess passwords if the ku is compromised)

Now take the 16-byte (for md5) ku, and form the concatenation ku^engineID^ku. Run that through MD5 to get kul, the localized user key (localized to a specific agent).

This calculation is done by the manager, which sends kul to the agent without revealing the password. (Typically, the actual ku is available to the agent at some initial point in the configuration process.)

This kul key cannot be used on another SNMP node, as its engineID will be different.

Summary:
    password => repeated to 1MB => take MD5 hash => ku
    MD5(ku^engineID^ku) => kul
Either side can do this calculation. However, normally the agent side keeps only kul.

Note that kul is never sent in the clear.

Authentication protocol

k1 and k2 are 64-byte extensions of the basic kul, computed deterministically from kul, as follows (Subramanian p 309):
    ipad = 64 bytes each consisting of 0x36
    opad = 64 bytes each consisting of 0x5c
    ekul = extended kul = kul padded with null bytes (0x00) to a length of 64 bytes
    k1 = ekul XOR ipad
    k2 = ekul XOR opad

We calculate:
    temp = md5(k2 ^ md5(k1 ^ message))
    fingerprint = temp truncated to 12 bytes

We place fingerprint in the message header; it can be recalculated at the receiving end to ensure authenticity. kul is never sent in the clear.

Why not just send md5(message ^ kul)? One concern is possible weaknesses in the md5 (or sha1) secure hashes.

Privacy:

DES key is 64 bits; the other 8 bytes of 16-byte key is used as preinitialization value. (Actually, DES keys are 56 bits)

How does this prevent the four risks:

DISCLOSURE: eavesdropping
MODIFICATION of messages or information. "man-in-the-middle" attacks
MASQUERADE: an endpoint pretending to be someone else.
MESSAGE STREAM MODIFICATION:

KEY Update, abstract version

1. The user generates a new password, eg "zanzibar"
2. User creates localized key newkul for zanzibar + specific agent
3. Generate 16-byte string rand, really random
4. Calculate delta as follows:
    temp := md5(oldkul^rand)
    delta := temp XOR newkul
5. Send rand and delta to the authoritative (agent) end.
    why can't temp be inferred?
    why can't oldkul be inferred?

The agent can use rand and delta and known oldkul to compute temp and thus newkul.

(This is from page 35++ of rfc3414, slightly simplified.)

When we do a key update, we actually send some encoding of (rand, delta) as a "keychange object". From rfc 3414:

The 'random' and 'delta' components are then concatenated as described above, and the resulting octet string is sent to the recipient as the new value of an instance of this object [ie a "keychange" object -- pld]

At the receiver side, when an instance of this [keychange] object is set to a new value, then a new value of K is computed as follows: ...

Note that this is yet another object-specific modification of the SET semantics: when you appear to be doing a simple assignment, you are in fact doing something rather more complex. (Other examples involved setting a testAndIncr semaphore, or a rowStatus object.)

usmUserTable

Agents keep usmUserTable with auth/priv info for each user. The index is user name (and local SNMP engineID, for proxy cases)

Agent starts out with one row in this table, configured manually. New users are created by cloning the row, and then updating the keys.

The original row is the "master" key, which may never be updated.

usmUserTable values
Note: some fields can't be read!

engineID
userName	string
securityName	string, may be same as userName
cloneFrom	a row-pointer referring to the original row from which this row was originally cloned; used for creating new users
userAuthProtocol	none / md5 / sha1
userAuthKeyChange	key-change string as above, used for the original clone key change. It encodes the (rand, delta) pair above. Note that setting a value to this field doesn't actually set the stored kul to the value sent; rather, it triggers the kul + (rand,delta) -> newkul update process above.
userOwnAuthKeyChange	like the preceding entry, except this one can only be changed if the userName in the message security header matches the userName object for this row. This is used for users making further changes to their own keys.
userPrivProtocol	none / DES
userPrivKeyChange	like userAuthKeyChange
userOwnPrivKeyChange	like userOwnAuthKeyChange
userPublic	How do we find out if a key change worked, if we can't read the new key value? Set this as well; either both were set or neither so just see if this was set. Helps if the Response to the keychange request was what got lost
userRowStatus	RowStatus column
userStorageType	volatile v nonvolatile storage

There is also a userSpinLock (a testAndIncr semaphore) for the entire table; it is not a column entry.

When doing key updates:

GET(userSpinLock.0) and save in sValue.
Generate the keyChange value based on the old (existing) secret key and the new secret key, let us call this kcValue. (This is the (rand, delta) data described above).
SET(userSpinLock.0=sValue, userAuthKeyChange=kcValue, userPublic=randomValue)

We verify that this worked with a GET request to see if userPublic changed to the randomValue specified.

Adding a new user:
Params: remote addr, new username, passwords, row to clone from

1. See if there already is a row for that user. If it exists but is not ACTIVE, then maybe another mgr is adding this user.

2. Create new row, by cloning, and put into CreateAndWait state.

3. Use master user to update auth/privacy keys. Note that in the future the user's own new password will work.

What if a user/password USR1 is compromised? This would have to be at the manager end. If so, we hopefully have the MASTER account to disable USR1 and then create credentials for a new USR2.

SNMP v3 key management

Note that the agent doesn't keep the password, and what it does keep has its own engineID as part of it (so it can't be used with other devices).

Basic passwd=>kul step (depends on engineID of agent)
password => repeated to 1MB => take MD5 hash => ku
MD5(ku^engineID^ku) => kul
This can be done by a manager after inquiring as to engineID, and can also be done by the agent!

Creating and Cloning Users

Agent initialization with master account <root,ARTICHOKE>:

1. new "agent" (no longer called that!) is configured with the master <userid,passwd> that is, <root,ARTICHOKE>. The agent does the passwd=>kul conversion (ARTICHOKE=>kul_arti) immediately, and stores only kul_arti. (note that kul_arti is specific to that agent, by virtue of the use of the agent's engineID; the "l" in "kul" means "local", as in "local to that agent".)

(technically you have to TRUST the agent software not to keep ARTICHOKE around. NetSNMP is not so well-behaved here!)

All but the most secure sites will use the same master password for large classes of devices (perhaps one master password per site, or per subnet, or per scope-of-management). Somebody has to keep a list of each agent and its master <userid,password>!

Compromise of one agent does not reveal ARTICHOKE, assuming only kul_arti was stored.

Now let us clone a new user account <ivan,ZANZIBAR>:

To set up this specific account, someone manually enters <root,ARTICHOKE> into the NMS or other software (hereinafter called the "manager"). The manager takes ARTICHOKE and generates the same kul_arti. This is used to authenticate messages to the agent. (kul_arti is used either as the DES encryption key or as the appended "salt" for MD-5/SHA1 authentication; either way, it is not sent in the clear).

The manager issues a kul_arti-authenticated command to "clone" the master <root, kul_arti> entry to create a new entry <ivan, kul_arti>, and then immediately changes the key, eg to <ivan,kul_zan>, using the normal key-change protocol. The clone and keychange can be done in one atomic step, or else the new row's rowStatus should remain createAndWait until the key change has been completed.

Cloning means to create the new row, using the new username as the key:
    SET(userCloneFrom = root, userStatus = createAndWait)

    SET(userAuthKeyChange = authKeyDelta, userPublic = random)

After verifying that the update "took" (ie that the userPublic field did in fact change), one does
    SET (userStatus = active)

Note that cloning an entry means creating a new row in a table All this requires a trusted admin to enter the master password ARTICHOKE into the manager, once. Just once.

4. User ivan can initiate key change at any time. This starts with the choice of a new password, say ZAKUSKI, and the communication of this to the agent which will update <ivan,kul_zan> to <ivan,kul_zak>. Ivan can do this to some or all of the agents he communicates with, BUT I cannot imagine a reason for doing it to just some.

5. The root account password can also be updated, under the direction of the root administrator, either for individual agents or whole scads of them.

Even if we use the same root password throughout the site, it sure beats using an SNMPv1 community string (ie a password) that cannot be changed remotely (and thus likely will not be changed in practice at all), and that is sent in the clear.

Password update can be done because concerns about human-user leakage (too many people know the password ZAKUSKI), or codebreaker leakage (a codebreaking algorithm is applied to a large set of packets). The latter is probably less of a concern than the former.

Initial NetSNMP USM configuration

See http://www.net-snmp.org/docs/README.snmpv3.html.

The official way is to run net-snmp-config --create-snmpv3-user, which generates the following dialog:

Enter a SNMPv3 user name to create:
master
Enter authentication pass-phrase:
saskatchewan
Enter encryption pass-phrase:
[press return to reuse the authentication pass-phrase]
novosibirsk
adding the following line to /var/net-snmp/snmpd.conf:
   createUser master MD5 "saskatchewan" DES novosibirsk
adding the following line to /usr/local/share/snmp/snmpd.conf:
   rwuser master

This did create the /var/net-snmp/snmpd.conf entry. However, when I ran
    snmpget -v 3 -u master -l authNoPriv -a MD5 -A saskatchewan localhost sysUpTime.0
I got
    authorizationError (access denied to that object)

It turned out that the entry rwuser master was in the wrong place; it needed to be put in /etc/snmp/snmpd.conf. And then it worked. (Actually I used rouser).

Note that use of the rwuser/rouser directives are not part of VACM, at least not apparently.

Also note that the version, username, security_level (-l), hash protocol (-a), and password (-A) can all be set in your per-user .snmp/snmp.conf file; having done this (except for the version) I can also just use

    snmpget -v 3 localhost sysUpTime.0

One can create multiple snmp users this way, bypassing the SNMPv3 clone-from process, or just one. To clone users, net-SNMP provides the snmpusm shell command; a typical example to clone user "pld" from the "master" account created above would be

snmpusm -v3 -u master -l authNoPriv -a MD5 -A saskatchewan localhost create pld master

snmpusm -v3 -u pld -l authNoPriv -a MD5 -A saskatchewan localhost passwd saskatchewan ramblers

The first command creates pld as a clone of master, still with password "saskatchewan", and the second sets pld's pasword to "ramblers".

Note that replacing localhost with another hostname allows you to use this command to clone a user on any SNMPv3 agent for which you have credentials.

Note also that NetSNMP is bad about hanging on to the original password. The snmpd.conf manual page states that the /var/net-snmp/snmpd.conf entry will be replaced with one containing just kul, but this does not seem to work:

This directive [the createUser line above] should be placed into the /var/net-snmp/snmpd.conf file instead of the other normal locations. The reason is that the information is read from the file and then the line is removed (eliminating the storage of the master password for that user) and replaced with the key that is derived from it. This key is a localized key, so that if it is stolen it can not be used to access other agents. If the password is stolen, however, it can be.

OpenNMS use of SNMPv3:
see opennms.org/index.php/SNMPv3_protocol_configuration