Commit c7ebbbba authored by David Johnson's avatar David Johnson

Pull in upstream changes, which now include most of ours.

parents 2b5fe3e6 206fb5d7
......@@ -17,6 +17,7 @@ tests/*xml
tests/*log
output.html
MANIFEST
README.md
setup.ini
docs/changelog.rst
docs/makefile.rst
......@@ -30,3 +31,4 @@ pyroute2.egg-info
benchmark.log
.tox
tests-workspaces/
pyroute2/config/version.py
Changelog
=========
* 0.5.12
* rtnl: implement team config pass <https://github.com/svinota/pyroute2/issues/699>
* ndb.auth: example auth modules for RADIUS and OpenStack / Keystone
* cli: syntax fixes
* 0.5.11
* ndb.report: filters and transformations
* ndb.objects.route: support basic MPLS routes management
* ndb.objects.route: support MPLS lwtunnel routes
* ndb.schema: reschedule events
* 0.5.10
* general: don't use pkg_resources <https://github.com/svinota/pyroute2/issues/677>
* iproute: fix Windows support
* netlink: provide the target field
* ndb: use the target field from the netlink header
* ndb: multiple SQL fixes, transactions fixed with the PostgreSQL backend
* ndb: multiple object cache fixes <https://github.com/svinota/pyroute2/issues/683>
* ndb.schema: drop DB triggers
* ndb.objects: fix object management within a netns <https://github.com/svinota/pyroute2/issues/628>
* ndb.objects.route: support route metrics
* ndb.objects.route: fix default route syntax
* 0.5.9
* ethtool: fix module setup
* 0.5.8
* ethtool: initial support <https://github.com/svinota/pyroute2/pull/675>
* tc: multimatch support <https://github.com/svinota/pyroute2/pull/674>
* tc: meta support <https://github.com/svinota/pyroute2/pull/671>
* tc: cake: add stats_app decoder <https://github.com/svinota/pyroute2/pull/662>
* conntrack: filter <https://github.com/svinota/pyroute2/pull/660>
* ndb.objects.interface: reload after setns
* ndb.objects.route: create() dst syntax
* ndb.objects.route: 'default' syntax
* wireguard: basic testing
* 0.5.7
* ndb.objects.netns: prototype
* ndb: netns management
* ndb: netns sources autoconnect (disabled by default)
* wireguard: basic support
* netns: fix FD leakage
* <https://github.com/svinota/pyroute2/issues/623>
* cli: Python3 fixes
* iproute: support `route('append', ...)`
* ipdb: fix routes cleanup on link down
* <https://github.com/svinota/pyroute2/issues/620>
* wiset: support "mark" ipset type
* 0.5.6
* ndb.objects.route: multipath routes
* ndb.objects.rule: basic support
* ndb.objects.interface: veth fixed
* ndb.source: fix source restart
* ndb.log: logging setup
* 0.5.5
* nftables: rules expressions
* <https://github.com/svinota/pyroute2/pull/592>
......
include python-pyroute2.spec
include setup.ini
include README*
include README.md
include README.make.md
include README.report.md
include README.license.md
include LICENSE*
include CHANGELOG.md
graft docs/html
......
......@@ -15,7 +15,7 @@ release := $(shell git describe | sed 's/-[^-]*$$//;s/-/.post/')
python ?= python
nosetests ?= nosetests
flake8 ?= flake8
setuplib ?= distutils.core
setuplib ?= setuptools
epydoc ?= epydoc
##
# Python -W flags:
......@@ -60,6 +60,7 @@ all:
clean: clean-version
@rm -rf dist build MANIFEST
@rm -f README.md
@rm -f docs-build.log
@rm -f docs/general.rst
@rm -f docs/changelog.rst
......@@ -80,11 +81,13 @@ clean: clean-version
@rm -rf pyroute2.egg-info
@rm -rf tests-workspaces
@rm -f python-pyroute2.spec
@rm -f pyroute2/config/version.py
@find pyroute2 -name "*pyc" -exec rm -f "{}" \;
@find pyroute2 -name "*pyo" -exec rm -f "{}" \;
setup.ini:
@awk 'BEGIN {print "[setup]\nversion=${version}\nrelease=${release}\nsetuplib=${setuplib}"}' >setup.ini
@awk 'BEGIN {print "__version__ = \"${release}\""}' >pyroute2/config/version.py
clean-version:
@rm -f setup.ini
......@@ -93,9 +96,8 @@ force-version: clean-version update-version
update-version: setup.ini
docs: force-version
@cp README.md docs/general.rst
@sed -i '1{s/.*docs\//.. image:: /;s/\ ".*/\n\ \ \ \ :align: right/}' docs/general.rst
docs: force-version README.md
@cp README.rst docs/general.rst
@cp README.make.md docs/makefile.rst
@cp README.report.md docs/report.rst
@cp CHANGELOG.md docs/changelog.rst
......@@ -152,7 +154,10 @@ upload: clean force-version docs
dist: force-version docs
@${python} setup.py sdist >/dev/null 2>&1
install: clean force-version
README.md:
@cat README.rst | python ./docs/conv.py >README.md
install: clean force-version README.md
${python} setup.py install ${root} ${lib}
# in order to get it working, one should install pyroute2
......
Pyroute2
========
Pyroute2 is a pure Python **netlink** library. It requires only Python stdlib,
no 3rd party libraries. The library was started as an RTNL protocol
Pyroute2 is a pure Python **netlink** library. The core requires only Python
stdlib, no 3rd party libraries. The library was started as an RTNL protocol
implementation, so the name is **pyroute2**, but now it supports many netlink
protocols. Some supported netlink families and protocols:
* **rtnl**, network settings --- addresses, routes, traffic controls
* **nfnetlink** --- netfilter API: **ipset**, **nftables**, ...
* **nfnetlink** --- netfilter API
* **ipq** --- simplest userspace packet filtering, iptables QUEUE target
* **devlink** --- manage and monitor devlink-enabled hardware
* **generic** --- generic netlink families
* **nl80211** --- wireless functions API (basic support)
* **taskstats** --- extended process statistics
* **acpi_events** --- ACPI events monitoring
* **thermal_events** --- thermal events monitoring
* **VFS_DQUOT** --- disk quota events monitoring
* **uevent** --- same uevent messages as in udev
Starting with 0.5.2 the library supports also PF_ROUTE sockets on BSD systems.
Netfilter API:
* **ipset** --- IP sets
* **nftables** --- packet filtering
* **nfct** --- connection tracking
Generic netlink:
* **ethtool** --- low-level network interface setup
* **wireguard** --- VPN setup
* **nl80211** --- wireless functions API (basic support)
* **taskstats** --- extended process statistics
* **acpi_events** --- ACPI events monitoring
* **thermal_events** --- thermal events monitoring
* **VFS_DQUOT** --- disk quota events monitoring
On the low level the library provides socket objects with an
extended API. The additional functionality aims to:
* Help to open/bind netlink sockets
* Discover generic netlink protocols and multicast groups
* Construct, encode and decode netlink and PF_ROUTE messages
Supported systems
-----------------
......@@ -28,26 +45,50 @@ and standard system tools.
Other platforms are not supported.
The simplest usecase
--------------------
NDB -- high level RTNL API
--------------------------
Key features:
The objects, provided by the library, are socket objects with an
extended API. The additional functionality aims to:
* Data integrity
* Transactions with commit/rollback changes
* State synchronization
* Multiple sources, including netns and remote systems
* Help to open/bind netlink sockets
* Discover generic netlink protocols and multicast groups
* Construct, encode and decode netlink and PF_ROUTE messages
A "Hello world" example:
Maybe the simplest usecase is to monitor events. Disk quota events::
.. code-block:: python
from pyroute2 import DQuotSocket
# DQuotSocket automatically performs discovery and binding,
# since it has no other functionality beside of the monitoring
with DQuotSocket() as ds:
for message in ds.get():
print(message)
from pyroute2 import NDB
ndb = NDB(log='on')
for record in ndb.interfaces.summary():
print(record.ifname, record.address, record.state)
for record in ndb.addresses.summary():
print(record._as_dict())
Get notifications about network settings changes with IPRoute::
(ndb
.interfaces
.create(ifname='br0', kind='bridge') # create a bridge
.add_port('eth0') # add ports
.add_port('eth1') #
.add_ip('10.0.0.1/24') # add addresses
.add_ip('192.168.0.1/24') #
.set('br_stp_state', 1) # set STP on
.set('state', 'up') # bring the interface up
.commit()) # commit pending changes
IPRoute -- Low level RTNL API
-----------------------------
Low-level **IPRoute** utility --- Linux network configuration.
The **IPRoute** class is a 1-to-1 RTNL mapping. There are no implicit
interface lookups and so on.
Get notifications about network settings changes with IPRoute:
.. code-block:: python
from pyroute2 import IPRoute
with IPRoute() as ipr:
......@@ -56,109 +97,40 @@ Get notifications about network settings changes with IPRoute::
for message in ipr.get():
print(message)
RTNetlink examples
------------------
More samples you can read in the project documentation.
Low-level **IPRoute** utility --- Linux network configuration.
The **IPRoute** class is a 1-to-1 RTNL mapping. There are no implicit
interface lookups and so on.
More examples:
Some examples::
.. code-block:: python
from socket import AF_INET
from pyroute2 import IPRoute
# get access to the netlink socket
ip = IPRoute()
# no monitoring here -- thus no bind()
# print interfaces
print(ip.get_links())
for link in ip.get_links():
print(link)
# create VETH pair and move v0p1 to netns 'test'
ip.link('add', ifname='v0p0', peer='v0p1', kind='veth')
idx = ip.link_lookup(ifname='v0p1')[0]
ip.link('set',
index=idx,
net_ns_fd='test')
ip.link('set', index=idx, net_ns_fd='test')
# bring v0p0 up and add an address
idx = ip.link_lookup(ifname='v0p0')[0]
ip.link('set',
index=idx,
state='up')
ip.addr('add',
index=idx,
address='10.0.0.1',
broadcast='10.0.0.255',
prefixlen=24)
# create a route with metrics
ip.route('add',
dst='172.16.0.0/24',
gateway='10.0.0.10',
metrics={'mtu': 1400,
'hoplimit': 16})
# create MPLS lwtunnel
# $ sudo modprobe mpls_iptunnel
ip.route('add',
dst='172.16.0.0/24',
oif=idx,
encap={'type': 'mpls',
'labels': '200/300'})
# create MPLS route: push label
# $ sudo modprobe mpls_router
# $ sudo sysctl net.mpls.platform_labels=1024
ip.route('add',
family=AF_MPLS,
oif=idx,
dst=0x200,
newdst=[0x200, 0x300])
# create SEG6 tunnel encap mode
# Kernel >= 4.10
ip.route('add',
dst='2001:0:0:10::2/128',
oif=idx,
encap={'type': 'seg6',
'mode': 'encap',
'segs': '2000::5,2000::6'})
# create SEG6 tunnel inline mode
# Kernel >= 4.10
ip.route('add',
dst='2001:0:0:10::2/128',
oif=idx,
encap={'type': 'seg6',
'mode': 'inline',
'segs': ['2000::5', '2000::6']})
# create SEG6 tunnel with ip4ip6 encapsulation
# Kernel >= 4.14
ip.route('add',
dst='172.16.0.0/24',
oif=idx,
encap={'type': 'seg6',
'mode': 'encap',
'segs': '2000::5,2000::6'})
ip.link('set', index=idx, state='up')
ip.addr('add', index=idx, address='10.0.0.1', prefixlen=24)
# release Netlink socket
ip.close()
The project contains several modules for different types of
netlink messages, not only RTNL.
Network namespace examples
--------------------------
Network namespace manipulation::
Network namespace manipulation:
.. code-block:: python
from pyroute2 import netns
# create netns
......@@ -168,27 +140,26 @@ Network namespace manipulation::
# remove netns
netns.remove('test')
Create **veth** interfaces pair and move to **netns**::
Create **veth** interfaces pair and move to **netns**:
.. code-block:: python
from pyroute2 import IPRoute
with IPRoute() as ipr:
# create interface pair
ipr.link('add',
ifname='v0p0',
kind='veth',
peer='v0p1')
ipr.link('add', ifname='v0p0', kind='veth', peer='v0p1')
# lookup the peer index
idx = ipr.link_lookup(ifname='v0p1')[0]
# move the peer to the 'test' netns:
ipr.link('set',
index='v0p1',
net_ns_fd='test')
ipr.link('set', index='v0p1', net_ns_fd='test')
List interfaces in some **netns**:
List interfaces in some **netns**::
.. code-block:: python
from pyroute2 import NetNS
from pprint import pprint
......@@ -209,18 +180,20 @@ Requirements
Python >= 2.7
The pyroute2 testing framework requirements:
The pyroute2 testing and documentaion framework requirements:
* flake8
* coverage
* nosetests
* sphinx
* aafigure
* netaddr
* dtcd (optional, https://github.com/svinota/dtcd)
Optional dependencies:
Optional dependencies for testing:
* eventlet
* mitogen
* bottle
* team (http://libteam.org/)
* mitogen -- for distributed rtnl
* psutil -- for ss2 tool
Links
-----
......@@ -230,4 +203,3 @@ Links
* bugs: https://github.com/svinota/pyroute2/issues
* pypi: https://pypi.python.org/pypi/pyroute2
* docs: http://docs.pyroute2.org/
* list: https://groups.google.com/d/forum/pyroute2-dev
......@@ -3,15 +3,25 @@ import json
import argparse
from pyroute2 import Console
from pyroute2 import Server
try:
from pyroute2.cli.auth.auth_keystone import OSAuthManager
except ImportError:
OSAuthManager = None
try:
from pyroute2.cli.auth.auth_radius import RadiusAuthManager
except ImportError:
RadiusAuthManager = None
argp = argparse.ArgumentParser()
for spec in (('-a', '[S] IP address to listen on'),
('-c', '[C] Command line to run'),
('-l', '[C,S] Log spec'),
('-m', 'set mode (C,S)'),
('-m', 'Set mode (C,S)'),
('-p', '[S] Port to listen on'),
('-r', '[C] Load rc file'),
('-s', '[C,S] Load sources from a json file')):
('-s', '[C,S] Load sources from a json file'),
('-x', '[S] Strict auth')):
argp.add_argument(spec[0], help=spec[1])
args = argp.parse_args()
commands = []
......@@ -25,10 +35,19 @@ if args.m in ('S', 'server'):
port = int(args.p)
else:
port = 8080
auth_plugins = {}
if OSAuthManager is not None:
auth_plugins['keystone'] = OSAuthManager
if RadiusAuthManager is not None:
auth_plugins['radius:cleartext'] = RadiusAuthManager
server = Server(address=args.a or 'localhost',
port=port,
log=args.l,
sources=sources)
sources=sources,
auth_strict=args.x,
auth_plugins=auth_plugins)
server.serve_forever()
else:
console = Console(log=args.l, sources=sources)
......
......@@ -23,8 +23,10 @@ import argparse
from socket import (AF_INET,
AF_UNIX
)
import collections
import psutil
try:
import psutil
except ImportError:
psutil = None
from pyroute2 import DiagSocket
from pyroute2.netlink.diag import (SS_ESTABLISHED,
SS_SYN_SENT,
......@@ -42,12 +44,18 @@ from pyroute2.netlink.diag import (SS_ESTABLISHED,
from pyroute2.netlink.diag import (UDIAG_SHOW_NAME,
UDIAG_SHOW_VFS,
UDIAG_SHOW_PEER)
try:
from collections.abc import Mapping
from collections.abc import Callable
except ImportError:
from collections import Mapping
from collections import Callable
# UDIAG_SHOW_ICONS,
# UDIAG_SHOW_RQLEN,
# UDIAG_SHOW_MEMINFO
class UserCtxtMap(collections.Mapping):
class UserCtxtMap(Mapping):
_data = {}
......@@ -111,6 +119,11 @@ class UserCtxtMap(collections.Mapping):
_recurs_path=_recurs_path)
def _enter_item(self, usr, flow, ctxt):
if not flow.pid:
# corner case of eg anonnymous AddressFamily.AF_UNIX
# sockets
return
sk_inode = int(self._parse_inode(flow))
sk_fd = flow.fd
......@@ -150,7 +163,7 @@ class UserCtxtMap(collections.Mapping):
raise RuntimeError("Not implemented")
class Protocol(collections.Callable):
class Protocol(Callable):
class Resolver:
@staticmethod
def getHost(ip):
......@@ -542,6 +555,9 @@ def prepare_args():
def run(args=None):
if psutil is None:
raise RuntimeError('ss2 requires python-psutil >= 5.0 to run')
if not args:
args = prepare_args()
......
......@@ -271,7 +271,9 @@ div.viewcode-block:target {
dl.method > dt {
margin-bottom: 1em;
padding: 1em;
border-left: solid 1px #ccc;
border-bottom: solid 1px #ccc;
border-radius: 10px;
}
dl.class > dt {
margin-bottom: 1em;
padding: 1em;
}
......@@ -18,18 +18,28 @@ In other sense any netlink socket is just an ordinary socket
with `fileno()`, `recv()`, `sendto()` etc. Of course, one
can use it in `poll()`.
There is an inheritance diagram of netlink sockets, provided
There is an inheritance diagram of Linux netlink sockets, provided
by the library:
.. inheritance-diagram:: pyroute2.iproute.linux.IPRoute
pyroute2.iproute.linux.IPBatch
pyroute2.iproute.linux.RawIPRoute
pyroute2.iproute.bsd.IPRoute
pyroute2.iproute.RemoteIPRoute
pyroute2.iwutil.IW
pyroute2.ipset.IPSet
pyroute2.netlink.uevent.UeventSocket
pyroute2.netlink.taskstats.TaskStats
pyroute2.netlink.generic.wireguard.WireGuard
pyroute2.netlink.generic.ethtool.NlEthtool
pyroute2.netlink.ipq.IPQSocket
pyroute2.netlink.nfnetlink.nfctsocket.NFCTSocket
pyroute2.netlink.nfnetlink.nftsocket.NFTSocket
pyroute2.netlink.event.EventSocket
pyroute2.netlink.event.acpi_event.AcpiEventSocket
pyroute2.netlink.event.dquot.DQuotSocket
pyroute2.netlink.event.thermal_event.ThermalEventSocket
pyroute2.netlink.devlink.DevlinkSocket
pyroute2.netlink.diag.DiagSocket
pyroute2.remote.RemoteSocket
pyroute2.remote.shell.ShellIPR
pyroute2.netns.nslink.NetNS
......@@ -132,20 +142,55 @@ type, one should inherit:
The messages hierarchy:
.. inheritance-diagram:: pyroute2.netlink.rtnl.ndmsg.ndmsg
pyroute2.netlink.rtnl.ndtmsg.ndtmsg
pyroute2.netlink.rtnl.tcmsg.tcmsg
pyroute2.netlink.rtnl.rtmsg.nlflags
pyroute2.netlink.rtnl.rtmsg.rtmsg_base
pyroute2.netlink.rtnl.rtmsg.rtmsg
pyroute2.netlink.rtnl.rtmsg.nh
pyroute2.netlink.rtnl.fibmsg.fibmsg
pyroute2.netlink.rtnl.ifaddrmsg.ifaddrmsg
pyroute2.netlink.rtnl.ifstatsmsg.ifstatsmsg
pyroute2.netlink.rtnl.ifinfmsg.ifinfmsg
pyroute2.netlink.rtnl.ifinfmsg.ifinfveth
pyroute2.netlink.rtnl.iw_event.iw_event
pyroute2.netlink.rtnl.nsidmsg.nsidmsg
pyroute2.netlink.rtnl.nsinfmsg.nsinfmsg
pyroute2.netlink.rtnl.rtgenmsg.rtgenmsg
pyroute2.netlink.devlink.devlinkcmd
pyroute2.netlink.diag.inet_addr_codec
pyroute2.netlink.diag.inet_diag_req
pyroute2.netlink.diag.inet_diag_msg
pyroute2.netlink.diag.unix_diag_req
pyroute2.netlink.diag.unix_diag_msg
pyroute2.netlink.event.acpi_event.acpimsg
pyroute2.netlink.event.dquot.dquotmsg
pyroute2.netlink.event.thermal_event.thermal_msg
pyroute2.netlink.taskstats.taskstatsmsg
pyroute2.netlink.taskstats.tcmd
pyroute2.netlink.generic.ethtool.ethtool_strset_msg
pyroute2.netlink.generic.ethtool.ethtool_linkinfo_msg
pyroute2.netlink.generic.ethtool.ethtool_linkmode_msg
pyroute2.netlink.generic.ethtool.ethtool_linkstate_msg
pyroute2.netlink.generic.ethtool.ethtool_wol_msg
pyroute2.netlink.generic.wireguard.wgmsg
pyroute2.netlink.ctrlmsg
pyroute2.netlink.genlmsg
pyroute2.netlink.nl80211.nl80211cmd
pyroute2.netlink.nfnetlink.ipset.ipset_msg
pyroute2.netlink.nfnetlink.nfgen_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_gen_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_chain_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_rule_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_set_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_table_msg
pyroute2.netlink.nfnetlink.nfctsocket.nfct_stats
pyroute2.netlink.nfnetlink.nfctsocket.nfct_stats_cpu
pyroute2.netlink.nfnetlink.nfctsocket.nfct_msg
pyroute2.netlink.ipq.ipq_mode_msg
pyroute2.netlink.ipq.ipq_packet_msg
pyroute2.netlink.ipq.ipq_verdict_msg
pyroute2.netlink.uevent.ueventmsg
:parts: 1
PF_ROUTE messages
......
.. _choice:
IPRoute or NDB
--------------
There are two different modules in the library, working on different
abstraction levels and employing different approaches. But since they
may be used to solve similar tasks, here is a short comparison that
should help you to make the choice.
:ref:`iproute`
* Decent coverage of the kernel functionality
* Supports TC
* No service threads by default
* No additional Python objects to represent RTNL objects
* Low-level API
* No RTNL objects state synchronisation
* Works only with one RTNL source
:ref:`ndb`
* Only a subset of RTNL functionality
* Doesn't support TC
* Thread-based architecture
* Creates Python objects to reflect RTNL objects: may consume a lot of memory
* High-level API with state synchroniztion
* Objects state integrity
* Objects relations
* Aggregates multiple RTNL sources -- systems, netns etc.
While IPRoute provides better RTNL functionality coverage, it operates on
a different level than NDB. IPRoute is a plain 1-to-1 mapping of the