Skip to content

GitLab

P
pyroute2
Commit c7ebbbba authored by David Johnson's avatar David Johnson
Browse files
Options

Pull in upstream changes, which now include most of ours.

parents 2b5fe3e6 206fb5d7
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 10272 additions and 1755 deletions
.gitignore
View file @ c7ebbbba
......@@ -17,6 +17,7 @@ tests/*xml
tests/*log
output.html
MANIFEST
README.md
setup.ini
docs/changelog.rst
docs/makefile.rst
......@@ -30,3 +31,4 @@ pyroute2.egg-info
benchmark.log
.tox
tests-workspaces/
pyroute2/config/version.py
CHANGELOG.md
View file @ c7ebbbba
Changelog
=========
* 0.5.12
* rtnl: implement team config pass <https://github.com/svinota/pyroute2/issues/699>
* ndb.auth: example auth modules for RADIUS and OpenStack / Keystone
* cli: syntax fixes
* 0.5.11
* ndb.report: filters and transformations
* ndb.objects.route: support basic MPLS routes management
* ndb.objects.route: support MPLS lwtunnel routes
* ndb.schema: reschedule events
* 0.5.10
* general: don't use pkg_resources <https://github.com/svinota/pyroute2/issues/677>
* iproute: fix Windows support
* netlink: provide the target field
* ndb: use the target field from the netlink header
* ndb: multiple SQL fixes, transactions fixed with the PostgreSQL backend
* ndb: multiple object cache fixes <https://github.com/svinota/pyroute2/issues/683>
* ndb.schema: drop DB triggers
* ndb.objects: fix object management within a netns <https://github.com/svinota/pyroute2/issues/628>
* ndb.objects.route: support route metrics
* ndb.objects.route: fix default route syntax
* 0.5.9
* ethtool: fix module setup
* 0.5.8
* ethtool: initial support <https://github.com/svinota/pyroute2/pull/675>
* tc: multimatch support <https://github.com/svinota/pyroute2/pull/674>
* tc: meta support <https://github.com/svinota/pyroute2/pull/671>
* tc: cake: add stats_app decoder <https://github.com/svinota/pyroute2/pull/662>
* conntrack: filter <https://github.com/svinota/pyroute2/pull/660>
* ndb.objects.interface: reload after setns
* ndb.objects.route: create() dst syntax
* ndb.objects.route: 'default' syntax
* wireguard: basic testing
* 0.5.7
* ndb.objects.netns: prototype
* ndb: netns management
* ndb: netns sources autoconnect (disabled by default)
* wireguard: basic support
* netns: fix FD leakage
* <https://github.com/svinota/pyroute2/issues/623>
* cli: Python3 fixes
* iproute: support `route('append', ...)`
* ipdb: fix routes cleanup on link down
* <https://github.com/svinota/pyroute2/issues/620>
* wiset: support "mark" ipset type
* 0.5.6
* ndb.objects.route: multipath routes
* ndb.objects.rule: basic support
* ndb.objects.interface: veth fixed
* ndb.source: fix source restart
* ndb.log: logging setup
* 0.5.5
* nftables: rules expressions
* <https://github.com/svinota/pyroute2/pull/592>
......
MANIFEST.in
View file @ c7ebbbba
include python-pyroute2.spec
include setup.ini
include README*
include README.md
include README.make.md
include README.report.md
include README.license.md
include LICENSE*
include CHANGELOG.md
graft docs/html
......
Makefile
View file @ c7ebbbba
......@@ -15,7 +15,7 @@ release := $(shell git describe | sed 's/-[^-]*$$//;s/-/.post/')
python ?= python
nosetests ?= nosetests
flake8 ?= flake8
setuplib ?= distutils.core
setuplib ?= setuptools
epydoc ?= epydoc
##
# Python -W flags:
......@@ -60,6 +60,7 @@ all:
clean: clean-version
@rm -rf dist build MANIFEST
@rm -f README.md
@rm -f docs-build.log
@rm -f docs/general.rst
@rm -f docs/changelog.rst
......@@ -80,11 +81,13 @@ clean: clean-version
@rm -rf pyroute2.egg-info
@rm -rf tests-workspaces
@rm -f python-pyroute2.spec
@rm -f pyroute2/config/version.py
@find pyroute2 -name "*pyc" -exec rm -f "{}" \;
@find pyroute2 -name "*pyo" -exec rm -f "{}" \;
setup.ini:
@awk 'BEGIN {print "[setup]\nversion=${version}\nrelease=${release}\nsetuplib=${setuplib}"}' >setup.ini
@awk 'BEGIN {print "__version__ = \"${release}\""}' >pyroute2/config/version.py
clean-version:
@rm -f setup.ini
......@@ -93,9 +96,8 @@ force-version: clean-version update-version
update-version: setup.ini
docs: force-version
@cp README.md docs/general.rst
@sed -i '1{s/.*docs\//.. image:: /;s/\ ".*/\n\ \ \ \ :align: right/}' docs/general.rst
docs: force-version README.md
@cp README.rst docs/general.rst
@cp README.make.md docs/makefile.rst
@cp README.report.md docs/report.rst
@cp CHANGELOG.md docs/changelog.rst
......@@ -152,7 +154,10 @@ upload: clean force-version docs
dist: force-version docs
@${python} setup.py sdist >/dev/null 2>&1
install: clean force-version
README.md:
@cat README.rst | python ./docs/conv.py >README.md
install: clean force-version README.md
${python} setup.py install ${root} ${lib}
# in order to get it working, one should install pyroute2
......
README.md → README.rst
View file @ c7ebbbba
Pyroute2
========
Pyroute2 is a pure Python **netlink** library. It requires only Python stdlib,
no 3rd party libraries. The library was started as an RTNL protocol
Pyroute2 is a pure Python **netlink** library. The core requires only Python
stdlib, no 3rd party libraries. The library was started as an RTNL protocol
implementation, so the name is **pyroute2**, but now it supports many netlink
protocols. Some supported netlink families and protocols:
* **rtnl**, network settings --- addresses, routes, traffic controls
* **nfnetlink** --- netfilter API: **ipset**, **nftables**, ...
* **nfnetlink** --- netfilter API
* **ipq** --- simplest userspace packet filtering, iptables QUEUE target
* **devlink** --- manage and monitor devlink-enabled hardware
* **generic** --- generic netlink families
* **nl80211** --- wireless functions API (basic support)
* **taskstats** --- extended process statistics
* **acpi_events** --- ACPI events monitoring
* **thermal_events** --- thermal events monitoring
* **VFS_DQUOT** --- disk quota events monitoring
* **uevent** --- same uevent messages as in udev
Starting with 0.5.2 the library supports also PF_ROUTE sockets on BSD systems.
Netfilter API:
* **ipset** --- IP sets
* **nftables** --- packet filtering
* **nfct** --- connection tracking
Generic netlink:
* **ethtool** --- low-level network interface setup
* **wireguard** --- VPN setup
* **nl80211** --- wireless functions API (basic support)
* **taskstats** --- extended process statistics
* **acpi_events** --- ACPI events monitoring
* **thermal_events** --- thermal events monitoring
* **VFS_DQUOT** --- disk quota events monitoring
On the low level the library provides socket objects with an
extended API. The additional functionality aims to:
* Help to open/bind netlink sockets
* Discover generic netlink protocols and multicast groups
* Construct, encode and decode netlink and PF_ROUTE messages
Supported systems
-----------------
......@@ -28,26 +45,50 @@ and standard system tools.
Other platforms are not supported.
The simplest usecase
--------------------
NDB -- high level RTNL API
--------------------------
Key features:
The objects, provided by the library, are socket objects with an
extended API. The additional functionality aims to:
* Data integrity
* Transactions with commit/rollback changes
* State synchronization
* Multiple sources, including netns and remote systems
* Help to open/bind netlink sockets
* Discover generic netlink protocols and multicast groups
* Construct, encode and decode netlink and PF_ROUTE messages
A "Hello world" example:
Maybe the simplest usecase is to monitor events. Disk quota events::
.. code-block:: python
from pyroute2 import DQuotSocket
# DQuotSocket automatically performs discovery and binding,
# since it has no other functionality beside of the monitoring
with DQuotSocket() as ds:
for message in ds.get():
print(message)
from pyroute2 import NDB
ndb = NDB(log='on')
for record in ndb.interfaces.summary():
print(record.ifname, record.address, record.state)
for record in ndb.addresses.summary():
print(record._as_dict())
Get notifications about network settings changes with IPRoute::
(ndb
.interfaces
.create(ifname='br0', kind='bridge') # create a bridge
.add_port('eth0') # add ports
.add_port('eth1') #
.add_ip('10.0.0.1/24') # add addresses
.add_ip('192.168.0.1/24') #
.set('br_stp_state', 1) # set STP on
.set('state', 'up') # bring the interface up
.commit()) # commit pending changes
IPRoute -- Low level RTNL API
-----------------------------
Low-level **IPRoute** utility --- Linux network configuration.
The **IPRoute** class is a 1-to-1 RTNL mapping. There are no implicit
interface lookups and so on.
Get notifications about network settings changes with IPRoute:
.. code-block:: python
from pyroute2 import IPRoute
with IPRoute() as ipr:
......@@ -56,109 +97,40 @@ Get notifications about network settings changes with IPRoute::
for message in ipr.get():
print(message)
RTNetlink examples
------------------
More samples you can read in the project documentation.
Low-level **IPRoute** utility --- Linux network configuration.
The **IPRoute** class is a 1-to-1 RTNL mapping. There are no implicit
interface lookups and so on.
More examples:
Some examples::
.. code-block:: python
from socket import AF_INET
from pyroute2 import IPRoute
# get access to the netlink socket
ip = IPRoute()
# no monitoring here -- thus no bind()
# print interfaces
print(ip.get_links())
for link in ip.get_links():
print(link)
# create VETH pair and move v0p1 to netns 'test'
ip.link('add', ifname='v0p0', peer='v0p1', kind='veth')
idx = ip.link_lookup(ifname='v0p1')[0]
ip.link('set',
index=idx,
net_ns_fd='test')
ip.link('set', index=idx, net_ns_fd='test')
# bring v0p0 up and add an address
idx = ip.link_lookup(ifname='v0p0')[0]
ip.link('set',
index=idx,
state='up')
ip.addr('add',
index=idx,
address='10.0.0.1',
broadcast='10.0.0.255',
prefixlen=24)
# create a route with metrics
ip.route('add',
dst='172.16.0.0/24',
gateway='10.0.0.10',
metrics={'mtu': 1400,
'hoplimit': 16})
# create MPLS lwtunnel
# $ sudo modprobe mpls_iptunnel
ip.route('add',
dst='172.16.0.0/24',
oif=idx,
encap={'type': 'mpls',
'labels': '200/300'})
# create MPLS route: push label
# $ sudo modprobe mpls_router
# $ sudo sysctl net.mpls.platform_labels=1024
ip.route('add',
family=AF_MPLS,
oif=idx,
dst=0x200,
newdst=[0x200, 0x300])
# create SEG6 tunnel encap mode
# Kernel >= 4.10
ip.route('add',
dst='2001:0:0:10::2/128',
oif=idx,
encap={'type': 'seg6',
'mode': 'encap',
'segs': '2000::5,2000::6'})
# create SEG6 tunnel inline mode
# Kernel >= 4.10
ip.route('add',
dst='2001:0:0:10::2/128',
oif=idx,
encap={'type': 'seg6',
'mode': 'inline',
'segs': ['2000::5', '2000::6']})
# create SEG6 tunnel with ip4ip6 encapsulation
# Kernel >= 4.14
ip.route('add',
dst='172.16.0.0/24',
oif=idx,
encap={'type': 'seg6',
'mode': 'encap',
'segs': '2000::5,2000::6'})
ip.link('set', index=idx, state='up')
ip.addr('add', index=idx, address='10.0.0.1', prefixlen=24)
# release Netlink socket
ip.close()
The project contains several modules for different types of
netlink messages, not only RTNL.
Network namespace examples
--------------------------
Network namespace manipulation::
Network namespace manipulation:
.. code-block:: python
from pyroute2 import netns
# create netns
......@@ -168,27 +140,26 @@ Network namespace manipulation::
# remove netns
netns.remove('test')
Create **veth** interfaces pair and move to **netns**::
Create **veth** interfaces pair and move to **netns**:
.. code-block:: python
from pyroute2 import IPRoute
with IPRoute() as ipr:
# create interface pair
ipr.link('add',
ifname='v0p0',
kind='veth',
peer='v0p1')
ipr.link('add', ifname='v0p0', kind='veth', peer='v0p1')
# lookup the peer index
idx = ipr.link_lookup(ifname='v0p1')[0]
# move the peer to the 'test' netns:
ipr.link('set',
index='v0p1',
net_ns_fd='test')
ipr.link('set', index='v0p1', net_ns_fd='test')
List interfaces in some **netns**:
List interfaces in some **netns**::
.. code-block:: python
from pyroute2 import NetNS
from pprint import pprint
......@@ -209,18 +180,20 @@ Requirements
Python >= 2.7
The pyroute2 testing framework requirements:
The pyroute2 testing and documentaion framework requirements:
* flake8
* coverage
* nosetests
* sphinx
* aafigure
* netaddr
* dtcd (optional, https://github.com/svinota/dtcd)
Optional dependencies:
Optional dependencies for testing:
* eventlet
* mitogen
* bottle
* team (http://libteam.org/)
* mitogen -- for distributed rtnl
* psutil -- for ss2 tool
Links
-----
......@@ -230,4 +203,3 @@ Links
* bugs: https://github.com/svinota/pyroute2/issues
* pypi: https://pypi.python.org/pypi/pyroute2
* docs: http://docs.pyroute2.org/
* list: https://groups.google.com/d/forum/pyroute2-dev
cli/pyroute2-cli
View file @ c7ebbbba
......@@ -3,15 +3,25 @@ import json
import argparse
from pyroute2 import Console
from pyroute2 import Server
try:
from pyroute2.cli.auth.auth_keystone import OSAuthManager
except ImportError:
OSAuthManager = None
try:
from pyroute2.cli.auth.auth_radius import RadiusAuthManager
except ImportError:
RadiusAuthManager = None
argp = argparse.ArgumentParser()
for spec in (('-a', '[S] IP address to listen on'),
('-c', '[C] Command line to run'),
('-l', '[C,S] Log spec'),
('-m', 'set mode (C,S)'),
('-m', 'Set mode (C,S)'),
('-p', '[S] Port to listen on'),
('-r', '[C] Load rc file'),
('-s', '[C,S] Load sources from a json file')):
('-s', '[C,S] Load sources from a json file'),
('-x', '[S] Strict auth')):
argp.add_argument(spec[0], help=spec[1])
args = argp.parse_args()
commands = []
......@@ -25,10 +35,19 @@ if args.m in ('S', 'server'):
port = int(args.p)
else:
port = 8080
auth_plugins = {}
if OSAuthManager is not None:
auth_plugins['keystone'] = OSAuthManager
if RadiusAuthManager is not None:
auth_plugins['radius:cleartext'] = RadiusAuthManager
server = Server(address=args.a or 'localhost',
port=port,
log=args.l,
sources=sources)
sources=sources,
auth_strict=args.x,
auth_plugins=auth_plugins)
server.serve_forever()
else:
console = Console(log=args.l, sources=sources)
......
cli/ss2
View file @ c7ebbbba
......@@ -23,8 +23,10 @@ import argparse
from socket import (AF_INET,
AF_UNIX
)
import collections
import psutil
try:
import psutil
except ImportError:
psutil = None
from pyroute2 import DiagSocket
from pyroute2.netlink.diag import (SS_ESTABLISHED,
SS_SYN_SENT,
......@@ -42,12 +44,18 @@ from pyroute2.netlink.diag import (SS_ESTABLISHED,
from pyroute2.netlink.diag import (UDIAG_SHOW_NAME,
UDIAG_SHOW_VFS,
UDIAG_SHOW_PEER)
try:
from collections.abc import Mapping
from collections.abc import Callable
except ImportError:
from collections import Mapping
from collections import Callable
# UDIAG_SHOW_ICONS,
# UDIAG_SHOW_RQLEN,
# UDIAG_SHOW_MEMINFO
class UserCtxtMap(collections.Mapping):
class UserCtxtMap(Mapping):
_data = {}
......@@ -111,6 +119,11 @@ class UserCtxtMap(collections.Mapping):
_recurs_path=_recurs_path)
def _enter_item(self, usr, flow, ctxt):
if not flow.pid:
# corner case of eg anonnymous AddressFamily.AF_UNIX
# sockets
return
sk_inode = int(self._parse_inode(flow))
sk_fd = flow.fd
......@@ -150,7 +163,7 @@ class UserCtxtMap(collections.Mapping):
raise RuntimeError("Not implemented")
class Protocol(collections.Callable):
class Protocol(Callable):
class Resolver:
@staticmethod
def getHost(ip):
......@@ -542,6 +555,9 @@ def prepare_args():
def run(args=None):
if psutil is None:
raise RuntimeError('ss2 requires python-psutil >= 5.0 to run')
if not args:
args = prepare_args()
......
docs/_static/custom.css
View file @ c7ebbbba
......@@ -271,7 +271,9 @@ div.viewcode-block:target {
dl.method > dt {
margin-bottom: 1em;
padding: 1em;
border-left: solid 1px #ccc;
border-bottom: solid 1px #ccc;
border-radius: 10px;
}
dl.class > dt {
margin-bottom: 1em;
padding: 1em;
}
docs/arch.rst
View file @ c7ebbbba
......@@ -18,18 +18,28 @@ In other sense any netlink socket is just an ordinary socket
with `fileno()`, `recv()`, `sendto()` etc. Of course, one
can use it in `poll()`.
There is an inheritance diagram of netlink sockets, provided
There is an inheritance diagram of Linux netlink sockets, provided
by the library:
.. inheritance-diagram:: pyroute2.iproute.linux.IPRoute
pyroute2.iproute.linux.IPBatch
pyroute2.iproute.linux.RawIPRoute
pyroute2.iproute.bsd.IPRoute
pyroute2.iproute.RemoteIPRoute
pyroute2.iwutil.IW
pyroute2.ipset.IPSet
pyroute2.netlink.uevent.UeventSocket
pyroute2.netlink.taskstats.TaskStats
pyroute2.netlink.generic.wireguard.WireGuard
pyroute2.netlink.generic.ethtool.NlEthtool
pyroute2.netlink.ipq.IPQSocket
pyroute2.netlink.nfnetlink.nfctsocket.NFCTSocket
pyroute2.netlink.nfnetlink.nftsocket.NFTSocket
pyroute2.netlink.event.EventSocket
pyroute2.netlink.event.acpi_event.AcpiEventSocket
pyroute2.netlink.event.dquot.DQuotSocket
pyroute2.netlink.event.thermal_event.ThermalEventSocket
pyroute2.netlink.devlink.DevlinkSocket
pyroute2.netlink.diag.DiagSocket
pyroute2.remote.RemoteSocket
pyroute2.remote.shell.ShellIPR
pyroute2.netns.nslink.NetNS
......@@ -132,20 +142,55 @@ type, one should inherit:
The messages hierarchy:
.. inheritance-diagram:: pyroute2.netlink.rtnl.ndmsg.ndmsg
pyroute2.netlink.rtnl.ndtmsg.ndtmsg
pyroute2.netlink.rtnl.tcmsg.tcmsg
pyroute2.netlink.rtnl.rtmsg.nlflags
pyroute2.netlink.rtnl.rtmsg.rtmsg_base
pyroute2.netlink.rtnl.rtmsg.rtmsg
pyroute2.netlink.rtnl.rtmsg.nh
pyroute2.netlink.rtnl.fibmsg.fibmsg
pyroute2.netlink.rtnl.ifaddrmsg.ifaddrmsg
pyroute2.netlink.rtnl.ifstatsmsg.ifstatsmsg
pyroute2.netlink.rtnl.ifinfmsg.ifinfmsg
pyroute2.netlink.rtnl.ifinfmsg.ifinfveth
pyroute2.netlink.rtnl.iw_event.iw_event
pyroute2.netlink.rtnl.nsidmsg.nsidmsg
pyroute2.netlink.rtnl.nsinfmsg.nsinfmsg
pyroute2.netlink.rtnl.rtgenmsg.rtgenmsg
pyroute2.netlink.devlink.devlinkcmd
pyroute2.netlink.diag.inet_addr_codec
pyroute2.netlink.diag.inet_diag_req
pyroute2.netlink.diag.inet_diag_msg
pyroute2.netlink.diag.unix_diag_req
pyroute2.netlink.diag.unix_diag_msg
pyroute2.netlink.event.acpi_event.acpimsg
pyroute2.netlink.event.dquot.dquotmsg
pyroute2.netlink.event.thermal_event.thermal_msg
pyroute2.netlink.taskstats.taskstatsmsg
pyroute2.netlink.taskstats.tcmd
pyroute2.netlink.generic.ethtool.ethtool_strset_msg
pyroute2.netlink.generic.ethtool.ethtool_linkinfo_msg
pyroute2.netlink.generic.ethtool.ethtool_linkmode_msg
pyroute2.netlink.generic.ethtool.ethtool_linkstate_msg
pyroute2.netlink.generic.ethtool.ethtool_wol_msg
pyroute2.netlink.generic.wireguard.wgmsg
pyroute2.netlink.ctrlmsg
pyroute2.netlink.genlmsg
pyroute2.netlink.nl80211.nl80211cmd
pyroute2.netlink.nfnetlink.ipset.ipset_msg
pyroute2.netlink.nfnetlink.nfgen_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_gen_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_chain_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_rule_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_set_msg
pyroute2.netlink.nfnetlink.nftsocket.nft_table_msg
pyroute2.netlink.nfnetlink.nfctsocket.nfct_stats
pyroute2.netlink.nfnetlink.nfctsocket.nfct_stats_cpu
pyroute2.netlink.nfnetlink.nfctsocket.nfct_msg
pyroute2.netlink.ipq.ipq_mode_msg
pyroute2.netlink.ipq.ipq_packet_msg
pyroute2.netlink.ipq.ipq_verdict_msg
pyroute2.netlink.uevent.ueventmsg
:parts: 1
PF_ROUTE messages
......
docs/choice.rst 0 → 100644
View file @ c7ebbbba
.. _choice:
IPRoute or NDB
--------------
There are two different modules in the library, working on different
abstraction levels and employing different approaches. But since they
may be used to solve similar tasks, here is a short comparison that
should help you to make the choice.
:ref:`iproute`
* Decent coverage of the kernel functionality
* Supports TC
* No service threads by default
* No additional Python objects to represent RTNL objects
* Low-level API
* No RTNL objects state synchronisation
* Works only with one RTNL source
:ref:`ndb`
* Only a subset of RTNL functionality
* Doesn't support TC
* Thread-based architecture
* Creates Python objects to reflect RTNL objects: may consume a lot of memory
* High-level API with state synchroniztion
* Objects state integrity
* Objects relations
* Aggregates multiple RTNL sources -- systems, netns etc.
While IPRoute provides better RTNL functionality coverage, it operates on
a different level than NDB. IPRoute is a plain 1-to-1 mapping of the