tagalog Package

filters Module

The filters Module contains the definitions for the filters supported by logtag and logship, as well as a number of functions used to construct the filter chains.

A filter is a generator function which takes an iterable as its first argument, and optionally takes additional configuration arguments. It returns a generator yielding the filtered log lines.

tagalog.filters.add_fields(iterable, **kw_fields)

Add fields to each item in iterable. Each key=value pair provided is merged into the @fields object, which will be created if required.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_fields(data_in, foo='bar', baz='qux')
>>> [x for x in data_out]
[{'@fields': {'foo': 'bar', 'baz': 'qux'}, '@message': 'one message'},
 {'@fields': {'foo': 'bar', 'baz': 'qux'}, '@message': 'another message'}]
tagalog.filters.add_source_host(iterable, override=False)

Add the FQDN of the current machine to the @source_host field of each item in iterable.

By default, existing @source_host fields will not be overwritten. This behaviour can be toggled with the override argument.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_source_host(data_in)
>>> [x for x in data_out]
[{'@source_host': 'lynx.local', '@message': 'one message'},
 {'@source_host': 'lynx.local', '@message': 'another message'}]
tagalog.filters.add_tags(iterable, *taglist)

Add tags to each item in iterable. Each tag is added to the @tags array, which will be created if required.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_tags(data_in, 'foo', 'bar')
>>> [x for x in data_out]
[{'@message': 'one message', '@tags': ['foo', 'bar']},
 {'@message': 'another message', '@tags': ['foo', 'bar']}]
tagalog.filters.add_timestamp(iterable, override=False)

Compute an accurate timestamp for each item in iterable, adding an accurate timestamp to each one when received. The timestamp is a usecond-precision ISO8601 string added to the @timestamp field.

By default, existing @timestamp fields will not be overwritten. This behaviour can be toggled with the override argument.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_timestamp(data_in)
>>> [x for x in data_out]
[{'@timestamp': '2013-05-13T10:37:56.766743Z', '@message': 'one message'},
 {'@timestamp': '2013-05-13T10:37:56.767185Z', '@message': 'another message'}]
tagalog.filters.build(description)

Build a filter chain from a filter description string

tagalog.filters.get(name, args=[])

Get a filter function from a filter name and a list of unparsed arguments

tagalog.filters.init_json(iterable)

Read lines of JSON text from iterable and parse each line as a JSON object. Yield dicts for each line that successfully parses as a JSON object. Unparseable events will be skipped and raise a warning.

>>> data_in = ['{"@message": "one message"}', '{"@message": "another message"}']
>>> data_out = init_json(data_in)
>>> [x for x in data_out]
[{u'@message': u'one message'}, {u'@message': u'another message'}]
tagalog.filters.init_txt(iterable)

Read lines of text from iterable and yield dicts with the line data stored in the @message field.

>>> data_in = ["hello\n", "world\n"]
>>> data_out = init_txt(data_in)
>>> [x for x in data_out]
[{'@message': 'hello'}, {'@message': 'world'}]
tagalog.filters.parse_lograge(iterable)

Attempt to parse each dict or dict-like object in iterable as if it were in lograge format, (e.g. “status=200 path=/users/login time=125ms”), adding key-value pairs to '@fields‘ for each matching item.

>>> data_in = [{'@message': 'path=/foo/bar status=200 time=0.060'},
...            {'@message': 'path=/baz/qux status=503 time=1.651'}]
>>> data_out = parse_lograge(data_in)
>>> [x for x in data_out]
[{'@fields': {'status': '200', 'path': '/foo/bar', 'time': '0.060'}, '@message': 'path=/foo/bar status=200 time=0.060'},
 {'@fields': {'status': '503', 'path': '/baz/qux', 'time': '1.651'}, '@message': 'path=/baz/qux status=503 time=1.651'}]
tagalog.filters.pipeline(*functions)

Construct a filter pipeline from a list of filter functions

Given a list of functions taking an iterable as their only argument, and which return a generator, this function will return a single function with the same signature, which applies each function in turn for each item in the iterable, yielding the results.

That is, given filter functions a, b, and c, pipeline(a, b, c) will return a function which yields c(b(a(x))) for each item x in the iterable passed to it. For example:

>>> def a(iterable):
...     for item in iterable:
...         yield item + ':a'
...
>>> def b(iterable):
...     for item in iterable:
...         yield item + ':b'
...
>>> pipe = pipeline(a, b)
>>> data_in = ["foo", "bar"]
>>> data_out = pipe(data_in)
>>> [x for x in data_out]
['foo:a:b', 'bar:a:b']

io Module

tagalog.io.lines(fp)

Read lines of UTF-8 from the file-like object given in fp, making sure that when reading from STDIN, reads are at most line-buffered.

UTF-8 decoding errors are handled silently. Invalid characters are replaced by U+FFFD REPLACEMENT CHARACTER.

Line endings are normalised to newlines by Python’s universal newlines feature.

Returns an iterator yielding lines.