beagle.datasources package

Submodules

beagle.datasources.base_datasource module

class beagle.datasources.base_datasource.DataSource[source]

Bases: object

Base DataSource class. This class should be used to create DataSources which are file based.

For non-file based data sources (i.e performing a HTTP request to an API to get some data). The ExternalDataSource class should be subclassed.

Each datasource requires the following annotations be made:

  1. name string: The name of the datasource, this should be human readable.
  2. transformer List[Transformer]: The list of transformers which you can send events from this datasource to.
  3. category string: The category this datasource outputs data to, this should be human readable.

Not supplying these three will not allow the class to get created, and will prevent beagle from loading.

Examples

>>> class MyDataSource(DataSource):
        name = "My Data Source"
        transformers = [GenericTransformer]
        category = "My Category"
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
to_graph(*args, **kwargs) → Any[source]

Allows to hop immediatly from a datasource to a graph.

Supports parameters for the to_graph() function of the transformer.

see :py:method:`beagle.transformers.base_transformer.Transformer.to_graph`

Examples

>>> SysmonEVTX('data/sysmon/autoruns-sysmon.evtx').to_graph(Graphistry, render=True)
Returns:Returns the outuput of the Backends .graph() function.
Return type:Any
to_transformer(transformer: Transformer = None) → Transformer[source]

Allows the data source to be used as a functional API. By default, uses the first transformer in the transformers attribute.

>>> graph = DataSource().to_transformer().to_graph()
Returns:A instance of the transformer class yielded to.
Return type:Transformer
class beagle.datasources.base_datasource.ExternalDataSource[source]

Bases: beagle.datasources.base_datasource.DataSource

This class should be used when fetching data from exteranl sources before processing.

Using a different class allows the web interface to render a different upload page when a data source requiring text input in favor of a file input is used.

Examples

See beagle.datasources.virustotal.generic_vt_sandbox_api.GenericVTSandboxAPI

beagle.datasources.fireeye_ax_report module

class beagle.datasources.fireeye_ax_report.FireEyeAXReport(ax_report: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events one by one from a FireEyeAX Report and sends them to the generic transformer.

The JSON report should look something like this:

{
    "alert": [
        {
            "explanation": {
                "malwareDetected": {
                    ...
                },
                "cncServices": {
                    "cncService": [
                        ...
                },
                "osChanges": [
                    {
                        "process": [...],
                        "registry": [...],
                        ...
                }
            }
        }
    ]
}

Beagle looks at the first alert in the alerts array.

Parameters:ax_report (str) – File path to the JSON AX Report, see class description for expected format.
category = 'FireEye AX'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'FireEye AX Report'
transformers = [<class 'beagle.transformers.fireeye_ax_transformer.FireEyeAXTransformer'>]

beagle.datasources.hx_triage module

class beagle.datasources.hx_triage.HXTriage(triage: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

A FireEye HX Triage DataSource.

Allows generation of graphs from the redline .mans files generated by FireEye HX.

Examples

>>> triage = HXTriage(file_path="/path/to/triage.mans")
category = 'FireEye HX'
events() → Generator[[dict, None], None][source]

Yields each event in the triage from the supported files.

metadata() → dict[source]

Returns basic information about the triage.

  1. Agent ID
  2. Hostname
  3. Platform (win, osx, linux)
  4. Triggering Alert name (if exists)
  5. Link to the controller the triage is from
Returns:Metadata for the submitted HX Triage.
Return type:dict
name = 'FireEye HX Triage'
parse_agent_events(agent_events_file: str) → Generator[[dict, None], None][source]

Generator over the agent events file. Converts each XML into a dictionary. Timestamps are converted to epoch time.

The below XML entry:

<eventItem uid="39265403">
    <timestamp>2018-06-27T21:15:32.678Z</timestamp>
    <eventType>dnsLookupEvent</eventType>
    <details>
    <detail>
        <name>hostname</name>
        <value>github.com</value>
    </detail>
    <detail>
        <name>pid</name>
        <value>12345</value>
    </detail>
    <detail>
        <name>process</name>
        <value>git.exe</value>
    </detail>
    <detail>
        <name>processPath</name>
        <value>c:\windows\</value>
    </detail>
    <detail>
        <name>username</name>
        <value>Bob/Schmob</value>
    </detail>
    </details>
</eventItem>

becomes:

{
    "timestamp": 1530134132,
    "eventType": "dnsLookupEvent",
    "hostname": "github.com",
    "pid": "12345",
    "process": "git.exe",
    "processPath": "c:\windows\",
    "username": "Bob/Schmob",
}
Parameters:agent_events_file (str) – The path to the file containing the agent events.
Returns:Generator over agent events.
Return type:Generator[dict, None, None]
parse_alert_files(temp_dir: str) → Generator[[dict, None], None][source]

Parses out the alert files from the hits.json and threats.json files

Parameters:temp_dir (str) – Folder which contains the expanded triage.
Yields:Generator[dict, None, None] – The next event found in the Triage.
transformers = [<class 'beagle.transformers.fireeye_hx_transformer.FireEyeHXTransformer'>]

beagle.datasources.procmon_csv module

class beagle.datasources.procmon_csv.ProcmonCSV(procmon_csv: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Reads events in one by one from a ProcMon CSV, and parses them into the GenericTransformer

category = 'Procmon'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Procmon CSV'
transformers = [<class 'beagle.transformers.procmon_transformer.ProcmonTransformer'>]

beagle.datasources.sysmon_evtx module

class beagle.datasources.sysmon_evtx.SysmonEVTX(sysmon_evtx_log_file: str)[source]

Bases: beagle.datasources.win_evtx.WinEVTX

Parses SysmonEVTX files, see beagle.datasources.win_evtx.WinEVTX

category = 'SysMon'
metadata() → dict[source]

Returns the Hostname by inspecting the Computer entry of the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Sysmon EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Parse a single record recursivly into a JSON file with a single level.

Parameters:
  • record (etree.ElementTree) – The current record.
  • name (str, optional) – Last records name. (the default is “”, which [default_description])
Returns:

dict representation of record.
Return type:

dict
transformers = [<class 'beagle.transformers.sysmon_transformer.SysmonTransformer'>]

beagle.datasources.win_evtx module

class beagle.datasources.win_evtx.WinEVTX(evtx_log_file: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Parses Windows .evtx files. Yields events one by one using the python-evtx library.

Parameters:evtx_log_file (str) – The path to the windows evtx file to parse.
category = 'Windows Event Logs'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Get the hostname by inspecting the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Windows EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Recursivly converts a etree.ElementTree record to a JSON dictionary with one level.

Parameters:
  • record (etree.ElementTree) – Current record to parse
  • name (str, optional) – Name of the current key we are at.
Returns:

JSON represntation of the event
Return type:

dict
transformers = [<class 'beagle.transformers.evtx_transformer.WinEVTXTransformer'>]

Module contents