Introduction
The bedrock of asset discovery in ThreatWatch is ‘twigs’ (short for ThreatWatch Inventory Gathering Script). twigs is a python based open source utility script maintained by ThreatWatch. twigs aims to provide a simple extensible interface to all types of assets and asset management systems in order to discover the metadata required by ThreatWatch to perform effective no-scan vulnerability management.
Please note that twigs is only designed to discover and inventory the metadata required for no-scan vulnerability assessments using ThreatWatch and it does not do any vulnerability assessment by itself. As of this writing, twigs can inventory hosts, AWS/Azure cloud assets, container images, open source dependencies from code repositories and ServiceNow CMDB. In the future, twigs will continue to extend to cover more types of assets and asset databases.
Requirements and installation
twigs can run on most Linux based systems (Redhat, CentOS, Ubuntu, Debian, Amazon Linux AMI) with Python 2.7. The github page for twigs is here and the latest online documentation is available here.
To install twigs simply run
$ sudo pip install twigs
on any suitable Linux system that needs to be discovered or which can be used to reach other systems that need to be discovered. Although twigs has very few python dependencies, you may also use python virtual environment to reduce dependency conflicts. A virtual environment for twigs for python 2.7 can be setup as follows:
$ python -m virtualenv --python=/usr/bin/python2.7 twigs_env_2_7
Basic CLI Options
The basic usage of twigs is as follows:
twigs [-h] [--handle HANDLE] [--token TOKEN] [--instance INSTANCE]
[--out OUT] [--scan {quick,regular,full}] [--email_report]
[--purge_assets]
{aws,azure,servicenow,repo,host,docker} ...
In most cases you will need the ‘–handle’ which is the ThreatWatch user/email, the ‘–token’ which is the API token issued for the user and the ‘–instance’ which is the name of the ThreatWatch instance where the discovery needs to be reported to. The API token can be obtained by logging into your ThreatWatch instance, navigating to ‘Profile->Key Management’ and generating an API token for yourself. In most cases ‘–instance’ will be the hostname of your dedicated ThreatWatch instance e.g. acme.threatwatch.io. ‘–instance’ defaults to threatwatch.io unless specified.
The ‘–handle’, ‘–token’ and ‘–instance’ can also be specified in the environment variables TW_HANDLE, TW_TOKEN and TW_INSTANCE respectively to keep the sensitive information from being recorded on command line history or being visible in process status as well as making the twigs CLI more convenient.
Discovery modes
twigs supports the following asset types/sources as modes for discovery:
host – Discover hosts locally or remotely using twigs as a lightweight agent for Red Hat, CentOS, Ubuntu, Debian, Amazon Linux AMI
aws – Use AWS inventory services to discover assets from an AWS cloud subscription. Additional steps are required to configure the AWS inventory service which are documented here.
azure – Use Azure inventory services to discover assets from an Azure cloud subscription. Additional steps are required to configure the Azure inventory service which are documented here.
docker – Discover and inventory container images locally or from a container repository.
repo – Point twigs to your local/remote code repository/directory to inventory open source dependencies.
servicenow – Discover assets from an existing ServiceNow CMDB instance.
Each one of these modes may require specific inputs other than the basic options e.g. AWS subscription details, ServiceNow instance details etc. Specific help on this can be obtained by giving ‘-h’ after any mode in the CLI. For e.g.
$ twigs aws -h
Basic local host discovery
To do a simple local host discovery, install twigs on the Linux host of your choice (Red Hat, CentOS, Ubuntu, Debian, Amazon Linux AMI) and run the host discovery mode (replace handle, token and instance with appropriate values for you).
$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host
You can schedule these twigs command using a suitable scheduler (cron) to have discovery run at a regular interval. This way twigs can be used as a lightweight agent for host discovery which can keep track of configuration drift on your host.
Trigger baseline vulnerability assessment and reporting
twigs will automatically request a baseline assessment for any asset that is discovered or updated. This request may be fulfilled by your instance based on resource availability. If you want to just update the asset without triggering automatic assessment, you may use the “–noscan” option
$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io --noscan host
You can also have the impact assessment report emailed back to the ‘–handle’ email id once it is completed.
$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io --email_report host
Security, privacy and anonymity of asset metadata
twigs offers a number of features to ensure the security of your asset inventory during discover and transit over the wire to your ThreatWatch instance. Because twigs is open source, its users can review and audit it to make sure it complies with their internal security toolset requirements. All communication between twigs and your ThreatWatch instance is outbound over secure http protocol only.
If you don’t want the asset inventory to be reported to a ThreatWatch instance but instead have it recorded locally in a CSV file, you can use the ‘–out’ option. This is useful in cases where users need to edit or anonymize the discovered assets before they are reported to ThreatWatch. In any case all assets discovered by twigs will be recorded in a file in the working directory named ‘out.csv’ even if ‘–out’ option is not given.
$ twigs --handle roadrunner@acme.com --out acmehost1.csv host
Also, if users want to edit, redact or anonymize the asset inventory they can do so using the ‘–out’ option before they upload it to their ThreatWatch instance. The anonymized asset CSV can then be uploaded to the ThreatWatch instance from the Web UI.
In addition to this, some discovery modes e.g. host, also allow overriding the asset ID or names using ‘–assetid’ and ‘–assetname’ options. For e.g.
$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --assetid myasset01 --assetname myasset
If users do not want the ThreatWatch instance to preserve the asset information beyond the baseline scan then the ‘–purge_assets’ option will remove the assets once the scan is complete and the report is emailed to the user. For e.g.
$ twigs --handle roadrunner@acme.com --token c8dddddd-eeee-eeee-eeee-aaca617649cc --instance acme.threatwatch.io host --scan quick --email_report --purge_assets
Finally, in the ‘host’ mode twigs supports an optional list of hosts/assets in a CSV file with credentials listed against each host/IP address. This file can be secured by using the ‘–secure’ option which asks for a password (interactively or on command line) and encrypts the credentials portion of the CSV file. For e.g.
$ twigs host --remote_hosts_csv assets.csv --secure
Windows discovery
The github repo for twigs has included a PowerShell script for discovering Windows hosts. This script can be run as a lightweight agent on the host itself or can be run remotely using PowerShell Remoting which is enabled by default on Windows Server 2012 systems. For more information check here.
The easy way to run windows_discovery.ps1 on a windows systems is,
C:\twigs\twigs\windows_discovery.ps1 --handle “roadrunner@acme.com” --token “c8dddddd-eeee-eeee-eeee-aaca617649cc” --instance acme.threatwatch.io
To run remotely,
Invoke-Command -ComputerName Server01, Server02 -FilePath C:\twigs\twigs\windows_discovery.ps1 --handle “roadrunner@acme.com” --token “c8dddddd-eeee-eeee-eeee-aaca617649cc” --instance acme.threatwatch.io
A future version of twigs will have windows discovery integrated with the twigs CLI itself.