Source code discovery

Overview

Twigs can discover your source code as an asset. You can discover a single git repo or local repo folder or a given git repository URL.

Supported package / dependency managers for source code discovery are:

  • Python [pip]
  • Java Script [npm, yarn]
  • Ruby [ruby]
  • Java [maven, gradle, jar]
  • .NET/C# [nuget]
  • DLL (* for vulnerability assessment only)
  • Rust [cargo]

There are multiple functionalities provided as below:

  • Identify vulnerabilities – This helps you identify any vulnerabilities in 3rd party libraries / packages used in your source code project. You can indicate to twigs whether you are interested in tracking vulnerabilities in direct (shallow level) or indirect (deep level) dependencies.
  • License compliance – You need to know how licensing of the open source components (libraries/packages) used in your software project impact you. For example – one cannot release a commercial software product built using open source components with a restrictive license
  • Code secrets – Twigs can be used to identify any secrets that are inadvertently embedded in your source code. Twigs can identify secrets using any or all of the three approach mentioned below:
    • Entropy – Detect secrets by automatically identifying high entropy strings in your source code.
    • Regular Expressions – twigs provides support for regular expressions for identifying standard secrets (like OAuth tokens, JWT tokens, etc.). You can specify your own custom regular expressions in a file if needed.
    • Common Passwords – twigs provides support for identifying common passwords from a top 500 common passwords list. You can provide your own common passwords file, if needed.
  • Static Application Security Testing (SAST) – You can perform SAST checks on your source code projects.
  • Infrastructure as Code (IaC) security tests on cloudformation, terraform, kubernetes, serverless, arm, terraform_plan, helm templates.

Prerequisites

For static code analysis (SAST) tests, twigs uses an open source tool called semgrep. For more information on semgrep and how to download it, refer to this link.

For security checks in your infrastructure-as-code (IaC) code, twigs uses an open source tool called checkov. For more information on checkov and how to download it, refer to this link.

twigs docker image bundles all these external tools so you don’t have to install them.

Steps

The steps involved to discover your source code as an asset are as below:

  • Open a new shell / terminal.
  • Check that twigs is installed and running properly by running below command:

twigs repo -h

  • Run the command as below:

twigs repo (--repo REPO_URL)[--branch BRANCH] [--type {pip,ruby,yarn,nuget,npm,maven,gradle,dll,jar,cargo}] [--level {shallow,deep}][--include_unused_dependencies] [--assetid ASSETID] [--assetname ASSETNAME][--secrets_scan] [--enable_entropy] [--regex_rules_file REGEX_RULES_FILE] [--check_common_passwords] [--common_passwords_file COMMON_PASSWORDS_FILE] [--include_patterns INCLUDE_PATTERNS] [--include_patterns_file INCLUDE_PATTERNS_FILE] [--exclude_patterns EXCLUDE_PATTERNS] [--exclude_patterns_file EXCLUDE_PATTERNS_FILE] [--mask_secret][--no_code][--sast][--iac_checks]

After discovery is complete, you can login into ThreatWorx Console to view the newly discovered asset from your source code.

Pro Tips:

  • Use the branch option to specify an optional branch of remote git repo (applicable for single git repo discovery only).
  • For certain dependency managers like npm, you can control the dependency levels to inspect by using the –level [shallow|deep] option.
  • twigs can determine the type of dependency manager by looking at the source code. But in case you want twigs to discover dependencies of a certain type, you can use the –type option.
  • For certain technologies (like npmjs), only used direct dependencies are considered by default (i.e. dependencies referenced in the source code). Use to the –include_unused_dependencies switch to consider all dependencies, note this may introduce false positives.
  • Secrets such as tokens, keys etc. embedded in the source code can be found using –secrets_scan option. A default set of regex rules are used for this which can be substituted for your regex rules file. A more advanced (but noisy) way is to use –enable_entropy to detect secrets based on text patterns.
  • The –mask_secret option will hide any secret strings that are collected from the source code.
  • The –no_code option will prevent any code snippet from being collected or transmitted as part of the discovery. The source file and line numbers will be collected and transmitted.
  • Common passwords embedded in source code can be scanned for using the –check_common_passwords option. You may provide your own list of common passwords as a simple text file using –common_passwords_file option.
  • The include_ and exclude_ options can be used to specify or skip certain file name patterns or directories while scanning the source code repository.

Examples

$ twigs repo --repo https://github.com/threatworx/twigs.git --secrets_scan --sast

Discover a public repository as an asset

$ twigs repo (--repo ~/my_workspace --type pip --secrets_scan --enable_entropy --check_common_passwords --mask_secret --no_code --sast --iac_checks

Discover a local code repository folder as an asset