Add Cisco FTD model

CHANGELOG.md

@@ -11,13 +11,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - vsololt: new model for VSOL GPON OLT (@Vantomas)
 - tplink: add simulation data and unit tests for the TP-Link DeltaStream DS-P7001-08 GPON OLT (@Vantomas)
 - device2yaml: add `-n`/`--newline` option to set the command line terminator (e.g. `-n "\r\n"`) for devices that submit a command only on a carriage return; the terminator is recorded as a `command_newline` key in the generated YAML (@Vantomas)
+- ftd: new model for Cisco FTD via HTTP (@mgrocock-cwcs)
 
 ### Changed
 - docker: set LANG=C.UTF-8. Fixes #3690 (@ytti)
 - routeros: remove intermittent POE `voltage_on_poe-in` comment (@hendrikbl)
 - tplink: use `\r\n` as the line terminator in pre_logout, required for the model unit tests to work (@Vantomas)
 - ssh: change max_window_size from 138k to 2MB to avoid triggering Mikrotik bug. Closes #3867 (@ytti)
 - ingate: redact secrets (private keys, passwords, secrets, passphrases, pre-shared keys, tokens and the SNMP community) when remove_secret is set (@thanegill)
+- http: support overriding port (@mgrocock-cwcs)
+- http: support DELETE method (@mgrocock-cwcs)
 
 ### Fixed
 - input/cli: raise an error when a requested login credential is missing. Fixes #3700 (@robertcheramy)

Dockerfile

@@ -63,6 +63,8 @@ RUN set -eux; \
       ruby-sqlite3 ruby-mysql2 ruby-pg ruby-sequel ruby-gpgme\
       # Gem dependencies for hooks
       ruby-aws-sdk ruby-xmpp4r \
+      # Gem dependencies for models
+      ruby-zip \
       # Gems needed by oxidized-web
       ruby-charlock-holmes ruby-haml ruby-htmlentities ruby-json \
       puma ruby-sinatra ruby-sinatra-contrib \

docs/Inputs.md

@@ -183,17 +183,19 @@ input:
 The HTTP input supports the following HTTP methods:
 - `:get`  - for GET requests
 - `:post` - for POST requests
+- `:delete` - for DELETE requests
 
 These methods are used internally by models that require HTTP-based 
-configuration retrieval. Models can use `get_http()` and `post_http()` methods 
-provided by the HTTP input.
+configuration retrieval. Models can use `get_http()`, `post_http()` and
+`delete_http()` methods provided by the HTTP input.
 
 Example usage in a model:
 
 ```ruby
 cfg :http do
-  post_response = post_http('/some/path', payload, 'Some-Extra-Header' => 'value')
-  get_response  = get_http('/some/path')
+  post_response   = post_http('/some/path', payload, 'Some-Extra-Header' => 'value')
+  get_response    = get_http('/some/path')
+  delete_response = delete_http('/some/path')
 end
 ```
 

docs/Model-Notes/FTD.md

@@ -0,0 +1,62 @@
+# Cisco FTD via HTTP
+
+Back up Cisco FTD firewalls via the [HTTP API](https://www.cisco.com/c/en/us/td/docs/security/firepower/ftd-api/guide/ftd-rest-api/ftd-rest-api-intro.html). This model uses the [configexport](https://www.cisco.com/c/en/us/td/docs/security/firepower/ftd-api/guide/ftd-rest-api/ftd-api-import-export.html#id_108006) method to export the configuration as a zip file, then extracts the JSON configuration from this file.
+
+## Configuration
+
+Ensure that the HTTP input is enabled in the Oxidized configuration, e.g.:
+
+```yaml
+input:
+  default: ssh, http
+```
+
+Oxidized will need to use the FTD's admin login. Set the username at the model level:
+
+```yaml
+models:
+  ftd:
+    username: admin
+```
+
+When integrating with LibreNMS, you may need to override the IP address used by Oxidized (e.g. if SNMP and the HTTP API are listening on different interfaces on your FTD). This can be done with a mapping rule, e.g.:
+
+```shell
+lnms config:set oxidized.maps.ip.hostname.+ '{"match": "HOSTNAME", "value": "IP"}'
+```
+
+(Where `HOSTNAME` is the hosname/IP address used when adding the device to LibreNMS, and `IP` is the IP address for the HTTP API.)
+
+The port for the HTTP API can be overridden with a variable. For example, create a mapping rule to assign the device to a group, e.g.:
+
+```shell
+lnms config:set oxidized.maps.group.hostname.+ '{"match": "HOSTNAME", "value": "GROUP"}'
+```
+
+(Where `HOSTNAME` is the hosname/IP address used when adding the device to LibreNMS, and `GROUP` is an appropriate group name.)
+
+Then set the port variable at the group level. You can also override the password here if necessary, e.g.:
+
+```yaml
+groups:
+  GROUP:
+    vars:
+      ftd_api_port: 8443
+    password: secret
+```
+
+## Variables
+
+The following variables can be used to control the behaviour of the model:
+
+- ftd\_api\_endpoint: URL path to the FTD API (default: /api/fdm/latest)
+- ftd\_api\_port: HTTPS port for the FTD API (default: 443)
+- ftd\_config\_filename: Filename to use for the configexport method call (default: oxidized.zip)
+- ftd\_polls: Number of times to poll the status of the configexport job (default: 10, minimum: 1)
+- ftd\_poll\_wait: Seconds to wait between polls (default: 10, minimum: 1)
+
+## Limitations
+
+If your FTDs are in an HA pair, then backups will only succeed on the active device.
+
+Back to [Model-Notes](README.md)

docs/Supported-OS-Types.md

@@ -54,7 +54,8 @@
 |                    |CatOS                         |[catos](/lib/oxidized/model/catos.rb)
 |                    |Cisco Catalyst Express        |[ciscoce](/lib/oxidized/model/ciscoce.rb)
 |                    |ExaLink Fusion (Nexus 3550-F) |[exalink](/lib/oxidized/model/exalink.rb)        |@obol89          |[ExaLink](Model-Notes/ExaLink.md)
-|                    |FireLinuxOS                   |[firelinuxos](/lib/oxidized/model/firelinuxos.rb)
+|                    |FTD via HTTP                  |[ftd](/lib/oxidized/model/ftd.rb)                |                 |[FTD](Model-Notes/FTD.md)
+|                    |FTD via SSH                   |[firelinuxos](/lib/oxidized/model/firelinuxos.rb)
 |                    |IOS                           |[ios](/lib/oxidized/model/ios.rb)                |@robertcheramy   |[IOS](Model-Notes/IOS.md)
 |                    |IOSXR                         |[iosxr](/lib/oxidized/model/iosxr.rb)
 |                    |NGA                           |[cisconga](/lib/oxidized/model/cisconga.rb)

lib/oxidized/input/http.rb

@@ -7,6 +7,7 @@ class HTTP < Input
     def connect(node)
       @node = node
       @secure = false
+      @port = nil
       @username = nil
       @password = nil
       @headers = {}
@@ -53,6 +54,11 @@ def post_http(path, body = nil, extra_headers = {})
       res.body
     end
 
+    def delete_http(path)
+      res = perform_http_request(path, method: :delete)
+      res.body
+    end
+
     def perform_http_request(path, method: :get, body: nil, extra_headers: {})
       uri = get_uri(path)
       http_method = method.to_s.upcase
@@ -84,13 +90,16 @@ def perform_http_request(path, method: :get, body: nil, extra_headers: {})
 
     def make_request(uri, ssl_verify, extra_headers = {}, method: :get, body: nil)
       Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https", verify_mode: ssl_verify) do |http|
-        req_class = if method == :get
+        req_class = case method
+                    when :get
                       Net::HTTP::Get
-                    elsif method == :post
+                    when :post
                       Net::HTTP::Post
+                    when :delete
+                      Net::HTTP::Delete
                     else
                       raise Oxidized::OxidizedError, "Unsupported HTTP method: #{method.inspect}. " \
-                                                     "Only :get and :post are supported"
+                                                     "Only :get, :post and :delete are supported"
                     end
         req = req_class.new(uri)
         @headers.merge(extra_headers).each { |header, value| req.add_field(header, value) }
@@ -115,6 +124,7 @@ def get_uri(path)
       path = URI.parse(path)
       uri_class = @secure ? URI::HTTPS : URI::HTTP
       uri_class.build(host:  @node.ip,
+                      port:  @port,
                       path:  path.path,
                       query: path.query)
     end

lib/oxidized/model/ftd.rb

@@ -0,0 +1,146 @@
+require 'json'
+require 'zip'
+
+class FTD < Oxidized::Model
+  class FTDError < Oxidized::OxidizedError; end
+
+  cfg_cb = lambda do
+    def login
+      payload = {
+        'grant_type' => 'password',
+        'username'   => @node.auth[:username],
+        'password'   => @node.auth[:password]
+      }.to_json
+
+      begin
+        body = post_http("#{@api_endpoint}/fdm/token", payload)
+        token = JSON.parse(body)
+        @headers['Authorization'] = "#{token['token_type']} #{token['access_token']}"
+      rescue StandardError => e
+        raise FTDError, "Login failed: #{e.message}"
+      end
+    end
+
+    def delete_config_file
+      delete_http("#{@api_endpoint}/action/configfiles/#{@config_filename}")
+    rescue StandardError => e
+      # Try to continue even if deletion fails.
+      logger.debug "Deleting config file failed: #{e.message}"
+    end
+
+    def schedule_config_export
+      payload = {
+        'type'                => 'scheduleconfigexport',
+        'diskFileName'        => @config_filename,
+        'doNotEncrypt'        => true,
+        'deployedObjectsOnly' => true
+      }.to_json
+
+      begin
+        body = post_http("#{@api_endpoint}/action/configexport", payload)
+        config_export = JSON.parse(body)
+        config_export['jobHistoryUuid']
+      rescue StandardError => e
+        raise FTDError, "Scheduling config export failed: #{e.message}"
+      end
+    end
+
+    def poll_job_status(job_id)
+      job_status = nil
+
+      @polls.times do
+        sleep(@poll_wait)
+
+        begin
+          body = get_http("#{@api_endpoint}/jobs/configexportstatus/#{job_id}")
+          job_status = JSON.parse(body)
+        rescue StandardError => e
+          # Keep polling if a poll fails.
+          logger.debug "Polling job status failed: #{e.message}"
+        end
+
+        break if %w[SUCCESS FAILED].include? job_status&.dig('status')
+      end
+
+      job_status
+    end
+
+    def check_job_status(job_status)
+      status = job_status&.dig('status')
+      return if status == 'SUCCESS'
+
+      message = if status == 'FAILED'
+                  job_status&.dig('statusMessage')
+                else
+                  job_status&.dig('error', 'messages', 0, 'description')
+                end
+
+      message ||= 'unknown error'
+      raise FTDError, "Config export job failed: #{message}"
+    end
+
+    def download_config_file
+      get_http("#{@api_endpoint}/action/downloadconfigfile/#{@config_filename}")
+    rescue StandardError => e
+      raise FTDError, "Downloading config file failed: #{e.message}"
+    end
+
+    def extract_config(config_file)
+      zipfile = Zip::File.open_buffer(config_file)
+      config = zipfile.read('full_config.txt')
+      JSON.parse(config)
+    rescue Zip::Error => e
+      raise FTDError, "Opening zip file failed: #{e.message}"
+    rescue Errno::ENOENT
+      raise FTDError, 'full_config.txt not found in zip file'
+    rescue JSON::ParserError => e
+      raise FTDError, "Parsing config JSON failed: #{e.message}"
+    rescue StandardError => e
+      raise FTDError, "Extracting config failed: #{e.message}"
+    end
+
+    login
+    delete_config_file # Delete any pre-existing config file, otherwise the config export will fail.
+    job_id = schedule_config_export
+    job_status = poll_job_status(job_id)
+    check_job_status(job_status)
+    config_file = download_config_file
+    delete_config_file
+    extract_config(config_file)
+  rescue FTDError => e
+    logger.debug e.message
+    false
+  end
+
+  cmd cfg_cb do |cfg|
+    def sort_list!(cfg, type, key)
+      cfg.each_with_index.select { |element, _| element['type'] == 'identitywrapper' and element['data']['type'] == type }.map(&:last).each do |i|
+        cfg[i]['data'][key].sort_by! { |element| element['id'] }
+      end
+    end
+
+    # generatedOn contains the timestamp of the config export. Delete it to avoid unnecessary differences.
+    cfg[0].delete('generatedOn')
+
+    # Some lists seem to change order between exports. Sort them to avoid unnecessary differences.
+    sort_list!(cfg, 'distinguishednamegroup', 'distiniguishedNames') # This needs to be 'distiniguishedNames', not 'distinguishedNames'.
+    sort_list!(cfg, 'geolocation', 'locations')
+
+    JSON.pretty_generate(cfg)
+  end
+
+  cfg :http do
+    @api_endpoint = vars(:ftd_api_endpoint) || '/api/fdm/latest'
+    @config_filename = vars(:ftd_config_filename) || 'oxidized.zip'
+    @polls = [vars(:ftd_polls)&.to_i || 10, 1].max
+    @poll_wait = [vars(:ftd_poll_wait)&.to_i || 10, 1].max
+
+    @secure = true
+    @port = vars(:ftd_api_port) || 443
+
+    @headers = {
+      'Accept'       => 'application/json',
+      'Content-Type' => 'application/json'
+    }
+  end
+end

oxidized.gemspec

@@ -41,6 +41,7 @@ Gem::Specification.new do |s|
   s.add_dependency 'net-ssh',              '~> 7.3'
   s.add_dependency 'net-telnet',           '~> 0.2'
   s.add_dependency 'psych',                '~> 5.0'
+  s.add_dependency 'rubyzip',              '>= 2.3.2', '< 4'
   s.add_dependency 'rugged',               '~> 1.6'
   s.add_dependency 'semantic_logger',      '>= 4.17', '< 5'
   s.add_dependency 'slop',                 '~> 4.6'