Implement Router

.gitignore

@@ -2,11 +2,16 @@
 *.swp
 .DS_Cache
 .DS_Store
+
+/devdata
 /vendor
+
 /example/e2e/logs
+
 # binaries
 /example/client/mlyc/mlyc
 /example/server/mly/mly
+
 /toolsv2/aerospike/aerospike
 /toolsv2/smasher/cmd/cmd
 /toolsv2/toolsv2

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "third_party/triton-common"]
+	path = third_party/triton-common
+	url = git@github.com:triton-inference-server/common.git

BREAKING_CHANGES.md

@@ -0,0 +1,13 @@
+# Breaking Changes
+
+See [wiki](https://github.com/viant/mly/wiki) for older entries and non-breaking changes.
+
+## `v0.19.x` to `v0.20.x`
+
+1. `example/server.RunApp` return changes from none to `error`.
+2. `service.New()` added parameter `tritonClients`.
+3. `service.NewWithPlatform()` removed.
+3. `service/endpoint/checker.SelfTest()` removed parameters `inputs_` and `outputs`.
+4. `service/endpoint/health.(*HealthHandler).RegisterHealthPoint()` second parameter changed.
+5. `service/endpoint/prometheus.Handler()` parameter changed.
+6. `service/endpoint.Build()` added parameter `tritonClients`.

CONFIG.md

@@ -2,62 +2,104 @@
 
 ## Server
 
-See [`service/endpoint/config.go`](service/endpoint/config.go).
-The server accepts configuration with the following options:
-
-* `Models` : list of models - see [`service/config/model.go`](service/config/model.go) for all options.
-  - `ID`: `string` - required - model ID, used to generate the URLs.
-  - `Debug`: `bool` - optional - enables further output and debugging.
-  - `URL`: `string` - required - model location source.
-    * to use S3, set environment variable `AWS_SDK_LOAD_CONFIG=true`
-    * to use GCS, set environment variable `GOOGLE_APPLICATION_CREDENTIALS=true`
-  - `DataStore`: `string` - optional - name of Datastore to cache, should match `Datastores[].ID`.
-  - `Transformer`: `string` - optional - name of model output transformer. See [#Transformer](#Transformer).
-  - `Batch`: optional - enables or overrides server-side batching configuration. See [`service/tfmodel/batcher/config/config.go`](service/tfmodel/batcher/config/config.go).
-  - `Test`: optional - enables a client request to send to self on start up.
+See [`service/endpoint/config.go`](service/endpoint/config.go) for a more thorough description.
+
+## Root Property `Models`
+
+This contains a list of models (i.e. TensorFlow SavedModel package, Triton Model, or Router) that mly will set up endpoints for.
+
+See [`service/config/model.go`](service/config/model.go) for all options.
+
+Properties:
+
+- `ID`: `string` - required - model ID, used to generate the URLs.
+- `Debug`: `bool` - optional - enables further output and debugging.
+- `URL`: `string` - required - model location source.
+  * to use S3, set environment variable `AWS_SDK_LOAD_CONFIG=true`
+  * to use GCS, set environment variable `GOOGLE_APPLICATION_CREDENTIALS=true`
+- `Location`: `string` - optional - where a copy of the models will be stored when loading the model. Defaults to the system temporary directory.
+- `Dir`: `string` - optional - any further path elements in `Location`. Mainly used if using a ZIP file with additional directories.
+- `DataStore`: `string` - optional - name of Datastore to cache, should match `Datastores[].ID`.
+- `Transformer`: `string` - optional - name of model output transformer. See [#Transformer](#Transformer).
+- `Batch`: optional - enables or overrides server-side batching configuration. See [`service/tfmodel/batcher/config/config.go`](service/tfmodel/batcher/config/config.go).
+- `UseDict`: `bool` - optional - if true, enables capabilities designed to shrink the cache key space by replacing out-of-vocabulary inputs from cache keys with a special token.
+- `Inputs`: used to further provide or define inputs, a list of `shared.Field`. For TensorFlow models, this is automatically populated, but further caching configurations need to be specified.
+  * `Name`: `string` - required - input name, only required if an entry is provided.
+  * `Index`: `int` - optional - used to maintain cache key ordering.
+  * `Auxiliary`: `bool` - optional - the input is permitted to be provided in an evaluation request. Mainly used for logging.
+  * `Wildcard`: `bool` - conditionally required - if enabled this input will not have a vocabulary for lookup. If `UseDict` is true and `Wildcard` is false, the service will refuse to start if it cannot guess the vocabulary extraction Operation.
+  * `Precision`: `int` - conditionally required - if the input is a float type and dictionary is enabled, this can be used to round the value to a lower precision which can improve cache hit rates. If `UseDict` is true, the service will refuse to start if it encounters a float input without a `Precision`.
+- `KeyFields`: `[]string` - optional - list of fields used to generate caching key (by default, all model inputs, sorted alphabetically). Can be used to order and add valid inputs that can be used as a cache key but not used as prediction input.
+- `Platform`: `string` - optional, defaults to `tensorflow`. Can be `tensorflow` or `triton`. See below for more information.
+- `Mode`: `string` - optional, defaults to `inference`. Can be `inference` or `router`. See below for more information.
+- `Auxiliary`: `[]string` - **deprecated**, optional - list of additional fields that are acceptable for eval server call. Deprecated, use `Field.Auxiliary`.
+- `Outputs`: `[]shared.Field` - optional - model outputs are automatically pulled from the model. Required for Triton backends.
+- `Test`: optional - enables a client request to send to self on start up.
     * `Test`: `bool` - if `true`, a client will generate a non-batch request with random values based on the model input signature.
-    * `Single`: `map[string]interface{}` - if present, will use the provided values for certain input keys, otherwise randomly generated based on model input signature.
+    * `Single`: `map[string]any` - if present, will use the provided values for certain input keys, otherwise randomly generated based on model input signature.
     * `SingleBatch`: `bool` - if `true`, a client will generate a batch request with random values based on the model input signature; if `Single` is set, values will be used for provided keys.
-    * `Batch`: `map[string][]interface{}` - if present, will be used to generate a batch of requests for the self-test.
-  - `Inputs`: optional - used to further provide or define inputs, a list of `shared.Field`.
-    * `Name`: `string` - required - input name, only required if an entry is provided.
-    * `Index`: `int` - optional - used to maintain cache key ordering.
-    * `Auxiliary`: `bool` - optional - the input is permitted to be provided in an evaluation request.
-    * `Wildcard`: `bool` - conditionally required - if enabled this input will not have a vocabulary for lookup; if `UseDict` is true, the service will refuse to start if it cannot guess the vocabulary extraction Operation. 
-    * `Precision`: `int` - conditionally required - if the input is a float type and dictionary is enabled, this can be used to round the value to a lower precision which can improve cache hit rates; if `UseDict` is true, the service will refuse to start if it encounters a float input without a `Precision`.
-  - `KeyFields`: `[]string` - optional - list of fields used to generate caching key (by default, all model inputs, sorted alphabetically). Can be used to order and add valid inputs that can be used as a cache key but not used as prediction input.
-  - `Auxiliary`: `[]string` - deprecated, optional - list of additional fields that are acceptable for eval server call. Deprecated, use `Field.Auxiliary`.
-  - `Outputs`: `[]shared.Field` - deprecated, optional - model outputs are automatically pulled from the model.
-
-* `Connection`: optional - list of external Aerospike connections.
-  - `ID`: `string` - required - connection ID
-  - `Hostnames`: `string` - required - Aerospike hostnames
-
-* `Datastores` : list of datastore caches
-  - `ID`: `string` - required - datastore ID (to be matched with `Models[].DataStores[].ID`)
-  - `Connection`: `string` - optional - connection ID
-  - `Namespace`: `string` - optional - Aerospike namespace
-  - `Dataset`: `string` - optional - Aerospike dataset 
-  - `Storable`: `string` - optional - name of registered `storable` provider
-  - `Cache`: optional - in-memory cache setting
-    * `SizeMB`: `int` - optional - cache size in MB
-
-* `Endpoint`: some special administrative options
-  - `Port`: `int` - optional - used in `addr` for `http.Server`, default `8080`.
-  - `ReadTimeoutMs`, `WriteTimeoutMs`: `int` - optional - additional settings for `http.Server`, default `5000` for both.
-  - `MaxHeaderBytes`: `int` - optional - additional settings for `http.Server`, default `8192` (`8 * 1024`).
-  - `WriteTimeout`: `int` - optional - maximum request timeout.
-  - `PoolMaxSize`, `BufferSize`: `int` - optional - controls implementation of `net/http/httputil`, default `512` and `131072` (`128 * 1024`), respectively.
-  - `MaxEvaluatorConcurrency`: `int` - optional - controls semaphore that prevents too many CGo goroutines from spawning, default `5000`.
-
-* `EnableMemProf`: `bool` - optional - enables endpoint for memory profiling.
+    * `Batch`: `map[string][]any` - if present, will be used to generate a batch of requests for the self-test.
+
+### Model Property `Platform`
+
+The `Platform` property specifies where inference actually happens.
+Currently supported values are `tensorflow` and `triton`, with `tensorflow` being the default if unspecified.
+
+If the `Platform` is `triton`, then mly will route requests to a configured [Triton Inference Server](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/index.html).
+
+### Model Property `Mode`
+
+The `Mode` property enables a specific inference mode.
+Currently supported values are `inference` and `router`, with `inference` being the default if unspecified.
+
+The `inference` mode operates as expected - the inference request is processed using the model in the backend and a prediction is generated.
+
+The `router` mode operates with a nuance - this will route input rows to a specific backend model.
+
+Currently, only the `Platform` `triton` is supported.
+
+## Root Property `Connections`
+
+Can be empty - a list of external Aerospike connections.
+
+Properties:
+
+- `ID`: `string` - required - connection ID
+- `Hostnames`: `string` - required - Aerospike hostnames
+
+## Root Property `Datastores`
+
+Can be empty - represent a list of caching data stores.
+
+Properties:
+
+- `ID`: `string` - required - datastore ID (to be matched with `Models[].DataStores[].ID`)
+- `Connection`: `string` - optional - connection ID
+- `Namespace`: `string` - optional - Aerospike namespace
+- `Dataset`: `string` - optional - Aerospike dataset
+- `Storable`: `string` - optional - name of registered `storable` provider
+- `Cache`: optional - in-memory cache setting
+  * `SizeMB`: `int` - optional - cache size in MB
+
+## Root Property `Endpoint`
+
+Contains some special administrative options.
+
+- `Port`: `int` - optional - used in `addr` for `http.Server`, default `8080`.
+- `ReadTimeoutMs`, `WriteTimeoutMs`: `int` - optional - additional settings for `http.Server`, default `5000` for both.
+- `MaxHeaderBytes`: `int` - optional - additional settings for `http.Server`, default `8192` (`8 * 1024`).
+- `WriteTimeout`: `int` - optional - maximum request timeout.
+- `PoolMaxSize`, `BufferSize`: `int` - optional - controls implementation of `net/http/httputil`, default `512` and `131072` (`128 * 1024`), respectively.
+- `MaxEvaluatorConcurrency`: `int` - optional - controls semaphore that prevents too many CGo goroutines from spawning, default `5000`.
+
+* `EnableMemProf`: `bool` - **deprecated**, optional - enables endpoint for memory profiling - use `ProfilerPort` instead
 * `EnableCPUProf`: `bool` - optional - enables endpoint for cpu profiling.
 * `AllowedSubnet`: `bool` - optional - restricts administrative endpoints to IP string prefixes.
   - Restricts the system configuration, memory profile, CPU profile, and health endpoints.
 * `ContinueOnRecover`: `bool` - optional - panics will not bubble up.
 
 ## Client
- 
+
 `mly` client does not come with an external config file.
 
 To create a client, use the following snippet:
@@ -75,7 +117,7 @@ See [`shared/client/option.go`](shared/client/option.go) for more options.
 
 Since v0.16.0, there have been added options for Aerospike behavior.
 
-Since each `shared/client.Service` instance operates on 1 model, there was an added option to share Aerospike connections via [`WithConnectionSharing()`](shared/client/option.go).
+Since each `shared/client.Service` instance is encapsulated for 1 model, there was an added option to share Aerospike connections via [`WithConnectionSharing()`](shared/client/option.go).
 
 Additionally, Aerospike client `Policy` override options were provided.
 Use `WithClientOptions()` to provide `shared/datastore/client.Option` options.

DEVTODO.md

@@ -0,0 +1,25 @@
+# Notes about Incoherent or Inconsistent Architecture and Design
+
+## 1 `service/domain`
+
+`service/domain.Input` and `service/domain.Ouput` have dependencies on TensorFlow but are part of the generalized `service/platform.PlatformEvaluator` interface.
+This is an unnecessary coupling and should be removed.
+The usage in the `service` module is mainly for type mapping from JSON to Go to Go-type for TensorFlow.
+
+
+## 2 `service/request.Request.Feeds`
+
+This seems too tailored towards TensorFlow inference.
+A common operation in Triton is to convert the offset-based slice data back into name-based slice data.
+Might see cognitive improvement if we reduce that back-and-forth.
+
+## 3 Input Validation
+
+Incorrect batch size results in panic.
+
+
+## 4 Triton Management Frequency
+
+Higher frequency model load frequency.
+Mainly useful during development, but may be useful in production environments where Triton server dies or is restarted?
+No, seems like that would be a separate issue in it of itself.