Skip to content

Expand query and database capabilities #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
moshloop wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Expand query and database capabilities #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6511de7
fix(db): Bound migration lock waits and retry contention
moshloop Jul 16, 2026
69d3ae7
feat(database): enable embedded PostgreSQL diagnostics
moshloop Jul 16, 2026
ddeadf3
feat(query): add OpenTelemetry trace querying via nested OpenSearch c…
moshloop Jul 17, 2026
4c9e633
feat(query): Add profile inheritance and OpenTelemetry trace support
moshloop Jul 17, 2026
a8e144b
feat(query): support OpenTelemetry connection mapping
moshloop Jul 17, 2026
459b41b
refactor(query): Migrate query chat to Captain service runtime
moshloop Jul 17, 2026
ccc0961
chore: update generated and lock files
moshloop Jul 17, 2026
2a60b0c
feat(fe): Add guided profile creation wizard
moshloop Jul 19, 2026
b93dcf6
feat(migrate): preserve unmanaged dependent views during schema changes
moshloop Jul 19, 2026
1669745
feat(dbtest): add shared test database resolution
moshloop Jul 19, 2026
abb7d2a
ci(db): run embedded and external database tests in CI
moshloop Jul 19, 2026
e45dd09
fix(query): Harden profile resolution and OpenTelemetry row handling
moshloop Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 38 additions & 6 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ jobs:
go-version: ["1.26"]
timeout-minutes: 30

# Exercises the embedded-postgres path on both platforms. The external-DB
# path is covered by e2e-external, which needs a Linux-only service container.
env:
COMMONS_DB_EMBEDDED_TEST: "1"

Comment on lines +40 to +44

@coderabbitai coderabbitai Bot Jul 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check for other workflows that might already cover this, and confirm db/embedded_test.go's env gating.
fd -e yml -e yaml . .github/workflows
rg -n 'COMMONS_DB_EMBEDDED_TEST' -g '*.go' -g '*.yml' -g '*.yaml'

Repository: flanksource/commons-db

Length of output: 276

e2e-tests still skips the embedded DB suite.

COMMONS_DB_EMBEDDED_TEST=1 is set here, but this job only runs ./e2e and ./e2e/helpers/.... The ./migrate/..., ./query/..., and ./db/... tests only run in e2e-external, so the embedded-only paths gated by COMMONS_DB_EMBEDDED_TEST never execute in CI. If the goal is to cover embedded PostgreSQL on both platforms, this needs a job that runs ./db/... with that env var set.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e.yml around lines 40 - 44, Add an embedded-PostgreSQL
test step or job to the e2e-tests workflow that runs the ./db/... suite while
COMMONS_DB_EMBEDDED_TEST=1 is set, ensuring it executes on both supported
platforms. Keep the existing ./e2e and helper test commands unchanged.

steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
Expand Down Expand Up @@ -116,11 +121,30 @@ jobs:
go fmt ./e2e/...
continue-on-error: false

e2e-linux:
name: E2E Tests (Linux - Detailed)
e2e-external:
name: E2E and DB Tests (external postgres)
runs-on: ubuntu-latest
timeout-minutes: 30

services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5

# COMMONS_DB_URL names the maintenance database; COMMONS_DB_CREATE defaults
# to true, so each test carves out and drops its own database on this server.
env:
COMMONS_DB_URL: postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable
CGO_ENABLED: 1

steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
Expand All @@ -135,19 +159,27 @@ jobs:
- name: Run E2E tests with verbose output
run: |
go run github.com/onsi/ginkgo/v2/ginkgo@v2.28.0 run -v -r --timeout=10m --poll-progress-after=1m ./e2e
env:
CGO_ENABLED: 1

- name: Run DB integration tests
run: go test ./migrate/... ./query/... ./db/... -timeout 15m

- name: Assert no scratch databases leaked
run: |
leaked=$(psql "$COMMONS_DB_URL" -tAc \
"SELECT count(*) FROM pg_database WHERE datname LIKE '%\_%\_%' AND datname NOT IN ('postgres','template0','template1')")
echo "leftover databases: $leaked"
test "$leaked" -eq 0

quality-gate:
name: Quality Gate
runs-on: ubuntu-latest
needs: [e2e-tests]
needs: [e2e-tests, e2e-external]
if: always()

steps:
- name: Check test results
run: |
if [ "${{ needs.e2e-tests.result }}" = "failure" ]; then
if [ "${{ needs.e2e-tests.result }}" = "failure" ] || [ "${{ needs.e2e-external.result }}" = "failure" ]; then
echo "E2E tests failed"
exit 1
fi
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/query.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,25 @@ permissions: read-all
jobs:
test:
runs-on: ubuntu-latest

services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5

# Without this the module's DB integration tests skip silently.
# COMMONS_DB_CREATE defaults to true, so each test gets its own database.
env:
COMMONS_DB_URL: postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable

steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
Expand Down
10 changes: 10 additions & 0 deletions Taskfile.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@ tasks:
cmds:
- go test -v ./...

test:integration:
desc: Run tests including DB integration tests (needs COMMONS_DB_URL, or set COMMONS_DB_EMBEDDED_TEST=1)
cmds:
- |
if [ -z "$COMMONS_DB_URL" ] && [ -z "$COMMONS_DB_EMBEDDED_TEST" ]; then
echo "set COMMONS_DB_URL to an external postgres, or COMMONS_DB_EMBEDDED_TEST=1 to embed one" >&2
exit 1
fi
- go test ./... -timeout 15m

tidy:
desc: Tidy go modules
cmds:
Expand Down
11 changes: 11 additions & 0 deletions cmd/query/connections/actions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,17 @@ func maskedConnection(c *models.Connection) resolvedConnection {
// testConnection probes the resolved URL: a TCP connect proves the host:port is
// reachable, and http/https URLs additionally report the response status.
func testConnection(ctx dbcontext.Context, c *models.Connection) testResult {
if c.Type == models.ConnectionTypeOpenTelemetry {
openTelemetry, err := dbconnection.NewOpenTelemetry(c)
if err != nil {
return testResult{OK: false, Message: err.Error()}
}
nested, err := openTelemetry.ResolveOpenSearch(ctx)
if err != nil {
return testResult{OK: false, Message: err.Error()}
}
return testConnection(ctx, nested)
}
if c.URL == "" {
return testResult{OK: false, Message: "connection has no URL to test"}
}
Expand Down
12 changes: 12 additions & 0 deletions cmd/query/connections/browser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,18 @@ func (h *connectionBrowserHandler) ServeHTTP(w http.ResponseWriter, r *http.Requ
http.Error(w, err.Error(), http.StatusNotFound)
return
}
if conn.Type == models.ConnectionTypeOpenTelemetry {
openTelemetry, resolveErr := dbconnection.NewOpenTelemetry(conn)
if resolveErr != nil {
http.Error(w, resolveErr.Error(), http.StatusUnprocessableEntity)
return
}
conn, resolveErr = openTelemetry.ResolveOpenSearch(h.ctx)
if resolveErr != nil {
http.Error(w, resolveErr.Error(), http.StatusUnprocessableEntity)
return
}
}
tail = strings.TrimSuffix(tail, "/")

switch {
Expand Down
10 changes: 10 additions & 0 deletions cmd/query/connections/info.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,16 @@ func discoverServer(ctx context.Context, connectionContext dbcontext.Context, co
info, err = discoverSQLServer(ctx, connectionContext, connection)
case models.ConnectionTypeOpenSearch:
info, err = discoverOpenSearch(ctx, connectionContext, connection)
case models.ConnectionTypeOpenTelemetry:
var openTelemetry dbconnection.OpenTelemetry
openTelemetry, err = dbconnection.NewOpenTelemetry(connection)
if err == nil {
var nested *models.Connection
nested, err = openTelemetry.ResolveOpenSearch(connectionContext)
if err == nil {
info, err = discoverOpenSearch(ctx, connectionContext, nested)
}
}
case models.ConnectionTypePrometheus:
info, err = discoverPrometheus(ctx, connectionContext, connection)
case models.ConnectionTypeRedis:
Expand Down
28 changes: 28 additions & 0 deletions cmd/query/connections/service.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"strings"

"github.com/flanksource/clicky"
dbconnection "github.com/flanksource/commons-db/connection"
dbcontext "github.com/flanksource/commons-db/context"
"github.com/flanksource/commons-db/models"
"github.com/google/uuid"
Expand Down Expand Up @@ -191,6 +192,9 @@ func createConnection(db *gorm.DB, body map[string]any) (*models.Connection, err
if err != nil {
return nil, err
}
if err := validateNestedConnection(db, c); err != nil {
return nil, err
}
c.ID = uuid.Nil // let the DB assign the id
if err := db.Create(c).Error; err != nil {
return nil, fmt.Errorf("create connection: %w", err)
Expand All @@ -209,6 +213,9 @@ func updateConnection(db *gorm.DB, id string, body map[string]any) (*models.Conn
if err != nil {
return nil, err
}
if err := validateNestedConnection(db, incoming); err != nil {
return nil, err
}
applyConnectionUpdate(existing, incoming)
if err := db.Save(existing).Error; err != nil {
return nil, fmt.Errorf("update connection: %w", err)
Expand All @@ -217,6 +224,27 @@ func updateConnection(db *gorm.DB, id string, body map[string]any) (*models.Conn
return existing, nil
}

func validateNestedConnection(db *gorm.DB, candidate *models.Connection) error {
if candidate.Type != models.ConnectionTypeOpenTelemetry {
return nil
}
openTelemetry, err := dbconnection.NewOpenTelemetry(candidate)
if err != nil {
return err
}
nested, err := dbcontext.HydrateConnectionByURL(dbcontext.New().WithDB(db, nil), openTelemetry.Connection)
if err != nil {
return fmt.Errorf("resolve nested OpenSearch connection: %w", err)
}
if nested == nil {
return fmt.Errorf("nested OpenSearch connection %q not found", openTelemetry.Connection)
}
if nested.Type != models.ConnectionTypeOpenSearch {
return fmt.Errorf("nested connection %q has type %q, expected %q", nested.Name, nested.Type, models.ConnectionTypeOpenSearch)
}
return nil
}

// deleteConnection removes a connection by id, erroring when absent.
func deleteConnection(db *gorm.DB, id string) error {
res := db.Where("id = ?", id).Delete(&models.Connection{})
Expand Down
28 changes: 28 additions & 0 deletions cmd/query/connections/service_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"testing"

"github.com/flanksource/commons-db/models"
"github.com/flanksource/commons-db/types"
"github.com/google/uuid"
)

Expand Down Expand Up @@ -41,6 +42,33 @@ func TestConnectionFromBodyRequiresNameAndType(t *testing.T) {
}
}

func TestValidateOpenTelemetryRequiresNestedOpenSearchConnection(t *testing.T) {
database := connectionInfoTestDB(t)
wrong := models.Connection{ID: uuid.New(), Name: "OS", Type: models.ConnectionTypeHTTP}
search := models.Connection{ID: uuid.New(), Name: "OS", Namespace: "team", Type: models.ConnectionTypeOpenSearch}
if err := database.Create(&wrong).Error; err != nil {
t.Fatal(err)
}
if err := database.Create(&search).Error; err != nil {
t.Fatal(err)
}
candidate := &models.Connection{
Name: "traces", Type: models.ConnectionTypeOpenTelemetry,
Properties: types.JSONStringMap{"connection": "connection://team/OS"},
}
if err := validateNestedConnection(database, candidate); err != nil {
t.Fatal(err)
}
candidate.Properties["connection"] = "connection://traces"
if err := validateNestedConnection(database, candidate); err == nil {
t.Fatal("expected self-referencing OpenTelemetry connection to fail")
}
candidate.Properties["connection"] = "https://example.test/OS"
if err := validateNestedConnection(database, candidate); err == nil {
t.Fatal("expected an HTTP URL ending in the unrelated connection name to fail")
}
}

func TestApplyConnectionUpdatePreservesSecretWhenBlank(t *testing.T) {
existing, _ := connectionFromBody(map[string]any{"name": "db", "type": "postgres", "password": "old"})
incoming, _ := connectionFromBody(map[string]any{"name": "db2", "type": "mysql"}) // no password
Expand Down
12 changes: 7 additions & 5 deletions cmd/query/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ module github.com/flanksource/commons-db/cmd/query
go 1.26.1

require (
github.com/flanksource/clicky v1.21.37-0.20260715094201-d847a9108f43
github.com/flanksource/clicky/aichat v1.21.36
github.com/flanksource/captain v0.0.12-0.20260715131932-f441c766f3fd
github.com/flanksource/clicky v1.21.38-0.20260716031053-78eecd7e914b
github.com/flanksource/clicky/aichat v1.21.38-0.20260716031053-78eecd7e914b
github.com/flanksource/clicky/valkey v1.21.34
github.com/flanksource/commons v1.53.1
github.com/flanksource/commons-db v0.1.15
Expand All @@ -15,6 +16,7 @@ require (
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
github.com/valkey-io/valkey-go v1.0.75
go.yaml.in/yaml/v3 v3.0.4
gorm.io/gorm v1.31.0
k8s.io/api v0.36.1
k8s.io/apimachinery v0.36.1
Expand Down Expand Up @@ -127,8 +129,7 @@ require (
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fergusstrange/embedded-postgres v1.34.0 // indirect
github.com/firebase/genkit/go v1.8.0 // indirect
github.com/flanksource/captain v0.0.9 // indirect
github.com/firebase/genkit/go v1.10.0 // indirect
github.com/flanksource/gomplate/v3 v3.24.82 // indirect
github.com/flanksource/is-healthy v1.0.88 // indirect
github.com/flanksource/kubectl-neat v1.0.4 // indirect
Expand Down Expand Up @@ -197,6 +198,7 @@ require (
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/jmespath/go-jmespath v0.4.1-0.20220621161143-b0104c826a24 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
Expand Down Expand Up @@ -272,6 +274,7 @@ require (
github.com/samber/lo v1.53.0 // indirect
github.com/samber/oops v1.21.0 // indirect
github.com/segmentio/asm v1.2.1 // indirect
github.com/segmentio/encoding v0.5.4 // indirect
github.com/sergi/go-diff v1.4.0 // indirect
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
github.com/shoenig/go-m1cpu v0.1.7 // indirect
Expand Down Expand Up @@ -329,7 +332,6 @@ require (
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
gocloud.dev v0.43.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
Expand Down
4 changes: 4 additions & 0 deletions cmd/query/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,8 @@ github.com/jmespath/go-jmespath v0.4.1-0.20220621161143-b0104c826a24 h1:liMMTbpW
github.com/jmespath/go-jmespath v0.4.1-0.20220621161143-b0104c826a24/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/joshdk/go-junit v1.0.0 h1:S86cUKIdwBHWwA6xCmFlf3RTLfVXYQfvanM5Uh+K6GE=
Expand Down Expand Up @@ -686,6 +688,8 @@ github.com/samber/oops v1.21.0 h1:18atcO4oEigNFuGXqr3NZWZ6P0XOSEXyBSAMXdQRxTc=
github.com/samber/oops v1.21.0/go.mod h1:Hsm/sKPxtCfPh0w/cE3xVoRfSiE1joDRiStPAsmG9bo=
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0=
github.com/segmentio/encoding v0.5.4/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0=
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI=
Expand Down
Loading
Loading