Add agentless Feature Flagging configuration source

[leoromanovsky]

Motivation

Java Feature Flagging needs an agentless configuration-source path so server SDKs can fetch UFC directly from a Datadog backend or customer-provided HTTP endpoint while preserving the existing Agent Remote Configuration path.

Changes

• Added canonical configuration-source settings: DD_FEATURE_FLAGS_CONFIGURATION_SOURCE, DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL, DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS, DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS, and DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS.
• Kept the existing DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED gate while adding DD_FEATURE_FLAGS_ENABLED as the canonical enablement variable.
• Starts Feature Flagging automatically for the default agentless source unless explicitly disabled with DD_FEATURE_FLAGS_ENABLED=false or DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=false.
• Split Feature Flagging startup across agentless, remote_config, and reserved offline sources.
• Added a UFC-over-HTTP polling service for agentless. It sends DD-API-KEY, supports extra request headers, ETag/304, retryable timeout/429/5xx handling, malformed-payload rejection, last-known-good preservation, and no overlapping polls.
• Appends /api/v2/feature-flagging/config/server-distribution when the configured agentless base URL is a root URL; preserves full endpoint URLs exactly.

Decisions

• DD_FEATURE_FLAGS_CONFIGURATION_SOURCE is SDK-local. It is not sent to the backend as a request header.
• agentless means “fetch UFC over HTTP.” The same SDK mode covers the Datadog default backend and customer-hosted endpoints.
• remote_config keeps the existing Agent + Remote Configuration path.
• offline is accepted as a reserved source but does not start a network-backed config service yet.
• DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS supports customer/backend-specific request headers. Config logging reports only header names, not values.

Class Shape

classDiagram
    direction LR
    class FeatureFlaggingSystem {
      +start()
      +stop()
      -createConfigurationSourceService()
    }
    class ConfigurationSourceService {
      <<interface>>
      +init()
      +close()
    }
    class RemoteConfigServiceImpl {
      +init()
      +close()
      +accept()
    }
    class UfcHttpConfigService {
      +init()
      +close()
      -pollOnce()
      -fetchAndApply()
    }
    class FeatureFlaggingGateway {
      +dispatch()
    }

    FeatureFlaggingSystem --> ConfigurationSourceService : selects one source
    ConfigurationSourceService <|.. RemoteConfigServiceImpl
    ConfigurationSourceService <|.. UfcHttpConfigService
    RemoteConfigServiceImpl --> FeatureFlaggingGateway : RC UFC
    UfcHttpConfigService --> FeatureFlaggingGateway : HTTP UFC

Loading

Customer Usage Example

Default Datadog-hosted agentless delivery:

export DD_API_KEY=<datadog-api-key>
# Optional: agentless is the default configuration source.
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Customer-hosted endpoint delivery uses the same agentless source. The only difference is the base URL and any required request headers:

export DD_API_KEY=<datadog-api-key>
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE=agentless
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL=https://flags.example.com/datadog/server-distribution?dd_env=prod
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_EXTRA_HEADERS='DD-APPLICATION-KEY=<app-key>,Fastly-Client=1'
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS=30
export DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS=2

java -javaagent:/path/to/dd-java-agent.jar -jar app.jar

Validation

System Tests

flowchart LR
    Tests["system-tests\nPARAMETRIC java"]
    JavaApp["java-test-client\nlocal dd-java-agent + dd-openfeature"]
    RC["dd-apm-test-agent\nRemote Config"]
    MockBackend["MockFFEAgentlessBackend\n/api/v2/feature-flagging/config/server-distribution"]
    UFC["flags-v1.json\nshared UFC fixture"]
    Eval["OpenFeature evaluate endpoint"]
    Assertions["assertions\nsource selection, auth, ETag/304, retry, timeout, malformed UFC, warm preservation, no overlap"]

    Tests --> JavaApp
    Tests --> RC
    Tests --> MockBackend
    RC -->|remote_config scenarios| JavaApp
    MockBackend -->|agentless scenarios| JavaApp
    UFC --> MockBackend
    JavaApp --> Eval
    Eval --> Assertions
    MockBackend --> Assertions

Loading

Evidence from the local Java system-tests run:

command: ./run.sh PARAMETRIC -L java tests/parametric/test_ffe/test_configuration_sources.py -q
library: java@1.64.0-SNAPSHOT+23c0b22999
result: 18 passed in 55.26s
mock route: /api/v2/feature-flagging/config/server-distribution
report: /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests/logs_parametric/reportJunit.xml

Dogfooding

flowchart LR
    Compose["ffe-dogfooding\nno datadog-agent compose graph"]
    JavaDogfood["app-java\nlocal dd-java-agent + dd-openfeature"]
    MockBackend["mock-agentless-backend\n/api/v2/feature-flagging/config/server-distribution"]
    Fixture["ffe-system-test-data\nufc-config.json"]
    Evaluator["evaluator dashboard\n/api/latest-results"]
    Evidence["evidence artifacts\napp health, evaluate response, mock status"]

    Compose --> JavaDogfood
    Compose --> MockBackend
    Fixture --> MockBackend
    MockBackend -->|DD-API-KEY + optional extra headers| JavaDogfood
    JavaDogfood -->|OpenFeature evaluation| Evaluator
    JavaDogfood --> Evidence
    MockBackend --> Evidence
    Evaluator --> Evidence

    classDef dogfood fill:#fff4df,stroke:#d97706,stroke-width:2px,color:#111827;
    class Compose,JavaDogfood,MockBackend,Evaluator,Evidence dogfood;

Loading

Evidence from the local Java dogfood run:

compose services: mock-agentless-backend, otlp-intake, app-go, app-java, app-nodejs, evaluator
configuration_source: unset, proving default agentless auto-starts
agentless_base_url: http://mock-agentless-backend:8092
provider: PROVIDER_READY

evidence: /tmp/ffe-agentless-evidence/20260709T012421Z-java-java_valid_control_after_trace_java_comments
fixture: valid_control
app evaluate: service=java, variant=on-value, evaluation_reason=STATIC
mock status: requests_total=1, max_in_flight=1, last_auth_present=true, last_status_code=200, last_if_none_match="\"ufc-v1\""

evidence: /tmp/ffe-agentless-evidence/20260709T012422Z-java-java_delayed_no_overlap_after_trace_java_comments
fixture: delayed_no_overlap
app evaluate: service=java, variant=on-value, evaluation_reason=STATIC
mock status: requests_total=1, max_in_flight=1, last_auth_present=true, last_status_code=200, last_if_none_match="\"ufc-v1\""

[dd-octo-sts]

🔴 Java Benchmark SLOs — Performance SLO breach detected

Suite	Status
Startup	🔴 breach

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	15.02 s	14.66 s	[+1.4%; +3.5%] (significantly worse)
startup:insecure-bank:tracing:Agent	13.95 s	13.67 s	[+1.2%; +2.9%] (significantly worse)
startup:petclinic:appsec:Agent	16.79 s	16.69 s	[-0.2%; +1.4%] (no difference)
startup:petclinic:iast:Agent	16.87 s	16.93 s	[-1.3%; +0.6%] (no difference)
startup:petclinic:profiling:Agent	16.10 s	16.71 s	[-7.9%; +0.5%] (no difference)
startup:petclinic:sca:Agent	16.93 s	16.79 s	[-0.0%; +1.8%] (no difference)
startup:petclinic:tracing:Agent	15.61 s	16.08 s	[-7.0%; +1.2%] (no difference)

Commit: 23c0b229 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.