Skip to content

Commit efbf211

Browse files
authored
feat(docker): map server.urls_to_pd and server.deploy_in_k8s from env (#3352)
The Server entrypoint maps a set of HG_SERVER_* variables onto the conf files, but nothing mapped the two keys that decide which address a Server registers in PD: server.urls_to_pd and server.deploy_in_k8s. GraphManager registers the urls_to_pd list only when deploy_in_k8s is true, and restserver.url otherwise. Without a mapping, the Helm chart rewrites rest-server.properties in a shell wrapper before it runs the entrypoint, and the TODO in docker-entrypoint.sh asked for these mappings.
1 parent a4da5d1 commit efbf211

5 files changed

Lines changed: 108 additions & 6 deletions

File tree

‎helm/hugegraph/templates/_helpers.tpl‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -946,7 +946,7 @@ start-hugegraph-pd.sh, start-hugegraph-store.sh, and hugegraph-server.sh).
946946
{{- $reservedEnv := dict
947947
"pd" (list "HG_PD_GRPC_HOST" "HG_PD_GRPC_PORT" "HG_PD_REST_PORT" "HG_PD_RAFT_ADDRESS" "HG_PD_RAFT_PEERS_LIST" "HG_PD_INITIAL_STORE_LIST" "HG_PD_INITIAL_STORE_COUNT" "HG_PD_DATA_PATH" "HG_PD_AUTH_SECRET_KEY" "JAVA_OPTS" "JAVA_OPTIONS")
948948
"store" (list "HG_STORE_PD_ADDRESS" "HG_STORE_GRPC_HOST" "HG_STORE_GRPC_PORT" "HG_STORE_REST_PORT" "HG_STORE_RAFT_ADDRESS" "HG_STORE_DATA_PATH" "JAVA_OPTS" "JAVA_OPTIONS")
949-
"server" (list "POD_IP" "HG_SERVER_SERVICE_ID" "HG_SERVER_GRAPHSPACE" "SERVICE_ID" "GRAPH_SPACE" "HG_SERVER_BACKEND" "HG_SERVER_PD_PEERS" "HG_SERVER_PD_REST_ENDPOINT" "STORE_REST" "HG_SERVER_INIT_STORE_ENABLED" "HG_SERVER_URLS_TO_PD" "HG_SERVER_STARTUP_TIMEOUT_S" "WAIT_STORAGE_TIMEOUT_S" "PD_AUTH_PASSWORD" "PASSWORD" "HG_SERVER_AUTH_TOKEN_SECRET" "JAVA_OPTS" "JAVA_OPTIONS")
949+
"server" (list "POD_IP" "HG_SERVER_SERVICE_ID" "HG_SERVER_GRAPHSPACE" "SERVICE_ID" "GRAPH_SPACE" "HG_SERVER_BACKEND" "HG_SERVER_PD_PEERS" "HG_SERVER_PD_REST_ENDPOINT" "STORE_REST" "HG_SERVER_INIT_STORE_ENABLED" "HG_SERVER_URLS_TO_PD" "HG_SERVER_DEPLOY_IN_K8S" "HG_SERVER_STARTUP_TIMEOUT_S" "WAIT_STORAGE_TIMEOUT_S" "PD_AUTH_PASSWORD" "PASSWORD" "HG_SERVER_AUTH_TOKEN_SECRET" "JAVA_OPTS" "JAVA_OPTIONS")
950950
"hubble" (list "HG_HUBBLE_PD_PEERS" "HG_HUBBLE_PD_SERVER" "HG_HUBBLE_PD_PASSWORD" "HG_HUBBLE_STORE_TARGETS" "HG_HUBBLE_SERVER_URL" "SPRING_DATASOURCE_URL") -}}
951951
{{- range $component, $reserved := $reservedEnv -}}
952952
{{- $componentValues := get $.Values $component | default dict -}}

‎helm/hugegraph/tests/server_discovery_test.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,17 @@ tests:
3939
path: spec.template.spec.containers[0].env[?(@.name=="HG_SERVER_URLS_TO_PD")].value
4040
value: http://graph.example.com:30080
4141

42+
# The wrapper always registers server.urls_to_pd; the image would let an
43+
# extraEnv value turn that off again or crash-loop the Pod on a bad flag.
44+
- it: refuses an operator extraEnv that would override the PD registration flag
45+
set:
46+
server.extraEnv:
47+
- name: HG_SERVER_DEPLOY_IN_K8S
48+
value: "false"
49+
asserts:
50+
- failedTemplate:
51+
errorPattern: "server.extraEnv must not set the chart-managed variable HG_SERVER_DEPLOY_IN_K8S"
52+
4253
- it: refuses an operator extraEnv that would shadow POD_IP
4354
set:
4455
server.extraEnv:

‎hugegraph-server/hugegraph-dist/docker/README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,22 @@ HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose-3pd-3store-3server.yml
133133
See [docker/README.md](../../../docker/README.md) for the full setup guide,
134134
environment variable reference, and troubleshooting.
135135

136+
With `usePD=true` (`HG_SERVER_USE_PD`), a Server registers its REST address in
137+
PD. By default that is `restserver.url` (`HG_SERVER_REST_URL`). To register a different list instead, for example the
138+
addresses clients reach through a Kubernetes Service, set both of these on the
139+
Server container:
140+
141+
| Variable | Property in `rest-server.properties` |
142+
|---|---|
143+
| `HG_SERVER_URLS_TO_PD` | `server.urls_to_pd`, a comma-separated list such as `http://hg-0.hg:8080,http://hg-1.hg:8080` |
144+
| `HG_SERVER_DEPLOY_IN_K8S` | `server.deploy_in_k8s`; `true` makes the Server register `server.urls_to_pd` |
145+
146+
`HG_SERVER_DEPLOY_IN_K8S` accepts the boolean spellings HugeConfig accepts
147+
(`true`/`false`, `yes`/`no`, `on`/`off`, `y`/`n`, `t`/`f`, any case). Any other
148+
value, `1` and `0` included, stops the container before either conf file is
149+
changed. Give each URL its own `http://` prefix; whitespace in the list is
150+
removed. Unset variables leave the properties as they are.
151+
136152
## 6. Process Supervision & Health Checks
137153

138154
All four HugeGraph Docker images (PD, Store, Server, Server-hstore) include

‎hugegraph-server/hugegraph-dist/docker/docker-entrypoint-test.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -505,5 +505,59 @@ done
505505
HG_SERVER_AUTH_TOKEN_SECRET="${fake_key}" bash ./docker-entrypoint.sh)
506506
[[ "$(cat "${TEST_HOME}/docker/init-store-locale")" == 'en_US.UTF-8' ]]
507507

508+
# Unset discovery variables leave the shipped defaults alone.
509+
if grep -qE '^server\.(urls_to_pd|deploy_in_k8s)' \
510+
"${TEST_HOME}/conf/rest-server.properties"; then
511+
echo "unset discovery variables wrote server.* keys" >&2
512+
exit 1
513+
fi
514+
515+
(
516+
cd "${TEST_HOME}"
517+
HG_SERVER_URLS_TO_PD=' http://server-0:8080, http://server-1:8080 ' \
518+
HG_SERVER_DEPLOY_IN_K8S=' TRUE ' \
519+
bash ./docker-entrypoint.sh
520+
)
521+
grep -qx 'server.urls_to_pd=http://server-0:8080,http://server-1:8080' \
522+
"${TEST_HOME}/conf/rest-server.properties"
523+
grep -qx 'server.deploy_in_k8s=true' "${TEST_HOME}/conf/rest-server.properties"
524+
525+
# A restart with new values replaces each key instead of appending another.
526+
(
527+
cd "${TEST_HOME}"
528+
HG_SERVER_URLS_TO_PD=http://server-2:8080 \
529+
HG_SERVER_DEPLOY_IN_K8S=off \
530+
bash ./docker-entrypoint.sh
531+
)
532+
[[ "$(grep -c '^server\.urls_to_pd' "${TEST_HOME}/conf/rest-server.properties")" -eq 1 ]]
533+
[[ "$(grep -c '^server\.deploy_in_k8s' "${TEST_HOME}/conf/rest-server.properties")" -eq 1 ]]
534+
grep -qx 'server.urls_to_pd=http://server-2:8080' \
535+
"${TEST_HOME}/conf/rest-server.properties"
536+
grep -qx 'server.deploy_in_k8s=off' "${TEST_HOME}/conf/rest-server.properties"
537+
538+
# HugeConfig rejects 0 and 1 for a boolean, so the entrypoint refuses them
539+
# before writing either conf file, the URL list included. The graph conf holds
540+
# backend=hstore by now, so a different backend makes its cmp able to fail.
541+
cp "${TEST_HOME}/conf/rest-server.properties" \
542+
"${TEST_HOME}/conf/rest-server.properties.before-k8s-flag"
543+
cp "${TEST_HOME}/conf/graphs/hugegraph.properties" \
544+
"${TEST_HOME}/conf/graphs/hugegraph.properties.before-k8s-flag"
545+
for invalid_k8s_flag in 1 0 enabled "t rue" "tr ue"; do
546+
if (
547+
cd "${TEST_HOME}"
548+
HG_SERVER_BACKEND=rocksdb \
549+
HG_SERVER_URLS_TO_PD=http://server-3:8080 \
550+
HG_SERVER_DEPLOY_IN_K8S="${invalid_k8s_flag}" \
551+
bash ./docker-entrypoint.sh
552+
); then
553+
echo "HG_SERVER_DEPLOY_IN_K8S '${invalid_k8s_flag}' unexpectedly succeeded" >&2
554+
exit 1
555+
fi
556+
cmp "${TEST_HOME}/conf/rest-server.properties.before-k8s-flag" \
557+
"${TEST_HOME}/conf/rest-server.properties"
558+
cmp "${TEST_HOME}/conf/graphs/hugegraph.properties.before-k8s-flag" \
559+
"${TEST_HOME}/conf/graphs/hugegraph.properties"
560+
done
561+
508562
echo "PASS: Docker entrypoint configures HStore discovery and authentication"
509563
bash "${SCRIPT_DIR}/test/test-docker-entrypoint-process.sh"

‎hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh‎

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -195,12 +195,27 @@ if [[ -n "${PASSWORD:-}" ]]; then
195195
AUTH_ADMIN_PASSWORD_ENCODED=$(encode_auth_prop_value "${PASSWORD}")
196196
fi
197197

198+
# Same accepted spellings as HG_SERVER_INIT_STORE_ENABLED below, checked
199+
# before the mapping so an invalid value leaves both conf files untouched.
200+
# URLs hold no whitespace, and the Server splits the list on bare commas, so
201+
# "a, b" would otherwise register an escaped space in front of the second URL.
202+
URLS_TO_PD=$(printf '%s' "${HG_SERVER_URLS_TO_PD:-}" | tr -d '[:space:]')
203+
# Only surrounding whitespace is trimmed, so a value such as "t rue" stays invalid.
204+
DEPLOY_IN_K8S="${HG_SERVER_DEPLOY_IN_K8S:-}"
205+
DEPLOY_IN_K8S="${DEPLOY_IN_K8S#"${DEPLOY_IN_K8S%%[![:space:]]*}"}"
206+
DEPLOY_IN_K8S="${DEPLOY_IN_K8S%"${DEPLOY_IN_K8S##*[![:space:]]}"}"
207+
DEPLOY_IN_K8S=$(printf '%s' "${DEPLOY_IN_K8S}" | tr '[:upper:]' '[:lower:]')
208+
case "${DEPLOY_IN_K8S}" in
209+
"" | y | t | yes | on | true | n | f | no | off | false) ;;
210+
*) log "ERROR: invalid HG_SERVER_DEPLOY_IN_K8S '${HG_SERVER_DEPLOY_IN_K8S}'"
211+
exit 1 ;;
212+
esac
213+
198214
# ── Map env → properties file ─────────────────────────────────────────
199-
# TODO: map server.urls_to_pd and server.deploy_in_k8s from the environment here, as PASSWORD is
200-
# mapped to auth.admin_pa below. Without them the Helm chart (helm/hugegraph) rewrites
201-
# rest-server.properties in a wrapper before exec'ing this script; drop that wrapper once both keys
202-
# have a mapping. A read-only root filesystem also needs the conf files on a writable path, since
203-
# set_prop in this script edits them in place; the chart refuses readOnlyRootFilesystem until then.
215+
# TODO: drop the Helm chart's (helm/hugegraph) rest-server.properties wrapper once its minimum
216+
# Server image has the server.urls_to_pd and server.deploy_in_k8s mappings below. A read-only root
217+
# filesystem also needs the conf files on a writable path, since set_prop in this script edits them
218+
# in place; the chart refuses readOnlyRootFilesystem until then.
204219
[[ -n "${HG_SERVER_BACKEND:-}" ]] && set_prop "backend" "${HG_SERVER_BACKEND}" "${GRAPH_CONF}"
205220
[[ -n "${HG_SERVER_PD_PEERS:-}" ]] && set_prop "pd.peers" "${HG_SERVER_PD_PEERS}" "${GRAPH_CONF}"
206221
[[ -n "${HG_SERVER_USE_PD:-}" ]] && \
@@ -219,6 +234,12 @@ fi
219234
set_prop "server.graphspace" "${HG_SERVER_GRAPHSPACE}" "${REST_SERVER_CONF}"
220235
[[ -n "${HG_SERVER_REST_URL:-}" ]] && set_prop "restserver.url" \
221236
"${HG_SERVER_REST_URL}" "${REST_SERVER_CONF}"
237+
# The address list this Server registers in PD, read only when
238+
# server.deploy_in_k8s is true; otherwise it registers restserver.url.
239+
[[ -n "${URLS_TO_PD}" ]] && set_prop "server.urls_to_pd" \
240+
"${URLS_TO_PD}" "${REST_SERVER_CONF}"
241+
[[ -n "${DEPLOY_IN_K8S}" ]] && set_prop "server.deploy_in_k8s" \
242+
"${DEPLOY_IN_K8S}" "${REST_SERVER_CONF}"
222243
[[ -n "${HG_SERVER_MIN_FREE_MEMORY:-}" ]] && set_prop "restserver.min_free_memory" \
223244
"${HG_SERVER_MIN_FREE_MEMORY}" "${REST_SERVER_CONF}"
224245
if [[ -n "${AUTH_TOKEN_SECRET_ENCODED}" ]]; then

0 commit comments

Comments
 (0)