The system consists of three cooperating processes running inside a VirtualBox VM, and an external client
running in WSL on the host machine. Figure 1 shows the overall architecture and the data flows between components.

- The Load Balancer is a Flask application running under a single-worker gunicorn server with gevent, which enables concurrent forwarding of requests without blocking. When a request arrives, the load balancer selects a worker container according to the configured algorithm, forwards the request via HTTP, and returns the response. It also records response times to Redis for use by the scaling controller.
- The Scaling Controller runs as an independent process. It periodically reads the metrics written by the load balancer from Redis, evaluates the configured scaling algorithm, and spawns or destroys worker containers via the Podman API. The current container list is stored in Redis so that both the load balancer and the scaling controller can read and update it consistently.
- The Worker Containers are Podman containers built from a common image. Each runs a single-threaded Python process (functions-framework) that receives a watermarking request, applies the watermark, and returns the result. Containers are created and destroyed dynamically by the scaling controller. The watermark image files are bind-mounted from the host into each container.
I have implemented two load balancing algorithms: one static Round Robin algorithm, and one dynamic, state based Least Connections algorithm. Since I am using a Flask server, all forwards are synchronous and blocking. Thus, the load balancer isn’t running as long as a forwarded request is pending. To overcome this, I use gevent to run the load balancer. Gevent provides asynchronous forwarding behaviour (using green-threads and I/O monkey-patching)[3]. Since the scaling controller runs as an independent process, the list of running containers (ContainerArray) is stored in the Redis server.
Algorithm 1 is used in many balancing and scheduling implementations. It maintains a list of running containers, and forwards incoming requests sequentially to all running containers.
Algorithm 1: Pseudocode for the Round Robin algorithm
input: request (an HTTP request to the /watermark endpoint)
1 containers = ContainerArray // list of running containers
2 idx = int // index of container which got the previous request
3 idx = (idx + 1) mod len(containers)
4 container = containers[idx]
5 forward(container, request)
While the Round Robin algorithm distributes request by its number, algorithm 2 distributes requests by the current load of each container. A container’s response is proportional to the size of the image and the size of the watermark it is processing. Thus, any container that is processing large images and watermarks will accumulate a longer backlog in the Round Robin algorithm. Thus, this algorithm steers new requests away from it, and toward idler containers. We maintain a dictionary mapping containers to the number of active requests it has, and each new request is sent to the container with the fewest pending requests.
Algorithm 2: Pseudocode for the Least Connections algorithm
input: request (an HTTP request to the /watermark endpoint)
1 containers: ContainerArray // list of running containers
2 selected container: Container
3 minimum = ∞
4 foreach container in containers do
5 if container.active connections < minimum then
6 minimum = container.active connections selected container = container
7 end
8 end
9 selected container.active connections += 1
10 forward(selected container, request)
11 selected container.active connections -= 1
I have implemented two scaling algorithms, both of which make scaling decision based on the previous response times, but with different calculations. Using the mean service time of approximately 50 ms per request measured in the first calibration experiment (table 2), SCALE_UP_THRESHOLD was set to 800 ms, which is roughly 16× the unloaded service time, corresponding to an in-flight queue depth of around 15 requests per container before scale-up is triggered. SCALE_DOWN_THRESHOLD was set to 500 ms (a queue depth of around 9), leaving a 300 ms gap to provide some lag between successive scaling decisions.
Algorithm 3 is a threshold-based scaling algorithm, where the decision to scale up or down is made based on the
average response time of the last 100 requests serviced by the load balancer. The algorithm is entirely reactive,
i.e., it acts only after the average response time has already crossed a threshold, meaning SLA degradation has already begun before a scaling decision is triggered. A cool down period of SCALING_COOLDOWN seconds
is enforced between successive scaling events to prevent thrashing: rapidly alternating between scale-up and scale-down decisions. The two thresholds, SCALE_UP_THRESHOLD and SCALE_DOWN_THRESHOLD, are deliberately asymmetric, to ensure that the gap between them provides hysteresis, so that the system does not immediately attempt to scale back down after scaling up. The thresholds were calibrated against the latency as measured by load balancer (which includes request forwarding overhead), rather than the raw container processing time.
Algorithm 3: Pseudocode for the Threshold-based scaling algorithm
1 containers: ContainerArray
2 average response time = mean(response times[:100]) // average response times of the last 100
requests across all containers
3 if time since last scale > SCALING_COOLDOWN_TIME and ¬IS SCALING then
4 if average response time > SCALE_UP_THRESHOLD and len(containers) < MAX_CONTAINERS
then
5 scale up()
6 end
7 else if average response time < SCALE_DOWN_THRESHOLD and len(containers) >
MIN_CONTAINERS then
8 scale down()
9 end
10 end
Algorithm 4 improves on the threshold-based approach by making proactive scaling decisions. Rather than
comparing the current average response time against a fixed threshold, it fits a linear regression to the most
recent response time measurements and uses the resulting slope to extrapolate a predicted response time
REGRESSION_LOOKAHEAD samples into the future. A positive slope indicates that response times are trending
upward, and if the predicted value exceeds SCALE_UP_THRESHOLD, the algorithm scales up before the threshold is
actually crossed. Conversely, a negative slope combined with a low predicted value triggers a scale-down. This
makes the algorithm better suited to gradually increasing loads (such as the ramp pattern), where the trend is
visible before the system is fully saturated. However, the algorithm requires a minimum number of observations
before the regression is reliable, introducing a cold-start period at the beginning of each experiment. It also
provides limited benefit for sudden burst traffic, where the load spike may arrive and subside faster than the
regression can detect and respond to the trend.
Algorithm 4: Pseudocode for the Regression-based scaling algorithm
containers: ContainerArray
rt slope = response time slope() predicted rt = avg rt + rt slope * REGRESSION_LOOKAHEAD
if time since last scale > SCALING_COOLDOWN_TIME and not IS SCALING then
if predicted rt > SCALE_UP_THRESHOLD and n < MAX_CONTAINERS then
scale up()
end
else if predicted rt > SCALE DOWN THRESHOLD and n > MIN CONTAINERS then
scale down()
end
end