Skip to content

About

FaaS Platform simulation using Python

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

Implementation

System Architecture

The system consists of three cooperating processes running inside a VirtualBox VM, and an external client running in WSL on the host machine. Figure 1 shows the overall architecture and the data flows between components. Figure 1: Architectural diagram of the implemented system

  1. The Load Balancer is a Flask application running under a single-worker gunicorn server with gevent, which enables concurrent forwarding of requests without blocking. When a request arrives, the load balancer selects a worker container according to the configured algorithm, forwards the request via HTTP, and returns the response. It also records response times to Redis for use by the scaling controller.
  2. The Scaling Controller runs as an independent process. It periodically reads the metrics written by the load balancer from Redis, evaluates the configured scaling algorithm, and spawns or destroys worker containers via the Podman API. The current container list is stored in Redis so that both the load balancer and the scaling controller can read and update it consistently.
  3. The Worker Containers are Podman containers built from a common image. Each runs a single-threaded Python process (functions-framework) that receives a watermarking request, applies the watermark, and returns the result. Containers are created and destroyed dynamically by the scaling controller. The watermark image files are bind-mounted from the host into each container.

Load Balancer

I have implemented two load balancing algorithms: one static Round Robin algorithm, and one dynamic, state based Least Connections algorithm. Since I am using a Flask server, all forwards are synchronous and blocking. Thus, the load balancer isn’t running as long as a forwarded request is pending. To overcome this, I use gevent to run the load balancer. Gevent provides asynchronous forwarding behaviour (using green-threads and I/O monkey-patching)[3]. Since the scaling controller runs as an independent process, the list of running containers (ContainerArray) is stored in the Redis server.

Round Robin

Algorithm 1 is used in many balancing and scheduling implementations. It maintains a list of running containers, and forwards incoming requests sequentially to all running containers.

Algorithm 1: Pseudocode for the Round Robin algorithm

input: request (an HTTP request to the /watermark endpoint)  
1 containers = ContainerArray // list of running containers
2 idx = int                   // index of container which got the previous request
3 idx = (idx + 1) mod len(containers)
4 container = containers[idx]
5 forward(container, request)

Least Connections

While the Round Robin algorithm distributes request by its number, algorithm 2 distributes requests by the current load of each container. A container’s response is proportional to the size of the image and the size of the watermark it is processing. Thus, any container that is processing large images and watermarks will accumulate a longer backlog in the Round Robin algorithm. Thus, this algorithm steers new requests away from it, and toward idler containers. We maintain a dictionary mapping containers to the number of active requests it has, and each new request is sent to the container with the fewest pending requests.

Algorithm 2: Pseudocode for the Least Connections algorithm

input: request (an HTTP request to the /watermark endpoint)
 1 containers: ContainerArray // list of running containers
 2 selected container: Container
 3 minimum = ∞
 4 foreach container in containers do
 5 if container.active connections < minimum then
 6 minimum = container.active connections selected container = container
 7 end
 8 end
 9 selected container.active connections += 1
10 forward(selected container, request)
11 selected container.active connections -= 1

Scaling Controller

I have implemented two scaling algorithms, both of which make scaling decision based on the previous response times, but with different calculations. Using the mean service time of approximately 50 ms per request measured in the first calibration experiment (table 2), SCALE_UP_THRESHOLD was set to 800 ms, which is roughly 16× the unloaded service time, corresponding to an in-flight queue depth of around 15 requests per container before scale-up is triggered. SCALE_DOWN_THRESHOLD was set to 500 ms (a queue depth of around 9), leaving a 300 ms gap to provide some lag between successive scaling decisions.

Threshold-Based

Algorithm 3 is a threshold-based scaling algorithm, where the decision to scale up or down is made based on the average response time of the last 100 requests serviced by the load balancer. The algorithm is entirely reactive, i.e., it acts only after the average response time has already crossed a threshold, meaning SLA degradation has already begun before a scaling decision is triggered. A cool down period of SCALING_COOLDOWN seconds is enforced between successive scaling events to prevent thrashing: rapidly alternating between scale-up and scale-down decisions. The two thresholds, SCALE_UP_THRESHOLD and SCALE_DOWN_THRESHOLD, are deliberately asymmetric, to ensure that the gap between them provides hysteresis, so that the system does not immediately attempt to scale back down after scaling up. The thresholds were calibrated against the latency as measured by load balancer (which includes request forwarding overhead), rather than the raw container processing time.

Algorithm 3: Pseudocode for the Threshold-based scaling algorithm

 1 containers: ContainerArray
 2 average response time = mean(response times[:100]) // average response times of the last 100
requests across all containers
 3 if time since last scale > SCALING_COOLDOWN_TIME and ¬IS SCALING then
 4      if average response time > SCALE_UP_THRESHOLD and len(containers) < MAX_CONTAINERS
then
 5      scale up()
 6      end
 7      else if average response time < SCALE_DOWN_THRESHOLD and len(containers) >
MIN_CONTAINERS then
 8          scale down()
 9      end
10 end

Regression-Based

Algorithm 4 improves on the threshold-based approach by making proactive scaling decisions. Rather than comparing the current average response time against a fixed threshold, it fits a linear regression to the most recent response time measurements and uses the resulting slope to extrapolate a predicted response time REGRESSION_LOOKAHEAD samples into the future. A positive slope indicates that response times are trending upward, and if the predicted value exceeds SCALE_UP_THRESHOLD, the algorithm scales up before the threshold is actually crossed. Conversely, a negative slope combined with a low predicted value triggers a scale-down. This makes the algorithm better suited to gradually increasing loads (such as the ramp pattern), where the trend is visible before the system is fully saturated. However, the algorithm requires a minimum number of observations before the regression is reliable, introducing a cold-start period at the beginning of each experiment. It also provides limited benefit for sudden burst traffic, where the load spike may arrive and subside faster than the regression can detect and respond to the trend.

Algorithm 4: Pseudocode for the Regression-based scaling algorithm

containers: ContainerArray
rt slope = response time slope() predicted rt = avg rt + rt slope * REGRESSION_LOOKAHEAD
if time since last scale > SCALING_COOLDOWN_TIME and not IS SCALING then
    if predicted rt > SCALE_UP_THRESHOLD and n < MAX_CONTAINERS then
        scale up()
    end
    else if predicted rt > SCALE DOWN THRESHOLD and n > MIN CONTAINERS then
        scale down()
    end
end

About

FaaS Platform simulation using Python

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages