Platform: Windows

Prerequisite: Basic Python, read this article and this article.

Introduction#

Today we will continue making the battle AI for game SvZ Defense. In this tutorial I will teach you how to read the level progress and leadership from game screenshots.

samurai-level-progress

Level progress (Samurai side) We want to read number 0 from this image.

samurai-leadership

Leadership (Samurai side) We want to read number 2 from this image.

zombie-level-progress

Level progress (Zombie side) We want to read number 0 from this image.

zombie-leadership

Leadership (Zombie side) We want to read number 6 from this image.

Observation#

The very first thing we notice is that the font and typeface of the digits are the same. This means that it’s a good situation to apply template matching technique.

Things are looking very similar as the reading coins amount problem. Except that in the coins amount the numbers are clearly white-background-black-text.

pachinko-digits

pachinko-digits2

where in this case the backgrounds are non-solid and the text have white or black color.

samurai-level-progress

samurai-level-progress2

samurai-leadership

samurai-leadership2

zombie-level-progress

zombie-level-progress2

zombie-leadership

zombie-leadership2

If we could come up with some way to convert them to white-background-black-text, we should be able to apply the same algorithm again to read the digits.

The Plan#

Fair warning: this is going to be more difficult than it looks. Please be patient and follow each step carefully as you move on.

1 . The first thing we want to do is actually quantize the image (reduce the number of colors). If we got lucky we might even make the background completely solid after this step.

progress_quantize_samurai

progress_quantize_zombie

leadership_quantize_samurai

leadership_quantize_zombie

2 . Since we only care about the black and white colors, in this step we can enhance the black and white colors in our images. After enhancing them intensively we will get something like:

progress_enhance_samurai

progress_enhance_zombie

leadership_enhance_samurai

leadership_enhance_zombie

3 . We only want to work with white-background-black-text images. After the last step our images are mostly black, white, and gray. Therefore we can just invert the color for the black-background-white-text images.

progress_converted_samurai

progress_converted_zombie

leadership_converted_samurai

leadership_converted_zombie

4 . Convert the images to binary and we will end up with something very similar to before.

progress_binary_samurai

progress_binary_zombie

leadership_binary_samurai

leadership_binary_zombie

5 . On this point forward, we will work with the blobs again but we can’t just yet apply the algorithm we have from before. We will want to make sure that the digits blobs are not broken and remove all blobs that are just noises. It doesn’t benefit your understanding to explain them here so I will put them in the later parts of the tutorial.

6 . Now we have met every requirement. Apply the template matching algorithm and read the digits.

Step 1️⃣#

Since this tutorial is going to be very long and potentially become hard to follow, this time I want to try something different.

samurai-read-digits-example

zombie-read-digits-example

Here are the two screenshots I used while I was developing the algorithm. This time I want you to first work with them before moving onto the real game. This ensures that we have the same environment and you will be adapting to your own environment much easier once your understand how it works.

Put the two screenshots in your project. Make sure your project have the following structure.

1
ai
2
|___ player_hp
3
|    |___ ...
4
|___ read_digit
5
|    |___ debug
6
|    |    |___ samurai-read-digits-example.png
7
|    |    |___ zombie-read-digits-example.png
8
|    |___ digits
9
|    |___ digit_recognizer.py
10
|    |___ reader.py
11
|___ config.toml
12
|___ ui_position.py

We will be adding more files for debugging purposes via code.

In reader.py, put

1
import os
2
import re
3
import cv2
4
import time
5
import glob
6
import numpy as np
7
import pytesseract
8
from PIL import Image
9
from typing import Tuple
10
from src.util.screen_getter import get_chosen_region, get_window_with_title
11
from src.ai.read_digit.digit_recognizer import Digit_Recognizer
12
from src.ai.ui_position import leadership_bound_samurai, level_progress_bound_samurai
13
from src.ai.ui_position import leadership_bound_zombie, level_progress_bound_zombie
14

15
script_dir = os.path.dirname(os.path.abspath(__file__))
16
pytesseract.pytesseract.tesseract_cmd = 'C:\Program Files\Tesseract-OCR\\tesseract.exe'
17

18
class Reader:
19
    def __init__(self, window, bound):
20
        self._window = window
21
        self._bound = bound
22
        self._digit_recognizer = Digit_Recognizer(os.path.join(script_dir, 'digits'))
23
        self.debug = False

In ui_position.py, add

1
leadership_bound_samurai = [56, 357, 88, 373]
2
level_progress_bound_samurai = [183, 51, 225, 69]
3
leadership_bound_zombie = [62, 357, 94, 373]
4
level_progress_bound_zombie = [188, 55, 230, 73]

In digit_recognizer.py, put

1
import os
2
import cv2
3
import glob
4
import numpy as np
5

6
class Digit_Recognizer:
7
    def __init__(self, digits_folder):
8
        def load_binary(path):
9
            img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
10
            _, binary_img = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY)
11
            return binary_img
12

13
        self.digit_template = {}
14

15
        # search for PNG files under digits folder
16
        png_files = glob.glob(os.path.join(digits_folder, "*.png"))
17

18
        # extract filenames without extension
19
        filenames = [os.path.splitext(os.path.basename(file))[0] for file in png_files]
20

21
        for filename in filenames:
22
            self.digit_template[filename] = load_binary(f'{digits_folder}/{filename}.png')
23

24
    def recognize(self, digit_image):
25
        def mse(array1, array2):
26
            r = np.mean((array1 - array2) ** 2)
27
            return r if r != 0 else 0.001
28

29
        similarities = []
30
        for key, value in self.digit_template.items():
31
            similarities.append(1 / mse(digit_image, value))
32

33
        result = 0
34
        highest = 0
35
        for i in range(len(similarities)):
36
            if similarities[i] > highest:
37
                highest = similarities[i]
38
                result = i
39

40
        return list(self.digit_template.keys())[result].split('_')[0]

NOTE
Digit_Recognizer is mostly the same as before, except that I changed the way to load templates. Now you only have to name the template image files correctly (same naming conversion as before) in order to use this class.

In config.toml, put

1
[digit_reader]
2
side = "samurai"  # samurai / zombie

NOTE
We won’t be using this file today.

In class Reader, add function

1
def extract(self):
2
    # region = get_chosen_region(self._window, self._bound)
3
    region = Image.open('debug/samurai-read-digits-example.png')
4
    region = region.crop(self._bound)
5
    processed_image = self._process_image(region)

NOTE
We will be using the example image for now and will be later changed to reading from emulator.

Now write the function to process the image. This will be steps 1 - 4. Let’s start with step 1: quantize the image. Create a new function in Reader.

1
@staticmethod
2
def _process_image(img):
3
    def rescale_and_quantize(image: Image.Image, scaling_factor=2, quantize_factor=3) -> Image.Image:
4
        new_size = (image.width * scaling_factor, image.height * scaling_factor)
5
        image = image.resize(new_size, Image.Resampling.BILINEAR)
6
        return image.quantize(colors=quantize_factor, method=Image.Quantize.FASTOCTREE)
7

8
    img = rescale_and_quantize(img, 2, 6)
9
    if self.debug:
10
        img.save('debug/quantized.png')
11
    return img

NOTE
The rescale_and_quantize() function rescales and quantizes a given PIL image. I decided to scale up by a factor of 2 because I found out that it increases the chance of successful recognitions. I also set quantize factor to 6. That’s also from experience. In the real life scenario, you will often need to try out different values in order for things to work.

Now let’s test the code.

1
if __name__ == '__main__':
2
    reader = Reader(None, leadership_bound_samurai)  # or 'level_progress_bound_samurai' for bound
3
    reader.debug = True
4
    reader.extract()

Run this code and verify that you have quantized.png

leadership_quantize_samurai if you used bound leadership_bound_samurai.

progress_quantize_samurai if you used bound level_progress_bound_samurai.

You can also test out the zombies side, but remember to change your test image path if you decide to do that.

HOORAY
🎉 Congratulations on finishing up step 1.

Step 2️⃣#

In this step we will enhance the black and white colors. We will enhance it to the extend that the resulting image will only contain white, black, and gray colors. In _process_image(), add

1
def enhance_black_and_white(image: Image.Image, factor=5) -> np.ndarray:
2
    grayscale = image.convert("L")
3
    image_array = np.array(grayscale, dtype=np.float32)
4
    image_array /= 255.0  # normalize
5
    image_array = (image_array - 0.5) * factor + 0.5  # increase contrast
6
    image_array = np.clip(image_array, 0, 1)  # keep values in valid range
7
    image_array = (image_array * 255).astype(np.uint8)  # convert back to 255 scale
8

9
    return image_array
10

11
img = rescale_and_quantize(img, 2, 6)
12
if self.debug:
13
    img.save('debug/quantized.png')
14
img = enhance_black_and_white(img, 5)
15
if self.debug:
16
    img.save('debug/enhance_black_and_white.png')
17
return img

NOTE
enhance_black_and_white() takes a PIL image and returns a cv2 image. Notice that we will be working with cv2 images on this point forward. It first converts the image to grayscale and normalize the image to the range between 0 and 1. After that it increases the contrast by the given factor. In the end it converts the range back to between 0 and 255.

Run the driver code again and verify you have a new debug image.

leadership_enhance_samurai

progress_enhance_samurai

HOORAY
🎉 Good job completing step 2!

Step 3️⃣#

This step is going to be a little tough. By this point we have images that are most white-background-black-text and black-background-white-text, but we only want white-background-black-text images. So the current task is really just to figure out if an image is black-background-white-text and invert it if yes.

To identify whether it has black-background, we can take the colors from the first row and the first column and calculate their mean. If it’s below the threshold the background would be black otherwise white.

Here is the code. Add it to _process_image()

1
def get_background_color(image: np.ndarray) -> Tuple[int, int, int]:
2
    # extract first row and first column
3
    first_row = image[0, :]  # all pixels in the first row
4
    first_col = image[:, 0]  # all pixels in the first column
5

6
    # combine both sets of pixels
7
    combined_pixels = np.hstack((first_row, first_col))
8

9
    # calculate the average color
10
    avg_color = np.mean(combined_pixels, axis=0).astype(int)
11

12
    # determine if it's closer to white (255, 255, 255) or black (0, 0, 0)
13
    threshold = np.array([127, 127, 127])
14
    # set to (255, 255, 255) if closer to white
15
    # otherwise (0, 0, 0)
16
    binary_color = (255, 255, 255) if np.all(avg_color > threshold) else (0, 0, 0)
17

18
    return binary_color
19

20
def convert_to_white_bg(image: np.ndarray) -> np.ndarray:
21
    bg_color = get_background_color(image)
22
    if bg_color != (255, 255, 255):
23
        image = cv2.bitwise_not(image)
24
    return image
25

26
img = rescale_and_quantize(img, 2, 6)
27
if self.debug:
28
    img.save('debug/quantized.png')
29
img = enhance_black_and_white(img, 5)
30
if self.debug:
31
    img.save('debug/enhance_black_and_white.png')
32
img = convert_to_white_bg(img)
33
if self.debug:
34
    img.save('debug/convert_to_white_bg.png')
35
return img

NOTE
get_background_color() calculates the mean of the colors in the first row and first column in the image. convert_to_white_bg() inverts the image color if it’s black background.

Verify you have the debug image.

leadership_converted_samurai

progress_converted_samurai

HOORAY
🎉 Great! Onto step 4.

Step 4️⃣#

This step is quite easy and we have done it before. Convert the image to binary. Add the following to _process_image().

1
def convert_to_binary(image: np.ndarray, threshold=30) -> np.ndarray:
2
    _, binary_array = cv2.threshold(image, threshold, 255, cv2.THRESH_BINARY)
3
    return binary_array
4

5
img = rescale_and_quantize(img, 2, 6)
6
if self.debug:
7
    img.save('debug/quantized.png')
8
img = enhance_black_and_white(img, 5)
9
if self.debug:
10
    img.save('debug/enhance_black_and_white.png')
11
img = convert_to_white_bg(img)
12
if self.debug:
13
    img.save('debug/convert_to_white_bg.png')
14
img = convert_to_binary(img, 30)
15
if self.debug:
16
    img.save('debug/convert_to_binary.png')
17
return img

Again, verify you have the debug image.

leadership_binary_samurai

progress_binary_samurai

HOORAY
🎉 Things are looking well.

Step 5️⃣#

🌶️ Be warned: this step is very long and complex, make sure you follow carefully.

At the first glance, it looks like that we can apply template matching algorithm on our current image. However, that’s not true. Since the background was not solid, the digits are often read ‘broken’. Here is what I mean by that.

Zoom in our ‘2’ here and you will find out that it’s made with 2 blobs.

leadership_binary_samurai_x3

leadership_binary_samurai_x3_colored

See the problem here? This means that instead digit ‘2’. It will be two blobs that makes no sense after you apply dfs. To visualize the problem:

leadership_binary_samurai_x3_problem

The solution: merge the blobs that belong to the same digit. Since we have the bounding box of each blobs, we can determine whether two blobs belong to the same digit by comparing their x values. It’s easier to explain with pictures. Let’s say we have two blobs that can form as a digit.

Situation 1:

merge-blobs-situation1

Situation 2:

merge-blobs-situation2

Situation 3:

merge-blobs-situation3

Situation 4:

merge-blobs-situation4

I want you to pay attention to their x-values. I hope you will agree with me that there’s overlaps in the x-values in all above situations. Whenever an overlap in x-value occurs, we want to merge two blobs.

EXERCISE
BTW, here is our ‘2’ example. Can you see which situation is this?

Ok, that was the most important problem I wanted to mention in this step. We still have other problems though, but not as hard to understand as that one.

Here is the plan:

1 . Read the blobs with bounding box. We have done this before.

2 . Although unlikely, we need to remove all blobs that touches the edge of the image. These blobs are nearly impossible to be digits and will mess up the result if they exist.

3 . We want to remove noises from the image. These are small blobs that are not very likely to be part of digits or just insignificant.

4 . We want to merge the blobs. Just like discussed previously.

5 . By this point each blob should represent a digit (or text). We want to filter out blobs that don’t meet the minimum height.

6 . We want to remove small blobs that are not likely to be valid by their area.

After we finish all these steps, we can treat the image the same as before and apply template matching.

Let’s start by finding the blobs along with their bounding boxes. We have already discussed the code before so I won’t explain again. If you want to review please read this. Add the following to Reader.

1
def _find_blobs_with_bounding_box(self, binary_img):
2
    result = []
3

4
    # visited array to track processed pixels
5
    visited = np.zeros_like(binary_img, dtype=np.uint8)
6

7
    # iterate through each pixel in the binary image
8
    h, w = binary_img.shape
9
    for y in range(h):
10
        for x in range(w):
11
            if binary_img[y, x] == 0 and visited[y, x] == 0:
12
                # extract the blob using DFS and get its bounding box
13
                blob_mask, min_x, max_x, min_y, max_y = self._dfs(x, y, binary_img, visited)
14
                # append the blob mask and bounding box to the list
15
                result.append((blob_mask, min_x, max_x, min_y, max_y))
16

17
    # sort blobs by their x-coordinate (left-to-right order)
18
    result.sort(key=lambda b: b[1])  # sort by min_x
19

20
    # return only the blobs
21
    return result
22

23
@staticmethod
24
def _dfs(x, y, binary_img, visited):
25
    assert binary_img[y, x] == 0, "starting position is not black"
26

27
    directions = [(-1, -1), (-1, 0), (-1, 1),
28
                    (0, -1), (0, 1),
29
                    (1, -1), (1, 0), (1, 1)]
30

31
    h, w = binary_img.shape
32
    stack = [(x, y)]
33
    min_x, max_x, min_y, max_y = x, x, y, y  # track bounding box
34

35
    # create a white background to store the blob
36
    blob_mask = np.full_like(binary_img, 255, dtype=np.uint8)
37

38
    while stack:
39
        cx, cy = stack.pop()
40
        if 0 <= cx < w and 0 <= cy < h and binary_img[cy, cx] == 0 and visited[cy, cx] == 0:
41
            # mark the pixel as visited
42
            visited[cy, cx] = 1
43
            # paint position (cx, cy) in blob_mask
44
            blob_mask[cy, cx] = 0
45

46
            # update bounding box
47
            min_x, max_x = min(min_x, cx), max(max_x, cx)
48
            min_y, max_y = min(min_y, cy), max(max_y, cy)
49

50
            # push all nearby 8 neighbors
51
            for dx, dy in directions:
52
                nx = cx + dx
53
                ny = cy + dy
54
                stack.append((nx, ny))
55

56
    return blob_mask, min_x, max_x, min_y, max_y

Next, use them in the function extract().

1
def extract(self):
2
    region = get_chosen_region(self._window, self._bound)
3
    processed_image = self._process_image(region)
4

5
    blobs_with_bounding_box = self._find_blobs_with_bounding_box(processed_image)

Next, we’ll be removing all blobs that touches the edge of the image. To visualize:

edge-touching-example

All red blobs will be removed. All black blobs will be preserved.

The code is surprisingly short since we have the bounding boxes.

1
@staticmethod
2
def _remove_edge_touching_blobs(blobs, image_width, image_height):
3
    # keep only blobs that do not touch any of the image boundaries
4
    filtered_blobs = [
5
        (blob_mask, min_x, max_x, min_y, max_y)
6
        for blob_mask, min_x, max_x, min_y, max_y in blobs
7
        if min_x > 0 and min_y > 0 and max_x < image_width and max_y < image_height
8
    ]
9

10
    return filtered_blobs

Add this to extract().

1
height, width = processed_image.shape[:2]
2
filtered_blobs = self._remove_edge_touching_blobs(blobs_with_bounding_box, width, height)

Next, we want to remove all the noises (tiny blobs) from the image. Since each blobs is a np array of 0s and 1s, this is also quite easy.

1
@staticmethod
2
def _remove_small_blobs(blobs, min_size):
3
    # keep only blobs with at least `min_size` black pixels (0)
4
    filtered_blobs = [
5
        (blob_mask, min_x, max_x, min_y, max_y)
6
        for blob_mask, min_x, max_x, min_y, max_y in blobs
7
        if np.sum(blob_mask == 0) >= min_size
8
    ]
9

10
    return filtered_blobs

NOTE
We get the sum of the 0s (black colors) in a blob and compare that with our threshold.

Add this to extract().

1
# removes small blobs that aren't likely to be part of digits
2
filtered_blobs = self._remove_small_blobs(filtered_blobs, 7)

Here comes the challenge part: merging the blobs.

First of all, add a new function in Reader.

1
def _merge_blobs(self, blobs, max_digit_width=23):

Inside _merge_blobs(), add

1
def get_merge_to(merge_list, blob):
2
    for i in range(len(merge_list)):
3
        x1i = blob[1]
4
        x1a = blob[2]
5
        x2i = merge_list[i][1]
6
        x2a = merge_list[i][2]
7

8
        if x1i <= x2i <= x1a <= x2a:
9
            if x2a - x1i <= max_digit_width:
10
                return i
11
        elif x2i <= x1i <= x2a <= x1a:
12
            if x1a - x2i <= max_digit_width:
13
                return i
14
        elif x2i <= x1i <= x1a <= x2a:
15
            if x2a - x2i <= max_digit_width:
16
                return i
17
        elif x1i <= x2i <= x2a <= x1a:
18
            if x1a - x1i <= max_digit_width:
19
                return i
20

21
    return -1

NOTE
This function determines the first mergable blob from the list for a given blob. This is also the code interpretation for the four situations we stated in the beginning of this step.

Inside _merge_blobs(), add another function

1
def merge_blobs(blob1, blob2):
2
    def crop_and_paste_blob(mask, blob_mask, min_x, max_x, min_y, max_y):
3
        # Step 1: Crop out the relevant blob region from its original mask
4
        cropped_blob = blob_mask[min_y:max_y + 1, min_x:max_x + 1]
5

6
        # Step 2: Compute position where the blob should be placed in merged_mask
7
        y_offset = min_y  # adjust for new coordinate system
8
        x_offset = min_x
9

10
        # ensure pasting does not go out of bounds
11
        paste_y1 = max(0, y_offset)
12
        paste_x1 = max(0, x_offset)
13
        paste_y2 = min(paste_y1 + cropped_blob.shape[0], mask.shape[0])
14
        paste_x2 = min(paste_x1 + cropped_blob.shape[1], mask.shape[1])
15

16
        # ensure cropping of `cropped_blob` to match pasting area
17
        crop_y1 = 0
18
        crop_x1 = 0
19
        crop_y2 = paste_y2 - paste_y1
20
        crop_x2 = paste_x2 - paste_x1
21

22
        # Step 3: Paste the cropped blob into merged_mask, keeping black pixels (0)
23
        mask[paste_y1:paste_y2, paste_x1:paste_x2] = np.minimum(
24
            mask[paste_y1:paste_y2, paste_x1:paste_x2],
25
            cropped_blob[crop_y1:crop_y2, crop_x1:crop_x2]
26
        )
27

28
        return mask
29

30
    # extract masks and bounding boxes
31
    mask1, min_x1, max_x1, min_y1, max_y1 = blob1
32
    mask2, min_x2, max_x2, min_y2, max_y2 = blob2
33

34
    # compute new bounding box that encloses both blobs
35
    new_min_x = min(min_x1, min_x2)
36
    new_max_x = max(max_x1, max_x2)
37
    new_min_y = min(min_y1, min_y2)
38
    new_max_y = max(max_y1, max_y2)
39

40
    # create an empty white mask (255) of the new size
41
    merged_mask = np.ones(mask1.shape[:2], dtype=np.uint8) * 255
42
    merged_mask = crop_and_paste_blob(merged_mask, mask1, min_x1, max_x1, min_y1, max_y1)
43
    merged_mask = crop_and_paste_blob(merged_mask, mask2, min_x2, max_x2, min_y2, max_y2)
44

45
    return merged_mask, new_min_x, new_max_x, new_min_y, new_max_y

NOTE
This is the actual function that does the merging. Basically, it creates a new image, then copy-and-pastes blob1 and blob2 into it.

Before we continue, add this function in Reader.

1
@staticmethod
2
def _clear_folder(path):
3
    files = glob.glob(os.path.join(path, '*'))
4
    for file in files:
5
        try:
6
            os.remove(file)
7
        except Exception as e:
8
            print(f"Error deleting {file}: {e}")

This function clears out all files under the given path. We will be using it for debugging purposes.

Go back to _merge_blobs(), add

1
if self.debug:
2
    os.makedirs('debug/merged', exist_ok=True)
3
    self._clear_folder('debug/merged')
4

5
# [(blob_mask, min_x, max_x, min_y, max_y)]
6
merged = []
7
processed = 0
8
while processed < len(blobs):
9
    curr = blobs[processed]
10

11
    merge_to = get_merge_to(merged, curr)
12
    if merge_to != -1:
13
        # merge curr to merged
14
        merged[merge_to] = merge_blobs(merged[merge_to], curr)
15
        if self.debug:
16
            cv2.imwrite(f'debug/merged/{processed}.png', merged[merge_to][0])
17
    else:
18
        # add curr to merged
19
        merged.append(curr)
20

21
    processed += 1
22

23
return merged

NOTE
We created a specific folder to store the images of merged blobs. After that, we iterate through all blobs and merge all mergable blobs. In each step, if the current blob has no mergable, add it to merged list, otherwise merge them and add that to merged. Repeat until we reach the count of blobs.

Don’t forget to use this function in extract().

1
# merge blobs (digits could be 'cut off' and we need to merge them)
2
filtered_blobs = self._merge_blobs(filtered_blobs)

If you run the driver code now, you should expect to see a new debug image.

debug2

The image won’t be saved to merged if you used bound ‘level_progress_bound_samurai’ because the digit in this image is connected.

The rest of this step is easy. Now we will remove all blobs that don’t meet the height requirement.

Add the following in Reader.

1
@staticmethod
2
def _filter_out_blob_masks_not_meeting_height(blobs, min_height=15):
3
    filtered_blobs = [
4
        (blob_mask, min_x, max_x, min_y, max_y)
5
        for blob_mask, min_x, max_x, min_y, max_y in blobs
6
        if max_y - min_y >= min_height
7
    ]
8

9
    return filtered_blobs

NOTE
Since we have bounding boxes, this is simple to do. The max_y subtracts min_y is the height of the blob.

Use this function in extract().

1
# removes blobs that aren't likely to be digits
2
filtered_blobs = self._filter_out_blob_masks_not_meeting_height(filtered_blobs)

The end of this step. We don’t need to add new function. Add the following in extract().

1
# removes blobs that aren't likely to be digits
2
filtered_blobs = self._remove_small_blobs(filtered_blobs, 30)

HOORAY
🎉 Almost there!

Step 6️⃣#

As said in the beginning, this step has no new stuff and we just need to apply template matching algorithm on our images.

Add the following in Reader.

1
@staticmethod
2
def _remove_blob_background(blob_with_bounding_box):
3
    blob_mask = blob_with_bounding_box[0]
4
    min_x = blob_with_bounding_box[1]
5
    max_x = blob_with_bounding_box[2] + 1
6
    min_y = blob_with_bounding_box[3]
7
    max_y = blob_with_bounding_box[4] + 1
8
    return blob_mask[min_y:max_y, min_x:max_x]

In extract(), add

1
blobs = []
2
for blob_with_bounding_box in filtered_blobs:
3
    # put the blob to the center of a 24x24 image
4
    blob = self._remove_blob_background(blob_with_bounding_box)
5
    blobs.append(self._edit_blob_canvas(blob, 24, 24))
6

7
if self.debug:
8
    os.makedirs('debug_digit', exist_ok=True)
9
    self._clear_folder('debug_digit')
10

11
count = 0
12
for blob in blobs:
13
    if blob[0]:  # if blob doesn't exceed our defined size
14
        if self.debug:
15
            cv2.imwrite(f'debug_digit/{count}.png', blob[1])
16
        count += 1
17

18
result = ''
19
for blob in blobs:
20
    if blob[0]:  # if blob doesn't exceed our defined size
21
        extracted_text = self._digit_recognizer.recognize(blob[1])
22
    else:
23
        extracted_text = pytesseract.image_to_string(blob[1], config='--psm 6')
24
        match = re.search(r'\d+', extracted_text)
25
        if match:
26
            extracted_text = match.group()
27
        else:
28
            extracted_text = '0'
29

30
    if extracted_text.isdigit():
31
        result += extracted_text
32

33
return int(result) if result else 0

Don’t forget that we haven’t yet filled the digits folder. You can use mine for now. Add these to digits.

Run the driver code with ‘leadership_bound_samurai’. You should see the result:

1
2

Run with ‘level_progress_bound_samurai’:

1
0

After you confirmed the result, change the top three lines in extract() to

1
def extract(self):
2
    region = get_chosen_region(self._window, self._bound)
3
    # region = Image.open('debug/samurai-read-digits-example.png')
4
    # region = region.crop(self._bound)

Next, change the driver code.

1
if __name__ == '__main__':
2
    chosen_window = get_window_with_title('BlueStacks App Player')
3
    reader = Reader(chosen_window, leadership_bound_samurai)
4
    reader.debug = True
5
    while True:
6
        print(reader.extract())
7
        time.sleep(0.5)

Open the game, play as Samurai. Check if the program is really reading the leadership value. If not, your bounding box is very likely to be different from mine. The easiest fix is to use the same setup as mine. In window_rescaler.py, make sure your boundary looks like this:

1
top, bottom, left, right = 0, 482, 0, 819

If not, adjust the values and run window_rescaler.py. If the problem still persists or you want to use your own setup, you will have to use mouse_coordinator.py to read the boundary for leadership and level progress. Good luck!

HOORAY
🎉 That’s the end of this tutorial! It should work for both samurai and zombie side as long as your bounding boxes are correct. Stay tunned for more!

💗 If you liked this blog, consider following me on GitHub.

🍯 Happy Coding 🍯