So, I looked on my abandoned projects, I found an interesting one, one where I want to develop a system to transform a screenshot of my Arknights inventory, and give me the list of how much items do I have.

I start with something like this

And the objective is to get something like:

[{ name: "Orundum", Amount: 124 }, { name: "LMD", Amount: 153800 }...]

So, I’ll start step by step, a baby mini-step that’d help us further down the line.

Given a cropped region, how do I know if it contains a proper image or not.

for example this is the comparison between bad crop, and a good crop:

Bad Crop

Bad Crop 2

Good Crop

Good Crop 2

So…, how do i do that?

Step 1: Definition.

So, let’s start with a definition, what is a good crop, and what is a bad crop.

generally speaking, a good crop should have colors on the inside, and no colors (white) on the outside. But, because the outside is whites (255, 255, 255), I think it’s easier to reason if I use the inverted values (0, 0, 0).

As starting point, I’ll be cheating and assume my crop area is the optimal size, the size of the icons, which is about 130x130 px.

Next, I’ll need to have a definition of what’s inside, and what’s outside.

A simple definition of it would be, by dividing the image into a region of 3x3 that looks like this

[ A B C ]
[ D E F ]
[ G H I ]

With this, the region “inside” would be the only one in the center (E), while the other region (A, B, C, D, F, G, H, I) is “outside”

However, if I applied the logic directly as is, the region will be divided into: