Lee ChanyangBackend Engineer

Language

← Back to project list

Personal Project · Vision Automation

In Progress

Screen-Recognition Automation Engine

I am developing a personal automation-runtime project that recognizes objects and states in screen environments that are difficult to control through APIs or selectors, executing actions only in verified states.

Role
Automation Systems Engineer
Responsibility
Problem definition and implementation of dataset tools, a detection runtime, and a guarded execution loop
Period
2026.05–Present · In Progress
Team
Solo
  • Python
  • OpenCV
  • YOLOv4-tiny
  • Tkinter
  • unittest
  • Windows API

Project Overview

The goal is an automation engine that observes the screen, makes a judgment, and executes safely

This is an ongoing personal project, not a presentation of a finished product. It shows the technical choices I made to reduce recurring screen-automation failures and the extent to which I verified them.

In real legacy business interfaces, VDI, native apps, and remote environments, DOM selectors or APIs may be unavailable or unreliable. Automation then becomes a problem of reliably identifying the current screen state, rather than simply clicking coordinates.

I therefore organized the project around Screen Perception, Dataset Flywheel, and Guarded Action Runtime rather than simple input automation. The intended structure reads objects and states from the screen and acts only when that assessment is sufficient.

Progression from Problem to ProductizationMermaid

Rendering the diagram.

Ownership

My direct responsibilities

  • Separated failure modes to understand why screen automation based on fixed coordinates and sleep timing breaks
  • Combined screen capture, manual labeling, YOLO training packaging, automatic labeling, and review into a single Detector Studio workflow
  • Separated target-specific detection models, coordinate correction, and movement/action timing into Profile and Strategy to manage variants without code branches
  • Implemented screen perception by combining ROI capture, template matching, YOLO detection, and HSV/connected-component-based state assessment
  • Implemented safeguards for coordinate actions using a state-based execution loop, stop_event, input release, a movement guard, and persisted execution budgets

Constraints

Constraints to address first

  • The project began as personal screen-automation experiments that are difficult to disclose. The public portfolio does not repackage the original use case as a product case study; it covers only reusable technical boundaries such as screen capture, labeling, detection, state assessment, and guarded execution loops.
  • Manipulation of third-party services, bypassing access restrictions, automated login/account operation, and deployable automated execution tools are excluded from the public, demo, and reuse scope.
  • Original domain names, screen images, model weights, coordinates/profiles, and operational data are not disclosed.
  • The current implementation is a prototype validated in a personal experimental environment, not a completed general-purpose RPA/QA product or commercial computer-vision platform.
  • Some live-screen tests depend on Windows GUI and specific rendering conditions, making them difficult to reproduce unchanged in CI.
  • Detection metrics are sensitive to datasets and execution environments, so I focus on structure and the improvement process rather than overstating them as headline outcomes.

Implementation Timeline

I divided the work into phases from problem discovery through datasets, perception, guarded execution, validation, and potential productization

Fixed coordinates, fixed waits, and single-image matching failed with even small changes in window position, rendering, frame rate, or UI transitions. I first established a structure that reads screen state before execution, then separated guarded execution and productization candidates into phases. Select a phase to see the problem identified at that stage and how it led to the architecture and verification criteria.

Phase 01 · Dataset Flywheel

Dataset Flywheel

Phase 01

I first built a collection, labeling, and automatic-label review loop to improve recognition

The first bottleneck in live-screen automation was not execution logic, but repeatedly collecting target samples and feeding corrected labels back into training.

Detector Studio Data FlywheelMermaid

Rendering the diagram.

  1. 01

    Requirements Analysis

    Screen objects were affected by background, size, occlusion, and brightness changes rather than behaving like static UI components. Manual coordinate corrections therefore did not produce cumulative improvement.

    • I automated YOLO label formatting and training packaging so accumulated samples could lead to model improvements.
    • I aimed to reduce repetitive labeling effort through a workflow in which people review only incorrect results after automatic labeling.
  2. 02

    Domain Modeling

    I implemented a workspace and Detector Studio workflow that separate captured images, manual labels, automatic labels, review results, and training packages.

    • Manual labeling results are converted into YOLO training input.
    • An existing model infers labels for unlabeled images, producing candidates for review; reviewed results feed back into the training set.
    • I separated target-specific detector directories so multiple detection targets could be explored in parallel.
  3. 03

    Implementation & Verification

    The current workflow is coupled to a specific training approach and Windows execution environment, so it lacks the portability and verifiability needed for a public tool.

    • Replace the training workflow with Ultralytics YOLO to align with the current ecosystem.
    • Create a synthetic demo dataset instead of using actual screens, providing a publicly shareable reproducible example.
    • Add label round-trip, training-packaging, and interruption-recovery tests.

Outcome

Verifiable outcomes

  • I separated screen-automation failure causes into fixed coordinates, fixed waits, false detections, and capture/rendering differences.
  • I implemented a Detector Studio workflow from screen capture to labeling, training packaging, automatic labeling, and review.
  • I managed detection targets and execution variants through 13 target-specific detector directories and 17 profiles, without branching the code.
  • I built a screen-perception runtime combining ROI capture, template matching, object detection, and color/brightness-based state assessment.
  • I added safeguards for coordinate actions through a state-based execution loop, stop_event, a movement guard, and atomic budget persistence.
  • 37 passing offline workflow tests verified basic state transitions in long-running execution loops.

Limitations

Limitations and unimplemented scope

  • The original repository is private, and a publicly shareable neutral-demo repository has not yet been separated out.
  • Detector Studio has an implemented workflow but lacks dedicated tests and a fully organized modern training path.
  • Some live-screen detection depends on Windows GUI, rendering methods, and datasets, and cannot currently be reproduced reliably in CI.
  • It does not yet include the permissions, audit logs, user approvals, and deployment packaging expected of a commercial RPA/QA product.
  • Detection-accuracy metrics are strongly affected by dataset bias and execution environments, so I did not use them as headline outcomes.