Personal Project · Vision Automation
In ProgressScreen-Recognition Automation Engine
I am developing a personal automation-runtime project that recognizes objects and states in screen environments that are difficult to control through APIs or selectors, executing actions only in verified states.
- Role
- Automation Systems Engineer
- Responsibility
- Problem definition and implementation of dataset tools, a detection runtime, and a guarded execution loop
- Period
- 2026.05–Present · In Progress
- Team
- Solo
- Python
- OpenCV
- YOLOv4-tiny
- Tkinter
- unittest
- Windows API
Project Overview
The goal is an automation engine that observes the screen, makes a judgment, and executes safely
This is an ongoing personal project, not a presentation of a finished product. It shows the technical choices I made to reduce recurring screen-automation failures and the extent to which I verified them.
In real legacy business interfaces, VDI, native apps, and remote environments, DOM selectors or APIs may be unavailable or unreliable. Automation then becomes a problem of reliably identifying the current screen state, rather than simply clicking coordinates.
I therefore organized the project around Screen Perception, Dataset Flywheel, and Guarded Action Runtime rather than simple input automation. The intended structure reads objects and states from the screen and acts only when that assessment is sufficient.
Rendering the diagram.
Ownership
My direct responsibilities
- Separated failure modes to understand why screen automation based on fixed coordinates and sleep timing breaks
- Combined screen capture, manual labeling, YOLO training packaging, automatic labeling, and review into a single Detector Studio workflow
- Separated target-specific detection models, coordinate correction, and movement/action timing into Profile and Strategy to manage variants without code branches
- Implemented screen perception by combining ROI capture, template matching, YOLO detection, and HSV/connected-component-based state assessment
- Implemented safeguards for coordinate actions using a state-based execution loop, stop_event, input release, a movement guard, and persisted execution budgets
Constraints
Constraints to address first
- The project began as personal screen-automation experiments that are difficult to disclose. The public portfolio does not repackage the original use case as a product case study; it covers only reusable technical boundaries such as screen capture, labeling, detection, state assessment, and guarded execution loops.
- Manipulation of third-party services, bypassing access restrictions, automated login/account operation, and deployable automated execution tools are excluded from the public, demo, and reuse scope.
- Original domain names, screen images, model weights, coordinates/profiles, and operational data are not disclosed.
- The current implementation is a prototype validated in a personal experimental environment, not a completed general-purpose RPA/QA product or commercial computer-vision platform.
- Some live-screen tests depend on Windows GUI and specific rendering conditions, making them difficult to reproduce unchanged in CI.
- Detection metrics are sensitive to datasets and execution environments, so I focus on structure and the improvement process rather than overstating them as headline outcomes.
Implementation Timeline
I divided the work into phases from problem discovery through datasets, perception, guarded execution, validation, and potential productization
Fixed coordinates, fixed waits, and single-image matching failed with even small changes in window position, rendering, frame rate, or UI transitions. I first established a structure that reads screen state before execution, then separated guarded execution and productization candidates into phases. Select a phase to see the problem identified at that stage and how it led to the architecture and verification criteria.
Dataset Flywheel
Outcome
Verifiable outcomes
- I separated screen-automation failure causes into fixed coordinates, fixed waits, false detections, and capture/rendering differences.
- I implemented a Detector Studio workflow from screen capture to labeling, training packaging, automatic labeling, and review.
- I managed detection targets and execution variants through 13 target-specific detector directories and 17 profiles, without branching the code.
- I built a screen-perception runtime combining ROI capture, template matching, object detection, and color/brightness-based state assessment.
- I added safeguards for coordinate actions through a state-based execution loop, stop_event, a movement guard, and atomic budget persistence.
- 37 passing offline workflow tests verified basic state transitions in long-running execution loops.
Limitations
Limitations and unimplemented scope
- The original repository is private, and a publicly shareable neutral-demo repository has not yet been separated out.
- Detector Studio has an implemented workflow but lacks dedicated tests and a fully organized modern training path.
- Some live-screen detection depends on Windows GUI, rendering methods, and datasets, and cannot currently be reproduced reliably in CI.
- It does not yet include the permissions, audit logs, user approvals, and deployment packaging expected of a commercial RPA/QA product.
- Detection-accuracy metrics are strongly affected by dataset bias and execution environments, so I did not use them as headline outcomes.