ν”Όλ“œλ‘œ λŒμ•„κ°€κΈ°
🧩 Runtime Snapshots #18 - Structured Runtime Perception: The Layer Agents Are Missing
Dev.toDev.to
AI/ML

λΈŒλΌμš°μ € μ—μ΄μ „νŠΈμ˜ 인지 ν•œκ³„ 극볡을 μœ„ν•œ Structured Runtime Perception 섀계

🧩 Runtime Snapshots #18 - Structured Runtime Perception: The Layer Agents Are Missing

Alechko2026λ…„ 6μ›” 29일5λΆ„advanced

Context

κΈ°μ‘΄ Browser AgentλŠ” Screenshot, Accessibility Tree, Raw DOM에 μ˜μ‘΄ν•˜λ©° UI μƒνƒœμ˜ λΉ„λŒ€μΉ­μ„± λ¬Έμ œμ— 직면함. λ‹¨μˆœ View λ°μ΄ν„°λ§ŒμœΌλ‘œλŠ” λ²„νŠΌμ˜ ν™œμ„±ν™” μ—¬λΆ€λ‚˜ λ‘œλ”© μƒνƒœ λ“± μ‹€μ‹œκ°„ λŸ°νƒ€μž„ μƒνƒœλ₯Ό μ •ν™•νžˆ νŒŒμ•…ν•˜μ§€ λͺ»ν•΄ λͺ¨λΈμ˜ νŒλ‹¨ 였λ₯˜κ°€ 빈번히 λ°œμƒν•¨.

Technical Solution

  • HTML λ§ˆν¬μ—…κ³Ό μ‹€μ œ λ Œλ”λ§ μƒνƒœ κ°„μ˜ 간극을 λ©”μš°λŠ” Structured Runtime Perception 계측 λ„μž…
  • SiFR(Structured Interface Representation)을 ν†΅ν•œ μ΅œμ ν™”λœ μƒνƒœ ν‘œν˜„μœΌλ‘œ λͺ¨λΈμ˜ μΆ”λ‘  νš¨μœ¨μ„± μ¦λŒ€
  • μ‹€μ œ μ‚¬μš©μž μ„Έμ…˜ λ‚΄μ—μ„œ μž‘λ™ν•˜λŠ” λΈŒλΌμš°μ € ν™•μž₯ ν”„λ‘œκ·Έλž¨ 및 MCP tools 기반의 μ‹€μ‹œκ°„ μƒνƒœ μΆ”μΆœ ꡬ쑰 섀계
  • λ‹¨μˆœ ν”½μ…€ 뢄석이 μ•„λ‹Œ 'ν˜„μž¬ μ‹€ν–‰ κ°€λŠ₯ν•œ μš”μ†Œ'와 'λ³€κ²½ 사항' μ€‘μ‹¬μ˜ μƒνƒœ μŠ€λƒ…μƒ· 생성 둜직 적용
  • 인증 및 κΆŒν•œμ΄ μœ μ§€λ˜λŠ” Live Session 연동을 톡해 APIκ°€ μ—†λŠ” λ ˆκ±°μ‹œ μ‹œμŠ€ν…œ 및 SaaS λŒ€μ‹œλ³΄λ“œ μ ‘κ·Όμ„± 확보

1. μ—μ΄μ „νŠΈ 섀계 μ‹œ DOM 전체가 μ•„λ‹Œ 'Actionable'ν•œ μš”μ†Œλ§Œ ν•„ν„°λ§ν•œ μƒνƒœ ν‘œν˜„μΈ΅μ„ κ΅¬μΆ•ν–ˆλŠ”κ°€

2. 정적 HTMLκ³Ό μ‹€μ œ λŸ°νƒ€μž„ μƒνƒœ(Disabled, Hidden, Loading)의 뢈일치λ₯Ό ν•΄κ²°ν•˜λŠ” 검증 둜직이 ν¬ν•¨λ˜μ—ˆλŠ”κ°€

3. μ™ΈλΆ€ 봇 계정이 μ•„λ‹Œ μ‹€μ œ μ‚¬μš©μž μ„Έμ…˜μ˜ μ»¨ν…μŠ€νŠΈλ₯Ό κ·ΈλŒ€λ‘œ μœ μ§€ν•˜λ©° μƒνƒœλ₯Ό μΆ”μΆœν•˜λŠ” ꡬ쑰인가

원문 읽기