OpenGlasses

中文文档 (Chinese)

An open-source voice-powered AI assistant for Ray-Ban and Oakley Meta smart glasses. 85+ built-in tools, multi-LLM support (cloud + on-device) with automatic model routing, a fully offline voice mode (on-device speech-to-text, AI, and voice), personas with simultaneous wake words, an in-lens HUD with hands-free task control on Ray-Ban Display glasses, an on-device knowledge graph, live translation, hands-free field-service guidance, real-time vision coaching, MCP tool servers, and CarPlay + Apple Watch companions — all controlled hands-free by voice.

Note: The Meta Wearables SDK is currently in developer preview. App Store distribution is pending approval — each user must build the app from source with their own Meta developer credentials.

Quick Start

1. Clone and generate the Xcode project — OpenGlasses.xcodeproj is not in git; each developer creates it locally with XcodeGen (avoids project.pbxproj merge conflicts):

   copy

   git clone https://github.com/straff2002/OpenGlasses.git
   cd OpenGlasses
   brew install xcodegen
   ./Scripts/generate-xcodeproj.sh
   open OpenGlasses.xcodeproj
   

   After git pull, run ./Scripts/generate-xcodeproj.sh again if project.base.yml changed. Meta credentials, team ID, and signing: Building from Source (optional ./Scripts/setup-local-dev.sh for a personal overlay).

2. Build on your iPhone from Xcode (⌘R) — set signing team if prompted

3. Add an AI model in Settings → AI Models (Anthropic, OpenAI, Gemini, or a local model)

4. Pair your Ray-Ban or Oakley Meta glasses via the Meta AI app

5. Say "Hey OpenGlasses" and ask anything

Features

Personas — Multiple AI Personalities

Each persona has its own wake word, AI model, and personality. All listen simultaneously.

Configure: Settings → Personas → Add. Pick a wake word, assign a model and prompt preset.

"Hey Siri" — Ask by Voice Without the Wake Word

OpenGlasses ships App Intents + Siri Shortcuts, so you can drive it straight from Siri — handy when the glasses' own "Hey Meta" is busy, or to start a query hands-free without waiting for the in-app wake word:

The first time, iOS surfaces these in the Shortcuts app and the Siri phrase picker (you can rename the phrase to anything you like). A persona name can ride in the phrase ("ask Claude…") because it's a fixed, resolvable choice; the question can't (iOS only lets App Shortcut phrases embed fixed choices, not free-form text), so it's asked two-step — Siri prompts and awaits your spoken reply. Consecutive asks within a few minutes continue the same conversation thread, and Siri shows the answer in a result card. The intent runs in the background and speaks the result — no need to bring the app forward. If Siri ever says OpenGlasses isn't running, enable Settings → Voice → Open App for Siri Questions to have it launch the app first.

On-Device Local LLM

Run AI models entirely on your iPhone — no internet, no cloud, no API keys.

1. Settings → AI Models → Add Model → pick "Local (On-Device)"
2. Download & Manage Models → download from HuggingFace
3. Select your downloaded model and tap Add

Recommended models:

Gemma 4 E2B is the default on-device agent — it runs automatically when no cloud model is configured. Models are stored persistently and work fully offline after download. Toggle Offline Mode in Settings → Tools to disable internet-dependent tools.

Self-Hosted Local Server (Ollama, llama.cpp, vLLM…)

Prefer to run a bigger model on a desktop or home server and keep everything on your own network? Point OpenGlasses at any OpenAI-compatible endpoint — no cloud, no API key:

1. Settings → AI Models → Add Model → pick "Custom (OpenAI-compatible)"
2. Set the Base URL to your server, e.g. http://your-mac.local:11434/v1 (Ollama), and leave the API Key blank
3. Tap Fetch models to list what the server has, or type a model ID (e.g. llava for vision). Turn on Vision to send glasses photos.

Works with Ollama, llama.cpp server, LM Studio, vLLM, and LocalAI. Your photos, voice, and conversations never leave the machine running the server.

Reaching the server: use the host's .local mDNS name (http://mymac.local:11434/v1) or a Tailscale address (*.ts.net, already allow-listed) rather than a raw 192.168.x.x IP — iOS App Transport Security can block cleartext http:// to a bare private IP, but allows .local, loopback, and the Tailscale exception.

Fully Offline Voice Mode

Run the entire voice loop on-device — nothing leaves your iPhone:

• Speech-to-text — an on-device SenseVoice recognizer (multilingual, ~240 MB). No audio is sent to any server.
• The AI — a local LLM (Gemma / Qwen via MLX, or Apple Intelligence).
• Text-to-speech — an on-device Kokoro neural voice (~185 MB) — natural, far better than the robotic system voice.

Speech recognition and the spoken voice are CPU/ONNX (not Metal), so they keep working even while the app is backgrounded. Choose the engines under Settings → Services → Speech Recognition / Voice Engine and download the models once — then you have a private assistant that works on a plane, in a tunnel, or anywhere with no signal. Each tier also degrades gracefully: with no model (or no cloud), it falls back to Apple Speech and the iOS voice.

85+ Native Tools

All voice-activated. Say what you need naturally — the AI picks the right tool.

Live Coach — Real-Time Vision Coaching

The glasses watch what you're doing and give short, spoken corrections on a loop — one tight sentence at a time, no repetition. Built-in domains: posture, cooking technique, guitar, climbing, sports tactics — or define your own.

Smart Capture

Point at a business card, receipt, or event flyer — OpenGlasses reads it on-device and offers to act.

Voice-Taught Skills

Teach the AI new behaviors at runtime — no code needed.

Object Memory

Remember where you put things. Uses GPS to calculate distance.

Live Translation

Continuous real-time translation of spoken foreign language.

Supports 25+ languages including Spanish, French, German, Japanese, Chinese, Korean, Arabic, and more.

Social Context

Build dossiers about people you meet.

Works alongside face recognition — when the AI recognizes someone, it can recall your notes about them.

On-Device Knowledge Brain

A private, on-device knowledge graph that quietly connects what you tell it — people, places, things, and how they relate — with zero cloud calls. Notes, social context, face encounters, and meeting summaries all feed it, and the AI can query the whole graph in one step.

Native-first — it works without any external gateway, and everything stays on the phone.

Barge-In

Interrupt the AI mid-sentence by saying any wake word. It stops immediately and starts listening to your new question.

Prompt Presets

Switch AI personality without reconfiguring. Built-in presets:

Create your own in Settings → System Prompt.

Custom Tools

Define new tools without writing code. Map to Siri Shortcuts or URL schemes.

Settings → Transparency → Custom Tools → Add:

• Shortcut tool: triggers a Siri Shortcut by name
• URL tool: opens a URL with parameter substitution

Example: a "log_water" tool that runs your "Log Water" shortcut when the AI decides you need it.

MCP Servers (Model Context Protocol)

Connect to any MCP-compatible tool server directly from your phone.

Settings → Transparency → MCP Servers → Add:

• Enter server URL + auth headers
• Tap "Discover Tools" — all tools auto-appear
• The AI can call them alongside native tools

Popular MCP servers: Home Assistant, Notion, GitHub, Slack, Todoist, and hundreds more.

Home Assistant Integration

Direct REST API control of your HA instance — works alongside or instead of HomeKit.

Settings → Services → Home Assistant:

• HA URL: e.g. http://192.168.1.100:8123
• Token: Long-Lived Access Token (HA → Profile → Security)

Voice commands: "Turn on the living room lights", "Set thermostat to 72", "Run the goodnight automation", "List all sensors"

Transparency & Privacy

See exactly what data the AI receives and what network calls are made.

The agentic path is hardened against prompt injection — untrusted content (web pages, scanned text, tool output) can't hijack the assistant into running sensitive tools. High-impact actions stay behind explicit confirmation and the agent-mode gate.

Camera & Streaming

• Voice-Activated Photo Capture — "take a picture" or "what's this?"
• QR/Barcode Scanner — "scan this code" (Vision framework, works offline)
• Live Camera Preview — real-time view of glasses POV
• Video Recording — MP4 with configurable bitrate
• RTMP Broadcasting — live stream to YouTube, Twitch, Kick
• WebRTC Browser Streaming — shareable URL for peer-to-peer viewing
• Privacy Filter — auto-blurs bystander faces

Ray-Ban Display HUD

On Ray-Ban Display glasses (the Meta frames with an in-lens display + Neural Band), OpenGlasses mirrors content into the heads-up display and lets you act on it hands-free. Additive and off by default (Settings → Hardware → Glasses Display). It's gated on the device's display capability — not the brand — so camera/audio frames like Ray-Ban Meta and Oakley Meta are simply unaffected.

• AI responses & live captions — spoken answers and the ambient-caption line appear in-lens as they happen.
• Notification & navigation cards — calendar and geofence reminders, plus turn-by-turn Navigation Assist guidance, rendered with icons and a safety treatment.
• Interactive task cards — run a workflow or a Field Assist procedure as a Now / Next card and complete steps with the Neural Band (Done / Skip / Back, or branch choices) or by voice ("next", "done", "skip", "back").

Built on Meta's on-device display design system, so contrast, colour, and legibility are tuned for the waveguide automatically.

Text-to-Speech

24 ElevenLabs voices (10 female, 14 male) with iOS fallback:

• Female: Rachel, Sarah, Matilda, Emily, Charlotte, Alice, Lily, Dorothy, Serena, Nicole
• Male: Brian, Adam, Daniel, George, Chris, Charlie, James, Dave, Drew, Callum, Bill, Fin, Liam, Thomas

Emotion-Aware TTS adjusts tone automatically — warmer for good news, calmer for instructions, concerned for warnings.

Realtime Modes

Smart Model Routing

Assign models to Fast, Balanced, and Best tiers, then let OpenGlasses pick per request — a quick local model for live coaching, your best cloud model for hard diagnostics. Or turn routing off and pin everything to one model.

Configure: Settings → AI Models → Model Routing.

CarPlay & Apple Watch

• CarPlay — hands-free voice assistant on your car's display.
• Apple Watch — companion app and widget for quick control and glanceable status.

Enterprise

Commercial features for teams and regulated industries. These are licensed separately from the open-source core — see License or contact Skunkworks NZ.

Field Assist — Guided Field Service

Hands-free, step-by-step guidance for technicians and other hands-busy work. Procedures branch on what you report or what the camera sees, surface safety reminders before each step, cite their source material, and write an audited session log you can export. Stuck? Escalate to a live remote expert with glasses video. Domain knowledge lives in vaults (e.g. refrigeration, HVAC, electrical) you author and extend yourself.

Medical Compliance

Professional-grade safeguards for clinical recordings, available as an in-app subscription.

• Encryption at rest — recordings and transcripts protected with NSFileProtectionComplete
• Biometric app lock — Face ID / Touch ID required on every launch
• Audit logging — every data-access event timestamped and exportable
• Medical export — FHIR R4, HL7, and PDF export to Epic, Cerner, and more
• Data retention — configurable auto-purge with secure deletion
• Leakage prevention — cloud tools disabled, excluded from iCloud backup
• International frameworks — HIPAA, GDPR, AU Privacy Act, NZ HIPC, PIPEDA, UK DPA

Requirements

• iOS 26+
• Xcode 26+ and XcodeGen (brew install xcodegen)
• Physical iPhone (Bluetooth, camera, microphone required)
• Ray-Ban or Oakley Meta smart glasses (paired via Meta AI app) — the in-lens HUD requires Ray-Ban Display
• At least one LLM: API key (Anthropic, OpenAI, Gemini, etc.) OR a downloaded local model

Building from Source

1. Clone

git clone https://github.com/straff2002/OpenGlasses.git
cd OpenGlasses

2. Meta Developer Credentials

1. Go to wearables.developer.meta.com
2. Create an account, organization, and app
3. Note your Meta App ID and Client Token
4. In Meta dashboard → iOS settings, enter your Apple Team ID, Bundle ID, and Universal Link URL

3. Configure Meta keys (Info.plist)

Put your Meta App ID, Client Token, and universal-link URL in the MWDAT section. Either:

• Recommended: run ./Scripts/setup-local-dev.sh, edit Config/Info/Info.personal.plist (gitignored), then ./Scripts/generate-xcodeproj.sh again; or
• Edit the shared template OpenGlasses/Info.plist if you are not using a personal overlay.

<key>MWDAT</key>
<dict>
    <key>AppLinkURLScheme</key>
    <string>https://YOUR-DOMAIN/YOUR-PATH</string>
    <key>MetaAppID</key>
    <string>YOUR_META_APP_ID</string>
    <key>ClientToken</key>
    <string>AR|YOUR_META_APP_ID|YOUR_CLIENT_TOKEN_HASH</string>
    <key>TeamID</key>
    <string>$(DEVELOPMENT_TEAM)</string>
</dict>

4. Universal Links

Host an apple-app-site-association file at https://YOUR-DOMAIN/.well-known/apple-app-site-association:

{
  "applinks": {
    "details": [{
      "appID": "YOUR_TEAM_ID.YOUR_BUNDLE_ID",
      "paths": ["/YOUR-PATH/*"]
    }]
  }
}

5. Enable Developer Mode

On iPhone: Meta AI app → Settings → About → tap version number 5 times → toggle Developer Mode on.

6. Build & Run

Same as Quick Start step 1. The repo ships project.base.yml plus optional project.local.yml; XcodeGen writes OpenGlasses.xcodeproj locally. Do not commit the generated project.

brew install xcodegen
./Scripts/generate-xcodeproj.sh
open OpenGlasses.xcodeproj

Xcode Cloud runs ./Scripts/generate-xcodeproj.sh in ci_scripts/ci_post_clone.sh (full app + watch + tests).

Default generate includes watch and unit tests. To build a slimmer project locally (iPhone + widget only):

cp .openglasses-generate.env.example .openglasses-generate.env   # gitignored
./Scripts/generate-xcodeproj.sh

Or one-off: OPENGLASSES_SKIP_WATCH=1 OPENGLASSES_SKIP_TESTS=1 ./Scripts/generate-xcodeproj.sh

Optional: personal signing & Meta config

Team ID, entitlements, and Meta keys differ per developer. Those settings live in gitignored files (never committed), merged on top of the shared spec via project.local.yml:

First-time setup from the templates:

./Scripts/setup-local-dev.sh

Edit project.local.yml (developmentTeam) and the files under Config/ as needed, then run ./Scripts/generate-xcodeproj.sh again.

If you only need Xcode’s automatic signing with the shared entitlements, skip the local overlay and set your team in Xcode after opening the generated project.

Select your iPhone, fix signing if prompted, and run (⌘R).

Configuration

All settings are in-app — no source code editing needed.

API Keys (Settings → AI Models)

Services (Settings → Services & Integrations)

Troubleshooting

Dependencies

Contributing

Contributions welcome! This is fully open-source. Fork, improve, submit PRs.

Key areas for contribution:

• New native tools
• Local model optimization
• Translation quality improvements
• Additional MCP server integrations
• UI/UX improvements

License

Business Source License 1.1 — free for non-commercial use. Commercial use requires a separate license from Skunk0 / Skunkworks NZ. Converts to Apache 2.0 on March 24, 2030. See LICENSE file for details.

Credits

Built by Skunk0 at Skunkworks NZ

Powered by Anthropic Claude, Meta Wearables SDK, Apple MLX, ElevenLabs, HaishinKit

Note: Independent open-source project, not affiliated with Meta or Anthropic.