AI Utah January: Orpheus and the 3 types of AI businesses

# Hi Y'all!

---

![](./images/IMG_9077.jpg)

---

![](./images/IMG_9066.jpg)

---

![](./images/IMG_9068.jpg)

---

![](./images/voze-heart.gif)

---

# Background: What we are building

---

![](./images/IMG_9078.PNG)

---

![](./images/IMG_9079.PNG)

---

# Now a digression

---

# The Three\* Types of AI businesses

---

## 1. Building Models for others (OpenAI, Anthropic, Google, Mistral, etc.)

---

## 2. Accelerated by frontier models

---

## 3. Building, or fine-tuning, models for internal use

---

# Voze is a Type 2 business\*\*

---

### \* This is my mental model... use it where helpful

### \*\* For now

---

---

# Why Does This Matter?

---

## When do we all transition from 2 to 3?

### Will we ever need to?

---

# Will the asymptote be high enough?

---

---

# What we've learned over the last 12 months

---

![](./images/transcript-process.png)

---

---

# Deepgram... hated silence

---

---

# Deepgram plus empty audio checks

---

# Multimodal LLMs have entered the chat

---

```
def get_prompt(client_type: str):
    prompts = {
        'tire': GEMINI_AUDIO_TRANSCRIBE_PROMPT
    }
    return prompts.get(client_type, GEMINI_AUDIO_TRANSCRIBE_PROMPT)
```

---

```
def transcribe_audio_gemini(file: StorageStreamDownloader, client_type: str=''):
    audio_bytes = file.read()

if audio_bytes:
        # Base configuration
        config = {
            'response_mime_type': 'application/json',
            'response_schema': TRANSCRIPTION_SCHEMA
        }

response = client.models.generate_content(
            model=gemini_transcription_model,
            contents=[
                get_prompt(client_type),
                types.Part.from_bytes(
                data=audio_bytes,
                mime_type='audio/m4a',
                )
            ],
            # generation_config=types.GenerationConfig(**config) if config else None
        )
        return response.text
    else:
        return ''
```

---

```
TRANSCRIPTION_SCHEMA = {
    "type": "object",
    "properties": {
        "transcript": {
            "type": "string",
            "description": "The transcribed text from the audio"
        },
        "confidence": {
            "type": "number",
            "description": "Confidence score of the transcription (0-1)"
        },
        "language": {
            "type": "string",
            "description": "Detected language of the audio"
        },
        "metadata": {
            "type": "object",
            "properties": {
                "duration_seconds": {"type": "number"},
                "speaker_count": {"type": "integer"},
                "has_background_noise": {"type": "boolean"}
            }
        }
    },
    "required": ["transcript"]
}
```

---

# Building tooling for the future

---

![](./images/orpheus.png)

---

![](./images/corrections.png)

---

![](./images/Rent-all.png)

---

![](./images/eemax.png)

---

![](./images/jacklo.png)

---

# Other random lessons learned

---

## Bitrate matters, at least a little

---

## Some audio is not really recoverable or usable

---

![](./images/remote/giphy-7b951be7.gif)

---

# What's next?

---

# ?s

---

## Daniel Sellers

### daniel[at]designfrontier.net

### designfrontier.net

### @daniel_sellers

### Voze.com