Skip to content

refactor: use one placeholder token per image in LLM inputs - #2191

Merged
leejet merged 1 commit into
masterfrom
refactor/single-image-placeholder
Oct 22, 2026
Merged

leejet merged 1 commit into
masterfrom
refactor/single-image-placeholder

Conversation

@leejet

@leejet leejet commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • Use one placeholder per image in prompts and LLM input IDs, expanding to full image embeddings during splicing.

Related Issue / Discussion

N/A

Additional Information

N/A

Checklist

@leejet
leejet merged commit fd21ecf into master Oct 22, 2026
10 checks passed
@leejet
leejet deleted the refactor/single-image-placeholder branch October 22, 2026 16:45
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b145b463-93d4-441b-8e7b-43d862c69b0e
📥 Commits

Reviewing files that changed from the base of the PR and between e779d4f and 33a9481.

📒 Files selected for processing: 5
  • examples/server/routes_sdcpp.cpp
  • src/conditioning/conditioner.hpp
  • src/model/diffusion/hidream_o1.hpp
  • src/model/te/llm.hpp
  • src/pipeline/request.cpp
+----------------+
| CSI: Git Diff. |
+----------------+
  \
   \     /)/)
        ( '')
        /  |
      o(___)
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

GabrieleLozupone added a commit to GabrieleLozupone/stable-diffusion.cpp that referenced this pull request Oct 22, 2026
Takes upstream up to fd21ecf, including the Vulkan fix for small BAR heaps
(leejet#2181, ggml c1f80e5). ggml now points at GabrieleLozupone/ggml
kokon-c1f80e5: upstream c1f80e5 plus the CUDA graphs commit.

Conflicts and follow-ups:
- server: /sdcpp/v1/text_gen registers through RuntimeRoutes like the other
  routes. The streaming reply generates after the handler returns, so it
  holds the model runtime lock itself while it runs (a model switch gets
  409 instead of unloading the context under it).
- capabilities: upstream's limits plus max_text_tokens / max_text_images.
- name conversion: upstream's language_model prefix rename plus the
  llama.cpp "output" -> lm_head mapping.
- text generation: leejet#2191 puts one <|image_pad|> per image in LLM inputs
  and expands it in splice_image_embeds(). generate_text() now does the
  same, with image indices in the expanded sequence, and the KV cache
  counts the expanded tokens.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant