Real rounded corners on client content (GLES/winit backend), via a custom shader

decoration.rs's round_top_corners only ever clipped the compositor's own
titlebar/border bitmap - its own doc comment already said why nothing more
had been done: clipping arbitrary client content needs a real per-pixel
mask, "a much bigger change than this cosmetic pass". That's this change,
for the one backend that can do it cheaply: the udev backend's
PixmanRenderer is software-only with no shader stage at all, but GlesRenderer
(winit) has a real custom-shader path (`compile_custom_texture_shader`,
`TextureShaderElement`) that a first look at smithay's higher-level
convenience APIs missed entirely.

crates/wayland/src/rounded_corners.rs: a GLSL fragment shader masking a
window's texture against a rounded-rect signed-distance field while
sampling it (the same technique cosmic-comp/niri use for GPU-side rounded
corners) - built by hand from the surface's own committed texture/view/
damage state (`RendererSurfaceState`'s public accessors), since no smithay
convenience wrapper builds a masked element at all (`CropRenderElement`
only crops to a rectangle). A decorated window rounds only its bottom two
corners - the top two are already rounded, on the titlebar's own CPU
bitmap, by decoration.rs, at the exact same `CORNER_RADIUS` (now
`pub(crate)`, shared between the two so the curve reads as one continuous
radius, not two different ones meeting at a seam) - an undecorated/CSD
window rounds all four, since its content is the window's whole visible
extent. Falls back to plain unrounded content on any failure (shader
didn't compile, no committed buffer yet, a single-pixel-buffer surface),
same "always show something over a prettier maybe-nothing" reasoning
cursor.rs's built-in-arrow fallback already uses. Deliberately scoped to a
window's *main* surface only, not subsurfaces - documented as a real, if
narrow, follow-up rather than attempted here.

`TextureShaderElement` only implements `RenderElement<GlesRenderer>`, not
the generic `RenderElement<R>` every `OverlayElement<R>` variant needs, so
it can't be added to that shared enum without breaking `OverlayElement<
PixmanRenderer>` (used identically by udev.rs) the moment a GLES-only
variant showed up in it. `WinitElement<=GlesRenderer>` (new, winit.rs-only)
wraps the existing enum as one variant instead of touching it - this is
the same lesson as the fullscreen-hiding investigation earlier this
session, just resolved cleanly this time: nesting a *foreign* generic
type inside your own hits real bound-resolution walls; wrapping your own
already-working type inside a new concrete-renderer enum doesn't, because
smithay's own `render_elements!` macro documents exactly this
`<=ConcreteRenderer>` form.

Config: `general.rounded_corners` (default `true`), `srd.window` unaffected
- this is a `general.*` compositor-behavior knob, not a per-window rule
action like `opacity`.

Verified live: shader compiles without error on this machine's real Mesa/
llvmpipe GL driver, and a decorated wezterm window's bottom-left and
bottom-right corners both show a real, smoothly anti-aliased curve on the
actual client-rendered pixels (not a compositor bitmap) in a host-session
screenshot - qualitatively sharper than `round_top_corners`' deliberate
hard cutoff, since a GPU shader can afford a ~2px smoothstep a CPU bitmap
pass isn't worth adding for. cargo build --workspace (all 9 crates), cargo
clippy --workspace (0 new warnings), cargo test --workspace (197 tests,
0 failed).

1 files changed, 7 insertions, 0 deletions

diff --git a/crates/core/src/manager.rs b/crates/core/src/manager.rs
index d45eaf6..08284c5 100644
--- a/crates/core/src/manager.rs
+++ b/crates/core/src/manager.rs
@@ -70,6 +70,12 @@ pub struct WindowManager {
     /// read from `general.resize_margin`. See [`crate::window::RESIZE_MARGIN`]'s
     /// doc comment for the default and why it's what it is.
     pub resize_margin: i32,
+    /// Whether a decorated window's content rounds its bottom two corners
+    /// to match the titlebar's own curve (an undecorated/CSD window rounds
+    /// all four). Read from `general.rounded_corners`. GLES-backend-only
+    /// in practice - see `crates/wayland/src/rounded_corners.rs` - so
+    /// this is a no-op on the udev/Pixman backend regardless of its value.
+    pub rounded_corners_enabled: bool,
     /// Default decoration colours and border width, read from `theme.colors.*`/
     /// `theme.decorations.*`. See `ThemeConfig`'s own doc comment.
     pub theme: ThemeConfig,
@@ -114,6 +120,7 @@ impl WindowManager {
             animation_duration_ms: 200,
             shadows_enabled: true,
             resize_margin: RESIZE_MARGIN,
+            rounded_corners_enabled: true,
             theme: ThemeConfig::default(),
             drag: None,
             resize: None,