Real rounded corners on client content (GLES/winit backend), via a custom shader

decoration.rs's round_top_corners only ever clipped the compositor's own
titlebar/border bitmap - its own doc comment already said why nothing more
had been done: clipping arbitrary client content needs a real per-pixel
mask, "a much bigger change than this cosmetic pass". That's this change,
for the one backend that can do it cheaply: the udev backend's
PixmanRenderer is software-only with no shader stage at all, but GlesRenderer
(winit) has a real custom-shader path (`compile_custom_texture_shader`,
`TextureShaderElement`) that a first look at smithay's higher-level
convenience APIs missed entirely.

crates/wayland/src/rounded_corners.rs: a GLSL fragment shader masking a
window's texture against a rounded-rect signed-distance field while
sampling it (the same technique cosmic-comp/niri use for GPU-side rounded
corners) - built by hand from the surface's own committed texture/view/
damage state (`RendererSurfaceState`'s public accessors), since no smithay
convenience wrapper builds a masked element at all (`CropRenderElement`
only crops to a rectangle). A decorated window rounds only its bottom two
corners - the top two are already rounded, on the titlebar's own CPU
bitmap, by decoration.rs, at the exact same `CORNER_RADIUS` (now
`pub(crate)`, shared between the two so the curve reads as one continuous
radius, not two different ones meeting at a seam) - an undecorated/CSD
window rounds all four, since its content is the window's whole visible
extent. Falls back to plain unrounded content on any failure (shader
didn't compile, no committed buffer yet, a single-pixel-buffer surface),
same "always show something over a prettier maybe-nothing" reasoning
cursor.rs's built-in-arrow fallback already uses. Deliberately scoped to a
window's *main* surface only, not subsurfaces - documented as a real, if
narrow, follow-up rather than attempted here.

`TextureShaderElement` only implements `RenderElement<GlesRenderer>`, not
the generic `RenderElement<R>` every `OverlayElement<R>` variant needs, so
it can't be added to that shared enum without breaking `OverlayElement<
PixmanRenderer>` (used identically by udev.rs) the moment a GLES-only
variant showed up in it. `WinitElement<=GlesRenderer>` (new, winit.rs-only)
wraps the existing enum as one variant instead of touching it - this is
the same lesson as the fullscreen-hiding investigation earlier this
session, just resolved cleanly this time: nesting a *foreign* generic
type inside your own hits real bound-resolution walls; wrapping your own
already-working type inside a new concrete-renderer enum doesn't, because
smithay's own `render_elements!` macro documents exactly this
`<=ConcreteRenderer>` form.

Config: `general.rounded_corners` (default `true`), `srd.window` unaffected
- this is a `general.*` compositor-behavior knob, not a per-window rule
action like `opacity`.

Verified live: shader compiles without error on this machine's real Mesa/
llvmpipe GL driver, and a decorated wezterm window's bottom-left and
bottom-right corners both show a real, smoothly anti-aliased curve on the
actual client-rendered pixels (not a compositor bitmap) in a host-session
screenshot - qualitatively sharper than `round_top_corners`' deliberate
hard cutoff, since a GPU shader can afford a ~2px smoothstep a CPU bitmap
pass isn't worth adding for. cargo build --workspace (all 9 crates), cargo
clippy --workspace (0 new warnings), cargo test --workspace (197 tests,
0 failed).

1 files changed, 1 insertions, 0 deletions

diff --git a/crates/wayland/src/lib.rs b/crates/wayland/src/lib.rs
index e8a50ef..7687812 100644
--- a/crates/wayland/src/lib.rs
+++ b/crates/wayland/src/lib.rs
@@ -49,6 +49,7 @@ mod lock;
 mod output_management;
 mod output_power;
 mod protocols;
+mod rounded_corners;
 mod screencopy;
 mod state;
 mod udev;